GotSpeech.NET

The online community for Microsoft Speech Server developers
Welcome to GotSpeech.NET Sign in | Help
in Search
CBrekeke Software

SAPI and Confidence Scores

Last post 10-04-2007, 5:21 PM by davewood_ms. 6 replies.
Sort Posts: Previous Next
  •  08-17-2007, 2:56 AM 3613

    SAPI and Confidence Scores

    Hi everyone.

    I've got a question about the confidence scores in the MS SAPI (the XP version). What do the confidence numbers actually mean? As far as I can see from the documentation, the higher the score, the more confident the API is about getting the word right. Is there a probabilistic interpretation? Are the scores log-likelihoods or some function of such?   

    Below is the output of MS SAPI given the utterance 'Leave': 

    Leave 33048.00
    We 15907.00
    We've 9689.00
    Believe 10061.00
    And -3608.00 leave 13883.00
    Lead 9695.00
    Leaves 8705.00
    The -45973.00 leave 13883.00
    And -3608.00 lead 9817.00
    The -45973.00 lead 9817.00
    Believed 4534.00
    Liu 12824.00

    Notice that the different outcomes don't appear to be sorted by confidence. Additionally, the scores can't possibly be log-likelihoods as they range over [-X, X].
    Thanks for your help.

    Enes 

  •  08-19-2007, 11:07 PM 3623 in reply to 3613

    Re: SAPI and Confidence Scores

    I'm not familiar with SAPI 5.1's implementation of confidence scores. Typically confidence scores are less than 1 meaning 0.9 indicates 90% certiainty.

    Where are you getting those values?


    Michael Dunn
    Senior Consultant

    Microsoft Consulting Services
    www.microsoft.com/services
    612.859.1899

    This posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm
  •  08-20-2007, 3:37 AM 3630 in reply to 3623

    Re: SAPI and Confidence Scores

    Thanks for the reply.

    The scores are taken from "SREngineConfidence" in the  structure:
    struct tagSPPHRASERULE

    {
    LPCWSTR *pszName;
    ULONG ulId;
    ULONG ulFirstElement;
    ULONG ulCountOfElements;
    const SPPHRASERULE *pNextSibling;
    const SPPHRASERULE *pFirstChild;
    float SREngineConfidence;
    signed char Confidence;
    };
    This apparently should return a positive quantity with higher values indicating 
    higher confidence as explained by the API, 
    url: http://msdn2.microsoft.com/en-us/library/ms717034.aspx 
    We are getting positive and negative values, and have no way of interpreting what 
    these numbers actually mean. Is 10.0 twice as good as 9.0 or ten times as good, or?
    Cheers,
    Enes 
  •  09-03-2007, 3:21 AM 3724 in reply to 3630

    Re: SAPI and Confidence Scores

    Hi,

    I am using the same parameters in my application. I'd like to share my experience with you.

    To decide the maximum value, you may try like this:

    1. Prepare a WAV file of some short speech;

    2. Train the recognizer by using the WAV file and its speech text; 

    3. Recognize the WAV file after training;

    4. Observe the value of SREngineConfidece, and you will get the maximum ones.

    Thanks

    Bill

     

  •  09-04-2007, 1:12 AM 3729 in reply to 3724

    Re: SAPI and Confidence Scores

    Hey Bill.

    Thanks for the reply. I can actually work out the maximum value but have no way of interpreting the scores themselves. At the moment, I find the minimum and maximum score, map them to the range [0,1] and interpret the scores as probabilities. This seems rather ad hoc as the relationship between the scores may not be linear at all. Unfortunately, the API is devoid of content on how to interpret the scores properly. A number of Google searches also came up with no answers ;(

    Cheers,

    Enes

  •  09-11-2007, 3:05 AM 3776 in reply to 3729

    Re: SAPI and Confidence Scores

    Hi,

    I think there are many factors affecting the confidence value.

    Those factors might be: acoustics, lexicon, linguistics, trainning process, times of usage etc.

    How do you think about these?

    Thanks

    Bill

  •  10-04-2007, 5:21 PM 3959 in reply to 3776

    Re: SAPI and Confidence Scores

    In principle, the SREngineConfidence score is a value between 0.0 and 1.0 {higher value meaning higher confidence}. But older versions of the SR engines like 5.1 don't honor this contract precisely, and I don't think the value can really be used with those engines. Only the Hi, Medium, and Low scores in the other Confidence field are usable.

    If I remember rightly, you need a more recent version of the SR engine, like the versions that ship with Microsoft Office 2003 or Vista to get a meaningful number in the SREngineConfidence field.

View as RSS news feed in XML