GotSpeech.NET

Hi everyone.

I've got a question about the confidence scores in the MS SAPI (the XP version). What do the confidence numbers actually mean? As far as I can see from the documentation, the higher the score, the more confident the API is about getting the word right. Is there a probabilistic interpretation? Are the scores log-likelihoods or some function of such?   

Below is the output of MS SAPI given the utterance 'Leave': 

Leave 33048.00
We 15907.00
We've 9689.00
Believe 10061.00
And -3608.00 leave 13883.00
Lead 9695.00
Leaves 8705.00
The -45973.00 leave 13883.00
And -3608.00 lead 9817.00
The -45973.00 lead 9817.00
Believed 4534.00
Liu 12824.00

Notice that the different outcomes don't appear to be sorted by confidence. Additionally, the scores can't possibly be log-likelihoods as they range over [-X, X].
Thanks for your help.

Enes 

I'm not familiar with SAPI 5.1's implementation of confidence scores. Typically confidence scores are less than 1 meaning 0.9 indicates 90% certiainty.

Where are you getting those values?

---

Michael Dunn
Senior Consultant

Microsoft Consulting Services
www.microsoft.com/services
612.859.1899

This posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm

Thanks for the reply.

The scores are taken from "SREngineConfidence" in the  structure:
struct tagSPPHRASERULE

{
    LPCWSTR                 *pszName;
    ULONG                   ulId;
    ULONG                   ulFirstElement;
    ULONG                   ulCountOfElements;
    const  SPPHRASERULE    *pNextSibling;
    const  SPPHRASERULE    *pFirstChild;
    float                   SREngineConfidence;
    signed char             Confidence;
 };

This apparently should return a positive quantity with higher values indicating 

higher confidence as explained by the API, 

url: http://msdn2.microsoft.com/en-us/library/ms717034.aspx 

We are getting positive and negative values, and have no way of interpreting what 

these numbers actually mean. Is 10.0 twice as good as 9.0 or ten times as good, or?

Cheers,

Enes 

Hi,

I am using the same parameters in my application. I'd like to share my experience with you.

To decide the maximum value, you may try like this:

1. Prepare a WAV file of some short speech;

2. Train the recognizer by using the WAV file and its speech text; 

3. Recognize the WAV file after training;

4. Observe the value of SREngineConfidece, and you will get the maximum ones.

Thanks

Bill

 

Hey Bill.

Thanks for the reply. I can actually work out the maximum value but have no way of interpreting the scores themselves. At the moment, I find the minimum and maximum score, map them to the range [0,1] and interpret the scores as probabilities. This seems rather ad hoc as the relationship between the scores may not be linear at all. Unfortunately, the API is devoid of content on how to interpret the scores properly. A number of Google searches also came up with no answers ;(

Cheers,

Enes

Hi,

I think there are many factors affecting the confidence value.

Those factors might be: acoustics, lexicon, linguistics, trainning process, times of usage etc.

How do you think about these?

Thanks

Bill

In principle, the SREngineConfidence score is a value between 0.0 and 1.0 {higher value meaning higher confidence}. But older versions of the SR engines like 5.1 don't honor this contract precisely, and I don't think the value can really be used with those engines. Only the Hi, Medium, and Low scores in the other Confidence field are usable.

If I remember rightly, you need a more recent version of the SR engine, like the versions that ship with Microsoft Office 2003 or Vista to get a meaningful number in the SREngineConfidence field.