Debugging Speech Server: Lesson Learned
If you've worked with computers long enough you know that there are times when you run across a problem that seems so elusive that you just want to attribute it to, "that's just the way it is", and move on.
Such was the case not too long ago when I ran across a problem using Speech Server. It got to the point where all I wanted to do was backup our databases, wipe clean the Speech Server, put on a fresh operating system, re-install Visual Studio 2005, SQL Server, prerequisites for Speech Server and Speech Server itself… not a very appealing option, but one I was really close to accepting. It was an extremely frustrating experience.
So the scenario is I have a laptop with a soft phone, calling across the local network to a Windows 2003 Server with Speech Server installed. I was able call the speech server and everything about the call was fine except for one important part. I could not hear anything. I knew the soft phone was being answered, because of the message on the soft phone that said I was connected. I could verify the connection was being made on the Speech Server because I could see the additional call show up through the Performance Monitor. I even used Wireshark to do a trace to show that the SIP traffic was going back and forth, and to see if there was some anomaly with the audio. But everything looked fine. There were no errors in any system or speech logs either on the Speech Server or my laptop. My natural inclination was to look into a firewall problem. After all, firewall problems always seem to be the culprit when it comes to SIP calls. After shutting down the firewall, however, I still had the problem.
Just for grins, I took a virtual machine that had Windows 2003 Server and Speech Server loaded, and ran the virtual machine on my laptop and attempted to make a call using the same scenario. I had a soft phone on my host operating system calling a Speech Server on a Virtual machine, all on my laptop. Everything worked fine, including speech.
My conclusion… my laptop was configured properly, my soft phone was configured properly and the problem with getting no audio had nothing to do with the laptop, and strangely was not working from the Speech Server.
So what had changed? The Speech Server I was using for development had been working fine before I had made changes to my application. With this new version of the app, it simply would not produce audio. Had I done something with the application to cause that? After testing the old version of the application, I still was not producing audio to my soft phone. I even tested the built in "Welcome to Microsoft Speech Server" app that comes with Speech Server and that didn't produce audio either.
Another thing that had changed was that we had installed .NET 3.5 on the Speech Server box to accommodate the latest WCF (Windows Communication Foundation). Could that have been the problem? To eliminate that possibility I also installed .NET 3.5 on the virtual machine I used earlier. After testing, it worked just fine. Audio passed to the soft phone without a problem.
So I still was not getting audio from our speech server to my laptop and I still haven't a clue why. Could it be that the problem was not the Speech Server but was my laptop? Seems unlikely since everything works fine when I used a virtual machine, but just in case I asked one of my colleagues to set their laptop up with a soft phone configured the way mine was. I then went to Speech Server to configure a trusted SIP peer. My colleague made the call and lo and behold, he received audio to his Soft Phone!
So now things pointed back to my laptop. But what could be the problem? What was different now as compared to two weeks ago when things were working fine?
It turns out the week prior to my discovering the problem I am describing here, we were at Microsoft going through a tech review of our application and one of the things that was needed was a demonstration of our application. To do that I used my laptop running a virtual machine with Speech Server that was all on my laptop. Because I was not going to be connected to a standard network, I had set up my laptop with a virtual network using a loopback adapter. In case you are not familiar with a loopback adapter, this is a piece of software that ships with Windows that allows you to configure a virtual network card that can be used to test network connections. It can also be used to setup a virtual machine that connects back to the host operating system through a virtual network without having a physical one.
Just like a real network card, you can have the loopback adapter connected or disabled. So for grins, I decided to look into the possibility that the loopback adapter was causing the problem. It turns out the loopback adapter was still connected as it was while I was at Microsoft. So I disabled it and tried my test again. IT WORKED! With the loopback adapter disabled, I was now getting audio from the Speech Server.
So there are three morals of this story. One, if you use a loopback adapter, make sure you shut it off before doing testing over a physical network. Two, don't just arbitrarily wipe a box clean and spend many hours getting that box up and running without exhausting other possibilities. If I had done that, I would have found out that the problem still existed, and that would have really been frustrating. Finally and probably most important, when debugging, always think about what changed since the last time it worked. Nine times out of ten, it is a change that seems to be benign that causes your problem.
To this day I am not really sure why the loopback adapter being connected causes a problem with audio being passed from Speech Server to a soft phone over a physical network, but it does. And maybe someone can help me understand that. This is just another one of those lessons learned that I wanted to share with you.