September was a really rough month for me. You probably noticed that I've been kind of quiet for the last month. Well that was mainly because of two things that happened.
I got sick on the 10th with some kind of flue bug that hit me hard. I only missed one day of work but sometimes the only reason that I didn't come home from work early was because I didn't feel up to driving. I still haven't kicked it completely but I'm back to about 98% of my old self. I can't recall ever being sick for such a long period of time. I kept at it because of the second thing that occurred.
As all of this was going on I had a huge project at work that was coming due. We changed the identity and access methods on our applications so I had to modify the data access routines from an old IVR application that has been in production for 4 years. I won't name the platform for this IVR system but the application is one huge state machine that I didn't write. I went away for training on this system 3 years ago and never did any coding with that language until this summer.
I finished the code changes on time and turned it over to QA for testing. They encountered some problems that were data related that we fixed by getting the data cleaned up and in a consistent state. Then QA came up with this weird issue where certain calls worked one way and other calls worked differently.
When an application reacts like this you have to suspect the data. So, I set about trying to determine what was different between the callers. We found two users that I could test with where one worked fine and the other exhibited the problem. After spending some time analyzing the database and the records returned by the different stored procedures I figured out what the difference was. I can't disclose what it was but I found that the bug only occurred in one really obscure set of circumstances. Armed with that information I turned to the debugger.
As I stated this application was one gigantic state machine containing 11 different modules. Debugging the call turned into a nightmare as I had to document the flow during a test call. Luckily the debugger lets you step through the call and see the contents of the different variables. Unfortunately it doesn't let you see all of the code as it goes from step to step (think of steps that are dragged onto a giant grid to show the logic flow). I had to remotely connect to the server and every time I needed to know more about one of the steps I had to switch to the development serve so that I could view the actual code. This was slow going and I did a lot of screen prints so that I could document what was happening.
I spent 6 hours in the debugger on one phone call stepping through the logic piece by piece until I finally found the problem. As it turns out the problem wasn't in the data interface module that I had modified but was in another module. It also turned out to be a logic error on the part of the original programmer and an issue that no one had complained about in the 4 years this app has been in production.
The problem occurred within an if statement that didn't cover all of the possible conditions. The if was written with the main condition being if var > 1 then do this else do something different. The intent was to determine if the caller had multiple tractors available (the else part) and on the surface it may have made sense to the original programmer. But a problem arose when the caller had only one tractor available. This should have been a case statement with conditions for 0 tractors, 1 tractor and 2 or more tractors available.
This was probably one of the most difficult debugging sessions I've ever encountered. I'm pretty good at attacking problems in a logical manner and eliminating things until I track down the bug. However tackling something like this while sick probably didn't help matters. The stress of all of this may also have contributed to my recovery being slowed.
So, that is why I have been somewhat quiet this last month. I've got several things I want to blog about but I'll spread them out over the next several days.
It sure feels good to be back in the saddle again and I would love to hear from you guys about your worst debugging experience.