DEV Community

Discussion on: What's the longest you've ever spent debugging a single bug?

Collapse
 
ecnepsnai profile image
Ian Spence • Edited

Probably close to a month.

I was working as an iOS developer at the time for a hardware company. The app I was working on could connect to the hardware device and you could configure it using my app over USB.

I was debugging why the connection would drop after sending too much data to the device. The protocol we were using, usbmux, was reverse engineered and turned into a C library (libusbmux). Troubleshooting that was moderately easy, but troubleshooting things on the iOS device was super difficult.

At the time Apple had not implemented network debugging, so I didn't have any way to actually debug my app while also having it connected to the device. It took SO MANY times of just logging out to a file until I finally caught a clue.

The entire protocol is TCP based, with a server on both end that handles sending the packets over USB. I noticed that the send buffers on the iOS device quickly filling up, despite the data reaching the hardware device. Knowing this I was able to finally identify that under certain situations libusbmux would not send TCP ACKs for received data, and the iOS device would think it never reached.

It took so long because:
1- I had no access to any actual debugging methods other than printing to a log file
2- Modifying the code on the hardware I was connecting to was very tedious and took forever (compiling the Linux kernel is slow!)

It also did not help that the people that implemented this feature had long since left the company, and the poor guy who wound up owning this feature just wanted nothing to do with it.

At one point I was just questioning my own sanity as I found myself, with the title of "Web Developer", working on an iOS app, debugging the TCP stack in the Linux kernel.