Matt’s Tidbits #35 - A strategy for debugging complex unit test failures

Last time I shared a quick tip on assertions in Kotlin. This week I want to share a strategy for debugging complex unit test failures.

Sometimes a unit test fails and it’s really obvious why it’s not working — perhaps you forgot to mock something, or you changed a simple method that now returns a different value, etc.

However, sometimes the scope of changes is so large (or the code is so complex) that it can be very difficult to pinpoint exactly what is causing the failure. To solve cases like this, I would like to share the following strategy:

1. First, save your changes and check out the latest version of the code where the test in question passes.
2. Next, use the debugger to step through the test a line at a time, and write down each (important) method call/variable state as you run through the whole test.
3. Now, re-apply your changes and step through the test again, performing the same documentation exercise as in step #2.
4. Now, compare each method call and variable state between the two versions. Look for any differences. Identify and challenge any assumptions you may have made — did you forget to call a method or set a variable, did you introduce unintended behavior when you changed the code, or does the test need to be changed?

Only through careful analysis and a thorough understanding of what the code used to do, what the code should do now, and what the test is actually testing will you be able to solve the problem correctly.

While performing this exercise myself recently, I discovered that I had left out a call to an important method — in this case it was called checkIfConnected(). In my (incomplete) understanding of the code, I was confident that the Bluetooth device was already connected (which it was), but unfortunately the checkIfConnected() method did more than just check if it was connected — it also initialized a bunch of other parts of the application. Adding that check back in fixed the test failure!

The morals of this story are:

1. Always understand the code before you try to change it.
2. Use any and all tools available to you (including pencil and paper) to help enhance your understanding and double-check your work to see if you missed something.
3. Always always always question your assumptions.
4. Be careful with how you name methods — if checkIfConnected() had been named initializeConnection(), I likely would not have removed what I thought was a superfluous statement!

I used pencil and paper to track this down, but I’m curious — does anyone have a software-based solution that they use to analyze and compare code flow? If you do, let me know in the comments below! And, please follow me on Medium if you’re interested in being notified of future tidbits.

This tidbit was discovered on September 19, 2019.

Comments

[Brice Favre]

Why don't you use git bisect to try to track what commit introduced the problem?

[Matthew Groves]

Thank you for the suggestion, Brice! In this particular case, I had made the change myself (as part of a large refactor), so finding which commit caused the problem was not the issue - it was more a matter of wading through a large pile of changes to find which one had been done incorrectly.

[Brice Favre]

Ok thanks for this precision and thanks for your article.