Ben Halpern

Posted on Jun 1, 2022

What are your debugging tips?

#discuss #productivity

Top comments (23)

Manuel Odendahl • Jun 1 '22 • Edited

Planning for debuggability

Find a way to reproduce reliably, and quickly, and if possible, in an automated fashion (either through a unit test, or through an automated interaction, say scripting a browser). This means planning for mocking components, so that when a bug happens, writing a reproducible automated piece of code is just minutes away

When designing your system, design for debugging tools. For example, redux allows you to use the redux devtools to log, replay, export everything that interacted with the store. State machine frameworks often offer similar functionality.

Do preemptive logging. If you are able to log inputs and outputs of your system, do so. This will allow you to catch which inputs led to a bug in production, potentially sidestepping the entire debugging process itself.

Leverage git bisect

Now that you can quickly identify when a bug happens, use git bisect (which you can now run fully automatically) to identify the first commit that introduces the bug. Look at the commit to see if anything comes up. Usually, it gives you a sense of which files were modified.

Focus on logging

Rely on printf debugging because it gives you debug logs to pour over, instead of single stepping slowly. You can also use automatic actions on breakpoints if you don't want to litter your code.

Use stacktrace logging and structured logging to augment the richness of your printf debugging. Log into a sqlite database, for example.

Log inputs and outputs of your system.

Ideally, you want to run your failing test case once, and figure out the bug just from looking at the logs.

Know your breakpoints/watchpoints

Set breakpoints and watchpoints to get an overview of what is going on. Be aware that breakpoints and watchpoints slow both you and the program down. Depending on what you are debugging, that might actually influence the behaviour of the system significantly too.

Single step when you think you know what might be causing issues, in order to inspect memory and get a look at the stack trace. If you think you know what might be causing the issue, use breakpoint conditionals and watchpoints to confirm your suspicion.

Explain the bug to someone else

Try to explain the bug to someone else, in the dumbest terms possible (don't make assumptions). Use a whiteboard or show the code if possible. Often the rubberduck effect will kick in, and the bug will magically solve itself.

Take a walk

After fighting a bit with the bug, actually take a walk. I used to be a smoker, and I can't even remember the number of times I solved a bug when going downstairs to smoke a cigarette. Now that I don't have the nicotine withdrawal, I have to actively remind myself to get my butt off the seat and go walk a little bit. Mild physical activity has proven link to solving problems in the subconscious.

Manuel Odendahl • Jun 1 '22

More crazy techniques that I've used in the past, especially to figure out weird edge cases:

This is something you should do anyway, but:
put your system under immense pressure, and introduce faults into different subsystems. Every piece of software is going to break at scale in some way, might as well figure it out early. Bugs often happen in the error path / through cascading failures in the sidepath.

Embedded software

In embedded, be very careful with debuggers. They often shred the performance and behaviour of the system under test. If you can afford it and EE put in the right connectors, use a full CPU instruction trace debugger.

Use logic analyzer + scope and as many traces as possible. Know your tools inside and out (trigger settings, bus decoding, recording, etc...). Make sure the impedance of the probe doesn't mess with the signals (see next point).

Don't underestimate the power of putting your finger on communication buses / pins / power lines (provided they are low voltage). If the line starts going haywire on the scope while holding a finger somewhere, you have faulty traces or are missing proper impedance or bus termination. Many a bug is actually a badly connected bus line / missing pull ups.

Also, don't forget to check that your device is actually powered on: often, your device will get backpowered through the debugging adapter in the first place, and barely operate because of the low current (very fun).

Embedded again: use LEDs a lot, potentially even build additional LED boards that you can attach to subsystems without influencing voltage too much.

Again in embedded, use a little circuit with an amplified piezo, and put it on different communication lines or gpios. You can tick certain loops of your program, or toggle a gpio in an interrupt. Connect the piezo, which will start clicking in frequency. The human ear is very very sensitive, and you will identify any change in frequency or irregular patterns without even have to look at the device or the scope.

Use musical notes

Use MIDI events to debug your code instead of printf. For example, webmidi is very easy to use in the browser. Connect it to a synthesizer and choose notes for different events (or samples). You can now listen to your code executing, and don't have to look at the debugger / log output. You can hear if an unexpected event happens. The human brain is strongly connected to audio, chances are you'll remember the "melody" of a bug happening, which will allow you to figure out the execution path taken.

Pascal Thormeier • Jun 1 '22

Woah, that midi technique sounds amazing, honestly! Are you using that on a regular basis?

Manuel Odendahl • Jun 1 '22

No it’s been quite a while honestly but I was thinking of bringing it back out and making a nice example!

Martin Muller 🇩🇪🇧🇷🇵🇹 • Jun 3 '22

Yeah agree the midi sound technique litterrly sounds super intereting!

Gabriel Pedroza • Jun 1 '22

Run the same code 4 more times to make sure it’s a real bug. If it is, it’s time to get the rubber ducky🐥

Ivan Pozderac • Jun 2 '22

If ducky can't help, I just ask this senior web developer for advice!

Parker Waiters • Jun 1 '22

Get systematic: Remove possible variables in the code until you can isolate the problem. This is why it's important to be disciplined with your version control so you can freely delete swaths of code without complicating your life by being in the middle of some huge edits when you're trying to do it.

Daniel Schulz • Jun 1 '22

I got a few:

Breakpoints for assignment problems, loggers for flow problems.
Document your bugs, you'll see them again.
Fix a bug, then write a test for it.
Does it break the product? It's a bug. Is it just inconvinient? It's a feature request.
Talking a walk outside is a valid bugfixing strategy.

crowdozer • Jun 9 '22 • Edited

My main two tips:

Do less debugging by designing modularly.

Functions shouldn't be doing 10 different complex things, break each complex task into its own function and then compose them together.

Instead of:

function doesSomething() {
    // ...
    // doing stuff..
    // ...
    const date = new Date()
    const dayOfWeek = date.getDay()
    const isTodayEvenDay = dayOfWeek % 2 == 0
    // ...
    // doing stuff..
    // ...
}

Consider doing this:

function isTodayEvenDay() {
    const date = new Date()
    const dayOfWeek = date.getDay()
    return dayOfWeek % 2 == 0
}

function doesSomething() {
    // ...
    // doing stuff..
    // ...
    const isEvenDay = isTodayEvenDay()
    // ...
    // doing stuff..
    // ...
}

This is a trivial example, but it's a mountain of loosely related "trivial" things that end up creating hard to debug (and difficult to test) nightmares.

We're all guilty of littering this kind of "trivial supporting logic" in the middle of our business logic. Maybe it's because you were just rapidly prototyping, or maybe someone decided it was a "trivial" amount of complexity so they left it. Then someone else added another "trivial" amount of complexity and... it turns into a lot of mental overhead required to develop and debug.

Reduce complexity. Reduce mental overhead.

Walk through it step by step.

I've seen a lot of people who start working on something complicated, and when it doesn't work, they just lock up. They don't know where to start. Why not start at the start? Or the end. It doesn't matter. Just pick somewhere and look at what's being passed around. Function arguments, return values, etc. Does the first step work correctly? Does the second step work correctly? Why is that value not what you expect? Where did it come from?

Pierre Bouillon • Jun 1 '22

Sleep on it

I can't even remember how many times I was stuck on a problem for hours and it just took some minutes to solve the next morning after some rest

Riccardo Bernardini • Jun 9 '22

Add as many "bug traps" as possible

My idea is that a bug should manifest itself immediately. Bugs that corrupt the internal state of your application that cause problems later, are the worst to hunt down since they can manifest themselves in a totally unrelated place (I am thinking, for example, to dangling pointer bugs).

Therefore, I generously spread my code with bug traps. Assertions, contracts, type invariants are definitively your friends here. If you have the tools, formal checking is a strong "proactive debugging" technique.

Beside that, my debugging approach is mostly based on debug logging to see where the program goes and when the problem arises. In most extreme cases, I resort to debuggers and breakpoints.

Ben Sinclair • Jun 2 '22

You can use any system you like, from scattering logging calls or print statements to a fully-featured debugger. What you're most often doing is watching the state of variables and state data at different points in your code.

You'll think something like, "this is a problem with X, so I should look at things that touch X". The tip here is that if it doesn't immediately jump out at you, then it's probably not anything to do with X and you should start browsing around for anything that looks suspicious. Don't get hyper-focussed on it thinking, "it must be a problem with this or that function".