Last Tuesday I lost about three hours to a regression in our checkout service. The cart total was off by a cent on certain promo combinations, and the only signal was a Slack ping from finance with a screenshot. No stack trace. No exception. Just wrong numbers.
I did what I always do first. I opened the diff for the last deploy, scrolled, squinted, and tried to feel my way to the bug. Forty minutes in, I had a mental shortlist of suspects and zero proof. The code I was sure caused it had been merged a week earlier and was already reverted in a different branch for unrelated reasons. I was, as my old tech lead used to say, debugging the file I wanted the bug to be in.
Then I remembered git bisect run.
I keep relearning this tool. It's been around forever. I've recommended it to juniors more times than I can count. And yet whenever I'm in the thick of a real outage, I forget it exists for the first hour, because my brain wants to read the diff like a detective novel instead of running an experiment.
Here's the workflow that finally stuck for me. I write a tiny script that exits 0 when the bug is absent and non-zero when it's present. For this regression, the script was twelve lines:
#!/usr/bin/env bash
set -e
go build ./cmd/checkout
./checkout-test --promo=BACK2SCHOOL --items=3 > /tmp/out
grep -q '"total":4297' /tmp/out
Then git bisect start HEAD v4.18.2, git bisect run ./check.sh, and walk to the kitchen for coffee. By the time I came back, git had narrowed it to a single commit. Not the one I'd been staring at. Not even close. It was a four-line change to how we round per-line tax that someone had landed quietly two weeks ago, with a benign-looking commit message about "consistency."
What surprised me wasn't the bug. The bug was small. What surprised me was how much faster the machine was at this than I was. I had been pattern-matching on author, recency, and which file looked spicy. Bisect doesn't care about any of that. It just keeps halving the search space until there's one commit left.
A few things I've learned about making this a habit instead of a heroic last resort:
The check script needs to be cheap. If your build is eight minutes, bisect across two hundred commits is a coffee break and a meeting and lunch. Mock the slow stuff. Build a smaller binary. I keep a bisect/ directory in some repos with pre-baked check scripts for known classes of bugs, so when something feels familiar I'm not writing the harness from scratch under pressure.
Skip commits that don't build. git bisect skip is your friend. You'll hit a few broken middle-of-refactor commits and that's fine; bisect handles them.
Don't trust your gut on the "good" commit. I once spent forty minutes bisecting before realizing my "known good" reference also had the bug, just less visibly. Now I always run the check script against the supposedly-good commit first, before bisect start. Two minutes of paranoia saves an hour of confusion.
Bisect across squash-merged PRs and you might land on a commit that touches forty files. That's still useful. The PR title alone usually points you somewhere. I've also started linking PR numbers in commit bodies more aggressively for exactly this reason.
The reason I keep forgetting this tool, I think, is that it feels like cheating. I want to understand the bug by reading the code, because that's the part that feels like real engineering. Bisect skips the understanding and goes straight to the answer. But the answer is what unblocks the team. Understanding can come after, in a calmer hour, with the offending commit already pinned to the wall.
The Tuesday bug ended up being a one-line fix. The git bisect run took eleven minutes including the coffee. The forty minutes I'd spent reading the diff before remembering the tool existed is the part I want back.
Next time I feel the urge to scroll through a deploy diff while a fire is burning, I'm going to write the check script first. Even if I'm sure I know where the bug is. Especially then.
Top comments (0)