Every war story you read ends with the fix. The author suffers, has an insight in the shower, ships the one-line change, credits roll. I've written that story. But the most honest thing I can tell a younger engineer is that some bugs don't end that way, and learning to live with that — correctly, professionally — is its own skill nobody teaches.
This is the one that got away.
The symptom
Every so often — rarely, unpredictably — a background job would process the same piece of work twice. Not always. Not on any schedule I could find. Maybe a handful of times a week against an enormous volume. Just often enough to be real, just rare enough to be impossible.
Downstream, a double-processed job meant a duplicated side effect, and duplicated side effects are the kind of thing that ranges from harmless to a customer emailing you a screenshot. So it mattered. And it had all the hallmarks of the genre engineers fear most: intermittent, non-reproducible, and — this is the cruel part — it got shyer the harder you looked.
The hunt, and why every tool failed
I did everything you're supposed to do.
I added logging. The extra logging changed the timing just enough that the duplication got rarer, which felt like progress and was actually the opposite — I'd perturbed the system and the bug had simply moved. This is the defining property of a heisenbug: observing it changes it. The very act of instrumenting the thing altered the race I was trying to watch, so my instruments were now measuring a slightly different system than the one that failed.
I tried to reproduce it in a controlled environment. Couldn't. In isolation, with synthetic load, everything behaved. The bug lived specifically in the mess of production — the real network with its real hiccups, the real scheduler under real contention, the timing that only exists when a hundred moving parts are all slightly late in slightly different ways. You cannot fit that into a test harness, because the mess is the cause, and the mess is exactly what a test harness removes.
I read the code. For hours, for days, across every path that touched a job. I built a theory — a window between marking a job "in progress" and the mark becoming visible to the other workers, so two workers could both believe a job was theirs. It's a plausible theory. It's the kind of thing that was probably happening. But I could never prove it was the thing, because I could never catch the bug in the act with enough fidelity to say "there, that, that's the moment." Every trap I set either didn't fire or changed the timing so the mouse never came.
I want to be honest about the feeling, because the feeling is part of the lesson: it's maddening. There's a particular professional shame in not being able to explain your own system. You start to doubt your competence. You stay up. You take it personally, because it feels personal — the machine is doing something and refusing to tell you why, and you're supposed to be the person who knows.
The move that actually mattered
The thing that saved me wasn't finding the bug. It was giving up on finding it, and asking a different question.
I'd been asking why does this job run twice? — a root-cause question, and the root cause was hiding successfully. The better question, the one I should have reached sooner, was: what if I stop trying to prevent it from running twice, and instead make running twice not matter?
That's idempotency, and it reframes the entire problem. If processing a job a second time produces the exact same result as processing it once — no duplicated side effect, because the work checks whether it's already been done and quietly no-ops — then I no longer care whether the mysterious double-execution happens. Let it happen. It's harmless now. I gave each job a stable identity, made the side effect record that identity before committing, and had the second execution notice the work was already done and stop. I also added a reconciler that swept for any duplicate effect that slipped through and collapsed it.
The duplication, as far as I know, still happens. Somewhere in there, rarely, a job still runs twice. I never fixed that. What I fixed was the consequence. The bug is still in the house; I just took away everything it could break.
What this taught me, which was more than a fix would have
For a long time I thought this was a failure — the one I couldn't crack. I've come around. Here's what it actually gave me.
Some causes live in emergent timing, and you will not always catch them. A distributed system's behavior isn't in any one file; it's in the interaction of many parts under conditions you can't fully recreate. Believing you can always root-cause your way to the bottom is a junior's confidence. Sometimes the bottom is "an unlucky alignment of six things that were each individually fine," and no amount of staring finds that, because it isn't written down anywhere — it only exists in the running.
Containment is a legitimate victory, not a consolation prize. The instinct that "real engineers find the root cause" is mostly true and occasionally a trap. When a bug resists every honest effort, making it harmless — idempotency, reconciliation, defense in depth — is not giving up. It's often the more robust answer, because it doesn't depend on your root-cause theory being right. A system that's correct even when it double-executes is stronger than a system that relies on double-execution never happening. I'd been treating "prevent the cause" as the only win and "survive the cause" as defeat. It's backwards. Surviving the cause is the sturdier engineering.
Know when to stop hunting. There's a point where another day of chasing a rare bug costs more than the bug does, and where the mature move is to contain it, write down everything you learned, and walk away with the itch unscratched. I still don't like that. I've made peace with it. The bug's still out there. It just can't hurt anyone anymore, and some nights that has to be enough.
Top comments (0)