It was a Tuesday, and I remember the commit message because it was four words long.
ci: reduce nightly noise
That was it. One line changed in the config. Retries from zero to two. I pushed it before lunch and the nightly went green that night for the first time in three weeks, and somebody in the QA channel posted a party emoji, and I felt like I had solved something.
I had not solved anything. I had turned off a smoke detector because I was tired of the beeping.
Why I did it
I want to be fair to the version of me that wrote that line, because he wasn't lazy and he wasn't stupid. He was drowning.
The nightly suite had about 400 specs. Somewhere between three and eight of them failed every single run, and never the same three. Monday it was two checkout specs. Tuesday it was a search spec and a profile spec. Wednesday it was checkout again but a different one.
Every morning I'd open the report, scan the failures, re-run the failed ones locally, watch them pass, and close the tab. That was forty minutes of my day, every day, producing nothing. And it wasn't just my forty minutes. The dev team had learned that a red nightly meant nothing, so they'd stopped looking at it entirely. Someone would ask in standup "is the nightly red for a real reason?" and the honest answer was "I don't know yet, give me an hour."
A suite that cries wolf a hundred and fifty times a month isn't a safety net. It's a chore. I knew that. What I got wrong was the fix.
I told myself retries were a triage tool. Let the flaky ones self-resolve, and the real failures, the ones that fail deterministically, will still come through red. Signal preserved, noise removed. That's the argument. I've heard four other engineers make it since, almost word for word, and every one of them believed it as sincerely as I did.
It's wrong, and it's wrong in a way that's worth doing the arithmetic on.
The arithmetic nobody does
Here's the thing about the failures I was calling "flaky." Almost none of them were random. Randomness has causes. A test that fails 30% of the time is not a test experiencing chaos. It's a test that is losing a race 30% of the time, deterministically, given the timing conditions on that runner.
So say I've got a genuine race condition in the app. The frontend fires a request, the modal renders before the response lands, and roughly 30% of the time the assertion catches the empty state.
With no retries, I detect that bug on 30% of runs. Roughly every third night, it goes red. Annoying, but visible. Eventually somebody investigates.
With retries: 2, the run only reports red if the test fails all three attempts:
0.30 × 0.30 × 0.30 = 0.027
2.7%. I went from seeing that bug every three nights to seeing it about once every five weeks. And when it does surface, it looks like an anomaly rather than a pattern: a one-off red in a sea of green, easy to dismiss.
Now do it for a subtler one. A bug that fires 5% of the time:
0.05 × 0.05 × 0.05 = 0.000125
One run in eight thousand.
That bug is now, for all operational purposes, invisible to me forever. But it still fires for one in twenty of my users. Nothing about the retry made the product more stable. It made my knowledge of the product less accurate, and that's the whole trade I made without realising I was making it.
The line I keep coming back to: your user does not get --retries 2.
The one that shipped
I'd love to tell you the bug that got through was some exotic distributed-systems thing. It wasn't. It was a double-submit on a payment form.
Under a specific timing window, the submit handler didn't disable in time and a fast double-click created two orders. We had a test that touched that flow. That test had been passing-on-retry, on and off, for months. In the report it was green. In the trend chart it was green. In every artifact anybody looked at, it was green.
Support found it before we did. Fourteen customers.
When I went back through the run history afterwards, and this is the part that still bothers me, the data was there. The runner had recorded every attempt. Attempt one failing, attempt two passing, over and over, for months. Nobody had built a report that surfaced it because nobody had asked for one, because the top-line number was green and green does not page anyone.
I hadn't lost the evidence. I'd just made sure nothing would ever show it to me.
What I actually think now
I'm not going to tell you to delete every retry. Some retries are legitimate, and the distinction is cleaner than I used to think:
Retry the infrastructure. Never retry the product.
A container failed to pull. The grid dropped your session. DNS hiccuped inside the cluster. Those are environmental, they're genuinely random, and retrying them costs you no information, because the thing that failed wasn't the thing you were testing. Retry those all day.
But a retry at the test level can't tell the difference between "the grid died" and "the app has a race condition." It swallows both identically. That's the design flaw. It's not a bad tool, it's an indiscriminate one, applied at the wrong layer.
Three things I'd do differently, in order of how much they'd have helped:
Make first-attempt failures a visible number. Not buried in a JSON artifact. On the dashboard, next to pass rate. If a test passes only on retry, that is a finding, not a formality. If I'd had a "passed on retry" count trending upward on a chart, the double-submit bug had months to catch my eye.
Quarantine instead of retry. A test that can't be trusted gets moved out of the gating suite and into a quarantine lane that still runs, still reports, but doesn't block anyone. The noise goes away, which was my real problem, but the information doesn't. Retries delete the information to remove the noise. Quarantine separates them.
Give every quarantined test an owner and an expiry. Otherwise quarantine is just a graveyard with better branding, and I've built one of those too.
The part I find hard to say
Something like that config flag is probably in your repo right now. Somebody added it during a bad sprint, for a good reason, and nobody has looked at it since.
I'm not writing this to tell you it's careless. I'm writing it because it took me eight months and fourteen affected customers to understand what I'd traded away, and the trade was invisible the entire time. That's what makes it worth talking about. Not that we did something dumb, but that we did something reasonable, and the tooling gave us no way to see the cost.
Go and look at what your suite reports for pass-on-retry. Not pass rate. Pass on retry.
If your stack can't tell you that number, that's the finding.
Top comments (2)
Love this. Those are the kinds of lessons that stick for life. Every experienced developer has a story like this. Keep building, keep breaking things, and keep learning; you’re on the right path. The journey is only getting started. 🚀💯
Super cool
Best I have read in a while now