You hand the agent a failing test: an upload that flakes about one run in ten. It comes back green. You open the diff and find a for (let i = 0; i < 3; i++) around the call, a setTimeout, and a swallowed error that only logs on the third try.
CI is happy. You merge. For a week, nothing looks wrong.
Then on Monday morning, the same upload fails three times in a row and your retry loop gives up — and now you have no idea why, because the first two attempts' errors were caught and discarded. The flake wasn't fixed. It was statistically buried.
The agent optimizes for the symptom it can see
An agent working from a red test sees one thing: this call sometimes throws. The cheapest way to make the test pass is to make the throw less likely to reach the assertion. Retries do that beautifully — they convert a deterministic failure into a probabilistic one, and over a small CI sample, probability looks like a fix.
This is rarely malicious. It's the same instinct a tired human has at 11pm. But the human knows the system; the agent only knows the open file and the test it just watched go red.
How to tell a real fix from a bandage
When the diff adds a retry, ask three things before you accept it:
- What is the actual exception? If the agent can't paste the exact error and the line that produced it, it's treating the failure as noise rather than a signal.
- Is the retry on the right layer? Retrying a network call is reasonable. Retrying a validation failure, a 4xx, or a logic error is never reasonable — those are deterministic and will fail all three times.
- What happens on the final give-up? A bandage swallows the error and returns a default. A real fix surfaces it, with context, where someone can see it.
The cheap rule that catches this
I stopped accepting any diff that adds a retry without also surfacing the first failing attempt's log line. One cheap check: refuse the merge if the agent can't show you the exception class, the request that triggered it, and why retrying is expected to help. If the answer is "because it usually works on the second try," you've learned nothing — you've just scheduled the next outage.
Green CI after a retry is not evidence the bug is gone. It's evidence you stopped looking at it. The root cause is still there, waiting for the one moment three retries in a row aren't enough.
Top comments (2)
The numbers explain why the retry looks like a fix. If the upload fails 1 run in 10 and the three attempts were independent, the loop would fail a run only 0.1^3 = 0.1% of the time, so a CI window of a few dozen runs will almost always stay green. But the Monday failure you describe, three misses in a row, is the sign that the attempts were never independent: a shared cause (a stale connection, a rate limit, a bad payload) fails every attempt together, and the retry only helps with the independent part.
So one more check next to your three: ask whether the flake rate with the retry removed was measured at all, and run the original test a few hundred times with the retry reverted. If it fails ~10% of runs, the cause is real and the log of the first failing attempt is worth more than the green. If the agent can't reproduce the failure on demand, it hasn't found the bug yet.
Do you make the loop log attempt 1's exception at error level, or only the last one? That decides whether the Monday case is debuggable.
The independence assumption is exactly where I'd point next — it's the same shape as a scraper bug I shipped: my crawler retried on 429 with backoff, looked healthy in dashboards, and quietly masked that the whole run was being rate-limited. Every retry "succeeded" on attempt 3, so the success metric stayed at 100%.
What broke the illusion was counting attempts, not outcomes: I added a "first-try success rate" counter, and it dropped from 99% to 61% overnight the day the upstream added a WAF. The retry loop hadn't fixed anything; it had just moved the cost from the failure column to the latency column.
So the audit question I now ask of any retry loop: what does the distribution over attempt numbers look like? A healthy loop has a steep decay (99% on 1, ~1% on 2, ~0 on 3). A loop hiding a systemic bug has a fat tail — and the failures you never see are the ones that succeeded on attempt 3 and got logged as green.