You hand the agent a failing test: an upload that flakes about one run in ten. It comes back green. You open the diff and find a for (let i = 0; i < 3; i++) around the call, a setTimeout, and a swallowed error that only logs on the third try.
CI is happy. You merge. For a week, nothing looks wrong.
Then on Monday morning, the same upload fails three times in a row and your retry loop gives up — and now you have no idea why, because the first two attempts' errors were caught and discarded. The flake wasn't fixed. It was statistically buried.
The agent optimizes for the symptom it can see
An agent working from a red test sees one thing: this call sometimes throws. The cheapest way to make the test pass is to make the throw less likely to reach the assertion. Retries do that beautifully — they convert a deterministic failure into a probabilistic one, and over a small CI sample, probability looks like a fix.
This is rarely malicious. It's the same instinct a tired human has at 11pm. But the human knows the system; the agent only knows the open file and the test it just watched go red.
How to tell a real fix from a bandage
When the diff adds a retry, ask three things before you accept it:
- What is the actual exception? If the agent can't paste the exact error and the line that produced it, it's treating the failure as noise rather than a signal.
- Is the retry on the right layer? Retrying a network call is reasonable. Retrying a validation failure, a 4xx, or a logic error is never reasonable — those are deterministic and will fail all three times.
- What happens on the final give-up? A bandage swallows the error and returns a default. A real fix surfaces it, with context, where someone can see it.
The cheap rule that catches this
I stopped accepting any diff that adds a retry without also surfacing the first failing attempt's log line. One cheap check: refuse the merge if the agent can't show you the exception class, the request that triggered it, and why retrying is expected to help. If the answer is "because it usually works on the second try," you've learned nothing — you've just scheduled the next outage.
Green CI after a retry is not evidence the bug is gone. It's evidence you stopped looking at it. The root cause is still there, waiting for the one moment three retries in a row aren't enough.
Top comments (4)
The numbers explain why the retry looks like a fix. If the upload fails 1 run in 10 and the three attempts were independent, the loop would fail a run only 0.1^3 = 0.1% of the time, so a CI window of a few dozen runs will almost always stay green. But the Monday failure you describe, three misses in a row, is the sign that the attempts were never independent: a shared cause (a stale connection, a rate limit, a bad payload) fails every attempt together, and the retry only helps with the independent part.
So one more check next to your three: ask whether the flake rate with the retry removed was measured at all, and run the original test a few hundred times with the retry reverted. If it fails ~10% of runs, the cause is real and the log of the first failing attempt is worth more than the green. If the agent can't reproduce the failure on demand, it hasn't found the bug yet.
Do you make the loop log attempt 1's exception at error level, or only the last one? That decides whether the Monday case is debuggable.
The independence assumption is exactly where I'd point next — it's the same shape as a scraper bug I shipped: my crawler retried on 429 with backoff, looked healthy in dashboards, and quietly masked that the whole run was being rate-limited. Every retry "succeeded" on attempt 3, so the success metric stayed at 100%.
What broke the illusion was counting attempts, not outcomes: I added a "first-try success rate" counter, and it dropped from 99% to 61% overnight the day the upstream added a WAF. The retry loop hadn't fixed anything; it had just moved the cost from the failure column to the latency column.
So the audit question I now ask of any retry loop: what does the distribution over attempt numbers look like? A healthy loop has a steep decay (99% on 1, ~1% on 2, ~0 on 3). A loop hiding a systemic bug has a fat tail — and the failures you never see are the ones that succeeded on attempt 3 and got logged as green.
One qualification to the "never retry a 4xx" rule:
429is a client-error status too, but it can be a legitimate retry candidate. HonorRetry-Afterwhen provided, keep the attempt/time budget bounded, and account for any replay risk in the operation.By contrast, resending the same invalid
400/422payload or the same unauthorized401request usually just repeats the defect. A409needs the API's documented conflict semantics; the status class alone isn't enough to choose a policy.I'd preserve the first failure and its request ID alongside later attempts. A dashboard that records only the final success makes it very hard to tell recovery from a loop that has merely hidden a growing failure rate. That distinction fits your main point about an agent "fixing" the symptom.
The three questions before accepting a retry are the right gate, and the one I keep adding is whether the first attempt's outcome is still unknown.
When the call is a network timeout, the client stopped waiting. That says nothing about whether the upload, charge, or write already landed on the other side. A loop that treats that as a throw and fires again under a new attempt can succeed twice: once in the dark, once on the retry. Green CI still looks like a fix.
What I want next to your first attempt log line is a parked state for unknown: keep the original request identity, refuse a second send until something reconciles the first, and only then decide retry or give up. Does your rule treat a timeout the same as a 5xx, or do you split them?