TL;DR. We let developers re-run a failed eval gate, up to three total runs, and merge on any green. Felt harmless. It is not. If a judge-scored gate catches a real regression with probability 0.7 per run, merge-on-any-green across three runs drops the catch rate to 0.7^3 = 0.343, because the regression ships the moment any single run passes. The same policy pushed our false-block rate near zero, which is why everyone loved it and nobody measured the other side. A retry is a second sample from the same noisy scorer, so aggregate the samples instead of letting the luckiest one decide.
The button
Our eval gate runs a judge-scored suite on every PR that touches a prompt or a retrieval config. I have written before about judge scores drifting between runs on unchanged output; the short version is same diff, 0.83 Friday, 0.78 Monday.
So we added the button. "Re-run eval gate." Up to three runs total, merge on any green. The team's false-block complaints stopped within a week. I counted that as a win and moved on.
The part I did not count: what the button does to a real regression.
The arithmetic
Two numbers describe any noisy gate. The chance it fails a clean PR (false block, call it f). The chance it fails a PR that genuinely regressed (catch rate, call it d).
Both react to retries the same way, and that is the problem. Under merge-on-any-green with k allowed runs:
- P(clean PR gets blocked) = f^k
- P(regressed PR gets blocked) = d^k
With f = 0.10 and d = 0.70 and k = 3, the false-block rate falls from 10 percent to 0.1 percent. Great. The catch rate falls from 70 percent to 34 percent. It used to catch seven regressions in ten. Now it catches about three, and the dashboard shows nothing, because a merged PR looks identical whether it merged on run one or run three.
Run it yourself:
def blocked(fail_prob: float, k: int) -> float:
"""P(all k runs fail) under merge-on-any-green."""
return fail_prob ** k
for k in (1, 2, 3, 5):
print(k, round(blocked(0.70, k), 3), round(blocked(0.10, k), 6))
Output for k = 1, 2, 3, 5: catch 0.7, 0.49, 0.343, 0.168. False blocks 0.1, 0.01, 0.001, 1e-05. Every extra allowed run trades a chunk of your remaining detection power for false-block reduction you mostly already had.
Why the policy works for tests and fails for judge scores
Google's testing blog documented the strongest version of retry-as-policy in their 2016 flaky-test writeup (testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html): a test can be marked flaky so that it reports "a failure only if it fails 3 times in a row." The same post is honest about the cost, noting that the mechanism trains developers to ignore flakiness in their own tests until the triple-fail threshold trips. They also report that roughly 1.5 percent of all test runs across their corpus come back flaky. A decade old, still the clearest thing written on the subject.
Here is why the policy is defensible for tests and not for judge scores. A conventional test is deterministic in intent; flakiness comes from the environment around it. Ports, clocks, race conditions. When a flaky test passes, the pass carries real information, because the assertion itself is exact and the code path demonstrably works when the environment cooperates.
A judge-scored eval inverts this. The scorer is the random variable, so a passing run is just another draw from the distribution that produced the failing run, and it deserves exactly the same trust. Taking the best of three draws and calling it the verdict is not noise handling. It is selecting the sample you liked.
One question worth asking your own CI today: how many of your merged PRs had at least one red eval run before the green one? If your system cannot answer that, you have no idea what your effective catch rate is.
What we run now
Three changes, in the order we made them.
Deterministic checks stay blocking and get no retry button. Schema validation, regex assertions, golden-string diffs, exit codes. A fail is a fail. This is most of the gate and it is the part that was never flaky to begin with.
The judge suite runs a fixed panel of n = 5 samples and gates on the aggregate, computed once. I argued nine days ago that a single judge score cannot hold a merge gate, and that has not changed. An aggregate over a fixed panel is a different object: the run-to-run variance that makes one score untrustworthy is exactly what the averaging shrinks. No re-run path exists for it. If someone wants another run, the new samples join the panel and the aggregate recomputes over all of them. Averaging shrinks the noise by a factor of sqrt(n). Best-of-n converts the noise into bias in the direction you wanted.
Retries exist only for infrastructure failures, and the runner distinguishes them by exit code. A timeout or a 429 re-runs automatically and silently. A low score never does.
The retry-rate metric came last and taught us the most. Retries per PR per week is now on the same dashboard as the pass rate. It is our flakiness number. The week it spikes, something in the judge path drifted, and we know before anyone starts arguing with the gate.
What I'd check first
- Count merged PRs in the last 30 days with a red-then-green eval history. That number times your per-run catch rate is roughly what you are leaking.
- Find every place a human can re-trigger a scored check. Each one lets the luckiest sample overrule the rest. Replace it with an aggregate or delete it.
- Split your runner's exit codes: infra failure, score failure, harness bug. Only the first class earns an automatic retry.
Top comments (0)