Three Green Reruns Prove Almost Nothing: Sizing a Flake-Detection Budget for Agent Patches

A merge gate that reruns a failing test three times is not testing for flakes. It is testing whether the flake happens to be loud.

Run the arithmetic before you argue about it. At a 2% per-run failure rate, three reruns catch the flake about 5.9% of the time. The other 94% of the time the gate prints green, the agent patch merges, and the flake ships with a clean receipt stapled to it.

This post is about that receipt. It gives you a small script, a decision table, and a gate policy for sizing how many reruns a patch actually needs — and for admitting when reruns are the wrong instrument entirely.

What a rerun is actually measuring

A rerun is a sample. One execution of a test is a Bernoulli trial: pass or fail, with some unknown probability p of failing on any given run. Rerunning k times and reporting green means you observed k passes in a row.

The probability that the gate catches a flake at all is:

detection(k, p) = 1 - (1 - p)^k

That is the whole model. It has one honest assumption — independence — and one uncomfortable consequence: the useful quantity is k, and nobody on your team is currently choosing k on purpose. It is chosen by whatever the CI config said in 2021.

The numbers most teams never print

Failure rate p per run	Reruns for 95% detection	Power at 3 reruns
5%	59	14.3%
2%	149	5.9%
1%	299	3.0%
0.5%	598	1.5%
0.1%	2,995	0.3%

Read the middle column slowly. A 1% flake — the kind people describe as "rare, probably infrastructure" — needs roughly 299 reruns to be caught 95% of the time. Three reruns catch it about 3% of the time.

So when a gate goes green after three reruns, the correct statement is not "the test is stable." It is "the test is stable at the resolution this gate can see." Those are different claims, and only one of them is evidence about the patch.

Step 1 — Estimate p, and admit how little you know

You rarely have a clean p. You have a history: 4 failures in 120 runs, say. The naive estimate is 3.3%, but with n = 120 the 95% Wilson interval on that estimate is wide — roughly 1.3% to 8.3%.

That width matters, because the budget is steeply nonlinear in p. Planning against 1.3% means ~230 reruns; planning against 8.3% means ~35. If you plan against the point estimate, you will under-provision the gate for the regimes that hurt.

Use the upper bound of the interval for the merge gate. Use the point estimate only for cost forecasting.

Step 2 — Invert the formula to get a rerun budget

Solving for k:

k = ceil( ln(1 - target) / ln(1 - p) )

With target = 0.95, that gives the middle column above. Two practical notes.

First, this is a budget, not a wish. If the required k exceeds what your wall-clock and runner bill can absorb, the answer is not "rerun more anyway." The answer is that reruns cannot reach the frequency band where this bug lives, and you need a different instrument.

Second, the marginal returns collapse fast. At p = 1%, going from 299 to 309 reruns buys about half a percentage point of detection. You are paying ten full test cycles for 0.5%.

Step 3 — The artifact: flake_budget.py

Standard-library Python 3.9+, no dependencies. It takes either raw rates or an observed fail/run count and prints the budget, the power you would have had one rerun short, and the marginal gain from the last rerun.

#!/usr/bin/env python3
"""flake_budget.py - size a rerun budget before trusting a green agent patch.

  python3 flake_budget.py --target 0.95 --cost 8 --rates 0.05 0.02 0.01 0.005
  python3 flake_budget.py --target 0.95 --cost 8 --observe 1 120
"""
import argparse
import math


def runs_for(p: float, target: float) -> int:
    """Reruns so P(>=1 failure) >= target, assuming independent runs."""
    if p >= 1.0:
        return 1
    if p <= 0.0:
        return -1  # unreachable at any k
    return math.ceil(math.log(1.0 - target) / math.log(1.0 - p))


def power(p: float, k: int) -> float:
    return 0.0 if p <= 0.0 else 1.0 - (1.0 - p) ** k


def wilson(fails: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    ph = fails / n
    d = 1.0 + z * z / n
    c = (ph + z * z / (2 * n)) / d
    h = z * math.sqrt(ph * (1 - ph) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--target", type=float, default=0.95)
    ap.add_argument("--cost", type=float, default=None, help="seconds per run")
    ap.add_argument("--rates", type=float, nargs="*", default=[])
    ap.add_argument("--observe", type=int, nargs=2, metavar=("FAILS", "RUNS"))
    args = ap.parse_args()

    rows = []
    if args.observe:
        fails, n = args.observe
        lo, hi = wilson(fails, n)
        print(f"observed {fails}/{n} -> p in [{lo:.4f}, {hi:.4f}] (95% Wilson)")
        rows.append(hi)  # plan against the conservative bound
    rows.extend(args.rates)
    if not rows:
        ap.error("pass --rates and/or --observe")

    print(f"\ntarget detection power: {args.target:.0%}")
    print(f"{'p(run)':>9} {'reruns':>7} {'power(k-1)':>11} {'gain':>6} {'secs':>8}")
    for p in rows:
        k = runs_for(p, args.target)
        before = power(p, k - 1)
        secs = f"{k * args.cost:.0f}" if args.cost else "-"
        print(f"{p:9.4f} {k:7d} {before:10.1%} {args.target - before:6.1%} {secs:>8}")


if __name__ == "__main__":
    main()

Run it against your own history before touching the CI config:

python3 flake_budget.py --target 0.95 --cost 8 --observe 1 120
# observed 1/120 -> p in [0.0015, 0.0455] (95% Wilson)
# target detection power: 95%
#    p(run)  reruns  power(k-1)   gain     secs
#    0.0455      65        93.9%   1.1%      520

Sixty-five reruns, about 8.7 minutes of runner time, to make a 4.5% flake visible 95% of the time. That may be a defensible price for one suspicious test. It is not a defensible price per agent patch when the agent is producing thirty patches a day.

Step 4 — Put the budget in the merge gate

A policy that survives contact with a real queue:

1. Compute the budget first. Every quarantined test carries its Wilson interval and required k. No number, no quarantine.
2. Compare k to the rerun cap. The cap is a fixed wall-clock number, not a habit. Three is fine as a cap; it is not fine as a justification.
3. If k > cap, refuse the green. Mark the patch unproven-on-flaky-test rather than passed. That is the whole point: the gate stops laundering absence of evidence into evidence of stability.
4. Convert, don't rerun. Turn the flaky path into a deterministic check — a property, an invariant, an injected clock, a locked fixture. A property check has k = 1 by construction and costs the same every time.
5. Escalate with a deadline. If conversion is not possible this week, the test goes to quarantine with an owner and an expiry, and the suite runs it out-of-band rather than gating merges on a coin flip.

Budget k for 95%	Reading	Gate action
≤ 10	cheap to observe	rerun to budget inside the gate
11–60	expensive but real	rerun on a nightly lane, block only on nightlies
61–300	beyond gate resolution	refuse green, open a conversion task
> 300	not a rerun problem	quarantine with expiry; property-check or delete

Where the free tier actually fits

Two parts of this loop are cheap and parallel: drafting the property check that replaces the rerun, and running the nightly sweep that measures p in the first place.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, and those two map onto exactly those two jobs — a model to read the flaky test and the patch diff and propose an invariant worth asserting, and a server to run the out-of-band sweep so the budget measurement does not sit on a developer laptop. Treat both availability claims as operator-supplied and re-verify them against current terms before you wire them into a required check; I have not benchmarked throughput, and a measurement lane that silently slows down is worse than no lane. The honest framing: use the free tier to buy the evidence, not to skip the gate. If you want to try the loop end to end, start with one quarantined test and one property it should have asserted all along.

Limitations worth stating out loud

Independence is doing heavy lifting. Flakes are frequently correlated with seed, test ordering, clock, or network state. Reruns on the same seed do not resample those axes, so the computed power is an upper bound, not a promise. Vary the axis you suspect, or the number lies to you politely.

Small n makes p a rumour. Below roughly 50 observations the interval is so wide that any budget derived from it is a guess with decimal places. Collect first, plan second.

Cost is not only wall-clock. A 300-rerun budget on a 90-second integration test is 7.5 hours per patch. That is a decision about the suite, not about the flake.

Who should not use this

Teams whose suite is small and fast, where a 3-rerun gate genuinely covers the flakes that matter — measure once, and if k ≤ 10 holds, the policy is overhead. Teams whose flakes are deterministic given a seed: the fix is a reproduction harness, not a budget. And anyone tempted to use the budget as a substitute for conversion — the script tells you the price of seeing a bug, never the price of fixing one.