Retries Are Not Evidence: A Sample-Size Gate for Agent Patches

Two of twenty reruns passed before the patch. Three of twenty passed after it. That is not a fix — it is the same coin landing on a different face.

Most agent-patch reviews still treat a green retry as proof. It is not proof of anything until you know how many trials produced it and how wide the uncertainty is. This article builds a small, reproducible gate for that: a trial ledger, Wilson intervals, a decision table, and a retry budget that stops the suite from manufacturing false confidence.

The conclusion first

A flaky test can only license a patch verdict when the two arms of the experiment — baseline and patched — are statistically separable at the sample size you actually ran. In my sizing table below, 20 trials per arm cannot separate a 50% pass rate from 85%, and cannot separate 90% from 95% at all. Sixty trials per arm separates the first pair and still struggles near the ceiling.

So the gate has three parts, and they are not interchangeable: the ledger decides whether a result is usable, the property check decides whether the patch is correct, and the budget decides when to stop paying for reruns.

Why a retry pass is mostly noise

Model a flaky test as a Bernoulli trial with pass probability p. If p = 0.5, then the chance that two retries both pass is 0.25. A quarter of unpatched runs look green on the second attempt. Nothing changed; you just sampled twice.

Retries convert a noisy signal into a clean-looking one, which is exactly the failure mode you want to avoid in automated merge gates. The fix is not more retries. It is recording every trial and refusing to attribute an outcome the data cannot support.

Step 1 — Record trials, not verdicts

Put the recorder in a plugin file that lives outside the tree the agent is allowed to edit. If the patched tree can rewrite the recorder, the ledger is evidence the patch produced about itself.

# flake_ledger.py — pytest plugin, kept outside the patched tree
import json, os, time, uuid

LEDGER = os.environ.get("FLAKE_LEDGER", "artifacts/flake.jsonl")
ARM = os.environ.get("FLAKE_ARM", "unset")        # "baseline" or "patched"
RUN = os.environ.get("FLAKE_RUN") or str(uuid.uuid4())[:8]

def pytest_runtest_logreport(report):
    if report.when != "call" and not report.failed:
        return
    row = {
        "run": RUN,
        "arm": ARM,
        "nodeid": report.nodeid,
        "when": report.when,
        "outcome": report.outcome,
        "ts": round(time.time(), 3),
    }
    os.makedirs(os.path.dirname(LEDGER), exist_ok=True)
    with open(LEDGER, "a") as fh:
        fh.write(json.dumps(row) + "\n")

Run each arm as a fixed number of independent pytest invocations, not one invocation with retries. Retries inside a single process share state — caches, ports, module-level fixtures — and correlated trials make every interval below too narrow.

for i in $(seq 1 60); do
  FLAKE_ARM=baseline FLAKE_RUN="b$i" pytest tests/test_retry_budget.py -p flake_ledger -q
done
for i in $(seq 1 60); do
  FLAKE_ARM=patched  FLAKE_RUN="p$i" pytest tests/test_retry_budget.py -p flake_ledger -q
done

Step 2 — Put an interval on the rate

A bare percentage hides its own sample size. A Wilson score interval does not, and it behaves sanely at 0 and 1 where the naive normal approximation collapses.

# analyze.py
import collections, json
from math import sqrt

Z = 1.959963985  # 95%

def wilson(k, n, z=Z):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = (z / denom) * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, centre - half), min(1.0, centre + half))

def load(path):
    runs = collections.defaultdict(dict)
    for line in open(path):
        if not line.strip():
            continue
        r = json.loads(line)
        if r["when"] == "call":
            runs[(r["arm"], r["run"])][r["nodeid"]] = r["outcome"]
    return runs

def rates(path, nodeid):
    per_arm = {"baseline": [0, 0], "patched": [0, 0]}
    for (arm, _run), seen in load(path).items():
        if nodeid not in seen:
            continue
        per_arm[arm][1] += 1
        per_arm[arm][0] += (seen[nodeid] == "passed")
    return {a: (k, n) for a, (k, n) in per_arm.items()}

Two computed examples, straight from wilson: ten passes out of twenty is (0.30, 0.70). Seventeen out of twenty is (0.64, 0.95). Those intervals overlap, so the experiment says nothing.

Step 3 — Decide with a table, not a feeling

def decide(k0, n0, k1, n1, min_trials=20):
    lo0, hi0 = wilson(k0, n0)
    lo1, hi1 = wilson(k1, n1)
    if n0 < min_trials or n1 < min_trials:
        return "underpowered", (lo0, hi0), (lo1, hi1)
    if hi0 < lo1:
        return "improved", (lo0, hi0), (lo1, hi1)
    if hi1 < lo0:
        return "regressed", (lo0, hi0), (lo1, hi1)
    return "no-detectable-change", (lo0, hi0), (lo1, hi1)

Verdict	Meaning	Gate action
underpowered	Fewer than 20 trials in an arm	Quarantine the test; block patch attribution
no-detectable-change	Intervals overlap	Patch may merge on other evidence, but cannot claim a flake fix
regressed	Patched rate is lower, separated	Block; the patch made the test less reliable
improved	Patched rate is higher, separated	Accept only if the property check also passes

The last row matters most. Statistical separation tells you the rate moved. It does not tell you why. An agent can raise a pass rate by loosening an assertion, and the interval will happily certify it.

Step 4 — Size the run before you start it

Baseline (k/n)	Patched (k/n)	n per arm	95% intervals	Separable?
10/20 (0.50)	17/20 (0.85)	20	[0.30, 0.70] / [0.64, 0.95]	No
30/60 (0.50)	51/60 (0.85)	60	[0.38, 0.62] / [0.74, 0.92]	Yes
18/20 (0.90)	19/20 (0.95)	20	[0.70, 0.97] / [0.76, 0.99]	No

Two lessons are visible in that table. First, doubling from 20 to 60 trials per arm is what turns a hunch into a finding. Second, near the ceiling the interval is squeezed by the boundary at 1.0, so distinguishing 90% from 95% needs far more trials than most suites can afford — which is a reason to say so out loud rather than to pretend.

Step 5 — Freeze the rate as a record with a budget

A freeze here is not a skip decorator and not an expiry lease. It is a stored claim: this test has rate p, measured over n trials on date t, with interval [lo, hi]. When a later run produces a rate outside that interval at a comparable n, the freeze is stale and the test goes back into measurement.

Enforce a hard cap so a broken harness cannot run forever:

# add to flake_ledger.py
BUDGET = int(os.environ.get("FLAKE_BUDGET", "2000"))
_count = 0

def pytest_runtest_logstart(nodeid, location):
    global _count
    _count += 1
    if _count > BUDGET:
        import pytest
        pytest.exit(f"flake budget exhausted at {BUDGET} trials", returncode=3)

Keep nodeid stable for the frozen suite. Renaming a test or adding a parametrization splits the ledger and silently resets the sample size, which is the quietest way to lose a measurement you already paid for.

Step 6 — Pair the rate with a property

The property check must be deterministic and must not depend on the flaky test. Sketch, with make_plan/run_plan standing in for your code:

def test_attempts_respect_budget_and_terminate():
    for seed in range(500):
        import random
        rng = random.Random(seed)
        plan = make_plan(rng)
        result = run_plan(plan)
        assert 1 <= result.attempts <= plan.budget

This runs 500 seeded cases in a fixed order and fails the same way every time. If an agent patch raises the flaky test's pass rate while breaking this invariant, the gate holds: the rate improved, the property regressed, and the patch stays out.

Where the two-arm run actually executes

Sixty trials per arm on a slow integration test is a long wall-clock job, and a laptop that sleeps mid-run corrupts the sample. This is the one place where tooling genuinely participates in the method rather than decorating it. I used MonkeyCode's free model access to draft and review the analyzer and to triage the ledger into categories — same test, renamed test, parametrized sibling — which is tedious string work rather than reasoning. The two-arm loop ran on MonkeyCode's free server option so the trial count did not depend on my machine staying awake.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the availability claims as the operator's, not as benchmarks. Free tiers change, throughput depends on your suite, and a shared machine adds variance you should measure rather than assume. Before trusting any hosted run, compare one arm's measured pass rate against the same test on your laptop; a large gap means the environment is an input you forgot to model.

Limitations, and who should not do this

• Wilson assumes independent trials. Mitigate with separate processes, randomized test order, and fresh temporary directories; correlated trials still bias the interval, just less obviously.
• Low-frequency tests stay underpowered forever. A test that runs once per pull request will not reach 60 trials in any useful window. Either batch it into a nightly arm or accept that you cannot attribute it.
• Slow tests make the math expensive. At four minutes per trial, 120 trials is eight hours. Compute the cost before committing to the table.
• Deterministic failures are out of scope. If a test fails at the same seed every time, this gate adds nothing; fix the test.
• Skip this if you have no historical outcome data. Retrofit the ledger first and collect for two weeks. A decision table with no rows is theater.
• Skip it if you need a merge decision in the next hour. This is a nightly or per-release gate, not an inline check.

The honest summary: retries buy you a cleaner-looking dashboard, not better evidence. If you adopt one thing from this, run the sizing table before the experiment — the trial count is the budget, and it should never be an afterthought.