Quarantine Is Not a Verdict: A Flake-Freeze Ledger for Agent Patch Gates

Conclusion first: a green test run on an agent patch is worth exactly as much as the set of tests you allowed to speak. Most teams quarantine flaky tests to keep CI tolerable, then quietly read the remaining green as proof of correctness. It is not proof. A quarantine list turns your gate into a filter with holes, and almost nobody records where the holes are.

This post lays out a three-layer strategy for patch review when the author is an agent: probe determinism on the base tree before judging the patch, freeze flaky tests in a ledger that expires, and refuse to issue PASS when the patch's changed lines are covered only by frozen tests. The code below is a reference implementation, not a benchmark result — I have not published timings for it, and you should not trust any you see without reproducing them yourself.

Why flakiness hurts agent patches more than human patches

A human author who hits a flaky test reruns it, mutters, and moves on. The flake costs them a minute.

An agent author does something worse: it optimizes against the signal. It sees red, edits, sees red again, and keeps editing until the signal turns green. Nothing in that loop distinguishes "my fix works" from "I made the test stop complaining." A test that fails one run in five is a perfect teacher for retries, sleeps, and loosened assertions.

That is the mechanism worth internalizing. Flakiness does not stay flakiness. Under an agent it converts into permanent weakening of the test suite, and the patch that does the weakening reviews as small and reasonable.

Step 1 — Measure determinism on the base tree, not the patch

Never judge a patched tree with a suite you have not characterized. Run the unpatched base K times against identical inputs and record per-test failure counts. Anything that is not 0/K or K/K is unstable and must not be used as evidence.

# flake_probe.py — characterize the suite before any patch is evaluated
import json, subprocess, sys, collections

K = int(sys.argv[1]) if len(sys.argv) > 1 else 7
TEST_CMD = ["python", "-m", "pytest", "-q", "--tb=no", "-p", "no:randomly"]

def run_once():
    proc = subprocess.run(TEST_CMD, capture_output=True, text=True)
    failed = set()
    for line in proc.stdout.splitlines():
        if line.startswith("FAILED"):
            failed.add(line.split(" ", 1)[1].split(" -")[0].strip())
    return failed

outcomes = collections.Counter()
for _ in range(K):
    for nodeid in run_once():
        outcomes[nodeid] += 1

ledger = []
for nodeid, failures in sorted(outcomes.items()):
    if 0 < failures < K:
        ledger.append({"test": nodeid, "failures": failures, "runs": K})

print(json.dumps({"k": K, "unstable": ledger}, indent=2))

Two caveats you should accept up front. First, K=7 gives you roughly 1/K resolution: a test that fails once in twenty runs will usually look clean. Second, a stable-looking test is only stable under the inputs you gave it — time, network, and filesystem order all hide outside the probe.

Step 2 — Freeze with an expiry date and an owner

A quarantine list without expiry is a permanent license to ignore a test. The ledger below forces every exclusion to be revisited, and forces somebody's name onto it.

{
  "frozen": [
    {
      "test": "tests/test_billing.py::test_retry_backoff",
      "observed": "3/7 fail on clean base",
      "reason": "wall-clock dependency in backoff assertion",
      "owner": "billing-oncall",
      "frozen_at": "2026-11-09",
      "expires": "2026-12-09"
    }
  ]
}

Enforce one rule mechanically: an expired entry is not silently dropped and not silently honored. It converts the gate verdict to INCONCLUSIVE until the test is either re-probed or actually fixed. That single rule is what keeps the ledger from becoming a graveyard.

Step 3 — Attribute changed lines to tests, and veto vacuous passes

The step most gates skip. If a patch touches refund.py, and the only tests that exercise refund.py are frozen, then a green run is not evidence — it is silence. You need per-test line coverage to detect that.

pytest --cov=src --cov-context=test -q
coverage json --show-contexts -o cov.json

With contexts enabled, each covered line carries the list of test node IDs that hit it. Cross-reference the patch's changed lines against that map and against the freeze ledger:

# attribution_veto.py — refuse PASS when the only witnesses are frozen
def veto(changed_lines, coverage, frozen):
    witnesses = set()
    for path, lines in changed_lines.items():
        for line in lines:
            witnesses |= set(coverage.get(path, {}).get(str(line), []))
    if not witnesses:
        return "INCONCLUSIVE", "no test covers the changed lines"
    if witnesses <= frozen:
        return "INCONCLUSIVE", "changed lines covered only by frozen tests"
    if frozen & witnesses:
        return "PASS_WITH_CAVEAT", f"{len(frozen & witnesses)} frozen tests cover changed lines"
    return "PASS", "all witnesses are stable"

Note the middle branch. Partial overlap is the common case, and it deserves its own label rather than being rounded up to PASS.

Step 4 — Put property checks where example tests are frozen

Freezing a test does not have to mean losing the behavior it half-checked. Convert the assertion into a seeded property so runs become reproducible by construction.

import random

def test_invoice_total_monotone():
    rng = random.Random(20261109)  # fixed seed: failures are replayable
    for _ in range(200):
        lines = [rng.randint(0, 10_000) for _ in range(rng.randint(0, 8))]
        assert total(lines + [1]) >= total(lines)

A fixed seed buys you reproducibility, not coverage. Rotate the seed on a schedule and keep the failing seed in the report so the next person can replay it exactly.

The decision table

Probe (base tree)	Attribution	Verdict
Patch touched lines have stable witnesses	full	PASS
Some witnesses frozen, some stable	partial	PASS_WITH_CAVEAT
All witnesses frozen	none stable	INCONCLUSIVE
No witness covers changed lines	uncovered	INCONCLUSIVE
Any freeze entry past expires	n/a	INCONCLUSIVE

INCONCLUSIVE must block merge the same way FAIL does. If it doesn't, the whole ledger is decoration.

Running the probe loop without owning the machine

Step 1 multiplies suite runtime by K, which is where this gets expensive. Repeated full-suite runs are exactly the workload that does not need your laptop.

One option is MonkeyCode's free model access and free server option: draft property checks from the diff with the free models, then run the K-run probe loop on the free server instead of locally. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check the project page for the current free allowance and server terms before you plan capacity around them — availability claims change, and you should verify rather than take my word for it.

Keep the gate logic itself in your repository. A verdict that depends on someone else's free tier is not a verdict.

Limitations, and who should not use this

• K runs bound, they do not prove. At K=7 you will miss low-rate flakes. Raise K for suites where a single false green is expensive.
• Coverage contexts cost memory and time. On large monorepos, expect the cov.json file itself to become the bottleneck.
• Wall-clock and network flakiness survives. The probe characterizes the environment you ran it in, nothing more.
• Free-tier compute has no SLA. If your merge gate must be attested by a runner you control, do not route it through an external free service.
• Not for you if your suite already exceeds ~10 minutes, since K repetitions make that unusable, or if you cannot enable per-test coverage contexts at all.

If you take one thing: the freeze list is a statement about evidence, not about inconvenience. Every entry should name who is blind because of it, and when they stop being blind.