Quarantine Is a Loan, Not a Fix: Attributing Agent-Patch Flakes Before You Freeze Them

A quarantined test is a loan against your future coverage, and most teams never write down the interest rate.

When an agent patch turns a test red and you freeze that test, you have not fixed anything. You have bought silence. The silence has a price: every later patch that touches the code the frozen test used to guard now ships unverified, and nobody notices because the suite is green.

This post gives three artifacts to make that loan explicit: a four-cell reproduction matrix that attributes the flake before you freeze it, a ledger entry that records what the freeze costs, and a CI gate that stops patches from spending coverage you no longer have.

Why agent patches hit this wall first

A human-authored flaky test is discovered slowly. One engineer sees it fail on a Tuesday, reruns it, shrugs, and moves on. An agent patch loop compresses that timeline: patches arrive faster than anyone triages red builds, so the cheapest local action is always to mark the test as flaky and continue.

That action is rational in the moment and corrosive over a month. Each freeze removes an assertion from the effective suite while the reported test count stays flat. You end up with a suite that looks complete and verifies a shrinking fraction of the diff.

The fix is not to ban freezing. It is to require an attribution and a replacement before the freeze is accepted.

Step 1: capture a transcript you can actually compare

Before running anything, pin the failure. A test name alone is not evidence; you need the node id, the seed, and the last line of output.

# Reproduce the red build with a fixed seed and a short traceback.
pytest -q --no-header -p randomly --randomly-seed=7 \
  tests/test_orders.py::test_settles_invoice \
  --tb=short 2>&1 | tee /tmp/flake-$(date +%s).log

If the test passes on the retry, you have a flake, not a failure. If it fails every time, stop here: that is a broken test, and a broken test should be fixed or deleted, never quarantined.

Step 2: run the four-cell reproduction matrix

The question you are answering is not is it flaky. It is what does the flake depend on. Four environments separate the common causes:

1. solo — the node alone, ordering plugin disabled. Tests isolation.
2. order — the scoped suite under several random seeds. Tests ordering sensitivity.
3. isolate — the node in a fresh process, so no prior test can leak state.
4. clock — a fixed timezone and a frozen wall clock. Tests time dependence.

#!/usr/bin/env python3
# flake_matrix.py - attribute a flaky pytest node before quarantining it.
import json, os, subprocess, sys

NODE = sys.argv[1]                  # tests/test_orders.py::test_settles_invoice
SCOPE = os.environ.get('SCOPE', 'tests')   # narrow this if the suite is slow
RUNS = int(os.environ.get('RUNS', '5'))


def run(extra_args, extra_env=None):
    env = {**os.environ, **(extra_env or {})}
    proc = subprocess.run(
        [sys.executable, '-m', 'pytest', '-q', '--no-header', *extra_args],
        capture_output=True, text=True, env=env,
    )
    tail = (proc.stdout or proc.stderr).strip().splitlines()[-1:]
    return {'rc': proc.returncode, 'tail': tail[0] if tail else ''}


def plugin_missing(cell):
    blob = (cell['tail'] or '').lower()
    return 'unrecognized arguments' in blob or 'no plugin' in blob


report = {}
report['solo'] = run([NODE, '-p', 'no:randomly'])
report['order'] = [
    run([SCOPE, '-p', 'randomly', '--randomly-seed=%d' % (seed + 1)])
    for seed in range(RUNS)
]
report['isolate'] = run([NODE, '-p', 'no:randomly', '--forked'])
report['clock'] = run([NODE, '-p', 'no:randomly'],
                      {'TZ': 'UTC', 'FREEZETIME': '2026-01-01T00:00:00Z'})

# An unsupported cell must report as unknown, never as a silent pass.
for name in ('isolate', 'clock'):
    if plugin_missing(report[name]):
        report[name] = {'rc': None, 'tail': 'unsupported: plugin not installed'}

print(json.dumps(report, indent=2))

--forked needs pytest-forked, and the clock cell only means something if your suite reads time through a freezable seam. Both cells degrade to rc: null rather than pretending to pass. That distinction is the whole point: an untested cell is not a green cell.

Step 3: read the matrix as a verdict, not a vibe

solo	order	isolate	clock	attribution	first move
pass	fail on any seed	pass	pass	ordering or shared state	reset shared fixtures; run that module under --forked in CI
pass	pass	fail	pass	process-level state	hunt the session or autouse fixture that mutates across tests
pass	fail on all seeds	fail	pass	leakage from earlier tests	add teardown; assert the invariant on entry, not just on exit
pass	pass	pass	fail	clock or timezone	inject a clock object; freeze it inside the test
pass	pass	pass	pass	not reproduced	raise RUNS to 20 before calling it flaky at all
fail	—	—	—	broken test	fix or delete; never quarantine

The fourth row is the one that costs teams the most. An order-dependent test and a state-leaking test look identical in CI, and they need different fixes. Quarantining both with the note flaky guarantees neither gets fixed.

Step 4: write the loan terms into a ledger

A freeze without a recorded obligation is permanent by default. Every quarantined test gets an entry that names the invariant it was protecting, a substitute check that protects it now, and the files it used to guard.

# quarantine.toml - one entry per frozen test, reviewed on expiry.
[[quarantine]]
test        = 'tests/test_orders.py::test_settles_invoice'
owner       = '@finley'
opened      = '2026-10-28'
expires     = '2026-11-11'          # 14 days, not 'until fixed'
attribution = 'order'                # from the matrix, not from memory
obligation  = 'invoice total equals sum of line items after discount'
substitute  = 'tests/properties/test_invoice_totals.py::test_total_is_additive'
guards      = ['src/billing/invoice.py', 'src/billing/discounts.py']
replay      = 'flake_matrix.py tests/test_orders.py::test_settles_invoice'

The substitute field is mandatory and it is not an issue link. A property check, an invariant assertion, or a narrower deterministic test counts. A vague promise to look into it does not.

Step 5: gate the diff, not just the clock

Expiry dates are easy to add and easy to ignore. The gate below enforces three rules, and the third one is the interesting one: a patch may not modify a file that a currently quarantined test was guarding.

# gate_quarantine.py - fail CI on an unpaid quarantine loan.
import datetime, pathlib, subprocess, sys, tomllib

ledger = tomllib.loads(pathlib.Path('quarantine.toml').read_text())
today = datetime.date.today()
changed = set(subprocess.run(
    ['git', 'diff', '--name-only', 'origin/main...'],
    capture_output=True, text=True, check=True,
).stdout.split())

problems = []
for entry in ledger.get('quarantine', []):
    if not entry.get('substitute'):
        problems.append(f"{entry['test']}: no substitute check")
    if datetime.date.fromisoformat(entry['expires']) < today:
        problems.append(f"{entry['test']}: expired on {entry['expires']}")
    touched = changed & set(entry.get('guards', []))
    if touched:
        problems.append(
            f"{entry['test']} is quarantined but the patch edits "
            f"{sorted(touched)}; restore the test or extend the substitute"
        )

if problems:
    print('\n'.join(problems))
    sys.exit(1)

print(f"quarantine ledger ok ({len(ledger.get('quarantine', []))} entries)")

Run it in the same job that runs the suite. It takes a second. It does not attempt to judge whether the patch is correct — it only refuses to let a patch hide behind a test that is currently asleep.

The gate is deliberately strict. Sometimes the right answer is to unquarantine the test on the spot, which is exactly the human decision you wanted to force.

Where a free tier genuinely fits

Two steps in this loop are repetitive and low-judgment, which makes them reasonable to offload.

First, drafting the substitute. Feed the failure transcript and the function under test to a model and ask for candidate invariants, not for a fix. You get a reviewable list of properties — additive totals, monotonic timestamps, idempotent retries — that a human accepts or rejects. The drafting is the slow part; the reviewing is the part worth your attention.

Second, the replay loop. Quarantine entries rot because nobody reruns the frozen test under multiple seeds. A small scheduled job that replays each entry and appends the seed and outcome to the ledger turns expires from a date someone guessed into a decision backed by runs.

MonkeyCode's free model access covers the first, and its free server option covers the second, without a spend request for either. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use it for exactly the two jobs above and not as a substitute for the attribution matrix, because a model cannot tell you whether your fixture leaks state.

Limitations and who should skip this

Calibrate the matrix before trusting it. Five seeds will miss a flake that reproduces once in forty runs; treat the fifth row of the verdict table as unreproduced, not as clean. If the clock cell reports unsupported, you have learned nothing about time dependence, and you should say so in the ledger entry.

Generated property candidates have their own failure mode: a plausible-looking assertion that is true by construction verifies nothing. Every substitute needs one negative check — mutate the value, confirm the property goes red. An unreviewed substitute is a worse loan than the freeze it replaced.

Skip this whole approach if any of the following is true.

• The suite is under roughly fifty tests. Fix the fixtures directly; the ledger is more overhead than the problem.
• You have no CI job that runs on every patch. A quarantine gate without a gate is a text file.
• More than a handful of tests are already frozen. You have a systemic fixture problem, and adding process to it will just document the decay.

Rollout checklist

1. Run flake_matrix.py on the current red node and paste the JSON into the ledger entry.
2. Write obligation as a sentence about behaviour, not about the test name.
3. Name a substitute you can point at today. If you cannot, the freeze is not ready.
4. Set guards to the files the test imports or asserts against.
5. Wire gate_quarantine.py into the existing test job and fail the build.
6. Review every entry on expiry; extend only with new replay evidence in the entry.

If you try this on a live agent-patch loop, start with one quarantined test and let the diff gate run in warn-only mode for a week. The number of patches it would have blocked is the number you were shipping unverified, and that number is the only argument you need for keeping it.