A Freeze With an Expiry: Rate-Based Triage for Flaky Tests After an Agent Patch

Core conclusion: "flaky" is a rate, not a label. Before you quarantine a test that an agent patch just destabilized, run it across a small environment matrix and estimate that rate — then write the quarantine down as a dated, owned liability with a verified replacement check attached.

Most freeze decisions are made from one bad CI log. One log gives you a single bit: it failed. It cannot tell you whether the test fails 3 times in 20 or 19 times in 20, and those two cases need opposite responses.

This post covers the probe, the decision rule, the ledger, and the coverage check that stops a freeze from quietly becoming a hole.

0. The precondition

This workflow assumes you have already established that the failure is not the patch's fault. If you have not done that yet, do attribution first; a freeze applied to a genuine regression is just a regression you chose to ignore.

1. The probe: three axes, sixteen runs

The goal is to sample the same test under conditions the patch may have made load-bearing. Two axes need no plugins at all: hash seed (dict and set ordering) and timezone. A third axis is isolation — run the test alone, then as part of its module — which catches order dependence without pytest-randomly.

# probe_determinism.py — 2 isolation modes x 4 seeds x 2 timezones = 16 runs
import json, os, subprocess, sys, xml.etree.ElementTree as ET
from pathlib import Path

PROBE = Path(".probe"); PROBE.mkdir(exist_ok=True)
SEEDS = ["0", "1", "2", "3"]
ZONES = [None, "Pacific/Kiritimati"]

def run(module, nodeid, seed, tz, tag):
    env = dict(os.environ, PYTHONHASHSEED=seed)
    if tz:
        env["TZ"] = tz
    xml = PROBE / f"{tag}.xml"
    args = [sys.executable, "-m", "pytest", module, "-q", "--junitxml", str(xml)]
    if nodeid:
        args += ["-k", nodeid]  # isolation mode
    subprocess.run(args, env=env, capture_output=True, timeout=600)
    return parse(xml)

def parse(xml):
    out = {}
    for case in ET.parse(xml).getroot().iter("testcase"):
        name = f"{case.get('classname')}::{case.get('name')}"
        bad = any(c.tag in ("failure", "error") for c in case)
        skip = any(c.tag == "skipped" for c in case)
        out[name] = "skip" if skip else ("fail" if bad else "pass")
    return out

if __name__ == "__main__":
    module, nodeid = sys.argv[1], (sys.argv[2] if len(sys.argv) > 2 else None)
    tally = {}
    for isolated in (False, True):
        for seed in SEEDS:
            for tz in ZONES:
                tag = f"{int(isolated)}-{seed}-{tz or 'local'}".replace('/', '_')
                for test, outcome in run(module, nodeid if isolated else None,
                                         seed, tz, tag).items():
                    row = tally.setdefault(test, {"pass": 0, "fail": 0, "skip": 0})
                    row[outcome] += 1
    Path(".probe/tally.json").write_text(json.dumps(tally, indent=2))
    print(json.dumps(tally, indent=2))

Sixteen runs on one module is cheap. Sixteen runs across a suite is not, which is why you scope the probe to the module that failed plus anything it shares a fixture with.

2. Read the rate with an interval

A rate from 20 samples is noisy, so attach a lower bound before you act on it. Wilson's interval behaves sanely at 0 and 1, unlike the normal approximation.

import math
def wilson_lb(k, n, z=1.96):
    if n == 0:
        return 0.0
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    t = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return max(0.0, (c - t) / d)

Plugging n = 20 into that function gives the decision table below. These are arithmetic examples from the formula, not measurements; run the probe to get your own.

Fails / 20	Wilson 95% lower bound	Reading	Action
0	0.00	deterministic pass	Not flaky here — look at the CI log's environment, not the test
1–2	under 0.15	rare, still nondeterministic	Quarantine and attach a replacement check
6	~0.15	genuinely intermittent	The textbook quarantine case
18	~0.70	mostly failing	Do not freeze; this is a broken test or a real defect
20	~0.84	deterministic fail	Patch regression. Revert or fix.

The useful part is the two deterministic rows. An agent patch that introduced a dependence on dict ordering will often show 20/20 fail under seed 1 and 20/20 pass under seed 0 — the per-seed breakdown, not the aggregate, is what names the cause.

3. The ledger is the freeze

A @pytest.mark.skip with a TODO comment is not a freeze. It has no owner, no clock, and no way to fail loudly. Put the freeze in a file the linter reads.

# quarantine.toml
[[quarantine]]
nodeid = "tests/test_billing.py::test_invoice_totals"
owner = "@finley"
opened = "2026-10-20"
expires = "2026-11-19"
reason = "flip 6/20, seed-dependent under PYTHONHASHSEED=1"
covered_by = "tests/properties/test_invoice_invariants.py::test_totals_are_additive"

The covered_by field is the whole point. A quarantine without a replacement assertion is a deletion with extra steps.

# lint_quarantine.py — run in CI; exit non-zero blocks the merge
import datetime, pathlib, subprocess, sys, tomllib

BUDGET = 3  # max open quarantines per top-level test directory

def main():
    ledger = pathlib.Path("quarantine.toml")
    if not ledger.exists():
        return 0
    entries = tomllib.loads(ledger.read_text()).get("quarantine", [])
    today = datetime.date.today()
    errors = []
    per_dir = {}
    for e in entries:
        if datetime.date.fromisoformat(e["expires"]) < today:
            errors.append(f"expired: {e['nodeid']}")
        for field in ("owner", "reason", "covered_by"):
            if not e.get(field):
                errors.append(f"missing {field}: {e['nodeid']}")
        check = subprocess.run(
            [sys.executable, "-m", "pytest", e["covered_by"], "-q"],
            capture_output=True)
        if check.returncode != 0:
            errors.append(f"replacement check failing: {e['covered_by']}")
        d = pathlib.Path(e["nodeid"].split("::")[0]).parts[0]
        per_dir[d] = per_dir.get(d, 0) + 1
    errors += [f"budget exceeded in {d}: {n}" for d, n in per_dir.items() if n > BUDGET]
    print("\n".join(errors) if errors else "quarantine ledger OK")
    return 1 if errors else 0

sys.exit(main())

Two properties matter here. Expiry is a hard stop enforced by CI, not a reminder. And the replacement check must actually pass on the current HEAD — otherwise the freeze is masking a second failure. tomllib requires Python 3.11+; on older interpreters use tomli.

4. Check for the silent zone

Quarantining a test reduces coverage. Sometimes it reduces a file's coverage to zero, and nobody notices because the suite is still green.

Run the suite twice with coverage json, once with the quarantined node IDs deselected and once without, then diff totals.percent_covered per file. Any file the patch touched that loses all covered lines is a silent zone: you now have changed code with no executing assertion at all. Either add an invariant check there or reject the patch.

coverage run -m pytest -q && coverage json -o .cov/with.json
coverage run -m pytest -q --deselect tests/test_billing.py::test_invoice_totals \
  && coverage json -o .cov/without.json
python scripts/diff_coverage.py .cov/with.json .cov/without.json --fail-on-zero

5. Where the repeated runs go

Sixteen runs per module multiplies fast. This is the part of the workflow where MonkeyCode's free model access and free server option are relevant: the probe matrix is embarrassingly parallel and stateless, and drafting a first version of the covered_by property check from the failing test's assertions is a good use of a model that a human then reviews line by line.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two cautions. Free availability is a property of your account today, not a guarantee — verify it before wiring anything into CI, and never make the merge gate depend on an external service. Keep the ledger, the probe script, and the lint in the repository; those are the durable artifacts.

6. Limits, and who should skip this

• Small suites. Under roughly 50 tests, the probe costs more attention than just fixing the flaky test. Fix it.
• Shared external dependencies. If the flakiness lives in a third-party API, the probe will show high flip rates across every axis and tell you nothing. Freezing the test does not freeze the service.
• Security or money-path assertions. Do not quarantine these. Fix them or block the patch.
• Rationale-free freezes. If you cannot write the reason field with a number in it, you do not yet know what you are freezing.

If you want to try the matrix, start with one module and sixteen runs. Scale after you have seen a per-seed breakdown change your mind about a test you were sure was flaky.