A Flake on an Agent Diff Is a Missing Cassette

Do not freeze a flake that sits on an agent diff. Convert it into a cassette, or leave the gate red.

A quarantine list is a human exception with a name and an owner. An agent does not inherit that exception. It will expand the skip set, rewrite the reason, and still look green.

This article is a merge policy, not a testing philosophy. The policy is narrow. If the patch touches a test and that test is unstable across reruns, the missing artifact is a pinned record of time, I/O, and entropy. A skip marker is not an artifact.

The leak, stated as a rule

Freeze lists assume a human will reopen the ticket. Agent patches do not reopen tickets. They optimize the visible gate.

Three failure modes show up on the same PR. The agent marks a racy test xfail. It stubs a clock that the suite still reads from the wall. It rewrites a fixture so the assertion no longer observes the old path. Each change can keep CI green. None of them prove the behavior.

Treat those as the same class of defect: unpinned nondeterminism on an agent-owned diff.

Three lanes, zero inherited skips

Keep the suite in three lanes. Name them in the repo, not in a wiki.

1. Property lane. Assertions that must hold for a generator or a seed range. Failures here are product bugs or generator bugs. They are never skips.
2. Cassette lane. Tests that read a recorded clock, filesystem, network, or RNG stream. The record is hashed. Drift is a diff, not a flake.
3. Merge lane. The intersection of tests the agent touched and tests the gate will rerun. This lane has a flake budget of zero. It does not consult pytest.ini skip lists, @flaky marks, or quarantine YAML.

The merge lane is the whole point. Human patches may still use a short quarantine with an owner. Agent patches may not.

Proposed classifier (unexecuted)

The workflow below is a local gate you can run against any agent PR. It does not claim production metrics. Label it as a proposal until you wire it to your runner.

# proposed: classify tests the agent touched, then refuse inherited skips
git diff --name-only origin/main...HEAD > /tmp/touched.txt
pytest -q --collect-only -q | tee /tmp/collected.txt
python lane_gate.py \
  --touched /tmp/touched.txt \
  --reruns 7 \
  --seed 20260912 \
  --forbid-marks flaky,quarantine,xfail_flake

lane_gate.py is a small classifier. It maps each node id to a file. It keeps tests whose files appear in the diff. It then reruns that subset with a fixed seed. Any mixed pass/fail outcome is a cassette gap, not a freeze candidate.

# lane_gate.py — proposed merge classifier, not a pytest plugin
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys
from pathlib import Path

FORBIDDEN = {"flaky", "quarantine", "xfail_flake"}

def load_lines(p: Path) -> list[str]:
    return [ln.strip() for ln in p.read_text().splitlines() if ln.strip()]

def node_file(nodeid: str) -> str:
    return nodeid.split("::", 1)[0]

def pytest_json(reruns: int, seed: int, expr: str) -> dict:
    cmd = [
        "pytest", "-q", "--tb=no",
        f"--randomly-seed={seed}",
        f"-k", expr or "not _disabled_",
        "--json-report", "--json-report-file=/tmp/report.json",
    ]
    # Reruns are sequential on purpose. Parallel reruns hide order bugs.
    reports = []
    for i in range(reruns):
        subprocess.run(cmd, check=False)
        reports.append(json.loads(Path("/tmp/report.json").read_text()))
    return {"runs": reports}

def classify(reports: dict, touched: set[str]) -> dict:
    outcomes: dict[str, set[str]] = {}
    for run in reports["runs"]:
        for t in run.get("tests", []):
            node = t["nodeid"]
            if node_file(node) not in touched:
                continue
            outcomes.setdefault(node, set()).add(t["outcome"])
    mixed = sorted(n for n, o in outcomes.items() if len(o) > 1)
    failed = sorted(n for n, o in outcomes.items() if o <= {"failed", "error"})
    passed = sorted(n for n, o in outcomes.items() if o == {"passed"})
    return {"mixed": mixed, "failed": failed, "passed": passed}

def cassette_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--touched", type=Path, required=True)
    ap.add_argument("--reruns", type=int, default=7)
    ap.add_argument("--seed", type=int, required=True)
    ap.add_argument("--cassettes", type=Path, default=Path("cassettes"))
    ap.add_argument("--forbid-marks", default="")
    args = ap.parse_args()

    touched = {p for p in load_lines(args.touched) if p.endswith(".py")}
    marks = {m.strip() for m in args.forbid_marks.split(",") if m.strip()}
    if marks & FORBIDDEN != marks:
        # Still enforce the built-in set even if the flag is narrowed.
        marks |= FORBIDDEN

    # Proposed: fail closed if the agent added a forbidden mark.
    diff = subprocess.check_output(["git", "diff", "origin/main...HEAD"], text=True)
    for mark in marks:
        if f"@pytest.mark.{mark}" in diff or f"pytest.mark.{mark}" in diff:
            print(f"blocked: agent diff introduces mark {mark}")
            return 2

    reports = pytest_json(args.reruns, args.seed, "")
    lanes = classify(reports, touched)
    Path("/tmp/lanes.json").write_text(json.dumps(lanes, indent=2))

    if lanes["mixed"]:
        print("blocked: mixed outcomes on agent-touched tests")
        for n in lanes["mixed"]:
            print(f"  cassette-gap {n}")
        return 3

    if args.cassettes.exists():
        ledger = {
            str(p): cassette_hash(p)
            for p in sorted(args.cassettes.rglob("*.json"))
        }
        Path("/tmp/cassette-ledger.json").write_text(json.dumps(ledger, indent=2))

    print(json.dumps({"ok": True, "passed": len(lanes["passed"]), "failed": len(lanes["failed"])}))
    return 0 if not lanes["failed"] else 1

if __name__ == "__main__":
    sys.exit(main())

The script is deliberately boring. It does not parse skip reasons. It does not expire a freeze. Mixed outcomes print as cassette-gap. That string is the review comment you want on the PR.

What a cassette must pin

A cassette is not a golden screenshot of the full process. Pin only the sources of nondeterminism the test actually read.

{
  "schema": "cassette.v1",
  "test": "tests/test_invoice.py::test_overdue_notice",
  "clock": "2026-09-12T00:00:00+00:00",
  "rng_seed": 20260912,
  "fs": {
    "/tmp/invoice/outbox": ["n-100.json", "n-101.json"]
  },
  "http": [
    {"method": "GET", "path": "/rates/usd", "status": 200, "body_sha256": "9c1d…"}
  ]
}

Store one file per test node. Hash the file in CI. If the agent rewrites the cassette, the ledger diff must explain which pin moved and why. A pin that disappears is a failed merge, even when the test passes.

Clock, RNG, filesystem listings, and HTTP bodies cover most Python service tests. Threads and wall-sleep do not belong in this lane until you have a deterministic scheduler. If you cannot pin it, it does not run on the merge lane.

Convert the leftover flake into a property

Some mixed outcomes are not I/O. They are hidden ranges: sort stability, retry counts, map iteration, floating windows.

Do not freeze those either. Write a property over the decoded cassette or over a generator the human owns. Keep the generator in a file the agent cannot edit on the same PR as production code. That split is the oracle. The test file can call the generator. It cannot redefine it.

# properties/overdue.py — human-owned, proposed
from datetime import datetime, timedelta, timezone

def overdue_windows(clock: datetime):
    # Inclusive lower bound, exclusive upper bound. No wall clock.
    start = clock.replace(tzinfo=timezone.utc)
    yield start, start + timedelta(days=1)
    yield start - timedelta(days=30), start

def notice_is_idempotent(send_notice, invoice_id: str, clock: datetime) -> None:
    a = send_notice(invoice_id, clock=clock)
    b = send_notice(invoice_id, clock=clock)
    assert a == b

Run properties with the same seed the classifier used. Seed drift across tools is another cassette gap. One integer, one file, one gate.

Where a free model and a spare server help

Generating cassette sketches by hand is slow when the agent touched twenty tests. A spare box is useful for the rerun loop because seven sequential pytest passes do not belong on a shared runner that bills per minute.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is enough to draft cassette candidates from a failing trace, and the free server option is enough to host lane_gate.py away from production CI. Treat both as a scratch space. Do not send customer fixtures there. Do not treat a generated cassette as merged until a human accepts the pins.

A useful prompt to the model is not "fix the flake." Ask it to list every unpinned source the traceback touched. Then you write the JSON. The model does not own the ledger.

Decision table

Symptom on agent-touched tests	Inherit freeze list?	Required artifact	Merge
Stable pass across N reruns, no new skip marks	No	Unchanged cassette hashes	Allow
Mixed pass/fail, clock or HTTP in traceback	No	New cassette pin + ledger diff	Allow only with pin
Mixed pass/fail, no I/O, order or retry variance	No	Human-owned property + fixed seed	Allow only with property
New @flaky / quarantine / xfail on touched tests	No	None — revert the mark	Block
Cassette file deleted or hash changed with no note	No	Ledger explanation	Block
Failure identical on every rerun	No	Product fix or oracle change	Block until red is explained

N is a local constant. Seven is a starting point for short unit tests. Long integration tests should reduce N and pin more, not freeze more.

Limitations

This gate does not make concurrent code deterministic. If the bug is a data race, a cassette of HTTP bodies will not save you. You need a thread sanitizer or a deterministic scheduler. Those are different tools.

It also does not replace hidden tests. The merge lane only sees tests the agent touched. Untouched flakes can still burn nightly CI. Keep a human quarantine for those if you must. Do not copy it onto the agent path.

Generated cassette sketches will over-pin. They will freeze response fields that should remain properties. That is why the ledger is reviewed as code. A pin on request_id is usually a mistake. A pin on status is usually not.

The classifier as written depends on pytest-json-report and pytest-randomly. If you do not want those plugins, emit JUnit and hash it yourself. The policy does not depend on the plugin names.

Who should not use this

Do not use a zero-inheritance freeze policy if your suite is already a skip list with no owners. You will block every agent PR and then disable the gate. Fix the suite first.

Do not use a public or shared scratch server for cassettes that contain customer payloads, auth headers, or production URLs. Redact before any off-CI rerun.

Do not use this as an auto-merge token. A green classifier means the agent did not hide a flake in the touched set. It does not mean the patch is correct. Correctness still needs an oracle the agent cannot edit on the same PR.

If your tests are browser-level and the flake is layout or font loading, pin a different artifact. This cassette schema is for service I/O. Visual tests need image hashes or accessibility trees, and those deserve their own ledger.

Close the loop on the next PR

Run the classifier on the next agent patch before you add another line to the freeze file. If the output says cassette-gap, pin time and I/O. If you cannot pin them, keep the test red.