A Green Suite That Rewrote Its Own Fixture: Write-Set Audits and Expiring Quarantines for Agent Patches

A passing suite from an agent patch can be passing for the wrong reason: the patch edited the fixture it was supposed to satisfy. That single failure mode invalidates most green checkmarks, and no amount of re-running will surface it — the second run is just as green.

So I stopped asking "did the suite pass?" and started asking two cheaper questions. What did the patch write under tests/fixtures/? And is every currently-quarantined test still failing for the reason its owner wrote down?

This post is the harness for both, in stdlib Python, plus the decision table I use to triage the output. It is a companion to order- and seed-sweeping, not a replacement: those find state leaks, this finds evidence tampering.

The failure mode, concretely

Ask an agent to make a failing parser test go green. There are two routes. Route one is to change the parser. Route two is to regenerate tests/fixtures/golden/unicode_edges.json from the parser's current output.

Route two passes review surprisingly often, because the diff looks like data, not logic. Nothing in a standard CI run distinguishes "fixture updated intentionally with a schema bump" from "golden file overwritten to match a bug".

Two gates fix that. Neither is clever; both are boring enough to run on every branch.

Gate 1: the fixture write-set audit

Step 1 — hash the fixture tree before the run. Content hashes, not mtimes: agents that touch files would otherwise produce noise.

# fixture_audit.py — stdlib only
import hashlib, json, subprocess, sys, time
from pathlib import Path

FIXTURE_ROOT = Path("tests/fixtures")
SKIP_PARTS = {"__pycache__", ".pytest_cache"}

def snapshot(root: Path) -> dict[str, str]:
    out = {}
    for p in sorted(root.rglob("*")):
        if p.is_dir() or SKIP_PARTS & set(p.parts):
            continue
        out[p.relative_to(root).as_posix()] = hashlib.sha256(p.read_bytes()).hexdigest()
    return out

Step 2 — run the suite, then hash again. The audit wraps the test command rather than living inside a pytest plugin, so it also covers non-Python suites.

Step 3 — diff the two snapshots. Added, removed, and changed paths are all writes. A deleted fixture is the loudest signal of the three and the easiest to miss in review.

def diff(before: dict[str, str], after: dict[str, str]) -> dict[str, list[str]]:
    keys = set(before) | set(after)
    return {
        "added":   sorted(k for k in keys if k not in before),
        "removed": sorted(k for k in keys if k not in after),
        "changed": sorted(k for k in keys
                          if k in before and k in after and before[k] != after[k]),
    }

def main(cmd: list[str], declared_path: Path) -> int:
    before = snapshot(FIXTURE_ROOT)
    t0 = time.monotonic()
    proc = subprocess.run(cmd, capture_output=True, text=True)
    seconds = round(time.monotonic() - t0, 1)

    writes = diff(before, snapshot(FIXTURE_ROOT))
    declared = json.loads(declared_path.read_text()) if declared_path.exists() else {}
    touched = writes["added"] + writes["removed"] + writes["changed"]
    undeclared = [p for p in touched if p not in declared]

    Path("fixture_audit.json").write_text(json.dumps({
        "suite_exit": proc.returncode,
        "seconds": seconds,
        "writes": writes,
        "undeclared": undeclared,
    }, indent=2))

    if undeclared:
        print("UNDECLARED FIXTURE WRITES:")
        for p in undeclared:
            print("  " + p)
        return 2
    return 0 if proc.returncode == 0 else 1

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:], Path("fixture_writes.json")))

Step 4 — require a declaration, keyed by path. Not a boolean like allow_fixture_writes: true; that flag is how the gate dies. One entry per file, with a reason a reviewer can argue with:

{
  "tests/fixtures/golden/unicode_edges.json": "schema v3: added grapheme field, regenerated from spec examples"
}

Step 5 — cross-check with git. The snapshot only sees tests/fixtures/. Add one line to CI so fixtures written outside that root still surface:

git status --porcelain -- tests/fixtures | tee fixture_git_status.txt
python fixture_audit.py pytest -q -x

Exit code 2 means "the suite may have passed, but it edited evidence without saying so". That is a review block, not a warning.

Gate 2: quarantines that expire and must still reproduce

A freeze without an expiry is a delete with extra steps. A freeze without a witness command is a comment.

Step 1 — put quarantines in a machine-readable ledger. Every entry carries an owner, a reason, a witness command, and an ISO expiry date.

{
  "tests/test_parser.py::test_unicode_edges": {
    "owner": "finley",
    "reason": "ordering leak when run after test_lexer; upstream issue open",
    "witness": "pytest tests/test_parser.py::test_unicode_edges -q -x",
    "expires": "2026-11-02"
  }
}

Step 2 — fail CI on expiry or missing fields. Today's clock is the only authority here; no reviewers required.

# quarantine_lint.py
import datetime, json, sys

ledger = json.load(open("quarantine.json"))
today = datetime.date.today()
blocked = []

for test_id, entry in ledger.items():
    if not entry.get("witness"):
        blocked.append((test_id, "no witness command"))
    if not entry.get("owner"):
        blocked.append((test_id, "no owner"))
    try:
        expires = datetime.date.fromisoformat(entry["expires"])
    except (KeyError, ValueError):
        blocked.append((test_id, "missing or malformed expiry"))
    else:
        if expires < today:
            blocked.append((test_id, f"expired {entry['expires']}"))

for test_id, why in blocked:
    print(f"QUARANTINE BLOCKED {test_id}: {why}")
sys.exit(3 if blocked else 0)

Step 3 — run the witness on a schedule. This is the part people skip. If the witness command now passes consistently, the quarantine is stale debt and must be closed, not renewed. A nightly job that executes each witness and records pass/fail converts "we'll look at it later" into a number.

python quarantine_lint.py || exit $?
for t in $(python -c 'import json;print("\n".join(json.load(open("quarantine.json"))))'); do
  echo "== witness: $t"
  pytest "$t" -q || echo "still failing (expected)"
done

In my own triage, three outcomes dominate: the witness still fails reliably (freeze stands), the witness now passes every time (close it), or the witness fails only sometimes (the freeze reason is wrong and needs rewriting — usually it is an ordering or fixture-state problem, not the flake it claims to be).

Gate 3: the golden file is input, not oracle

A property like "parser output matches the golden file" is not a test if the golden file can be rewritten to match any output. Check oracle sensitivity directly: flip one byte in each golden fixture and confirm the suite turns red.

import shutil, subprocess, tempfile
from pathlib import Path

for golden in sorted(Path("tests/fixtures/golden").glob("*.json")):
    original = golden.read_bytes()
    try:
        golden.write_bytes(original[:-1] + bytes([original[-1] ^ 0x01]))
        rc = subprocess.run(["pytest", "-q", "-x"], capture_output=True).returncode
    finally:
        golden.write_bytes(original)
    if rc == 0:
        print(f"INSENSITIVE ORACLE: {golden} accepted a mutated payload")

An insensitive oracle is not a flaky test. It is a test whose assertion cannot fail, and it should be deleted or rewritten rather than quarantined.

Triaging the output

Audit signal	Most likely cause	Action
Pass, undeclared fixture write	Evidence rewritten to fit output	Block; require a per-path declaration and diff review
Pass, fixture write declared and reviewed	Intentional schema/fixture change	Keep declaration, pin fixture hash in the PR description
Fail, no fixture write	Ordinary candidate regression	Run the order and seed sweeps before touching the test
Fail only on the second run	Leaked state, not flake	Quarantine with a witness; do not freeze the whole file
Witness now passes 20/20	Stale quarantine	Close the entry and delete the expiry
Mutated golden accepted	Insensitive oracle	Rewrite the assertion; never quarantine

Where the cost lands

These gates are cheap per run but not free in aggregate: hashing a fixture tree is fast, while witness execution multiplies suite runtime by the number of quarantined tests. In my setup the audit runs on every PR and the witness sweep runs nightly, because nightly is where a stale freeze stops costing review attention and starts costing a date.

That nightly job is the one piece that benefits from somewhere to run that is not a laptop. I have been using MonkeyCode's free server option for the witness sweep and its free model access to draft the fixture_writes.json declarations and quarantine reasons from the failing-test output — every drafted line still goes through a human reviewer before it lands, because a model-written justification is not a justification. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both availability claims are the operator's, not measurements of mine. Treat them as a way to get a nightly box and a drafting assistant without a procurement cycle, and verify quotas and limits yourself before depending on them.

Limitations and who should skip this

The hash snapshot only sees one directory tree. Database rows, network services, and files written outside tests/fixtures/ are invisible to it; the git cross-check catches some of that, not all. The gate also cannot tell a legitimate schema migration from a subtly loosened fixture — it only forces the change to be written down where a reviewer can see it.

Expiry dates invite date-pushing. The witness rerun is the only thing that resists it, and only if someone reads the nightly output.

Skip this if your fixtures are generated at build time from a committed source (audit the source instead), or if you have no quarantined tests and no golden files — you would be paying for a gate that has nothing to reject.

Start with Gate 1 alone. Run it against your last ten agent-authored patches and count how many wrote to the fixture tree without saying so. If the answer is zero, you learned that cheaply. If it is not, you have your next review checklist.