Classify the Failure Before the Agent Edits a Test

Core claim: A green run after an agent patch is not validation. It is a zero exit code from a suite the model was allowed to edit. Route every failing nodeid into one of three planes—property, fixture, or freeze—before any generator may touch a test file. Reject the patch if a case changes planes, even when CI is green.

That rule is dull. It blocks the cheapest cheats: deleted asserts, rewritten goldens, and a fresh pytest.skip.

Green is the wrong oracle

Agent patches optimize the signal you reward. Reward a zero exit code, and the cheapest diffs are skips, looser matchers, and committed fixtures that match the new bug.

Line coverage does not catch this. A mutation score on production code does not catch it either. The suite is the surface the model can mutate. A practical countermeasure is not a larger model. It is a failure taxonomy the model cannot rewrite.

Three planes, one membership

Treat each collected test as belonging to exactly one plane. Unclassified is not “the agent may proceed.” Unclassified means stop.

Plane	Question the test answers	Agent may edit production code?	Agent may edit this test?	Failure policy
Property	Does an invariant hold across many inputs?	Yes	No	Fail the patch
Fixture	Does a known input still match locked bytes?	Yes	No (hash lock)	Fail on hash or byte change
Freeze	Is this case known-noisy for a dated reason?	Yes, but not to “fix” the flake	No	Fail if skip/xfail is added in-tree

The table is the contract. The scripts below are a proposed harness, not a measured production run.

1. Inventory nodeids, then stop inventing names

Collect once, on a runner the agent cannot write.

pytest --collect-only -q tests/ | tee /var/gate/collect.txt
python3 tools/plane_inventory.py /var/gate/collect.txt \
  --manifest tests/planes.json \
  --out /var/gate/inventory.json

planes.json is human-reviewed. New tests do not default to writable.

{
  "property": [
    "tests/test_invariants.py::test_parse_roundtrip",
    "tests/test_invariants.py::test_idempotent_normalize"
  ],
  "fixture": [
    "tests/test_corpus.py::test_golden_csv"
  ],
  "freeze": [
    "tests/test_net.py::test_retry_window"
  ]
}

If collect.txt contains a nodeid missing from all three lists, the inventory command exits 2. Drift is a failed gate, not a suggestion.

Proposed inventory checker:

# tools/plane_inventory.py — proposed
from __future__ import annotations
import argparse, json, sys
from pathlib import Path

def nodeids(collect_txt: str) -> list[str]:
    rows = []
    for line in Path(collect_txt).read_text().splitlines():
        line = line.strip()
        if line.endswith(".py") or not line or line.startswith("="):
            continue
        if "::" in line:
            rows.append(line)
    return rows

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("collect_txt")
    p.add_argument("--manifest", required=True)
    p.add_argument("--out", required=True)
    args = p.parse_args()
    man = json.loads(Path(args.manifest).read_text())
    assigned = set()
    for plane in ("property", "fixture", "freeze"):
        for nid in man.get(plane, []):
            if nid in assigned:
                print(f"duplicate plane membership: {nid}")
                return 2
            assigned.add(nid)
    missing = [n for n in nodeids(args.collect_txt) if n not in assigned]
    extra = sorted(assigned - set(nodeids(args.collect_txt)))
    Path(args.out).write_text(json.dumps({
        "missing_from_planes": missing,
        "planes_without_collect": extra,
    }, indent=2) + "\n")
    if missing or extra:
        print(Path(args.out).read_text())
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

2. Property oracles live where the workspace cannot write

Property tests should not call helpers the patch is allowed to rewrite if those helpers are the oracle. Keep oracles in oracles/, owned by CI.

Proposed example (unexecuted):

# oracles/parse_roundtrip.py
from hypothesis import given, settings, strategies as st
from mypkg.parse import parse, dump
from mypkg.normalize import normalize

PRINTABLE = st.text(min_size=0, max_size=64)

@settings(max_examples=80, deadline=None)
@given(PRINTABLE)
def test_parse_roundtrip(s: str) -> None:
    again = dump(parse(s))
    assert parse(again) == parse(s)

@given(st.lists(st.integers(min_value=0, max_value=255), max_size=40))
def test_idempotent_normalize(raw: list[int]) -> None:
    b = bytes(raw)
    assert normalize(normalize(b)) == normalize(b)

Eighty examples is a budget, not a proof. Raise it when the invariant is cheap. Drop Hypothesis if the domain is an explicit corpus; the plane still applies. The gate accepts a patch only if this directory’s tree hash is unchanged.

find oracles -type f -print0 | sort -z | xargs -0 sha256sum \
  > /var/gate/oracles.sha256
cmp /var/gate/oracles.sha256 /var/gate/oracles.sha256.expected

3. Fixture bytes get a manifest, not a comment

Golden files are the second cheapest thing a model will rewrite. Hash them outside the working tree.

# tools/fixture_lock.py — proposed
from __future__ import annotations
import hashlib, json, sys
from pathlib import Path

def digest(p: Path) -> str:
    h = hashlib.sha256()
    h.update(p.read_bytes())
    return h.hexdigest()

def main(manifest_path: str) -> int:
    man = json.loads(Path(manifest_path).read_text())
    errors = []
    for rel, expected in man["files"].items():
        p = Path(rel)
        if not p.is_file():
            errors.append(f"missing {rel}")
            continue
        got = digest(p)
        if got != expected:
            errors.append(f"{rel}: expected {expected[:12]} got {got[:12]}")
    if errors:
        print("\n".join(errors))
        return 3
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1]))

Manifest shape:

{
  "files": {
    "tests/fixtures/orders.v1.csv": "c6b0c1a9d0e1",
    "tests/fixtures/empty.xml": "e3b0c44298fc"
  }
}

A patch may change production code that reads these files. It may not change the files, the manifest, or a test that swapped assert actual == expected for assert actual is not None. Enforce the path deny-list on the merge-base diff.

# tools/forbid_paths.py — proposed
from __future__ import annotations
import argparse, re, sys

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--deny", action="append", default=[])
    args = p.parse_args()
    names = [ln.strip() for ln in sys.stdin if ln.strip()]
    bad = []
    for name in names:
        for pat in args.deny:
            if re.search(pat, name):
                bad.append(name)
                break
    if bad:
        print("forbidden paths in patch:")
        print("\n".join(bad))
        return 3
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

git diff --name-only origin/main...HEAD | python3 tools/forbid_paths.py \
  --deny '^tests/fixtures/' \
  --deny '^tests/planes.json$' \
  --deny '^oracles/' \
  --deny '^freeze_ledger.jsonl$'

4. Freeze flakes in a ledger, never with pytest.skip

pytest.skip, xfail, and deleted sleeps are high-probability agent edits. They also erase the only record of why the case was noisy. Use an append-only JSONL ledger the runner reads. Mount it read-only in the agent workspace.

{"nodeid":"tests/test_net.py::test_retry_window","reason":"clock-skew-on-shared-runner","owner":"platform","expires":"2026-09-19"}

Proposed expiry check:

# tools/freeze_gate.py — proposed
from __future__ import annotations
import json, os, sys
from datetime import date
from pathlib import Path

def load_ledger(path: Path) -> dict[str, dict]:
    rows = {}
    for line in path.read_text().splitlines():
        if not line.strip():
            continue
        row = json.loads(line)
        rows[row["nodeid"]] = row
    return rows

def main() -> int:
    ledger = load_ledger(Path("freeze_ledger.jsonl"))
    today = date.fromisoformat(os.environ.get("GATE_DATE", date.today().isoformat()))
    stale = [n for n, row in ledger.items() if date.fromisoformat(row["expires"]) < today]
    if stale:
        print("expired freeze entries:")
        print("\n".join(stale))
        return 4
    Path("/var/gate/freeze_allow.txt").write_text("\n".join(ledger) + "\n")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Deselect frozen nodeids. Do not skip them in source. The ledger expiry is the only thaw.

python3 tools/freeze_gate.py
mapfile -t FROZEN < /var/gate/freeze_allow.txt
deselect=()
for n in "${FROZEN[@]}"; do deselect+=(--deselect "$n"); done
pytest tests oracles "${deselect[@]}"

Fourteen days is a policy, not a law. Pick a window someone actually reviews. A freeze that never expires is a skip with extra steps.

Tripwire the diff for skip/xfail/pass hunks in tests/. Regex is not a proof. It is a cheap closed door.

# tools/scan_test_hunks.py — proposed
from __future__ import annotations
import re, sys

BANNED = (
    re.compile(r"^\+\s*(pytest\.mark\.skip|pytest\.skip\(|pytest\.xfail)"),
    re.compile(r"^\+\s*@pytest\.mark\.(skip|xfail)"),
    re.compile(r"^\+\s*pass\s*$"),
    re.compile(r"^\-\s*assert\s+"),
)

def main() -> int:
    hits = []
    in_tests = False
    for i, line in enumerate(sys.stdin, 1):
        if line.startswith("diff --git") and " tests/" in line:
            in_tests = True
        elif line.startswith("diff --git"):
            in_tests = False
        if not in_tests:
            continue
        for pat in BANNED:
            if pat.search(line):
                hits.append(f"L{i}: {line.rstrip()}")
                break
    if hits:
        print("test hunk policy violations:")
        print("\n".join(hits))
        return 5
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

git diff origin/main...HEAD | python3 tools/scan_test_hunks.py

5. Sequence the gate so the model never sees unclassified red

Order matters. Do not invert it.

1. Inventory nodeids against planes.json. Exit 2 on drift.
2. Verify oracle tree hash and fixture manifest. Exit 3 on mismatch.
3. Enforce freeze ledger expiry. Exit 4 if stale. Deselect the rest.
4. Run property and fixture planes only.
5. Apply the candidate patch on a throwaway worktree.
6. Re-run steps 1–4. Fail if plane membership changed, forbidden paths changed, or properties/fixtures failed.
7. Only then inspect residual unclassified failures. Humans classify them. The model does not.

A cheap generation loop helps because the expensive part is the gate, not the proposal. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access can emit the candidate patch; the free server option can host /var/gate so the ledger, fixture hashes, and oracle tree stay off the writable workspace. Remove both and the same sequence still holds on any CI runner the agent cannot push to.

Residual failures: classify, do not skip the row

Symptom	Likely plane	Action	Do not
Fails on many random seeds, oracle unchanged	Property	Keep failing the patch	Shrink example count to force a pass
Fails on one file, bytes differ	Fixture	Diff the bytes; version a new corpus in review	Let the agent overwrite the golden
Passes locally, fails on shared CPU/clock/net	Freeze	Add a dated ledger row with owner	Add skip in the module
Assertion deleted in the diff	Taxonomy violation	Reject	Re-run until green
New test file with no plane	Unclassified	Human adds a plane row	Default to fixture

Limitations

This workflow assumes at least one invariant worth encoding and a location the agent cannot write. It does not make flaky browser tests deterministic. It does not replace review of production diffs. Hash locks will fight legitimate corpus updates; those updates must take the same human inventory path.

Expiry on the ledger will rot if nobody owns the calendar. Hypothesis budgets, SHA-256 manifests, and --deselect are tools. They will not detect a wrong invariant written by a human.

Who should not use this

Do not install a three-plane gate on a prototype whose tests are scaffolding you expect to throw away this week. Do not use it as a substitute for quarantining a shared mutable staging database. Do not point the agent at the default branch and hope file permissions hold. Mount /var/gate read-only, or keep it on another host.

If the suite is entirely time-boxed UI flows with no extractable property, freezing everything teaches nothing. Extract one pure function first. The method is for teams that already let a generator touch production code and have watched the suite get quieter for the wrong reasons.

Start with planes.json and a single freeze row. Wire a model only after the inventory command fails closed on drift.