Classify the Test Diff Before You Trust an Agent Patch

An agent patch that also edits tests is not one change. It is two. The first is a production hypothesis. The second is a possible rewrite of the oracle that will judge it.

Green CI reports a boolean. It does not tell you which of those two edits happened. A test-diff policy does. Classify every test hunk before you merge the production hunk. Freeze flakes with a signature and an expiry. Keep property checks outside the tree the agent can write.

This is a method, not a measured case study. The scripts below are a proposed workflow. Adapt the heuristics to your runner and your CODEOWNERS rules.

The failure class

Agents optimize for a green run. That objective is compatible with deleting an assertion, widening a timeout, rewriting a golden file, or adding pytest.mark.skip. Each of those can make a broken production change look finished.

Flaky tests make the invitation explicit. A flake is cheaper to "stabilize" than to diagnose. Skipping it hides a race. Raising a sleep hides a deadlock. Updating a snapshot hides a contract break. The suite gets quieter. The product does not get safer.

The countermeasure is not "never let an agent touch tests." Additive coverage is useful. The countermeasure is a policy that labels each test hunk, then refuses the labels that relax an oracle.

Four labels

Use exactly four labels. More labels become a discussion. Fewer labels hide mixed hunks.

1. ALLOW — the hunk only adds checks, fixtures, or examples. It does not relax an existing oracle.
2. FREEZE — the test is flaky or environment-bound. Record a signature, an expiry, and a property that still has to hold.
3. REJECT — the hunk deletes, skips, xfail-marks, or weakens an existing check.
4. PROPERTY — production behavior must be pinned by a check the agent cannot edit in the same PR.

A single file can produce several labels. Classify at hunk granularity, then take the strictest label as the file verdict. REJECT beats FREEZE. FREEZE beats ALLOW. PROPERTY is not a verdict; it is a required companion run.

Decision table

Test-diff signal	Typical intent	Label	Required companion
New assert / new example / new fixture file	Additive coverage	ALLOW	None
Comment, import sort, rename with identical AST	Noise	ALLOW	AST-equal check
Deleted assert / deleted expected exception	Oracle removed	REJECT	Restore or explain out of band
Added skip, xfail, or pytest.mark.flaky	Oracle muted	REJECT unless freeze ledger already lists it	Freeze record
Timeout or retry count increased	Timing oracle relaxed	REJECT or FREEZE	Signature timing + expiry
Golden / snapshot / fixture bytes changed	Expected output rewritten	REJECT or PROPERTY	Fixture digest lock
Assertion right-hand side loosened (== to truthy, exact to regex)	Oracle weakened	REJECT	Property check on the old predicate
Test body rewritten, assertion count unchanged	Unknown	PROPERTY	External invariant must pass
Known flake, same failure signature as ledger	Stabilization theater	FREEZE	Property id + expiry ≤ 14 days

The table is the artifact. The classifier below only approximates it. A human still reads REJECT and mixed-PROPERTY rows.

Workflow

Run this after the agent opens a PR, before you look at the production diff.

1. List test hunks. Restrict to paths you treat as oracles: tests/**, **/testdata/**, **/fixtures/**, **/*.snap.
2. Parse each hunk. Count deleted assertions, added skips, timeout deltas, and fixture-byte changes. Do not trust the commit message.
3. Look up the freeze ledger. A skip is legal only when the test id is already frozen, unexpired, and signed.
4. Lock fixture digests. If a golden file changes, require a matching PROPERTY run or REJECT.
5. Run properties from a read-only plane. Directory invariants/ is not in the agent's write set. CI fails if that plane is dirty in the same PR.
6. Expire freezes. A freeze older than its expires field is a skip list. Fail the build.
7. Only then review production code. If the test-diff verdict is REJECT, stop. The production hypothesis is untested, not proven.

The order matters. Reviewing production first trains you to excuse the test edits that made it green.

Artifact: classifier, ledger, and a property plane

The classifier is deliberately boring. It reads git diff and prints labels. It does not call a model.

#!/usr/bin/env python3
"""test_diff_policy.py — classify oracle hunks in an agent PR.

Proposed workflow. Heuristics are incomplete; see Limitations.
"""
from __future__ import annotations

import hashlib
import json
import re
import subprocess
import sys
from datetime import date
from pathlib import Path

ORACLE_RE = re.compile(r"(tests/|testdata/|fixtures/|\.snap$|\.golden$)")
ASSERT_RE = re.compile(r"^[-+].*\b(assert|expect\(|self\.assert)")
SKIP_RE = re.compile(r"^\+.*\b(pytest\.mark\.(skip|xfail)|self\.skipTest)")
TIMEOUT_RE = re.compile(r"timeout\s*=\s*(\d+)")


def git_diff(base: str) -> str:
    return subprocess.check_output(
        ["git", "diff", "-U0", base, "--", "."], text=True
    )


def iter_hunks(diff: str):
    path, buf = None, []
    for line in diff.splitlines():
        if line.startswith("diff --git"):
            if path and ORACLE_RE.search(path):
                yield path, buf
            path = line.split(" b/")[-1].strip()
            buf = []
        else:
            buf.append(line)
    if path and ORACLE_RE.search(path):
        yield path, buf


def classify_hunk(path: str, lines: list[str]) -> str:
    deleted_assert = 0
    added_assert = 0
    added_skip = 0
    old_t, new_t = [], []
    for line in lines:
        if ASSERT_RE.search(line):
            if line.startswith("-"):
                deleted_assert += 1
            elif line.startswith("+"):
                added_assert += 1
        if SKIP_RE.search(line):
            added_skip += 1
        if line.startswith("-") and (m := TIMEOUT_RE.search(line)):
            old_t.append(int(m.group(1)))
        if line.startswith("+") and (m := TIMEOUT_RE.search(line)):
            new_t.append(int(m.group(1)))
    if added_skip:
        return "REJECT"
    if deleted_assert > added_assert:
        return "REJECT"
    if new_t and old_t and max(new_t, default=0) > max(old_t, default=0):
        return "FREEZE"
    if path.endswith((".snap", ".golden")) and any(
        line.startswith(("+", "-")) and not line.startswith(("+++", "---"))
        for line in lines
    ):
        return "PROPERTY"
    if deleted_assert == added_assert and deleted_assert > 0:
        return "PROPERTY"
    return "ALLOW"


def load_ledger(path: Path) -> dict:
    if not path.exists():
        return {"freezes": []}
    return json.loads(path.read_text())


def expired(ledger: dict, today: date) -> list[str]:
    bad = []
    for row in ledger.get("freezes", []):
        if date.fromisoformat(row["expires"]) < today:
            bad.append(row["test_id"])
    return bad


def fixture_digest(path: Path) -> str:
    return "sha256:" + hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> int:
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    ledger = load_ledger(Path("invariants/freeze_ledger.json"))
    stale = expired(ledger, date.today())
    if stale:
        print("EXPIRED_FREEZE", *stale, sep="\n")
        return 2
    verdicts = []
    for path, lines in iter_hunks(git_diff(base)):
        label = classify_hunk(path, lines)
        verdicts.append((label, path))
        print(f"{label:8} {path}")
    if any(v == "REJECT" for v, _ in verdicts):
        return 1
    if any(v in {"FREEZE", "PROPERTY"} for v, _ in verdicts):
        print("COMPANION required: invariants/ must be clean and green")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Pair it with a ledger the agent cannot rewrite without a CODEOWNERS review.

{
  "freezes": [
    {
      "test_id": "tests/test_parser.py::test_trailing_comma[worker-4]",
      "signature": "order-dependent",
      "reason": "fails when pytest-xdist worker count > 1",
      "expires": "2026-09-20",
      "property_id": "prop_parser_trailing_comma",
      "fixture_digest": "sha256:6b1c0e2a9d4f8a11"
    }
  ]
}

Signature values should be a closed set: timing, order-dependent, network, rng, fs-clock. A free-text reason is for humans. The signature is for the policy. If an agent "fixes" an order-dependent freeze by inserting time.sleep(2), the classifier still sees a timeout delta and returns FREEZE or REJECT. The sleep is not a diagnosis.

The property plane is a separate directory. Example invariant for the freeze above:

# invariants/prop_parser_trailing_comma.py
# Read-only to the agent. Proposed check, not a production suite.
from parser import parse_config


def prop_trailing_comma_is_syntax_error(src: str) -> None:
    """A trailing comma in a mapping is never silently accepted."""
    if not src.strip().endswith(",}") and not src.strip().endswith(",]"):
        return
    try:
        parse_config(src)
    except ValueError:
        return
    raise AssertionError("trailing comma parsed without ValueError")


CASES = ["{a: 1,}", "[1, 2,]", "{a: 1}", "[]"]

if __name__ == "__main__":
    for src in CASES:
        prop_trailing_comma_is_syntax_error(src)
    print("property ok", len(CASES))

CI glue is a few commands. Keep the property plane on a path the agent PR cannot modify, or fail if git diff touches it.

#!/usr/bin/env bash
set -euo pipefail
BASE="${1:-origin/main}"

if git diff --name-only "$BASE" -- invariants/ | grep -q .
then
  echo "invariants/ is dirty in this PR; property plane must be read-only"
  exit 3
fi

python test_diff_policy.py "$BASE"
python -m pytest invariants/ -q

That is the whole loop. Test-diff policy, then properties, then humans. Not the reverse.

Where generation happens

The policy does not care which model produced the patch. It cares whether the oracle moved. Generate candidates in a throwaway branch. Run the classifier locally. If you need a sandbox with free model access and a free server option to produce those branches, MonkeyCode is one open-source option to try. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The policy still belongs in your repo either way. A green remote run is not a substitute for the four labels.

Limitations

The classifier is syntactic. assert result == 3 rewritten as assert result is not None keeps the assertion count constant and will often land in PROPERTY, not REJECT. That is why PROPERTY exists. It is also why a human still reads mixed rows.

Fixture hashing does not understand semantic equivalence. Reordering JSON keys changes the digest. Pretty-printing a snapshot changes the digest. That is intended. If you want canonicalization, add it as an explicit step in invariants/, not as a silent ignore in the classifier.

Freeze ledgers rot. An expiry of 14 days is a suggestion, not a finding. A freeze without an expiry is a skip list with extra JSON. A freeze whose property_id is missing is a skip list with a comment. Who should not use this: throwaway prototypes, generated-only trees with no invariants, and teams that let the same PR edit invariants/ and src/ together. If tests are the product, this policy is the wrong shape.

The method also does not replace mutation testing, caller mapping, or boundary partitions. It answers one question only: did the agent move the oracle? Ask that question first. Then decide whether the production diff is worth reading.