Judge Agent Patches on Tests They Never Saw

A green run on tests an agent was allowed to read is not a merge signal. It is a training-set score. Split the suite first. Judge the patch on a held-out oracle, on fixtures the agent did not generate, and on a flake ledger it cannot edit.

Agent patches overfit in a specific way. They do not only chase coverage. They also rewrite the exam. A copied constant, a fixture dumped from the new code, and a widened skip all produce the same CI color. The color is not the defect. Shared information is.

This workflow treats tests like a train and holdout split. The agent sees a public suite. Merge reads a blind suite. Property checks live in the blind suite and are sourced from a spec file, not from the diff. Fixtures are locked to a schema hash committed before the agent starts. Flakes go to a human-signed ledger. None of those files belong in the agent's workspace. The commands below are a proposed gate, not a production report.

What the split actually protects

Visible tests are documentation the agent may use. Held-out tests are the judge. If a check can be read while the patch is being written, it is not an oracle. It is a prompt.

That rule is stricter than "add property tests." Property tests in the same tree the agent edits can still echo the implementation. The split is about information flow, not about test style.

Three failure modes show up when the split is missing:

1. The agent copies a literal from src/ into tests/ and asserts equality with itself.
2. The agent regenerates a fixture from the new behavior, then proves the new behavior.
3. The agent quarantines a failing test as flaky, and CI returns to green.

The gate below is aimed at those three. It is not a substitute for code review.

1. Commit the split before the agent runs

Keep a manifest at tests/split.json. Commit it on the default branch. Do not let the agent patch this file.

{
  "visible_globs": ["tests/public/**/*.py"],
  "heldout_globs": ["tests/heldout/**/*.py"],
  "spec_path": "docs/behavior_spec.md",
  "fixture_schema": "tests/heldout/fixtures/schema.json",
  "flake_ledger": "tests/flake_ledger.json",
  "agent_idents": ["agent@", "bot@", "coder@local"]
}

Prepare a workspace that physically omits held-out paths. A sentence in the prompt is not a boundary. Files the model can open will leak.

# tools/prepare_agent_workspace.py
from __future__ import annotations

import json
import shutil
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]


def load_split() -> dict:
    return json.loads((ROOT / "tests/split.json").read_text())


def is_heldout(rel: str, globs: list[str]) -> bool:
    rel = rel.replace("\\", "/")
    for g in globs:
        prefix = g.split("**", 1)[0].rstrip("/")
        if rel == prefix or rel.startswith(prefix + "/"):
            return True
    return False


def main(dest: Path) -> None:
    split = load_split()
    blocked = set(split["heldout_globs"])
    secret_files = {split["flake_ledger"], split["fixture_schema"]}
    if dest.exists():
        shutil.rmtree(dest)
    for src in ROOT.rglob("*"):
        if not src.is_file() or ".git" in src.parts:
            continue
        rel = str(src.relative_to(ROOT)).replace("\\", "/")
        if is_heldout(rel, list(blocked)) or rel in secret_files:
            continue
        out = dest / rel
        out.parent.mkdir(parents=True, exist_ok=True)
        shutil.copy2(src, out)
    print(f"workspace {dest} omits held-out tests, ledger, and fixture schema")


if __name__ == "__main__":
    main(Path(sys.argv[1]))

python tools/prepare_agent_workspace.py /tmp/agent-src
# agent may edit /tmp/agent-src only
git -C /tmp/agent-src diff > /tmp/agent.patch

Repos that already mix every test in one folder should move files once. Do not invent a second copy of the same assertion and call it held-out.

2. Derive property checks from the spec, not the patch

Held-out properties must not import new literals from the patched module. They read bounds from docs/behavior_spec.md or from a committed spec_constants.py that the agent is not allowed to change.

A proposed check for a discount function:

# tests/heldout/test_discount_properties.py
from decimal import Decimal

from hypothesis import given, strategies as st

from pricing import discount

# Sourced from docs/behavior_spec.md, not from src/pricing.py
MAX_RATE = Decimal("0.40")
MIN_QTY = 1
MAX_QTY = 10_000


@given(
    qty=st.integers(min_value=MIN_QTY, max_value=MAX_QTY),
    rate=st.decimals(min_value=Decimal("0"), max_value=MAX_RATE, places=2),
)
def test_discount_never_exceeds_spec_cap(qty, rate):
    price = Decimal("19.99")
    off = discount(price, qty, rate)
    assert Decimal("0") <= off <= price * MAX_RATE

Then reject held-out files that mention names introduced only in the patch. That is an AST pass over the diff, not a coverage pass.

# tools/oracle_independence.py
from __future__ import annotations

import ast
import subprocess
import sys
from pathlib import Path


def changed_symbols(base: str) -> set[str]:
    diff = subprocess.check_output(
        ["git", "diff", "-U0", base, "--", "src"], text=True
    )
    names: set[str] = set()
    for line in diff.splitlines():
        if not line.startswith("+") or line.startswith("+++"):
            continue
        try:
            tree = ast.parse(line[1:])
        except SyntaxError:
            continue
        for node in ast.walk(tree):
            if isinstance(node, ast.Assign):
                for target in node.targets:
                    if isinstance(target, ast.Name):
                        names.add(target.id)
    return names


def names_in(path: Path) -> set[str]:
    tree = ast.parse(path.read_text())
    return {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)}


def main(base: str, heldout: Path) -> int:
    introduced = changed_symbols(base)
    leaked = []
    for py in heldout.rglob("*.py"):
        overlap = names_in(py) & introduced
        if overlap:
            leaked.append((py, sorted(overlap)))
    for py, names in leaked:
        print(f"oracle leak {py}: {names}")
    return 1 if leaked else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], Path("tests/heldout")))

The parser on single + lines is a tripwire. It will miss multi-line assignments. Pair it with review.

A model can propose extra properties. Feed it the spec file only. Do not feed it the diff. If the proposal imports a symbol that exists only in the patch, drop the proposal.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access is enough to draft those spec-only properties. The free server option is the place to execute pytest tests/heldout so the editing session is not also the judge. The split file in git is still the actual control.

3. Lock held-out fixtures to a schema hash

Do not let the agent emit JSON blobs that become the expected output. Commit a schema. Generate samples with a checked-in script. Hash the schema in the gate.

# tools/fixture_lock.py
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> int:
    schema = Path("tests/heldout/fixtures/schema.json")
    lock = Path("tests/heldout/fixtures/schema.sha256")
    actual = sha256(schema)
    expected = lock.read_text().strip()
    if actual != expected:
        print(f"schema hash mismatch {actual} != {expected}")
        return 1
    data_dir = Path("tests/heldout/fixtures/data")
    for sample in sorted(data_dir.glob("*.json")):
        json.loads(sample.read_text())
    return 0


if __name__ == "__main__":
    sys.exit(main())

Regenerate samples only from tools/gen_heldout_fixtures.py, which reads the schema and the spec constants. If the agent patch includes tests/heldout/fixtures/data, fail the gate. Fixture writes are a human path.

4. Put flakes on a human-signed ledger

A freeze is not a skip in the test file. It is a row in tests/flake_ledger.json committed by a human identity.

{
  "entries": [
    {
      "nodeid": "tests/public/test_cache.py::test_ttl_expiry",
      "traceback_sha256": "a3f1c0ffeeexamplehash0000000000000000000000000000000000000000",
      "opened_by": "human",
      "ticket": "QA-214"
    }
  ]
}

The gate checks three things:

1. No new pytest.mark.skip, xfail, or timeout deltas in the agent diff.
2. The flake ledger's last git author does not match agent_idents.
3. Every remaining skip in the public suite has a ledger row.

# tools/flake_ledger_gate.py
from __future__ import annotations

import json
import re
import subprocess
import sys
from pathlib import Path

SKIP_RE = re.compile(r"pytest\.mark\.(skip|xfail)|timeout\s*=")


def last_author(path: str) -> str:
    return subprocess.check_output(
        ["git", "log", "-1", "--format=%ae", "--", path], text=True
    ).strip()


def diff_touches_skips(base: str) -> bool:
    diff = subprocess.check_output(["git", "diff", base, "--", "tests"], text=True)
    return any(
        SKIP_RE.search(line) for line in diff.splitlines() if line.startswith("+")
    )


def main(base: str, split_path: str) -> int:
    split = json.loads(Path(split_path).read_text())
    ledger = split["flake_ledger"]
    author = last_author(ledger)
    if any(tok in author for tok in split["agent_idents"]):
        print(f"flake ledger last authored by agent ident {author}")
        return 1
    if diff_touches_skips(base):
        print("agent diff introduces skip/xfail/timeout markers")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], "tests/split.json"))

Do not auto-expire rows in this version of the gate. Expiry without a reproduced traceback is another way to go green without evidence. Re-open a row when a human re-runs the nodeid with a fixed seed and records a new hash.

5. Run the blind suite off the agent's machine

Sequence:

1. Build /tmp/agent-src without held-out files.
2. Collect the patch.
3. Apply the patch to a clean checkout that still contains tests/heldout.
4. Run oracle_independence.py, fixture_lock.py, and flake_ledger_gate.py.
5. Run pytest tests/heldout -q --basetemp=/tmp/heldout-run.
6. Merge only if every step returns 0.

git apply --check /tmp/agent.patch
git apply /tmp/agent.patch
python tools/oracle_independence.py HEAD
python tools/fixture_lock.py
python tools/flake_ledger_gate.py HEAD
pytest tests/heldout -q --basetemp=/tmp/heldout-run

Step 5 should not execute on the same workspace the agent used. Shared temp directories leak fixtures. A separate runner is the point: same git SHA, same seed, no agent files left on disk. Use that runner when local CI is the machine that also hosted the editing session.

Decision table

Signal	Merge?	Reason
Public suite green, held-out not run	No	Training-set score only
Held-out green, new literals in held-out tests	No	Oracle copied the patch
Held-out green, fixture files in the diff	No	Exam rewritten
skip/xfail added, ledger unchanged	No	Quarantine without a human row
Ledger edited by an agent ident	No	Judge captured by the author
All gates 0, held-out green on a clean runner	Yes, pending review	Blind score plus review

Limits

The AST leak check misses multi-line assignments and refactors that rename a constant before copying it. Hypothesis budgets can hide shrinking failures if you do not pin a seed. A spec that is just a restatement of the code will produce held-out tests that still cannot fail. Small libraries with a handful of tests cannot split without starving the agent of examples. Write more public tests first.

This approach does not prove absence of security defects, race conditions, or performance cliffs. It only blocks a class of information-leak green bars.

Who should not use it

Skip the split if there is no written spec. End-to-end tests against a shared staging account cannot be held out; they are already non-deterministic. A single person who both writes the prompt and signs the ledger makes the ledger a prop.

The next file to commit is tests/split.json, not another model preset. Draft properties from the spec. Run the suite the agent never saw on a machine it does not control. Neither step replaces the split.