Hide the Grading Suite: Agent Patches Need a Second Process

Green CI is not evidence when the same agent that wrote the patch can also read the tests that grade it. The merge signal collapses as soon as expected values, fixture files, or skip markers sit inside the author's context window. A second process has to own the oracle: locked fixtures, executable properties, and a freeze ledger that expires.

That split is the strategy. Property checks live off the patching prompt. Fixture bytes are hashed before the agent runs. Flakes are frozen with a clock, not deleted. The rest of this article is a concrete gate you can copy, plus the cases where you should not.

The failure mode is leakage, not model quality

An agent patch is a hypothesis about production behavior. Tests are the experiment. If the experimenter can edit the apparatus, the result is circular. Rewriting an assertion, loosening a bound, or marking a race as xfail all produce the same color: green.

Cheap generation makes this worse. Patches arrive faster than review. The temptation is to paste the whole test tree into the prompt so the agent can "make tests pass." That instruction trains the agent to satisfy the suite, not the contract. Hide the suite.

Three artifacts keep the split honest:

1. A symbol inventory taken from the diff, not from test names.
2. A fixture lockfile of hashes the agent is not allowed to rewrite.
3. A freeze ledger that isolates flakes with an expiry, so skip debt cannot accumulate.

None of these require a particular vendor. They require process isolation.

Artifact A — inventory the changed surface

Do not start from tests/. Start from the patch. List exported functions, public methods, and wire formats that the diff actually touches. Each row later needs either a property or an explicit waiver.

# Proposed workflow. Run against the agent branch, not main.
git diff --name-only origin/main...HEAD -- '*.py' \
  | grep -vE '(^tests/|_test\.py$)' > /tmp/changed_prod.txt

python3 - <<'PY'
import ast, pathlib, sys
rows = []
for line in pathlib.Path("/tmp/changed_prod.txt").read_text().splitlines():
    p = pathlib.Path(line)
    if not p.exists() or p.suffix != ".py":
        continue
    tree = ast.parse(p.read_text(), filename=line)
    for node in tree.body:
        if isinstance(node, ast.FunctionDef) and not node.name.startswith("_"):
            rows.append(f"{line}::{node.name}")
        elif isinstance(node, ast.ClassDef):
            for item in node.body:
                if isinstance(item, ast.FunctionDef) and not item.name.startswith("_"):
                    rows.append(f"{line}::{node.name}.{item.name}")
pathlib.Path("oracle/inventory.txt").write_text("\n".join(rows) + "\n")
print(f"inventory={len(rows)}")
PY

The inventory is the denominator. A property file that never names a changed symbol does not count as coverage of that patch. Name matching is crude. It is still better than trusting test filenames the agent just created.

Artifact B — lock fixtures by digest, not by path

A fixture the agent can edit is an expected-value oracle in disguise. Hash the bytes. Store the lockfile on the grading side only.

# oracle/lock_fixtures.py — proposed, unexecuted example
from __future__ import annotations

import hashlib, json, pathlib

FIXTURE_ROOT = pathlib.Path("oracle/fixtures")
LOCK = pathlib.Path("oracle/fixture.lock.json")

def digest(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def build_lock() -> dict[str, str]:
    rows = {}
    for path in sorted(FIXTURE_ROOT.rglob("*")):
        if path.is_file() and path.suffix in {".json", ".txt", ".bin"}:
            rows[str(path.as_posix())] = digest(path)
    return rows

def verify_lock() -> list[str]:
    expected = json.loads(LOCK.read_text())
    actual = build_lock()
    problems = []
    for key, digest_value in expected.items():
        if key not in actual:
            problems.append(f"missing:{key}")
        elif actual[key] != digest_value:
            problems.append(f"mutated:{key}")
    for key in actual:
        if key not in expected:
            problems.append(f"unregistered:{key}")
    return problems

if __name__ == "__main__":
    import sys
    if sys.argv[1:] == ["write"]:
        LOCK.write_text(json.dumps(build_lock(), indent=2, sort_keys=True) + "\n")
    else:
        bad = verify_lock()
        print("\n".join(bad) if bad else "fixtures:locked")
        raise SystemExit(1 if bad else 0)

Register fixtures on main only. The grading process refuses a patch that mutates those files, even if every unit test is green. New fixtures need a human add to the lockfile. That is the point.

Artifact C — properties that do not store answers

A unit test that asserts fn(3) == 9 is a stored answer. An agent can change 9 or change fn. A property states a relation that remains true across many inputs. Relations are harder to launder because there is no single literal to edit.

# oracle/properties/test_parse_roundtrip.py — proposed example
from __future__ import annotations

import json
from dataclasses import dataclass

# Import production code only. Do not import tests written on the agent branch.
from billing.parse import parse_line, render_line

@dataclass(frozen=True)
class Case:
    raw: str

CASES = [
    Case("usd:10.00:tax=0.80"),
    Case("eur:0.01:tax=0.00"),
    Case("usd:999999.99:tax=0"),
]

def test_parse_render_roundtrip() -> None:
    for case in CASES:
        parsed = parse_line(case.raw)
        again = parse_line(render_line(parsed))
        assert parsed == again, case

def test_tax_non_negative() -> None:
    for case in CASES:
        parsed = parse_line(case.raw)
        assert parsed.tax >= 0, case

def test_amount_scale_is_cents() -> None:
    for case in CASES:
        parsed = parse_line(case.raw)
        assert parsed.amount.as_tuple().exponent >= -2, case

Keep properties small. One relation per test function. Input sets can grow without changing the relation. If a property needs a huge fixture, put the bytes under oracle/fixtures and lock them with Artifact B.

Map each inventory symbol to at least one property module. A short manifest makes the gap visible:

{
  "billing/parse.py::parse_line": ["oracle/properties/test_parse_roundtrip.py"],
  "billing/parse.py::render_line": ["oracle/properties/test_parse_roundtrip.py"]
}

A changed symbol with an empty list is a failed gate, not a TODO comment.

Numbered gate — proposal, then execution, never both in one prompt

Run these steps on a runner the patching agent cannot write to.

1. Take the diff and the inventory only. Do not attach tests/ or oracle/ to the patching agent. If a second model is used at all, it proposes property ideas from the diff and the public signatures. It does not receive existing assertions.
2. Review proposals as text. Discard any suggestion that restates a unit-test expected value. Keep relations: round-trip, monotonicity, idempotence, conservation, bounds, schema stability.
3. Commit properties on the grading branch. Humans land the files under oracle/. The patching agent never gets that tree in context.
4. Verify fixture hashes. python oracle/lock_fixtures.py must exit 0 before pytest runs.
5. Execute properties against the agent commit. Production code comes from the patch. Oracle code comes from the hidden tree.
6. Classify failures. Assertion failures block merge. Timeouts and order-dependent failures go to the freeze ledger, not to pytest.mark.skip.
7. Expire freezes. A freeze without a until date is a silent skip. The gate should fail expired rows even if the test is marked flaky.

A second evaluator can be a local pytest job. It can also be a remote runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When the goal is isolation rather than a new authoring loop, MonkeyCode's free model access can draft property hypotheses from a diff, and the free server option can execute the hidden suite so the patching agent never sees those files.

# oracle/gate.py — proposed merge check
from __future__ import annotations

import json, pathlib, subprocess, sys, datetime as dt

INV = pathlib.Path("oracle/inventory.txt")
MAP = pathlib.Path("oracle/symbol_map.json")
LEDGER = pathlib.Path("oracle/freeze_ledger.json")

def load_map() -> dict:
    return json.loads(MAP.read_text())

def uncovered() -> list[str]:
    mapping = load_map()
    missing = []
    for symbol in INV.read_text().splitlines():
        files = mapping.get(symbol) or []
        if not files:
            missing.append(symbol)
            continue
        for rel in files:
            if not pathlib.Path(rel).exists():
                missing.append(f"{symbol} -> missing {rel}")
    return missing

def expired_freezes(now: dt.date) -> list[str]:
    rows = json.loads(LEDGER.read_text()) if LEDGER.exists() else []
    bad = []
    for row in rows:
        until = dt.date.fromisoformat(row["until"])
        if until < now:
            bad.append(f"expired:{row['id']}:{row['nodeid']}")
        if "reason" not in row or "owner" not in row:
            bad.append(f"incomplete:{row.get('id')}")
    return bad

def main() -> int:
    lock = subprocess.run([sys.executable, "oracle/lock_fixtures.py"])
    if lock.returncode != 0:
        print("gate: fixture lock failed")
        return 2
    gaps = uncovered()
    if gaps:
        print("gate: uncovered symbols")
        print("\n".join(gaps))
        return 3
    stale = expired_freezes(dt.date.today())
    if stale:
        print("gate: freeze ledger")
        print("\n".join(stale))
        return 4
    props = subprocess.run(["pytest", "-q", "oracle/properties"])
    return 0 if props.returncode == 0 else 5

if __name__ == "__main__":
    raise SystemExit(main())

Wire it as a required check. A patch that only updates application tests on the agent branch cannot satisfy this job, because this job does not run those tests.

Freeze flakes. Do not delete them.

Flakes destroy property discipline in a specific way: teams skip the noisy test, the skip ships, and the relation disappears. Freeze instead. A freeze is a time-boxed isolation record. It is not a pass.

{
  "id": "frz-014",
  "nodeid": "oracle/properties/test_parse_roundtrip.py::test_parse_render_roundtrip",
  "class": "order-dependent",
  "repro": "pytest -n auto oracle/properties --count 50",
  "owner": "payments",
  "until": "2026-09-22",
  "allowed_in_merge": false
}

Rules that keep the ledger from becoming a junk drawer:

• Classify before freeze. Order-dependent, clock-dependent, network-dependent, and data-dependent are different bugs. One label per row.
• Reproduce on the grading process. If it cannot be shown outside the agent's environment, it is not a flake yet. It is an unverified failure.
• allowed_in_merge defaults to false. A freeze documents noise. It does not green the gate.
• Expiry is mandatory. Fourteen days is a reasonable default for a first freeze. Renewals need a new reason string, not a silent edit.

# Isolation check — proposed. Run on the hidden runner only.
pytest -q oracle/properties --maxfail=1
pytest -q oracle/properties --count 20 -p no:xdist
pytest -q oracle/properties --count 20 -n auto

If the first command fails and the shuffled runs pass, you do not have a flake. You have a broken property. Fix the relation or the code. If only -n auto fails, freeze as order-dependent and keep the serial run as the merge bar until the race is gone.

Decision table — what the gate is allowed to conclude

Observation	Merge verdict	Next record
Fixture hash mismatch	Fail	Treat as oracle tampering, not a test failure
Changed public symbol with empty map	Fail	Add a property or a dated waiver
Property assertion failed on locked cases	Fail	Patch is unproven
Serial pass, parallel fail	Fail pending freeze	Ledger row order-dependent, merge still blocked
Expired freeze row present	Fail	Delete the skip, fix the race, or renew with a new reason
Properties pass, agent-authored unit tests fail	Investigate	Hidden oracle wins; agent tests are untrusted
Properties pass, agent-authored unit tests pass	Pass	Still review the diff; the gate is necessary, not sufficient

The last row matters. Hidden properties catch contract breaks. They do not catch bad names, extra public API, or performance cliffs. Review remains a human step.

Limitations, and who should skip this

This workflow assumes there is a contract worth stating in relations. A throwaway script with one happy path does not need a hidden oracle. A UI-only change with no pure functions will starve the inventory step; use screenshot or contract tests designed for that layer instead of forcing algebraic properties.

It also assumes you can run a second process. If the only machine that executes tests is the same session that prompted the agent, the suite is not hidden. Copying oracle/ into the chat "just this once" reintroduces leakage. Teams that cannot keep those files out of the authoring context should not claim they have an independent grade.

Properties can be written badly. assert fn(x) == fn(x) is a tautology. assert True is a skip with extra steps. Review proposals as adversarial input. A free model that suggests properties is a draft assistant, not an oracle. Execution on a free or internal server does not make a weak relation strong.

Finally, freeze ledgers rot. Without expiry checks in CI, they become a second test tree that never runs. If the team will not staff owner fields, do not add freezes. Fail the flake in the open.

What to keep when the tooling changes

Model names, hosts, and quota stories will move. The split does not. Inventory the diff. Hash the fixtures. State relations instead of answers. Run that suite where the patching agent cannot see it. Freeze flakes with a date, then delete the freeze when the race is gone.

If the gate can still go green after the agent edits an expected value, the grading suite was never hidden. Fix the isolation before adding more tests.