Build a Write-Protected Test Plane for Agent Patches

An agent patch that also edits tests cannot be scored by those tests. The score is circular. Split the suite into a write-protected plane the diff cannot touch, freeze flakes so they cannot be healed by rewriting expectations, and only then accept or reject the candidate.

This article proposes a concrete gate. It is a procedure, not a production incident report. The gate is small enough to run on a laptop or on a free remote runner. It does not require a named model, a quota, or a hardware claim.

The contamination problem

A human patch and an agent patch fail in different ways. A human usually leaves tests alone unless the contract changed. An agent often lands code and tests in the same diff. That pairing looks complete. It is not evidence.

If src/fee.py and tests/test_fee.py move together, the new assertion may describe the new code and nothing else. Green then means “self-consistent,” not “still correct.” Short sentence: circular scores are not scores.

The fix is not “run more tests.” The fix is to decide, before the patch is applied, which files are allowed to change and which files are the measuring instrument.

Three planes, one rule

Keep three directories and two lock files. The names are conventional. The rule is not.

1. Contaminated plane — tests/in_diff/. Tests that arrive inside the patch. Useful as documentation. Never used as the accept/reject signal.
2. Protected plane — tests/protected/. Property checks, contract tests, and golden relations. This tree is read-only for the agent. A patch that touches it is rejected before scoring.
3. Quarantine plane — tests/flaky.freeze.json. Tests that have been observed as unstable. They do not fail the gate and they do not pass the gate. They are frozen until a human expires the entry.

The single rule: only the protected plane may vote. Everything else is commentary.

Decision table

Use this table as the artifact. Apply it to every path in the candidate diff. vote means the file may contribute to the accept/reject bit. ignore means keep the file for review. reject_diff means stop; do not score.

Path pattern	In the agent diff?	Action	Why
src/**	yes	continue	code under test
tests/in_diff/**	yes	ignore	contaminated evidence
tests/protected/**	yes	reject_diff	instrument was rewritten
tests/fixtures.lock.json	yes	reject_diff	fixture identity changed
tests/flaky.freeze.json	yes	reject_diff	quarantine was edited
tests/protected/**	no	vote	only legal signal
listed in flaky.freeze.json	n/a	ignore	no pass, no fail
untracked test outside both planes	yes	reject_diff	ungoverned assertion

The last row matters. Agents invent helper tests in odd folders. Ungoverned assertions are a second contamination channel. Fail closed on unknown test paths.

Proposed workflow

The steps below are a proposed local procedure. Label them as unexecuted until you run them on your tree. Commands assume a git worktree and pytest. Adapt the runner if you do not use pytest.

1. Inventory the instrument

List every file that is allowed to vote. Commit that list. Do not generate it from the patch.

find tests/protected -type f \( -name 'test_*.py' -o -name '*_test.py' \) \
  | sort > tests/protected.manifest
git add tests/protected.manifest

If the manifest itself appears in a later diff, treat it like a protected file. The catalog of voters is part of the instrument.

2. Lock fixtures by digest, not by path

Path locks are not enough. An agent can keep tests/protected/fixtures/invoice.json and rewrite its bytes. Hash the payloads.

# tests/tools/lock_fixtures.py  (proposed helper)
from __future__ import annotations

import hashlib
import json
from pathlib import Path

FIXTURE_ROOT = Path("tests/protected/fixtures")
LOCK_PATH = Path("tests/fixtures.lock.json")

def digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def build_lock() -> dict[str, str]:
    rows = {}
    for path in sorted(FIXTURE_ROOT.rglob("*")):
        if path.is_file():
            rows[path.as_posix()] = digest(path)
    return rows

def main() -> None:
    LOCK_PATH.write_text(json.dumps(build_lock(), indent=2) + "\n")

if __name__ == "__main__":
    main()

Regenerate the lock only in a human commit. The gate compares the working tree to the committed lock. A mismatch is a failed instrument, not a failed unit test.

3. Freeze flakes with an expiry, not a skip

pytest.mark.skip is a hole. An agent can delete the mark or invert the assertion. Keep freeze records outside the test body.

{
  "tests/protected/test_tax_rounding.py::test_half_even": {
    "reason": "order-dependent float on parallel workers",
    "frozen_on": "2026-09-04",
    "expires_on": "2026-09-18",
    "owner": "tax-platform"
  }
}

A freeze that is past expires_on is an error in the gate, not a silent skip. Expired freezes must be re-triaged. That prevents a permanent blind spot.

4. Reject diffs that touch the instrument

# tests/tools/protected_plane_gate.py  (proposed helper)
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

PROTECTED_PREFIXES = (
    "tests/protected/",
    "tests/protected.manifest",
    "tests/fixtures.lock.json",
    "tests/flaky.freeze.json",
    "tests/tools/",
)

def changed_files(base: str = "HEAD") -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", base],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]

def violates(path: str) -> bool:
    return any(path == p or path.startswith(p) for p in PROTECTED_PREFIXES)

def main() -> int:
    bad = [p for p in changed_files() if violates(p)]
    if bad:
        sys.stderr.write("protected plane touched:\n")
        for p in bad:
            sys.stderr.write(f"  {p}\n")
        return 2
    freeze = json.loads(Path("tests/flaky.freeze.json").read_text())
    print(f"ok: {len(freeze)} frozen ids, instrument untouched")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Exit 2 means “do not score.” Exit 0 means “the instrument is intact; now you may run it.”

5. Run only the protected plane, minus frozen ids

python tests/tools/protected_plane_gate.py || exit 2
python tests/tools/check_fixture_lock.py || exit 2

IGNORE=$(python - <<'PY'
import json
from pathlib import Path
data = json.loads(Path("tests/flaky.freeze.json").read_text())
print(" ".join(f"--deselect={k}" for k in data))
PY
)

pytest tests/protected -q $IGNORE --maxfail=1

tests/in_diff/ can still run as a non-blocking report. Its failures are review notes. Its passes are not a merge signal.

6. Property checks live in the protected plane

Do not assert exact bytes of agent output. Assert relations the patch is not allowed to redefine. Example property, labeled as a template:

# tests/protected/test_fee_properties.py
import pytest
from hypothesis import given, strategies as st

from billing.fee import apply_fee  # system under test; not imported from the patch tests

@given(
    amount=st.decimals(min_value="0.01", max_value="1000000", places=2),
    bps=st.integers(min_value=0, max_value=500),
)
def test_fee_never_exceeds_amount(amount, bps):
    charged, net = apply_fee(amount, bps)
    assert charged >= 0
    assert net == amount - charged
    assert charged <= amount

The property is the contract. The agent may change the implementation. It may not change this file. If the contract itself must change, that is a human commit to the protected plane, reviewed as a spec change, not as part of the agent patch.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is relevant only on the candidate side of this workflow. A cheap generator is useful because this gate expects volume: many patches, one instrument. The model does not get to write tests/protected/, tests/fixtures.lock.json, or tests/flaky.freeze.json. If it does, the gate in step 4 rejects the diff before pytest starts.

The free server option is relevant on the instrument side. Scoring should not share a working tree with the editor that produced the patch. A separate runner reduces accidental local state: dirty env vars, leftover fixtures, a pytest plugin the agent enabled in conftest.py. Check the candidate out in a clean worktree on that server, run the three commands in step 5, and copy back three artifacts only: gate exit code, pytest report, and the fixture-lock comparison.

That split is the point. Generation can be noisy. Measurement must be boring.

If you already keep evaluation off the laptop that produced the diff, running the protected plane on a free server is one way to keep those two roles from collapsing into the same shell.

What this does not prove

The protected plane proves that the candidate did not rewrite the instrument and that the remaining properties still hold. It does not prove the product is right. A weak property set will green a wrong patch. That is a spec hole, not a runner hole.

Frozen tests hide behavior. A freeze older than its expires_on must fail the gate. If your team never expires rows, you have built a skip list with extra JSON.

Free model output is still a hypothesis. Nondeterministic generation is expected. That is why the model is not in the scoring path. Do not average pass rates across regenerations and call the mean a verdict. One candidate, one worktree, one protected run.

Who should not use this

Do not use this layout if your tests are inseparable from the implementation — generated snapshots with no independent relation, or UI scripts whose only oracle is a screenshot. You need a property or a contract first. The plane cannot protect an empty instrument.

Do not use it as a substitute for review of src/**. A patch can satisfy charged <= amount and still bill the wrong customer. Protected tests constrain; they do not replace diff reading.

Do not use it to launder agent-authored tests into the protected plane after they go green. Promotion of a test from tests/in_diff/ to tests/protected/ is a spec change. It needs the same review as a hand-written contract.

Minimal checklist

Copy this into the PR template if you adopt the procedure:

1. protected_plane_gate.py exits 0.
2. Fixture lock matches HEAD.
3. No freeze is past expires_on.
4. pytest ran tests/protected only, with frozen ids deselected.
5. tests/in_diff/ was not used as the merge bit.

If any box is unchecked, the candidate is unscored. Unscored is not failed and not passed. It is not ready.