Write-Block the Test Tree Before an Agent Patch Can Go Green

A green suite after an agent patch is not evidence if the agent could edit tests/. Treat src/ as the only writable surface. Derive the contract from an AST hash of public signatures, bind fixtures by content digest, and freeze flakes by seed plus fixture digest. Anything weaker lets the patch rewrite the oracle that was supposed to judge it.

This is a merge-gate design, not a model-quality claim. The gate is deterministic. The agent is not.

The failure mode the suite will not show you

Agent patches fail in a narrow, repeatable way. They change behavior and then change the assertion that would have caught the change. CI stays green. The diff looks industrious. Reviewers who skim test files last never see that the oracle moved.

Name-stable tests do not fix this. An agent that can rewrite a snapshot, a fixture JSON, or a parametrize list has already escaped the hypothesis you think you are testing. The cheapest control is mechanical: the patch commit must not contain a write under tests/.

Property checks still matter. They just cannot be authored inside the same writable tree as the patch.

Policy in three layers

1. Write-block. git diff --name-only of the agent commit must not include tests/** or the flake ledger. Fail closed.
2. Contract hash. Parse src/ with ast. Hash public function signatures from HEAD and from the patch. If the hash moved, require an explicit allow file, not a silent test edit.
3. Fixture digest + seed freeze. Load fixtures by SHA-256. Record flakes as (nodeid, seed, fixture_digest) with an expiry. A freeze never matches a renamed file that kept the old nodeid but swapped bytes.

The order is the method. Skip layer 1 and layers 2–3 become theater.

Proposed local layout

Label the following as a proposed layout, not a production report.

repo/
  src/
  tests/                 # agent cannot write
  ci/
    contract.allow       # human-edited
    fixtures.lock        # path -> sha256
    flake-ledger.jsonl   # seed freezes
    gate_agent_patch.py

Keep the agent workspace able to edit src/ only. Run the gate on a clean checkout of the proposed commit. Do not import patched modules to compute the contract. Import executes agent code.

Step 1 — Reject any test-tree write

# ci/gate_agent_patch.py  (proposed)
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

BLOCKED_PREFIXES = ("tests/", "ci/fixtures.lock", "ci/flake-ledger.jsonl")

def changed_paths(base: str, head: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...{head}"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]

def assert_test_tree_readonly(base: str, head: str) -> None:
    blocked = [
        p for p in changed_paths(base, head)
        if p.startswith(BLOCKED_PREFIXES) or p in BLOCKED_PREFIXES
    ]
    if blocked:
        raise SystemExit("test-tree write blocked:\n" + "\n".join(blocked))

Run it as python ci/gate_agent_patch.py origin/main HEAD. A patch that “fixes” tests is a failed patch. Move the intended test change to a human follow-up commit.

Step 2 — Hash the contract from AST, not from imports

Public names are the merge surface. Defaults, argument names, and annotations are part of that surface. Body edits that preserve the signature should not require an allow-file update. Signature edits should.

import ast
from pathlib import Path

def public_signatures(root: Path) -> list[str]:
    rows: list[str] = []
    for path in sorted(root.rglob("*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
        rel = path.as_posix()
        for node in tree.body:
            if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
                continue
            if node.name.startswith("_"):
                continue
            args = [a.arg for a in node.args.args]
            returns = ast.unparse(node.returns) if node.returns else ""
            rows.append(f"{rel}::{node.name}({','.join(args)})->{returns}")
    return rows

def contract_digest(root: Path) -> str:
    payload = "\n".join(public_signatures(root)).encode()
    return hashlib.sha256(payload).hexdigest()

Compare contract_digest(Path("src")) at base and head by checking out each tree into a temporary directory, or by parsing blobs from git show. If the digest differs, ci/contract.allow must list the new digest. The allow file is human-edited. The agent cannot touch it under the write-block.

That split is the entire point. A behavior fix that preserves the public signature stays inside the agent’s budget. An API change leaves the test tree, which the agent cannot update, and the gate stays red until a person writes properties for the new surface.

Step 3 — Content-address fixtures

Directory identity is not identity. Two files named invoice.json can disagree in a single field and still satisfy a test that reads “the fixture path.” Lock bytes.

def load_lock(path: Path) -> dict[str, str]:
    return json.loads(path.read_text(encoding="utf-8"))

def assert_fixtures(lock: dict[str, str]) -> None:
    for rel, expected in sorted(lock.items()):
        blob = Path(rel).read_bytes()
        actual = hashlib.sha256(blob).hexdigest()
        if actual != expected:
            raise SystemExit(f"fixture drift {rel}: {actual} != {expected}")

Regenerate ci/fixtures.lock only from a human-owned command, for example python -m ci.hash_fixtures tests/fixtures. Put that command in docs, not in the agent prompt. If a patch needs a new fixture, it needs a person.

Step 4 — Property checks that never see the agent prompt

Write properties against the pre-patch public names. Keep them small enough to fail on one invariant. The example below is a proposed Hypothesis test, not a measured production suite.

# tests/test_parse_invoice_properties.py  (proposed)
from hypothesis import given, settings, strategies as st

from src.invoices import parse_invoice, InvoiceError

@settings(max_examples=80, deadline=None)
@given(st.integers(min_value=0, max_value=10_000))
def test_cents_round_trip(cents: int) -> None:
    payload = {"cents": cents, "currency": "USD"}
    inv = parse_invoice(payload)
    assert inv.cents == cents
    assert parse_invoice(inv.to_dict()).cents == cents

@given(st.integers(max_value=-1))
def test_negative_cents_rejected(cents: int) -> None:
    try:
        parse_invoice({"cents": cents, "currency": "USD"})
    except InvoiceError:
        return
    raise AssertionError("negative cents accepted")

Do not ask the agent to “add tests until green.” That instruction recreates the oracle-move failure. If coverage is thin, a person extends tests/ in a separate commit that the write-block will accept because it is not an agent patch.

Step 5 — Freeze flakes by seed, not by display name

Display names churn. Seeds do not. A freeze that keys only on test_cents_round_trip will hide a new failure after a fixture swap. Key on the triple.

{"nodeid": "tests/test_parse_invoice_properties.py::test_cents_round_trip", "seed": 417221, "fixture_digest": "a1b2...", "expires": "2026-09-22", "reason": "tz-boundary on CI image"}

Proposed enforcement:

1. Re-run the failing node with HYPOTHESIS_VERBOSITY=verbose (or your runner’s seed flag) until the seed is known.
2. Refuse a freeze that omits fixture_digest.
3. Drop the row on expires. A freeze without expiry is a skipped test with extra ceremony.
4. Never freeze a failure that reproduces with the same triple on a second local run.

from datetime import date

def freeze_allows(nodeid: str, seed: int, fixture_digest: str, today: date) -> bool:
    for line in Path("ci/flake-ledger.jsonl").read_text().splitlines():
        row = json.loads(line)
        if row["nodeid"] != nodeid:
            continue
        if int(row["seed"]) != seed:
            continue
        if row["fixture_digest"] != fixture_digest:
            continue
        if date.fromisoformat(row["expires"]) < today:
            continue
        return True
    return False

If the same nodeid fails on a new seed, it is not the frozen flake. It is a new defect. Log it as such.

Where a free model and a free server fit

The gate above does not need a vendor. The agent that produces src/ diffs does.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here. Use the free server as an isolated write workspace for src/. Keep tests/, ci/fixtures.lock, ci/contract.allow, and ci/flake-ledger.jsonl off that writable volume, or mount them read-only. Pull the resulting src/ diff back into CI and run gate_agent_patch.py locally. The model never receives a mandate to edit the oracle.

Do not treat a free endpoint as a source of test truth. Endpoint availability changes. Assertions must not.

Decision table

Observation	Gate result	Next action
tests/ in the agent diff	fail	strip test edits; re-run
contract digest unchanged, properties red	fail	reject patch; do not freeze
contract digest changed, no allow-file row	fail	human updates allow file and properties
fixture bytes ≠ lock	fail	human regenerates lock or rejects
same nodeid+seed+digest, before expiry	skip once	calendar the expiry
same nodeid, new seed	fail	treat as new bug

Print this table in the CI log. Reviewers should not have to reconstruct the policy from chat.

Commands to keep in the job

git fetch origin main
python ci/gate_agent_patch.py origin/main HEAD
python -m pytest tests -q --maxfail=1

Add --seed (or equivalent) on the pytest invocation so a red property can be replayed without guessing. Persist the seed in the job log. The ledger is useless if the seed is not recorded on the first failure.

Limitations

The write-block assumes tests are not generated from src/ at commit time. Protobuf stubs, snapshot UIs, and golden-file renderers often require test-tree writes as part of a legitimate patch. This gate will fail those workflows until snapshots move behind a human-owned regenerator.

AST hashing as written ignores class methods, decorators, and re-exports. Extend the visitor before you apply it to a public object API. It also ignores numeric default values unless you add them to the signature string. That is a deliberate first cut, not a complete contract language.

Seed freezes do not distinguish infrastructure flakes from logic bugs. They only bound the skip. If a freeze row is re-added with a later expiry and the same triple, you are hiding a defect. Count re-freezes in CI and fail the job when a triple is revived more than once.

Who should not use this

Do not use a write-blocked test tree if the agent’s deliverable is the test file: teaching kernels, kata generators, or migration tools that must rewrite assertions. Do not use seed freezes on tests that are not deterministic given a seed. Do not import patched code to “verify” the contract hash; that reintroduces execution of untrusted bytes into the gate.

Small scripts with no public function surface get little from an AST digest. A single integration check plus the write-block is enough there.

If you already generate src/ patches with free model access on a free server, drop gate_agent_patch.py into CI before you widen the agent’s write set. The ledger format can change. The read-only test tree should not.