Fence Fixture and Freeze Paths Before You Score an Agent Patch

An agent patch that edits production code and also edits fixtures, goldens, or the flake freeze file is scoring itself. The job can be green for reasons that have nothing to do with the defect. Fence those paths in CI. Score the patch on properties and on input-only fixtures the patch was not allowed to touch.

That rule is small. It removes a class of false passes that mixed test and code diffs hide.

What the agent must not write

Three path classes belong to the test system, not to the patch:

1. Input fixtures and golden files under tests/fixtures/ and tests/goldens/.
2. Property generators and recorded seeds under tests/properties/.
3. The freeze ledger tests/freeze.jsonl, which records flakes CI actually reproduced.

If git diff --name-only origin/main...HEAD lists any of those paths, fail the merge job before tests run. Running tests after the oracle moved tells you nothing about the patch.

This is not a style preference. It is a write barrier.

A scoring order that survives generated code

Use one order. Do not skip steps when the patch looks small.

1. Classify paths in the merge diff.
2. Reject the patch if oracle, generator, or freeze paths changed.
3. Run property checks with an explicit minimum hit count.
4. Run input-only fixtures against the patched code.
5. Allow a freeze only when CI reproduces a flake on a recorded seed and shuffle, then append the ledger itself.

Properties still matter. Fixtures still matter. A freeze still exists. The change is ownership: the agent proposes code. CI owns evidence.

Decision table

Diff touches	Property hits	Fixture result	Freeze writer	Merge
src/ only	>= N distinct	pass	n/a	allow
src/ only	< N	pass	n/a	reject
src/ + tests/goldens/	any	any	any	reject
src/ + tests/freeze.jsonl	any	any	agent	reject
src/ only, one fixture flakes	>= N	flake reproduced by CI	CI ledger	allow with freeze
src/ only, flake not reproduced	>= N	fail	none	reject

N is a job parameter, not a feeling. Pick it from the generator, not from wall-clock luck.

Step 1 — Classify the diff

The script below is a local, unexecuted template against your own default branch. It does not need a vendor action. Wire it as a required check and stop on exit code 2.

#!/usr/bin/env python3
"""classify_agent_diff.py — fail if the patch touches oracle paths."""
from __future__ import annotations

import subprocess
import sys

ORACLE_PREFIXES = (
    "tests/fixtures/",
    "tests/goldens/",
    "tests/properties/",
)
ORACLE_FILES = {"tests/freeze.jsonl"}


def changed_files(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def oracle_hits(files: list[str]) -> list[str]:
    hits: list[str] = []
    for rel in files:
        if rel in ORACLE_FILES or any(rel.startswith(p) for p in ORACLE_PREFIXES):
            hits.append(rel)
    return hits


def main() -> int:
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    files = changed_files(base)
    blocked = oracle_hits(files)
    print("changed:")
    for rel in files:
        print(f"  {rel}")
    if blocked:
        print("blocked oracle paths:")
        for rel in blocked:
            print(f"  {rel}")
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it before pytest:

python3 classify_agent_diff.py origin/main

Exit 2 means the patch touched evidence. Do not continue. A later green job would be contaminated.

Step 2 — Property checks with a hit counter

A property that never observes an input is not a check. Count distinct inputs that reached the assertion. Fail the job when the count is below N, even if no assertion failed.

The module name app.parse is a stand-in. Replace it with the package under test. Keep the seed and the minimum hit count in the environment so two jobs can be compared.

# tests/properties/test_parse_budget.py
from __future__ import annotations

import os
import random
from collections.abc import Iterator

from app.parse import parse_query

MIN_HITS = int(os.environ.get("PROPERTY_MIN_HITS", "50"))
SEED = int(os.environ.get("PROPERTY_SEED", "20260921"))


def queries(seed: int, n: int) -> Iterator[str]:
    rng = random.Random(seed)
    alphabet = "abcxyz0123 _-*"
    for _ in range(n):
        k = rng.randint(0, 24)
        yield "".join(rng.choice(alphabet) for _ in range(k))


def test_parse_query_never_raises_and_records_hits() -> None:
    hits: set[str] = set()
    for q in queries(SEED, MIN_HITS):
        result = parse_query(q)
        assert result is not None
        assert isinstance(result.terms, list)
        hits.add(q)
    assert len(hits) >= MIN_HITS, f"hits={len(hits)} min={MIN_HITS} seed={SEED}"

Export both variables from CI. Print both in the job log. A pass with a silent default seed is not comparable across runs.

export PROPERTY_SEED=20260921
export PROPERTY_MIN_HITS=50
pytest tests/properties/test_parse_budget.py -q

Step 3 — Input-only fixtures

Store inputs. Do not store expected blobs next to them if an agent can edit both in one diff. Derive expected values from a constraint the patch cannot change in the same change set, or from a prior release artifact that lives outside the branch.

must_include is a constraint, not a full golden transcript. Full transcripts drift. Agents rewrite them. Constraints stay small and stay fenced.

# tests/fixtures/test_queries.py
from pathlib import Path
import json
from app.parse import parse_query

FIXTURE_DIR = Path(__file__).parent / "queries"


def test_each_input_fixture_parses() -> None:
    paths = sorted(FIXTURE_DIR.glob("*.json"))
    assert paths, "fixture directory empty"
    for path in paths:
        payload = json.loads(path.read_text())
        assert "q" in payload and "must_include" in payload
        result = parse_query(payload["q"])
        for token in payload["must_include"]:
            assert token in result.terms, f"{path.name} missing {token}"

Example fixture file, checked in by a human, never by the patch under review:

{"q": "foo bar", "must_include": ["foo", "bar"]}

Step 4 — CI writes the freeze ledger

Do not accept a freeze hunk in the agent commit. If a fixture flakes, CI reproduces it. A five-iteration loop is enough to start. Optional pytest plugins can replace the loop later; they are not required for the rule.

seed="${PROPERTY_SEED:-20260921}"
fails=0
for i in 1 2 3 4 5; do
  PYTHONHASHSEED=$((seed + i)) pytest tests/fixtures/test_queries.py -q
  status=$?
  if [ "$status" -ne 0 ]; then
    fails=$((fails + 1))
  fi
done
echo "repro_fails=${fails} seed=${seed}"

Record the outcome in a ledger CI appends on the default branch, not in the pull request. Label the next script as CI-bot only.

# ci/append_freeze.py — run only on the CI bot account
from __future__ import annotations

import json
import os
import time
from pathlib import Path

LEDGER = Path("tests/freeze.jsonl")


def main() -> None:
    record = {
        "test_id": os.environ["FREEZE_TEST_ID"],
        "seed": os.environ["PROPERTY_SEED"],
        "hashseed_base": os.environ.get("PYTHONHASHSEED", ""),
        "reproduced": os.environ["FREEZE_REPRODUCED"] == "1",
        "writer": "ci",
        "ts": int(time.time()),
    }
    if record["writer"] != "ci":
        raise SystemExit("refusing non-ci writer")
    if not record["reproduced"]:
        raise SystemExit("refusing freeze without reproduction")
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(record, sort_keys=True) + "\n")


if __name__ == "__main__":
    main()

A freeze without reproduced=true, a seed, and writer=ci is not a freeze. It is a skip list the patch smuggled in.

Job log shape

Emit one JSON object per stage. Humans can read it. Machines can grade it without scraping pytest banners.

{"event":"path_fence","blocked":[],"ok":true}
{"event":"property","seed":20260921,"hits":50,"min":50}
{"event":"fixture","passed":12,"failed":0,"flaked":0}
{"event":"freeze","writer":"ci","reproduced":false}

Grade the objects, not the job color. A missing path_fence object is a fail. A freeze object with writer other than ci is a fail.

Where a free model and a free server fit

Generated patches need a cheap loop that still obeys the fence. A laptop can run classify_agent_diff.py. Overnight candidate volume usually needs a remote job.

MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two pieces are enough for this workflow: the model proposes a src/ diff, the server runs the path fence and the property/fixture jobs, and the freeze ledger stays off-limits to the model. Do not treat that loop as a merge. Treat it as a filter that discards contaminated diffs before paid CI or a human review queue.

If you already run the fence on your own runners, skip the product. The scripts above do not depend on it.

Limitations

Path fences do not detect a tautology inside an existing property file the agent did not touch. They also miss a fixture that was weak before the patch. The barrier is about who writes evidence in the same diff, not about the quality of old tests.

A freeze ledger on the default branch can still rot. Reproduction today does not bind the test forever. Re-run the shuffle on a schedule. Drop records that no longer flake.

PROPERTY_MIN_HITS can be gamed if the generator emits duplicates. Count distinct inputs, not loop iterations. The set in the sample test is the control. A raw counter is not.

This workflow assumes you can split src/ from tests/ in the repository. A tree that colocates generated snapshots inside src/ needs a tighter prefix list. Update ORACLE_PREFIXES before you enable the exit-2 gate.

Branch protection has to match the scripts. If anyone can push to tests/freeze.jsonl on the default branch, the ledger is another oracle.

Who should not use this

Do not use a hard path fence if humans routinely land fixture updates in the same commit as production fixes and you have no way to split them. The gate will block legitimate work. Split the commits first.

Do not use CI-authored freezes if you cannot protect tests/freeze.jsonl with branch rules. An unprotected ledger is another oracle the next patch will edit.

Do not point a free server at secrets, private customer fixtures, or production data. Generated-query alphabets are enough for the parse example. Real traffic traces belong in a locked store the agent cannot read or write.

What to merge

Merge a patch when the diff stays inside production paths, property hits meet N, fixtures pass, and any freeze was appended by CI after a reproduced shuffle. Everything else is a contaminated green.

Keep the classifier, the hit counter, and the ledger writer in the same required job group. One missing piece restores the old failure mode: the patch writes the evidence, then the evidence blesses the patch.