Agent Patches Need a Second Checkout for Property Checks

A single checkout cannot both apply an agent patch and certify it. If the model can edit tests, fixtures, or skip markers in the same tree, green is not evidence. It is a rewrite.

Split the job. Keep a second, read-only checkout as the oracle. Hash fixtures there. Record flakes as freeze records with a seed and an environment digest. Do not convert those flakes into xfail.

This article is a proposed harness, not a production case study. Commands and modules below are labeled examples. Adapt paths and merge bases to your repo.

Shared-tree pytest is the wrong unit

Agent pipelines often run one pytest on the patched worktree. That is cheap. It is also the hole.

Equality tests die when expected files sit in the patch. Fixture drift dies when the agent updates golden JSON. Flakes die when someone adds @pytest.mark.xfail and the gate learns nothing about replay.

You do not need a new framework. You need a second tree, a digest, and a ledger the patch cannot append to from inside the model’s write paths.

Policy table: what may vote

The table is a policy proposal. It is not a measured fleet result.

Observation	Source tree	Merge vote	Agent write allowed
Property check on locked inputs	reference checkout	yes	no
Fixture digest change vs origin/main	reference checkout	block	no
New skip / xfail / timeout in the patch diff	patched checkout (inspect only)	block	n/a
Non-deterministic fail that replays on a stored seed	freeze ledger	no vote; record only	no
Assertion whose expected file is in the patch write-set	discarded	no	ignore

Property checks vote. Fixture mismatch blocks. Freeze records explain noise. They do not pass the gate.

1. Pin a reference checkout

Create a worktree from the merge base, not from the agent branch. The oracle must not see patch-only files.

BASE=$(git merge-base HEAD origin/main)
git fetch origin main
git worktree add --detach /tmp/oracle-ref "$BASE"
# Example: refuse later writes from the generator workspace
chmod -R a-w /tmp/oracle-ref/tests

If your agent runs as the same OS user, chmod is not a security boundary. It is a footgun guard. Run the generator in a separate user, container, or host if the patch is untrusted.

Collect the write-set from the patched tree only:

git diff --name-only "$BASE"...HEAD > /tmp/write-set.txt

Any path under tests/ that appears in /tmp/write-set.txt is untrusted as an oracle. You may still inspect it for skip-marker deltas. You must not execute it as a pass signal.

2. Digest fixtures the patch cannot edit

Hash the reference fixture directory. Store the digest next to the freeze ledger, not in the agent branch.

# digest_tree.py — example helper, not a shipped tool
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def file_digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def tree_digest(root: Path) -> dict:
    files = sorted(p for p in root.rglob("*") if p.is_file())
    rows = {str(p.relative_to(root)): file_digest(p) for p in files}
    blob = json.dumps(rows, sort_keys=True).encode()
    return {
        "root": str(root),
        "sha256": hashlib.sha256(blob).hexdigest(),
        "files": rows,
    }


if __name__ == "__main__":
    payload = tree_digest(Path(sys.argv[1]))
    json.dump(payload, sys.stdout, indent=2)
    sys.stdout.write("\n")

python3 digest_tree.py /tmp/oracle-ref/tests/fixtures > /tmp/fixture-digest.json
# Compare against the last accepted digest on origin/main
diff -u ci/fixture-digest.json /tmp/fixture-digest.json

A digest mismatch is a block, not a rewrite opportunity. If fixtures must change, that change lands in a human-authored commit on the reference branch first. The agent patch then rebases onto it.

3. Run properties from the reference tree

Keep properties small and oracle-owned. The example target is a pure function. Replace it with your real module imported from the patched code while inputs stay in the reference tree.

# properties_clamp.py — example; label as unexecuted until you wire your package
from __future__ import annotations

import random
from dataclasses import dataclass


def clamp(x: float, lo: float, hi: float) -> float:
    if lo > hi:
        lo, hi = hi, lo
    return min(max(x, lo), hi)


@dataclass(frozen=True)
class Case:
    x: float
    lo: float
    hi: float


def cases(seed: int, n: int = 200) -> list[Case]:
    rng = random.Random(seed)
    out = []
    for _ in range(n):
        lo = rng.uniform(-10, 10)
        hi = rng.uniform(-10, 10)
        x = rng.uniform(-20, 20)
        out.append(Case(x, lo, hi))
    return out


def check(case: Case) -> None:
    y = clamp(case.x, case.lo, case.hi)
    lo, hi = sorted((case.lo, case.hi))
    assert lo <= y <= hi, (case, y)
    assert clamp(y, case.lo, case.hi) == y, (case, y)

Run that module with PYTHONPATH pointing at the patched package build, and with cwd or fixture paths pointing at /tmp/oracle-ref. Two directories. One process for generation, one for oracle.

# Example layout. Patch code is built from HEAD; inputs stay on the reference tree.
PYTHONPATH="$PWD/src" python3 - <<'PY'
from properties_clamp import cases, check
seed = 17
for case in cases(seed):
    check(case)
print("properties_ok", seed)
PY

If a property needs files, read them from /tmp/oracle-ref/tests/fixtures only. If the patched tree also has that path, ignore it.

4. Inspect skip deltas; do not execute them

Scan the write-set for outcome-changing markers. A new skip is a failed check in this policy.

python3 - <<'PY'
from pathlib import Path
markers = ("pytest.mark.skip", "pytest.mark.xfail", "pytest.skip(", "pytest.xfail(")
changed = Path("/tmp/write-set.txt").read_text().splitlines()
blocked = []
for rel in changed:
    p = Path(rel)
    if p.suffix != ".py" or not p.exists():
        continue
    text = p.read_text(errors="replace")
    if any(m in text for m in markers):
        blocked.append(rel)
if blocked:
    raise SystemExit("skip_or_xfail_in_write_set:\n" + "\n".join(blocked))
print("no_skip_markers_in_write_set")
PY

This is static inspection. It will false-positive on comments. Tighten it with AST if your suite is large. Do not weaken it by executing the patched test module as a voter.

5. Freeze the observation, not the test

A flake is an observation that does not replay under a stored seed and digest. It is not permission to skip.

# freeze_ledger.py — example JSONL registrar
from __future__ import annotations

import argparse
import hashlib
import json
import os
import time
from pathlib import Path

LEDGER = Path(os.environ.get("FREEZE_LEDGER", "freeze.jsonl"))


def env_digest() -> str:
    parts = [
        os.uname().sysname,
        os.uname().release,
        os.environ.get("PYTHONHASHSEED", ""),
    ]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]


def record(test_id: str, seed: int, status: str, fixture_sha: str) -> None:
    row = {
        "ts": int(time.time()),
        "test_id": test_id,
        "seed": seed,
        "status": status,  # "flake" | "replay_ok" | "replay_fail"
        "fixture_sha": fixture_sha,
        "env": env_digest(),
    }
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row, sort_keys=True) + "\n")


def latest(test_id: str) -> dict | None:
    rows = []
    if LEDGER.exists():
        for line in LEDGER.read_text().splitlines():
            row = json.loads(line)
            if row["test_id"] == test_id:
                rows.append(row)
    return rows[-1] if rows else None


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("cmd", choices=("record", "show"))
    p.add_argument("--test", required=True)
    p.add_argument("--seed", type=int, default=0)
    p.add_argument("--status", default="flake")
    p.add_argument("--fixture-sha", default="")
    args = p.parse_args()
    if args.cmd == "record":
        record(args.test, args.seed, args.status, args.fixture_sha)
        return
    print(json.dumps(latest(args.test), indent=2))


if __name__ == "__main__":
    main()

Rules for the ledger:

1. Append only. Never delete a flake row to make CI green.
2. A flake row does not vote pass. It parks the test out of the merge predicate until replay_ok lands against the same fixture_sha.
3. If fixture_sha changes, old freeze rows are stale. Re-run properties. Do not inherit a freeze across fixture edits.
4. Cap parked tests. If the ledger grows without replay_ok, stop merging agent patches into that package. The suite is telling you the oracle is under-specified, not that the gate should relax.

FIXTURE_SHA=$(python3 -c "import json; print(json.load(open('/tmp/fixture-digest.json'))['sha256'])")
python3 freeze_ledger.py record --test properties_clamp --seed 17 --status flake --fixture-sha "$FIXTURE_SHA"
python3 freeze_ledger.py show --test properties_clamp

Timeouts belong in the same ledger with status=flake and a recorded duration budget. They are not silent skips.

Where an isolated generator helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The generator should not share a filesystem with /tmp/oracle-ref. MonkeyCode's free model access and free server option are relevant here as an isolated place to produce the patch tarball or git bundle. You still pull that artifact into a local or CI runner that owns the reference checkout, the digest, and the freeze ledger.

Do not paste oracle source, fixture bytes, or ledger contents into the generation prompt if the model can emit patches against those files. The second checkout is wasted if the prompt reconstitutes the tests.

This workflow does not depend on a particular model name, quota, or hardware profile. Those details change. The split does not.

Merge predicate (example)

pass := properties_ok
        AND fixture_digest_unchanged
        AND no_skip_markers_in_write_set
        AND freeze_ledger has zero rows with status=replay_fail
        AND parked flake count <= MAX_PARKED

# freeze rows with status=flake do not contribute to pass

MAX_PARKED is a team constant. Start at 0 if the package is small. Raise it only with a written owner for replay work. A freeze ledger that only grows is a skip list with extra JSON.

Limitations

The harness does not stop a model that is given write access to the reference worktree. chmod and extra directories fail closed only if CI enforces the paths.

Static skip detection misses dynamic pytest.skip in helpers. Property examples here cover a pure function. I/O, clocks, and network still need injected clocks and recorded seeds, or they will flood the ledger.

Digest comparison is byte-level. Canonical JSON reordering will look like fixture drift. Normalize before hashing, or you will block harmless encodings.

The freeze ledger is not a flaky-test product. It will hide a real regression if you park the test and keep merging. Replay is mandatory. Parking without an owner is just xfail with a timestamp.

Who should not use this

Do not use a second checkout if your tests already live in a separate immutable repo and agents cannot open PRs against it. You already have the split.

Do not use freeze records on safety-critical paths, payments, or authz. Parked observations are unacceptable there. Fail the patch.

Do not use this policy to launder a red suite. If properties cannot run from origin/main today, fix the reference tree first. An agent patch cannot donate an oracle you do not already trust.

Close

Execute properties from a tree the patch cannot write. Hash fixtures before the model runs. Record flakes as replayable rows. Leave xfail out of the write-set.

The merge question is then small: did the oracle pass on locked inputs, and did the ledger stay within its park budget? Everything else is noise the agent is incentivized to edit.