Path-Fence Agent Patches: Properties, Fixtures, and Flakes Stay Read-Only

An agent patch is not verified if it can edit the evidence. Property checks, fixtures, and flaky-test freezes only constrain a model when those files sit outside the writable diff. A green suite after a test rewrite is a classification error, not a pass.

This article specifies a path fence. The agent may change src/. It may not change oracle/. The merge job hashes the oracle tree and rejects any patch that touches it, recreates a frozen path, or deletes a property.

The failure the fence is built for

Agents optimize the reward they can see. If tests live in the same checkout as production code, a cheap strategy is to drop an assertion, skip a flake, or regenerate a golden file from the new output. The production function can stay wrong. The suite stays green.

Flaky tests make the error cheaper. A skip looks like hygiene. A rewritten fixture looks like an update. Neither proves the patch preserved behavior. The fence treats those edits as a different class of change: test mutation, not a product fix.

What stays human-owned

Keep three file classes under oracle/, committed by people, never by the proposer:

1. Properties. Invariants that must hold for any legal input, not one recorded example.
2. Fixtures. Content-addressed inputs and expected digests, not “latest actual output.”
3. Freezes. Flaky cases moved out of the agent-writable tree with an owner and an expiry. Not pytest.mark.skip inside the agent’s diff.

The scoring host mounts oracle/ read-only. The proposer’s workspace mounts src/ read-write. If the two mounts share a working copy, the fence still fails the merge when the diff includes an oracle path.

1. Split the tree before the first agent run

Do this once, on a human commit. Do not let the model propose the split.

repo/
  src/                 # agent may write
  oracle/
    LOCK               # sha256 of the oracle tree
    properties/        # human invariants
    fixtures/          # sha256-named inputs + expected digests
    frozen/            # ownership-transfer flakes
      manifest.json
  tools/
    oracle_fence.py    # merge gate; agent may not write this either

Put tools/oracle_fence.py on the same deny list as oracle/. A model that can rewrite the gate can rewrite the evidence. The allowlist is the production tree only.

2. Lock the oracle with a manifest hash

The lock file is not a changelog. It is a digest of every path under oracle/ except LOCK itself. Recompute it in CI. Compare it to HEAD. Any drift without a human-reviewed oracle commit is a failed gate.

# tools/oracle_fence.py — labeled example, not a production SLA
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

DENY_PREFIXES = ("oracle/", "tools/oracle_fence.py")
ORACLE_ROOT = Path("oracle")
LOCK_PATH = ORACLE_ROOT / "LOCK"


def git_changed_paths(base: str, head: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...{head}"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def tree_digest(root: Path) -> str:
    h = hashlib.sha256()
    paths = sorted(
        p for p in root.rglob("*") if p.is_file() and p.name != "LOCK"
    )
    for path in paths:
        rel = path.as_posix().encode()
        h.update(b"\x00" + rel + b"\x00" + path.read_bytes())
    return h.hexdigest()


def fence(base: str, head: str) -> int:
    changed = git_changed_paths(base, head)
    blocked = [p for p in changed if p.startswith(DENY_PREFIXES)]
    if blocked:
        print("FENCE: agent diff touched oracle or gate:")
        for p in blocked:
            print(f"  - {p}")
        return 2

    actual = tree_digest(ORACLE_ROOT)
    expected = LOCK_PATH.read_text().strip()
    if actual != expected:
        print("FENCE: oracle/LOCK mismatch")
        print(f"  lock={expected}")
        print(f"  tree={actual}")
        return 3

    frozen = json.loads((ORACLE_ROOT / "frozen" / "manifest.json").read_text())
    src_hits = [p for p in changed if p.startswith("src/")]
    for item in frozen["cases"]:
        banned = item["retired_src_path"]
        if banned in src_hits or banned in changed:
            print(f"FENCE: patch recreates frozen path {banned}")
            return 4
    return 0


if __name__ == "__main__":
    sys.exit(fence(sys.argv[1], sys.argv[2]))

Run it against the merge range, not against a dirty worktree the agent still holds.

python tools/oracle_fence.py origin/main HEAD
echo "$?"   # 0 = fence hold; non-zero = do not merge

A non-zero status is terminal for the patch. Do not “retry the model” until a human decides whether the oracle should move.

3. Write properties that the src tree cannot see as skippable tests

A property is a function from generated input to a Boolean. It does not import production test helpers that the agent can stub. Keep the checker in oracle/properties/ and call production symbols only through the public API.

The C++ fragment below is a labeled example for a bounded parser. It is not a claim about a shipping binary.

// oracle/properties/parse_roundtrip.cpp — example invariant
#include "api/parse.hpp"
#include <cstdint>
#include <span>
#include <vector>

bool prop_parse_rejects_oversize(std::span<const uint8_t> in, std::size_t cap) {
    if (in.size() <= cap) return true;           // vacuously out of this property
    auto r = api::parse(in, cap);
    return !r.ok && r.written == 0;              // must not emit a partial object
}

bool prop_roundtrip_under_cap(std::span<const uint8_t> in, std::size_t cap) {
    if (in.size() > cap) return true;
    auto r = api::parse(in, cap);
    if (!r.ok) return true;                      // rejection is allowed
    auto out = api::encode(r.value);
    return out == std::vector<uint8_t>(in.begin(), in.end());
}

Drive those Booleans from a generator the agent does not seed. Pin the generator seed in the scoring job, not in src/. If the patch needs a new exception, a human adds a property or a fixture. The model does not.

4. Address fixtures by digest, not by filename

Name files after the hash of the input bytes. Store the expected digest beside them. A patch that changes output without a human oracle commit will fail the comparison even if the agent invents a new filename.

oracle/fixtures/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08.in
oracle/fixtures/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08.expected

# tools/check_fixtures.py — labeled example
import hashlib
from pathlib import Path

def check_fixtures(run_src_api) -> int:
    root = Path("oracle/fixtures")
    failures = 0
    for inp in sorted(root.glob("*.in")):
        expected = Path(str(inp)[:-3] + ".expected").read_text().strip()
        digest = hashlib.sha256(inp.read_bytes()).hexdigest()
        if digest != inp.stem:
            print(f"corrupt fixture name: {inp}")
            return 5
        actual = hashlib.sha256(run_src_api(inp.read_bytes())).hexdigest()
        if actual != expected:
            print(f"fixture miss {inp.stem}: {actual} != {expected}")
            failures += 1
    return 1 if failures else 0

Do not accept a “fixture refresh” flag on the agent job. Refresh is an oracle commit. It needs a reviewer who can say the new digest is intended.

5. Freeze flakes by ownership transfer, not by skip

When a test is flaky, do not leave it in src/ with a skip marker the next patch can delete. Copy the case into oracle/frozen/, record who owns the freeze, and delete the original path so the agent cannot resurrect it as a fake fix.

{
  "cases": [
    {
      "id": "parse_timeout_udp_1472",
      "retired_src_path": "src/tests/test_parse_timeout.cpp",
      "owner": "protocols",
      "reason": "repro rate < 0.02 on default runner; not an agent skip",
      "expires_unix": 1764000000,
      "repro_cmd": "ctest -R parse_timeout_udp --repeat-until-fail 50"
    }
  ]
}

The scoring host never runs frozen cases as merge evidence. A separate human-owned job can still run repro_cmd after expiry. If expires_unix is in the past and the path is still frozen, the human job fails. That failure is not assigned to the agent patch. It is assigned to the freeze owner.

Numbered freeze rule:

1. Observe flake on the scoring host with a pinned seed and pinned machine image.
2. Record repro_cmd and the observed rate. Do not guess.
3. Move the file into oracle/frozen/ and delete retired_src_path.
4. Update oracle/LOCK on a human commit.
5. Reject any later agent diff that recreates retired_src_path or edits manifest.json.

6. Where a free proposer belongs

The proposer and the scorer must not share write access to oracle/. If you need a proposer isolated from that tree, MonkeyCode’s free model access and free server option can generate a src/-only diff. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the fence, the fixture checker, and oracle/LOCK on a host the model cannot reach. Feed the model a pack that omits oracle/ and tools/oracle_fence.py. Score the returned patch with the commands above. Isolation is the feature. Model brand is not the evidence.

Decision table

Diff content	Fence result	Next action
src/** only, properties hold, fixtures match	pass	human review of the production change
any oracle/** path	fail (code 2)	discard patch; open a human oracle PR if needed
oracle/LOCK drift without oracle files in the diff	fail (code 3)	stop; working copy is corrupt or incomplete
recreate retired_src_path	fail (code 4)	discard; freeze still owns the case
fixture digest mismatch	fail (checker)	do not refresh from the agent; inspect production change
skip marker added under src/	fail if your style gate bans skips	freeze by transfer instead

Limitations

The fence does not prove the properties are strong. A tautological property that returns true for every input will stay green. Humans still have to write invariants that can fail.

The digest lock does not detect semantic fixture rot when the expected digest was updated for the wrong reason. It only detects unauthorized edits. Review of oracle commits remains a people process.

Ownership-transfer freezes hide coverage. If too many cases sit in oracle/frozen/, the merge signal gets weaker. Set a cap per owner. Expire aggressively. Do not use freezes as a parking lot for tests the team does not want to run.

Clock skew can mis-fire expires_unix. Pin CI time or store expiry as a commit SHA plus a human ticket, not as a wall clock on an untrusted runner.

This workflow also assumes a two-tree repo. Monorepos that generate tests from src/ need an extra deny rule for generated output. If the generator is agent-writable, the fence is incomplete.

Who should not use this

Skip the fence if the repository is a single file with no distinct test tree. Skip it if no human will own oracle/LOCK. Skip it if every failure is already deterministic and the agent cannot reach the test directory. Skip it for exploratory spikes that will not merge.

Teams that let the same process both propose and merge should not adopt a subset of these files as decoration. A read-only oracle/ that the merge job never hashes is documentation, not a gate.

The core rule stays small. Score production diffs against properties, fixtures, and freezes the agent cannot write. If the evidence moved, you did not score the patch. You scored a new test suite.