Put a Write Boundary Between the Agent Patch and the Oracle

An agent patch that can edit its own tests is not under test. Green CI then records agreement with a rewritten oracle, not preservation of the behavior you meant to keep. The practical fix is a write boundary: the model may change production code, and it may add tests in a sandbox directory, but it cannot own the property checks, the fixtures, or the flake freeze that decide merge.

The failure is oracle capture

Agent patches fail in a different way than human patches. They often make the suite green by moving the goalposts. A skipped test, a loosened assertion, a regenerated golden file, and a deleted property all look like progress in a diff that is scored on CI color.

That is oracle capture. The patch under evaluation becomes the author of the evidence used to accept it. Any later claim that "the tests passed" is then a statement about the new oracle, not about the old contract.

The rest of this article is a testing strategy that assumes capture is the default risk. It does not assume a trusted model. It assumes a trusted directory layout, a small guard script, and a human-owned freeze.

Three layers the patch does not own

Keep three oracle layers outside the agent's write set. Each layer answers a different question.

Property checks answer which invariants still hold for any input in a class. They are small, side-effect free, and hostile to one-off fixtures the model could memorize. They live in tests/oracle/properties/.

Fixtures answer whether a known input still produces the agreed bytes, status, or shape. They are boring on purpose. They live in tests/oracle/fixtures/ with a lockfile of hashes. Regeneration is a human pull request.

The flake freeze answers which tests are allowed to be silent, for how long, and who owns the silence. It is a data file, not a decorator the patch can sprinkle into test modules. It lives at tests/oracle/flake-freeze.toml.

If the agent can edit any of those three, the other two become theater.

A four-step write boundary

1. Declare path classes in version control, not in chat instructions.
2. Reject any agent commit that touches oracle paths, even if the model claims it is "fixing tests."
3. Compare assertion weight on existing test files against the merge base; fail on net loss.
4. Run property checks and fixture replay after the write check, never before. A green run on a captured oracle is discarded.

The order matters. Cheap path and assertion checks go first. Generative properties go second. Fixture replay goes third, because bytes on disk are the slowest signal and the easiest to poison if you run them while the lockfile is still writable.

Artifact: oracle_guard.py

The following script is a complete, labeled example. It is meant to run in CI against origin/main (or any merge base you pass). It does not execute the suite. It only enforces ownership of the oracle.

#!/usr/bin/env python3
"""Reject agent patches that capture the oracle or weaken existing tests."""

from __future__ import annotations

import argparse
import hashlib
import re
import subprocess
import sys
from pathlib import Path

ORACLE_PREFIXES = (
    "tests/oracle/properties/",
    "tests/oracle/fixtures/",
    "tests/oracle/flake-freeze.toml",
    "tests/oracle/fixtures.lock",
)

SANDBOX_PREFIX = "tests/agent_sandbox/"

WEAK_LINE = re.compile(
    r"assert\s+True\b|pytest\.mark\.(skip|xfail)|self\.skipTest\(|unittest\.skip",
    re.M,
)
ASSERT_LINE = re.compile(r"\bassert\b|self\.assert|pytest\.raises", re.M)


def git(*args: str) -> str:
    return subprocess.check_output(["git", *args], text=True).strip()


def changed_files(base: str) -> list[str]:
    out = git("diff", "--name-only", f"{base}...HEAD")
    return [line for line in out.splitlines() if line]


def file_at(rev: str, path: str) -> str:
    try:
        return git("show", f"{rev}:{path}")
    except subprocess.CalledProcessError:
        return ""


def assert_weight(src: str) -> int:
    return len(ASSERT_LINE.findall(src)) - len(WEAK_LINE.findall(src))


def check_write_boundary(files: list[str]) -> list[str]:
    errors = []
    for path in files:
        if any(path == p or path.startswith(p) for p in ORACLE_PREFIXES):
            errors.append(f"oracle path edited by patch: {path}")
    return errors


def check_assertion_weight(base: str, files: list[str]) -> list[str]:
    errors = []
    for path in files:
        if not path.startswith("tests/") or path.startswith(SANDBOX_PREFIX):
            continue
        if path.startswith("tests/oracle/"):
            continue
        old, new = file_at(base, path), file_at("HEAD", path)
        if old and assert_weight(new) < assert_weight(old):
            errors.append(
                f"assertion weight dropped in {path}: "
                f"{assert_weight(old)} -> {assert_weight(new)}"
            )
    return errors


def check_lockfile() -> list[str]:
    lock = Path("tests/oracle/fixtures.lock")
    root = Path("tests/oracle/fixtures")
    if not lock.exists() or not root.exists():
        return ["missing fixtures.lock or fixtures directory"]
    listed = {}
    for line in lock.read_text().splitlines():
        if not line.strip() or line.startswith("#"):
            continue
        digest, rel = line.split(None, 1)
        listed[rel] = digest
    errors = []
    for path in sorted(root.rglob("*")):
        if not path.is_file():
            continue
        rel = path.as_posix()
        digest = hashlib.sha256(path.read_bytes()).hexdigest()
        if rel not in listed:
            errors.append(f"unlocked fixture: {rel}")
        elif listed[rel] != digest:
            errors.append(f"fixture hash mismatch: {rel}")
    for rel in listed:
        if not Path(rel).is_file():
            errors.append(f"lock entry missing on disk: {rel}")
    return errors


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", default="origin/main")
    args = parser.parse_args()
    files = changed_files(args.base)
    errors = []
    errors += check_write_boundary(files)
    errors += check_assertion_weight(args.base, files)
    errors += check_lockfile()
    if errors:
        print("oracle guard failed:")
        for item in errors:
            print(f"  - {item}")
        return 1
    print(f"oracle guard passed for {len(files)} changed path(s)")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it before pytest, not after.

python3 oracle_guard.py --base origin/main
pytest tests/oracle/properties tests/oracle/fixtures tests/agent_sandbox

A labeled CI fragment:

# .github/workflows/oracle.yml
name: oracle-boundary
on: [pull_request]
jobs:
  guard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python3 oracle_guard.py --base origin/main
      - run: pytest -q tests/oracle tests/agent_sandbox

Property checks that live outside the patch

Property checks should not require a golden output the model can overwrite. They should relate two observations of the same function. The example below is a contract for a parser the agent is allowed to patch. The test file itself is oracle-owned.

# tests/oracle/properties/test_parse_contract.py
from src.config_parse import dump, parse

SAMPLES = (
    {},
    {"region": "us-east-1"},
    {"region": "us-east-1", "retries": 0},
    {"nested": {"a": 1, "b": [1, 2, 3]}},
)


def test_dump_parse_preserves_keys_and_types():
    for sample in SAMPLES:
        roundtrip = parse(dump(sample))
        assert set(roundtrip) == set(sample)
        for key, value in sample.items():
            assert type(roundtrip[key]) is type(value)


def test_parse_rejects_non_object_payloads():
    for bad in ("", "[]", "null", "1", "\"x\""):
        try:
            parse(bad)
        except ValueError:
            continue
        raise AssertionError(f"expected ValueError for {bad!r}")

That is not a full property-based library. It is a fixed sample set plus two relations: key-set identity after roundtrip, and type identity after roundtrip. Expand the sample set in an oracle PR when you discover a new class of input. Do not let the patch invent the samples that would make it pass.

If you later add a generative library, keep the generator in tests/oracle/ as well. A generator that lives next to the production diff will eventually emit only values the patch already handles.

Fixture hashes the agent cannot refresh

Store fixtures as raw files. Store their SHA-256 digests in tests/oracle/fixtures.lock. The guard script above treats a hash mismatch as a failed boundary, not as a hint to regenerate.

# tests/oracle/fixtures.lock
# sha256  path
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  tests/oracle/fixtures/empty.json

A human updates both files together. The agent does not. If a legitimate format change requires new bytes, that is a product decision, and it belongs in a second pull request that contains no production-code "fix" from the same patch.

A freeze file the agent cannot grow

Flaky tests still exist. The mistake is encoding the freeze as pytest.mark.skip inside a module the agent can edit. Put the freeze in oracle data instead, and teach the runner to read it.

# tests/oracle/flake-freeze.toml
# Example file. Expiry is a calendar date, not a hope.

[[freeze]]
nodeid = "tests/slow/test_queue.py::test_drain_under_load"
reason = "timing-sensitive on shared runners"
owner = "platform"
expires = "2026-09-20"

A small runner hook (labeled example) refuses expired rows and applies silence only from this file:

# tests/oracle/freeze_plugin.py
from __future__ import annotations

import datetime as dt
from pathlib import Path

try:
    import tomllib
except ImportError:  # Python < 3.11
    import tomli as tomllib

FREEZE_PATH = Path("tests/oracle/flake-freeze.toml")


def load_freeze() -> dict[str, dict]:
    data = tomllib.loads(FREEZE_PATH.read_text())
    today = dt.date.today().isoformat()
    active = {}
    for row in data.get("freeze", []):
        if row["expires"] < today:
            raise RuntimeError(
                f"expired freeze for {row['nodeid']}; delete it or fix the test"
            )
        active[row["nodeid"]] = row
    return active


def pytest_collection_modifyitems(config, items):
    freeze = load_freeze()
    for item in items:
        if item.nodeid in freeze:
            item.add_marker("skip")

Policy: an agent commit that adds a freeze row is already rejected by the write boundary. A human may add a row with an owner and a short expiry. After expiry, CI fails closed. Silence without an owner is not an allowed state.

Decision table

Signal in the agent diff	Merge rule
src/ only	Allowed. Run properties + fixtures. Freeze unchanged.
New file under tests/agent_sandbox/	Allowed. Treat as extra evidence, never as the only evidence.
Edit under tests/oracle/	Reject. Oracle changes go in a separate human PR.
Net drop in assertion weight	Reject. That is capture, even if CI is green.
Fixture bytes change without lock + human PR	Reject.
New skip/xfail not listed in the freeze	Reject.
Freeze expiry in the past	Reject the build, not the product code in isolation.

Where a free coding server fits

Generating a candidate patch and evaluating it against a frozen oracle are different jobs. They should not share a writable checkout. One workable split is to let a coding agent propose a diff on an isolated server, then pull that diff into a read-bound CI job that runs oracle_guard.py and the oracle suite.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that isolated proposal step. The oracle still lives in your repository, and the guard script above does not depend on any vendor. If you already have a sandbox, keep it. The boundary is the method.

Limitations

The guard is path-based. A model that edits src/ to no-op a function the properties never call will still go green. Properties have to cover the invariants you care about. The boundary does not invent them.

Hash-locked fixtures freeze bytes, not meaning. A whitespace-only change looks like a behavioral change. That is the point of a lockfile, and it is also why fixture updates need a human.

Assertion weight is a heuristic. Replacing three weak asserts with one strong pytest.raises can trip the check. When that happens, update the test in a human PR and record why the weight dropped.

The freeze file fails closed on expiry. That can block unrelated work if nobody owns the flaky test. Assign owner to a real rotation, or do not freeze the test.

This strategy assumes you can keep tests/oracle/ out of the agent's tool write-set. If your agent loop uses a single writable workspace with no path policy, the script will only catch capture after the fact, in CI. That is better than nothing. It is worse than a sandbox that cannot open oracle files.

Who should not use this

Do not install a write boundary if the tests themselves are the product under change and the agent is supposed to refactor the suite. The policy will fight the task.

Do not use hash-locked fixtures for tests whose output is intentionally non-deterministic (clocks, random IDs, network order) unless you have already stripped those fields. You will freeze noise.

Do not use a calendar freeze as a substitute for quarantine infrastructure if you cannot fail the build when a date passes. An ignored expiry is a skip list with extra ceremony.

Skip the sandbox-directory idea if you will treat agent-authored tests as equivalent to oracle tests. Extra tests are optional evidence. They are not a contract.

The core rule is small enough to repeat. The patch proposes behavior. The oracle, which the patch cannot write, accepts or rejects it. Property checks, locked fixtures, and a human-owned flake freeze are three ways to keep that sentence true.