Stop Agent Patches From Owning Both the Code and the Oracle

An agent patch is not mergeable because the suite is green. It is mergeable when the production diff cannot rewrite the oracle, property checks consume fixtures the agent cannot touch, and every flake lives in a dated ledger instead of in the test file. Shared authorship of code and assertion is the failure mode. The rest of this article is a filesystem contract that makes that split enforceable in CI.

Green is cheap when the generator writes src/ and tests/ in one turn. Assertions echo new constants. Golden files move to match the bug. Instability is silenced in the module rather than recorded. None of those fail a job whose only gate is pytest exiting 0.

What the contract actually checks

This is a policy on paths and token overlap. It is not a review slogan. The table is the whole design.

Symptom	What the diff did	What CI must require
New assert equals a literal from the patch	Oracle and implementation share a constant	Properties load frozen inputs; tests may not reuse patch string literals
Fixture JSON rewritten in the same PR	Expected output moved to the bug	oracle/ is write-denied for the generator
Skip, xfail, or a stretched timeout	Flake hidden in the suite	Flake id must exist in flake_ledger.json with expires_on

If a change cannot be expressed against those three rows, it does not belong in an agent write set.

Layer 1: an oracle directory the generator cannot write

Put receipts outside the write set. HTTP transcripts, golden files, and captured return values belong in oracle/. The generator may read them. The merge job must reject any patch that edits that tree.

Proposed layout:

oracle/
  receipts/
    parse_invoice_v3.json
    rate_limit_429.json
  CODEOWNERS
src/
  billing/
    parse.py
tests/
  properties/
    test_parse_invariants.py
  ledger/
    flake_ledger.json
scripts/
  check_agent_contract.py
  enforce_flake_ledger.py
  record_receipt.py

CODEOWNERS can assign humans. That is not enough. Ownership files do not run. The merge job still has to diff the path.

# proposed CI fragment; label as unexecuted
set -euo pipefail
CHANGED=$(git diff --name-only origin/main...HEAD)
if echo "$CHANGED" | grep -E '^oracle/'; then
  echo "contract fail: write set includes oracle/"
  exit 1
fi

Receipts are data. They are not tests. A receipt records an input, a constraint block, and the commit that produced it. Properties interpret receipts. If the generator wants a new receipt, a human records it from a known HEAD, not from the model's self-check.

# scripts/record_receipt.py — proposed helper, not a measured run
from __future__ import annotations

import json
import subprocess
from pathlib import Path

ORACLE = Path("oracle/receipts")

def record(name: str, payload_input: dict, constraints: dict) -> None:
    head = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
    body = {
        "baseline": head,
        "input": payload_input,
        "constraints": constraints,
    }
    target = ORACLE / name
    target.write_text(json.dumps(body, indent=2, sort_keys=True) + "\n")
    print(f"wrote {target} at {head}")

Call it from a maintainer shell, then commit oracle/ in a PR that contains no src/ edits. Mixing those two write sets is how expected output tracks the bug.

Layer 2: properties that consume receipts, not patch literals

Example-based tests written in the same turn as the production function are a second copy of the function. Property checks avoid that copy only if they are constrained. They load frozen receipts. They assert invariants that existed before the patch. They fail when a new test file pastes string literals from the production diff.

The invariants should be boring. Conservation. Sign. Currency identity. Rejection flags. They should not restate a newly invented tax rate. If the generator changes the rate, the receipt still holds the prior constraint, and the job fails until a human updates oracle/.

# tests/properties/test_parse_invariants.py
# proposed, unexecuted example
from __future__ import annotations

import json
from pathlib import Path

import pytest

ORACLE = Path(__file__).resolve().parents[2] / "oracle" / "receipts"


def load_receipt(name: str) -> dict:
    payload = json.loads((ORACLE / name).read_text())
    if "input" not in payload or "constraints" not in payload:
        raise AssertionError(f"receipt {name} missing input/constraints")
    return payload


@pytest.mark.parametrize("receipt_name", sorted(p.name for p in ORACLE.glob("*.json")))
def test_totals_are_non_negative_and_conserved(receipt_name: str) -> None:
    receipt = load_receipt(receipt_name)
    from src.billing.parse import parse_invoice

    result = parse_invoice(receipt["input"])
    constraints = receipt["constraints"]

    assert result.total >= 0
    assert result.total == sum(result.lines)
    assert result.currency == constraints["currency"]
    if constraints.get("must_reject"):
        assert result.rejected is True

Add an AST overlap check so a new test cannot photocopy the patch. Short tokens collide, so the script ignores strings shorter than six characters. Tune that threshold per repo. The point is mechanical, not stylistic.

# scripts/check_agent_contract.py
from __future__ import annotations

import ast
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]


def changed_files() -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", "origin/main...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def string_literals(path: Path) -> set[str]:
    tree = ast.parse(path.read_text())
    found: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Constant) and isinstance(node.value, str) and len(node.value) >= 6:
            found.add(node.value)
    return found


def main() -> int:
    changed = changed_files()
    oracle_hits = [p for p in changed if p.startswith("oracle/")]
    if oracle_hits:
        print("contract fail: write-denied path edited:", oracle_hits)
        return 1

    src_files = [ROOT / p for p in changed if p.startswith("src/") and p.endswith(".py")]
    test_files = [ROOT / p for p in changed if p.startswith("tests/") and p.endswith(".py")]
    if not src_files:
        return 0

    src_lits: set[str] = set()
    for path in src_files:
        src_lits |= string_literals(path)

    leaked = []
    for path in test_files:
        overlap = string_literals(path) & src_lits
        overlap = {s for s in overlap if "/" not in s and not s.startswith("src.")}
        if overlap:
            leaked.append((str(path), sorted(overlap)[:8]))

    if leaked:
        print("contract fail: tests reuse string literals from the patch")
        for item in leaked:
            print(" ", item)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

A clean overlap report is not proof the tests are strong. It only proves the tests are not a paste of the patch. Keep mutation testing or a human design review if you need strength. This script does not provide it.

Layer 3: a flake ledger with calendar expiry

Do not freeze flakes inside the test module. A skip, an xfail, or a raised timeout is a patch to the suite. Put the record in tests/ledger/flake_ledger.json and make CI the only consumer that honors it.

{
  "schema": "flake_ledger.v1",
  "entries": [
    {
      "id": "billing.parse.currency-roundtrip",
      "nodeid": "tests/properties/test_parse_invariants.py::test_totals_are_non_negative_and_conserved[rate_limit_429.json]",
      "reason": "intermittent rounding on half-even at scale=4",
      "first_seen": "2026-09-01",
      "expires_on": "2026-09-22",
      "owner": "billing-maintainers",
      "max_runs": 3
    }
  ]
}

Enforcement rules, in order:

1. A test may be quarantined only if its nodeid is in the ledger and expires_on is still in the future relative to the job date.
2. The generator write set cannot include tests/ledger/flake_ledger.json. Humans add rows. CI does not rewrite them.
3. max_runs is the only retry knob. The production test file stays unmarked.
4. When expires_on is past, the job fails closed even if the test passed this time. An expired row is unresolved debt, not a quiet success.

# scripts/enforce_flake_ledger.py
from __future__ import annotations

import json
from datetime import date
from pathlib import Path

LEDGER = Path("tests/ledger/flake_ledger.json")


def main() -> int:
    data = json.loads(LEDGER.read_text())
    today = date.today()  # article written 2026-09-12; sample row expires 2026-09-22
    required = {"id", "nodeid", "reason", "first_seen", "expires_on", "owner", "max_runs"}
    expired = []
    seen_ids: set[str] = set()
    for row in data["entries"]:
        missing = required - set(row)
        if missing:
            print("contract fail: ledger row missing", sorted(missing))
            return 1
        if row["id"] in seen_ids:
            print("contract fail: duplicate flake id", row["id"])
            return 1
        seen_ids.add(row["id"])
        if int(row["max_runs"]) < 1:
            print("contract fail: max_runs must be >= 1", row["id"])
            return 1
        if date.fromisoformat(row["expires_on"]) < today:
            expired.append(row)
    if expired:
        print("contract fail: expired flake ledger rows")
        for row in expired:
            print(f"  {row['id']} expired {row['expires_on']} owner={row['owner']}")
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The sample window is a policy choice, not a measured flake half-life. Do not treat ten days as a benchmark. A rolling 90-day date on every row is a skip list with extra JSON. Keep windows short and owners real.

Workflow: generation stays off the oracle volume

Order matters. Skip a step and the contract becomes a prompt.

1. Branch from main. Record git rev-parse HEAD as the receipt baseline. Do not start the generator on a dirty tree.
2. Mount oracle/ read-only in the generation workspace. Bind-mount it if you run locally. If you lack a local sandbox, run generation on an isolated server that never holds write credentials for that volume.
3. Request a src/ diff plus, if needed, new files under tests/properties/. Refuse test files that open any path except oracle/receipts/.
4. Run python scripts/check_agent_contract.py, then the property suite, then python scripts/enforce_flake_ledger.py.
5. If a test is unstable, do not patch the test. Open a ledger row with an owner and an expiry. Merge nothing until the three commands exit 0.

# proposed local sequence
git checkout -b agent/parse-currency
git rev-parse HEAD
python scripts/check_agent_contract.py
pytest tests/properties -q
python scripts/enforce_flake_ledger.py

Free model access and a free server option are enough for step 2 when the repository is non-secret and the oracle is mounted from CI artifacts rather than from a laptop checkout.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode exposes free model access and a free server option that can host that isolated generation step. The contract does not depend on either one. A local container with a read-only bind mount is the same design. Do not put credentials, customer receipts, or private weights on a shared free runner. Availability is not a durability claim, a quota claim, or a hardware claim.

Decision matrix for the next patch

Patch shape	Allow?	Required artifact
src/ only, properties still pass on existing receipts	Yes	contract script + property job
src/ plus a new property file that only reads oracle/	Yes	AST overlap check must be clean
src/ plus an edited oracle/ receipt	No	human-recorded receipt in a follow-up PR
Test-only diff that adds literals from the production hunk	No	rewrite as a property on a receipt
Ledger row with no owner or no expires_on	No	reject at schema check
Expired ledger row, test currently passing	No	human PR that deletes the row after a documented rerun

Limitations, and who should not use this

The write-deny rule assumes path-level CI and a generator that cannot push to oracle/. If the agent commits through a credential that can edit that tree, the contract is theater. Put the check on the merge job, not in the prompt.

Property checks need invariants you can state without the new code. Conservation, idempotence, monotonicity, round-trip, deny-by-default authorization. If the change is a one-off copy edit with no invariant, this layout is overhead. Use a human-authored example test and skip generation.

The AST literal overlap check is coarse. Shared error codes and enum names will collide. Maintain an allowlist if your domain language is dense. Do not treat a clean report as coverage.

Flake ledgers fail if expiry is ceremonial. Teams that cannot name an owner for a row should not run a generator against that suite. Quarantine without an owner is how skips come back under a new filename.

Do not use a free shared server for repositories that contain secrets, production receipts, or regulated data. If the runner disappears mid-job, the oracle is safe only if it never left CI storage.

This contract does not replace mutation testing or a human design review. It only splits authorship of the oracle from authorship of the patch. Keep the ledger expiry in CI. The generator can propose src/. It should never be the author of what correct looks like.