Seal the Oracle Before You Merge an Agent Patch

An agent patch is evidence only if production code is the writable tree. Fixtures, generators, and flake policy have to live outside that tree. If the model can rewrite expected values, loosen a schema, or convert a flake into skip, a green run measures edit access, not behavior.

This article proposes an unexecuted harness: schema-lock fixtures, run properties from a sealed generator, and replace permanent skips with a numbered flake budget. The goal is a merge rule you can audit in a diff, not a feeling that CI looked calm.

The failure mode is oracle capture

Agent patches fail tests in ordinary ways. Timeouts, off-by-one bounds, missing fields, and retry races show up in the same files a human would touch. The unusual failure is what happens next. The patch set often includes the test.

A snapshot file changes by three bytes. A pytest marker becomes skip. An assertion on sort order becomes an assertion on len(). CI goes green. The production function still drifts. You cannot treat that result as a hypothesis test, because the oracle moved with the hypothesis.

Two path classes have to exist in the repo. src/ is writable. sealed/ is not. Review rejects any agent diff that writes into sealed/, even when the commit message calls it cleanup.

1. Split the tree before the model runs

Keep three directories with different write rules. Do not encode the rules only in a chat prompt. Encode them in the merge gate.

1. src/ — production code the agent may patch.
2. sealed/schema/ — fixture JSON Schema and allow-lists for field names.
3. sealed/gen/ — property generators, seed lists, and flake-budget policy.

Proposed layout:

repo/
  src/
  tests/                 # thin runners only; no expected payloads
  sealed/
    schema/
      order_v3.json
    gen/
      properties.py
      seeds.txt
    policy/
      flake_budget.yaml
      writable.txt

writable.txt is the allow-list the gate reads. Anything else that appears in git diff --name-only against main is a reject, including deletions under sealed/.

# sealed/policy/writable.txt
src/

A proposed check, labeled as unexecuted:

# sealed/gen/writable_gate.py
from pathlib import Path

ALLOW = Path("sealed/policy/writable.txt").read_text().splitlines()
ALLOW = [line.strip() for line in ALLOW if line.strip() and not line.startswith("#")]

def classify(paths):
    sealed, other = [], []
    for p in paths:
        if p == "sealed/policy/writable.txt" or p.startswith("sealed/"):
            sealed.append(p)
        elif any(p == a or p.startswith(a) for a in ALLOW):
            other.append(p)
        else:
            sealed.append(p)  # tests that gained assertions still need review
    return sealed, other

def gate(diff_names):
    sealed, prod = classify(diff_names)
    if sealed:
        raise SystemExit(f"oracle capture: {sealed}")
    if not prod:
        raise SystemExit("empty production diff")

Run it on the merge SHA, not on the working tree the agent left behind.

git diff --name-only origin/main...HEAD > /tmp/diff_names.txt
python sealed/gen/writable_gate.py

If you evaluate candidate patches from a free model on a free server, point that server at this gate first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product claim used here is only that free model access and a free server option exist; this harness does not depend on a named model, a quota, or a hardware profile.

2. Lock fixture shape, not a golden byte string

Byte-for-byte snapshot locks look strict. They are also the first thing an agent rewrites. Schema locks are stricter in the direction that matters. The fixture may gain a new instance in a human-owned corpus. It may not gain a new field, drop a required key, or switch a type.

Proposed schema for an order fixture:

{
  "$id": "sealed/schema/order_v3.json",
  "type": "object",
  "additionalProperties": false,
  "required": ["order_id", "currency", "items", "total_cents"],
  "properties": {
    "order_id": {"type": "string", "minLength": 1},
    "currency": {"type": "string", "enum": ["USD", "EUR", "JPY"]},
    "items": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["sku", "qty", "unit_cents"],
        "properties": {
          "sku": {"type": "string", "minLength": 1},
          "qty": {"type": "integer", "minimum": 1},
          "unit_cents": {"type": "integer", "minimum": 0}
        }
      }
    },
    "total_cents": {"type": "integer", "minimum": 0}
  }
}

Validate every fixture file on every run. Do not load fixtures from src/ or from /tmp the agent created.

# sealed/gen/schema_lock.py
import json
from pathlib import Path
import jsonschema

SCHEMA = json.loads(Path("sealed/schema/order_v3.json").read_text())

def load_fixture(path: Path):
    data = json.loads(path.read_text())
    jsonschema.validate(data, SCHEMA)
    return data

A useful extra invariant lives next to the schema, not in the test the agent can see:

def total_matches_items(order: dict) -> bool:
    summed = sum(i["qty"] * i["unit_cents"] for i in order["items"])
    return summed == order["total_cents"]

If the production patch changes how totals are computed, this property fails. If the agent instead edits total_cents in a fixture under tests/, the writable gate already failed. That is the point of splitting ownership.

3. Keep the generator and the seed list sealed

Property checks are only as honest as their generator. If the agent can shrink the domain — drop zero-qty rejection, drop JPY, drop 64-bit totals — the property still passes and means less.

Put the generator in sealed/gen/properties.py. Put seeds in sealed/gen/seeds.txt. The runner in tests/ may import those modules. It may not copy them.

Proposed generator (unexecuted example):

# sealed/gen/properties.py
from dataclasses import dataclass
import hashlib

CURRENCIES = ("USD", "EUR", "JPY")

@dataclass(frozen=True)
class Order:
    order_id: str
    currency: str
    items: tuple
    total_cents: int

def _int(seed: str, salt: str, modulo: int) -> int:
    digest = hashlib.sha256(f"{seed}:{salt}".encode()).digest()
    return int.from_bytes(digest[:8], "big") % modulo

def generate(seed: str) -> Order:
    n_items = 1 + _int(seed, "n", 4)
    items = []
    for i in range(n_items):
        qty = 1 + _int(seed, f"qty{i}", 5)
        unit = _int(seed, f"unit{i}", 10_000)
        items.append((f"sku-{i}", qty, unit))
    total = sum(q * u for _, q, u in items)
    currency = CURRENCIES[_int(seed, "ccy", len(CURRENCIES))]
    return Order(order_id=seed, currency=currency, items=tuple(items), total_cents=total)

def holds(order: Order, compute_total) -> bool:
    got = compute_total(order)
    return got == order.total_cents and got >= 0

Seeds stay boring and explicit:

# sealed/gen/seeds.txt
seed-0001
seed-0002
seed-0003
seed-jp-overflow
seed-single-item

The thin test file under tests/ only wires production code to the sealed module. That file can change for import paths. It cannot change seeds.

# tests/test_order_properties.py
from pathlib import Path
from sealed.gen.properties import generate, holds
from src.order import compute_total

SEEDS = Path("sealed/gen/seeds.txt").read_text().splitlines()
SEEDS = [s for s in SEEDS if s and not s.startswith("#")]

def test_total_property():
    failures = []
    for seed in SEEDS:
        order = generate(seed)
        if not holds(order, compute_total):
            failures.append(seed)
    assert failures == [], failures

Re-seeding is a human change to sealed/gen/seeds.txt. An agent that adds a seed which avoids the failing branch is performing oracle capture by another name.

4. Replace skip with a flake budget

Flaky tests are real. Network leases, clock edges, and shared temp directories produce intermittent red. A permanent skip is still oracle capture. It removes a check from the set the patch is scored against.

Give each named test a budget record instead:

# sealed/policy/flake_budget.yaml
version: 1
default:
  retries: 0
  max_fail_rate: 0.0
  expires: null
tests:
  tests/test_retry_window.py::test_lease_renewal:
    retries: 2
    window_runs: 40
    max_fail_rate: 0.05
    expires: "2026-09-21"
    owner: "platform-runtime"
    note: "lease clock skew on shared runner"

Rules for the budget, all enforced in code:

1. Unknown tests have retries: 0. No implicit quarantine.
2. expires is required when retries > 0. A missing date is a gate error.
3. After expiry, the record must be deleted or the test must pass with retries: 0.
4. Fail rate is counted across window_runs, not across a single lucky retry.
5. Agents cannot add, extend, or delete records under sealed/policy/.

Proposed evaluator:

# sealed/gen/flake_budget.py
from datetime import date
import yaml

def load_policy(path):
    return yaml.safe_load(path.read_text())

def evaluate(name, results, policy, today):
    """results: list[bool] oldest-to-newest, True means pass."""
    conf = policy["tests"].get(name, policy["default"])
    retries = conf.get("retries", 0)
    expires = conf.get("expires")
    if retries and not expires:
        return "reject", "retries require expires"
    if expires and date.fromisoformat(expires) < today:
        return "reject", f"budget expired {expires}"
    if not results:
        return "reject", "no runs"
    if retries == 0:
        return ("pass", "clean") if results[-1] else ("fail", "no budget")
    window = results[-conf.get("window_runs", len(results)):]
    fail_rate = 1.0 - (sum(window) / len(window))
    if fail_rate > conf.get("max_fail_rate", 0.0):
        return "fail", f"fail_rate={fail_rate:.3f}"
    if results[-1] or any(results[-(retries + 1):]):
        return "pass", "inside budget"
    return "fail", "retries exhausted"

Store results in CI artifacts keyed by test name and SHA. Do not store them in the patch. A flake budget that the agent can reset by deleting a cache is not a budget.

Decision table

Diff touches	Fixture schema	Property on sealed seeds	Flake record	Merge
src/ only	valid	holds	unchanged, unexpired	allow
src/ only	valid	holds	expired	reject
src/ + tests/ assertion edit	valid	holds	unchanged	human review
src/ + sealed/schema/	any	any	any	reject
src/ + new skip	any	skipped property	missing	reject
empty src/, tests only	valid	holds	any	reject
src/ only	valid	fails one seed	inside budget	reject property, do not skip

Human review in row three is intentional. A developer may retarget a unit test. An automated patch may not do that in the same commit as the production change.

5. Run order on a local machine or a free server

The sequence is the same in both places. Changing the host must not change the seed list.

1. Compute git diff --name-only origin/main...HEAD.
2. Run writable_gate.py. Exit non-zero on sealed-path writes.
3. Validate every fixture against sealed/schema/.
4. Run test_order_properties.py with seeds.txt unchanged.
5. Evaluate flake_budget.yaml against the stored window, using today's date.
6. Publish a single report: paths classified, seeds failed, budgets consumed.

python sealed/gen/writable_gate.py
python -m pytest tests/test_order_properties.py -q
python sealed/gen/flake_budget.py --today 2026-09-07

If the candidate patch was produced by a free model, keep generation and evaluation on different write mounts. The evaluator process should see sealed/ as read-only. That is a filesystem rule, not a prompt instruction.

Limitations

Schema locks do not encode business meaning beyond types, required keys, and a few invariants you remembered to write. A patch can still compute a wrong discount that happens to keep total_cents equal to qty * unit_cents.

Sealed seeds are a finite set. They are better than an agent-chosen subset. They are not a proof. Adding seeds is a human maintenance task, and stale seeds rot in the same way stale unit tests rot.

Flake budgets hide infra problems if expires is always extended by a human who is tired of red CI. The expiry field only helps if someone treats an extension as an incident, not as paperwork.

The writable-tree rule blocks mixed commits. That will annoy legitimate refactors that move a function and its unit test together. Those refactors should stay human-authored, or land as two commits with two different gates.

This harness is proposed and unexecuted here. It does not report throughput, cost, or model quality. It does not claim that any vendor endpoint stays free.

Who should not use this

Do not use a sealed oracle if the project has no production/test split yet. You will spend the week arguing about path names.

Do not use a flake budget on tests that corrupt data or spend money. Retries are for non-destructive checks. A flaky billing charge is a defect, not a budget line.

Do not apply the writable-tree gate to generated code that is the product, such as a compiler's golden corpus, unless that corpus is already reviewed by a separate owner. Sealing the wrong tree freezes the wrong oracle.

If the only available check is an end-to-end UI path with no schema and no seed, this method will not create those things for you. Write the schema first.

What to keep when you throw the rest away

Production diffs and oracle diffs must not share a commit from an agent. Fixture shape belongs in a schema the patch cannot edit. Properties belong to a generator and a seed file the patch cannot edit. Flakes get a numbered budget with an expiry date, not a skip marker.

Those four sentences are the whole strategy. The Python above is one way to make them fail closed.