Keep the Oracle Outside the Patch: An Invariant Inventory

Agent patches fail merge gates for reasons snapshot equality cannot name. The useful split is not pass versus fail. It is invariant miss versus fixture drift versus unreproducible noise. An invariant inventory kept outside the patch write-set can name those three. Agent-authored tests cannot.

This article proposes a merge workflow, not a measured production study. The artifact is a checked-in inventory, a seed-pinned property runner, a fixture hash lock, and a freeze keyed by (property_id, seed, fixture_hash). Treat the code as a labeled example. Do not treat it as a benchmark.

Core rule

Production code in a patch may change. Tests that arrive in the same patch are specimens of intent. They are not oracles. Oracles live in a separate inventory with pinned seeds and hashed fixtures. If a flake appears, freeze that triple. Do not xfail the test name.

That rule is mechanical. It does not require a story about model quality.

Why equality suites collapse

An agent that edits src/ and tests/ together can keep every assertion green by rewriting the expected value. The suite still “passes.” The behavior still moved.

A second failure mode is the inverse. The agent leaves tests untouched, but a golden file in tests/fixtures/ is regenerated as a side effect of formatting, key order, or timestamps. The gate goes red. Nothing about the invariant changed.

A third mode is noise. A property uses an unpinned RNG, a clock, or a network lease. Re-running the same commit does not reproduce the red. Teams then skip, xfail, or bump a timeout. That hides the signal the next patch needs.

Snapshot tests report a boolean. The inventory has to report a class.

Artifact 1: the inventory file

Keep oracles in a file the agent is not asked to edit in the same change as production code. YAML is enough.

# invariants/inventory.yaml
version: 1
properties:
  - id: parse_roundtrip
    module: app.codec
    fn: parse
    seed: 174221
    fixture: tests/fixtures/parse_corpus.json
    checks:
      - roundtrip
      - no_extra_keys
  - id: window_bounds
    module: app.window
    fn: clamp
    seed: 90210
    fixture: tests/fixtures/window_cases.json
    checks:
      - lo_le_hi
      - output_in_range
freezes:
  # empty on a clean tree

The inventory is the spec. Fixtures are inputs. Seeds make iteration order stable. Freezes are exceptions with an expiry, not a permanent mute.

Artifact 2: hash the fixtures, not the assertion text

Hash bytes. Do not hash prettier dumps. A lock file records the digest the inventory currently trusts.

# tools/fixture_lock.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

LOCK = Path("invariants/fixture.lock.json")


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def rebuild(entries: dict[str, str]) -> None:
    LOCK.write_text(json.dumps(entries, indent=2, sort_keys=True) + "\n")


def verify(entries: dict[str, str]) -> list[str]:
    drift = []
    for rel, expected in entries.items():
        actual = digest(Path(rel))
        if actual != expected:
            drift.append(f"{rel}: expected {expected[:12]} got {actual[:12]}")
    return drift

Command to refresh the lock when a human intends a fixture change:

python tools/fixture_lock.py --rebuild
git add invariants/fixture.lock.json tests/fixtures/

A patch that changes fixture bytes without a lock update is fixture drift. Block it. A patch that changes the lock without an inventory note is also drift. Block that too.

Artifact 3: seed-pinned properties

The runner loads one property, pins the seed, and applies named checks. Hypothesis is optional. A small loop is enough to show the contract.

# tools/run_inventory.py
from __future__ import annotations

import importlib
import json
import random
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable

import yaml

CHECKS: dict[str, Callable[[Any, Any], None]] = {}


def check(name: str):
    def wrap(fn):
        CHECKS[name] = fn
        return fn
    return wrap


@check("roundtrip")
def roundtrip(fn, sample):
    assert fn(fn(sample, reverse=True), reverse=False) == sample


@check("no_extra_keys")
def no_extra_keys(fn, sample):
    out = fn(sample)
    assert set(out).issubset(set(sample) | {"ok", "err"})


@check("lo_le_hi")
def lo_le_hi(fn, sample):
    lo, hi, x = sample["lo"], sample["hi"], sample["x"]
    y = fn(lo, hi, x)
    assert lo <= hi
    assert lo <= y <= hi


@check("output_in_range")
def output_in_range(fn, sample):
    lo, hi, x = sample["lo"], sample["hi"], sample["x"]
    y = fn(lo, hi, x)
    assert min(lo, hi) <= y <= max(lo, hi)


@dataclass(frozen=True)
class Result:
    prop_id: str
    status: str  # pass | invariant_miss | fixture_drift | noise | frozen
    detail: str


def load_samples(path: Path, seed: int) -> list[Any]:
    data = json.loads(path.read_text())
    rng = random.Random(seed)
    order = list(range(len(data)))
    rng.shuffle(order)
    return [data[i] for i in order]


def run_one(prop: dict, freeze_key: tuple[str, int, str] | None) -> Result:
    fixture = Path(prop["fixture"])
    seed = int(prop["seed"])
    key = (prop["id"], seed, fixture.as_posix())
    if freeze_key == key:
        return Result(prop["id"], "frozen", "triple is in freeze list")
    mod = importlib.import_module(prop["module"])
    fn = getattr(mod, prop["fn"])
    try:
        samples = load_samples(fixture, seed)
    except OSError as exc:
        return Result(prop["id"], "fixture_drift", str(exc))
    try:
        for sample in samples:
            for name in prop["checks"]:
                CHECKS[name](fn, sample)
    except AssertionError as exc:
        return Result(prop["id"], "invariant_miss", f"{name}: {exc}")
    return Result(prop["id"], "pass", "ok")

Re-run the same commit twice. If the first run is red and the second is green at the same seed and digest, classify noise. Do not classify pass.

python tools/run_inventory.py --repeat 2 --json > /tmp/inv1.json
python tools/run_inventory.py --repeat 2 --json > /tmp/inv2.json
python tools/classify_delta.py /tmp/inv1.json /tmp/inv2.json

Decision table

Observed signal	Class	Merge vote	Required next edit
Inventory properties pass; fixtures digest-match; agent tests also green	likely safe	allow inventory lane	none
Agent edits tests/ in the same patch as src/	specimen	those tests do not vote	keep inventory lane
Fixture bytes change; fixture.lock.json unchanged	fixture drift	block	restore bytes or rebuild lock with a note
Lock changes; inventory id / checks unchanged	fixture drift	block	add a property note or revert lock
Pinned seed, same digest, assertion red on every repeat	invariant miss	block	fix production or change inventory in a follow-up
Pinned seed, same digest, red then green across repeats	noise	block	freeze the triple with an expiry
Unpinned RNG or live clock in a property	protocol error	block	pin seed; remove the clock
Model-proposed property added in the same PR as the patch	untrusted	no vote	queue for human inventory review

The table is the gate. A single pytest exit code is not.

Numbered workflow

1. Freeze the oracle location. Put invariants/ in code owners or a path filter the merge bot treats as human-gated. The agent may read it. The agent should not write it in the same patch as production code.
2. Record fixture digests. Run the lock tool on every inventory fixture. Commit invariants/fixture.lock.json.
3. Pin every seed. If a property cannot run with a seed, it does not belong in the voting inventory. Move it to a manual lab script.
4. Run the inventory as its own lane. Do not fold it into the agent’s pytest selection. A separate process and a separate junit file keep the classes readable.
5. Classify reds with two repeats. One red is not a class. Two repeats at the same seed and digest distinguish miss from noise.
6. Freeze triples, not names. A freeze record must include property_id, seed, fixture path, reason, and expires_on. When it expires, the property votes again. Silent xfail is not an expiry.
7. Promote specimens later. Agent-written tests can be copied into the inventory after a human rewrites them as properties with pinned seeds. Same PR promotion is the failure mode this workflow exists to stop.

Example freeze record:

freezes:
  - property_id: window_bounds
    seed: 90210
    fixture: tests/fixtures/window_cases.json
    reason: noise
    expires_on: "2026-09-26"
    repeats: ["red", "green"]

The date above is an example expiry one week after a 2026-09-19 classification. Replace it with the calendar date of the freeze. Do not copy it as a default TTL for every flake.

Where a free model and a free server belong

The inventory still needs candidate properties. Writing them by hand for every touched module is slow. Generating them inside the same patch is unsafe.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode’s free model access is useful as a proposal step, not as a voter. Feed the production diff, not the agent’s new tests, and ask for candidate id / checks pairs. Land those candidates in invariants/proposed/ on a branch that cannot satisfy the merge predicate. A human either rejects them or rewrites them into inventory.yaml with a seed and a fixture hash.

MonkeyCode’s free server option is useful as the witness host for tools/run_inventory.py. Keep that host out of the merge vote. The merge lane runs the human inventory. The witness host re-runs the same inventory on a second machine so a local cache or a tainted pytest plugin cannot mint a green. If the two lanes disagree, classify noise or environment drift. Do not average the exit codes.

Do not assume a particular model name, quota, GPU, or uptime. Those are not specified here. The workflow only needs “a model that can propose invariants” and “a second place to run the runner.”

Classification script (proposal)

# tools/classify_delta.py
from __future__ import annotations

import json
import sys
from pathlib import Path


def load(path: str) -> dict[str, str]:
    rows = json.loads(Path(path).read_text())
    return {row["prop_id"]: row["status"] for row in rows}


def classify(a: str, b: str) -> str:
    if a == b == "pass":
        return "pass"
    if a == b == "invariant_miss":
        return "invariant_miss"
    if a == b == "fixture_drift":
        return "fixture_drift"
    if {a, b} <= {"pass", "invariant_miss", "noise"} and a != b:
        return "noise"
    return "protocol_error"


def main(p1: str, p2: str) -> None:
    left, right = load(p1), load(p2)
    ids = sorted(set(left) | set(right))
    for prop_id in ids:
        c = classify(left.get(prop_id, "missing"), right.get(prop_id, "missing"))
        print(f"{prop_id}\t{c}")


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Exit policy for CI, also a proposal:

• pass on every id: inventory lane green.
• any invariant_miss or fixture_drift: fail the job.
• any noise: fail the job and require a freeze triple or a seed fix.
• any protocol_error: fail the job; the runner, not the product, is wrong.

Limitations

This workflow does not prove functional completeness. An inventory with two properties will not catch a third invariant nobody wrote down. Proposed properties from a free model inherit that gap. They also inherit whatever the prompt omitted.

Seed pinning removes one noise source. It does not freeze time. If a check reads datetime.now(), repeats will still disagree. Pin clocks in the check, or keep that check out of the inventory.

Hashed fixtures do not encode semantics. Two JSON files can differ in key order and hash differently while representing the same map. Canonicalize before hashing, or the lock will thrash.

Freeze triples expire. If nobody retries the property on expires_on, the suite quietly reintroduces a flake into the vote. That is intentional. A freeze without expiry is an xfail with extra YAML.

Who should not use this

Do not use this inventory as a merge gate if the repo has no human owner for invariants/. An unowned oracle becomes another file the agent rewrites.

Do not use it if the project cannot run the same runner twice. One-shot integration tests that mutate cloud state are not seed-pinnable. Keep them off the inventory lane.

Do not use a free witness host for secrets, production data, or licensed corpora. A second server is for replay of pinned fixtures you already decided to publish in the repo.

Do not promote model-proposed checks in the same patch that claims to implement them. That recreates the specimen-as-oracle bug with nicer YAML.

What to keep when you delete the product names

The method still holds without a particular vendor. Split oracles from patch specimens. Hash fixtures. Pin seeds. Repeat once. Freeze the triple that still wobbles. Let proposed properties wait for a human.

If you need a second process that only proposes inventory rows and a second host that only replays them, MonkeyCode’s free model access and free server option fit that split. Review the inventory diff as carefully as the production diff. The oracle is the part that outlives the patch.