Capture the Oracle Before the Agent Runs

An agent patch is a change to code you already run. The tests the model adds describe the change it wanted. They do not describe the module that is in production today. The useful order is the reverse: capture current I/O, lock the fixtures that produced it, run property checks that do not need a perfect expected value, and freeze flaky node ids so the agent cannot delete or rewrite them. A green run on tests written in the same diff is not a gate.

This is a proposed workflow, not a production war story. Paths and names below are local conventions. Adapt them. Do not treat the snippets as measured results.

Why post-patch unit tests miss drift

Agents optimize for the assertions they just wrote. If the new test encodes the new behavior, both sides of the diff agree. That agreement is cheap. It is also circular.

Silent drift shows up in three other places. Return values that still type-check but no longer match last week's outputs. Fixtures that were edited to make a new path pass. Flaky tests removed so the suite looks stable. None of those failures appear in a pytest summary that only lists tests added in the patch.

The gate below treats the pre-patch module as the oracle. New tests can still land. They cannot be the only evidence.

What you freeze, in order

Four artifacts live under .gate/. Keep them in review. The agent may read them. It should not be the author of the last write.

1. oracle.json — hashed I/O pairs from functions you are willing to pin.
2. fixtures.lock.json — SHA-256 of every input file the oracle used.
3. relations.py — metamorphic checks. Same input class, different related inputs, a relation on outputs.
4. flake_freeze.json — pytest node ids the agent is not allowed to edit or delete.

Property checks sit in layer 3 because they survive when you do not have a single golden output. Fixture hashes sit in layer 2 because an agent that rewrites the CSV is not testing the same function. The freeze file sits last because flakes are a process problem. They are not a prompt the model should solve by erasure.

1. Capture the oracle on the current tree

Run capture on main, or on the merge base, before the agent is allowed to write. The recorder imports a callable, feeds it frozen inputs, and stores a canonical JSON form of the result.

# .gate/oracle_capture.py
from __future__ import annotations

import hashlib
import importlib
import json
from pathlib import Path
from typing import Any, Callable


def canonical(value: Any) -> str:
    return json.dumps(value, sort_keys=True, default=str, separators=(",", ":"))


def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def load_callable(spec: str) -> Callable[..., Any]:
    module_name, attr = spec.rsplit(":", 1)
    module = importlib.import_module(module_name)
    return getattr(module, attr)


def capture(spec: str, cases: list[dict[str, Any]], out: Path) -> None:
    fn = load_callable(spec)
    rows = []
    for case in cases:
        output = fn(*case.get("args", []), **case.get("kwargs", {}))
        payload = canonical(output)
        rows.append(
            {
                "id": case["id"],
                "args": case.get("args", []),
                "kwargs": case.get("kwargs", {}),
                "output_sha256": sha(payload),
                "output_preview": payload[:240],
            }
        )
    out.write_text(json.dumps({"spec": spec, "rows": rows}, indent=2) + "\n")

A case file is boring on purpose. Boring is reviewable.

{
  "spec": "billing.quote:compute_quote",
  "cases": [
    {"id": "empty-cart", "args": [[]], "kwargs": {}},
    {"id": "single-item", "args": [{"sku": "A1", "qty": 2, "cents": 499}], "kwargs": {}},
    {"id": "tax-exempt", "args": [{"sku": "A1", "qty": 1, "cents": 499}], "kwargs": {"tax_exempt": true}}
  ]
}

python -m gate.oracle_capture \
  --cases .gate/cases/quote.json \
  --out .gate/oracle.json

After capture, oracle.json is an invariant of the unpatched tree. The patch job compares new outputs to these hashes. A mismatch is not an automatic reject. It is a required entry in a human-edited allowed_drift.json. No entry, no merge.

2. Hash the fixtures the oracle consumed

If the agent can edit the input file, the oracle is theater. Lock bytes, not filenames.

# .gate/lock_fixtures.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("tests/fixtures")

def file_sha(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def lock(out: Path) -> None:
    items = []
    for path in sorted(ROOT.rglob("*")):
        if path.is_file():
            items.append({"path": str(path.as_posix()), "sha256": file_sha(path)})
    out.write_text(json.dumps({"files": items}, indent=2) + "\n")

The gate rehashes the same paths on the patched tree. Any changed hash without a matching row in allowed_drift.json fails the job. Fixture edits then show up as a review comment, not as a quieter test.

3. Property-check relations, not exact answers

Exact oracles rot when the domain has more than one correct shape. Metamorphic relations still constrain the patch. They do not claim to know the one true JSON.

Four relations cover a large share of agent mistakes on pure-ish functions. Add only relations you can state in one sentence.

1. Idempotent: f(f(x)) equals f(x) for a declared subset of inputs.
2. Monotone: raising qty does not lower total_cents.
3. Permute-insensitive: shuffling line items does not change the total.
4. Round-trip: encode then decode returns a value equal under canonical().

# .gate/relations.py
from __future__ import annotations

from billing.quote import compute_quote, parse_quote, render_quote


def assert_idempotent(cart: list[dict]) -> None:
    once = compute_quote(cart)
    twice = compute_quote(once["normalized_cart"])
    assert once["total_cents"] == twice["total_cents"]


def assert_monotone_qty(item: dict) -> None:
    low = compute_quote([{**item, "qty": 1}])
    high = compute_quote([{**item, "qty": 3}])
    assert high["total_cents"] >= low["total_cents"]


def assert_permute_insensitive(cart: list[dict]) -> None:
    left = compute_quote(cart)
    right = compute_quote(list(reversed(cart)))
    assert left["total_cents"] == right["total_cents"]


def assert_round_trip(cart: list[dict]) -> None:
    rendered = render_quote(compute_quote(cart))
    parsed = parse_quote(rendered)
    assert parsed["total_cents"] == compute_quote(cart)["total_cents"]

Run them as ordinary pytest functions. Keep the examples small. A free server can loop extra random carts later. The relation file stays short so a reviewer can reject a bad property in one pass.

Hypothesis-style shrinking is optional. If you add it, pin the seed in CI. An unseeded property that fails once a week will get frozen, and then you are back to a flake list.

4. Freeze flakes so the agent cannot tidy them away

Flakes are not a patch task. A model that sees xfailed or skipped tests will often delete them, mark them skip, or rewrite the assertion until the race disappears from the log. That hides the bug. It does not rank it.

Put node ids in a freeze file owned by humans.

{
  "nodeids": {
    "tests/test_quote.py::test_concurrent_tax_table": {
      "reason": "race on shared tax cache",
      "owner": "billing",
      "status": "open"
    },
    "tests/test_quote.py::test_legacy_coupon_clock": {
      "reason": "depends on local TZ",
      "owner": "billing",
      "status": "open"
    }
  }
}

The gate inspects git diff for the patch and fails if a frozen node id is removed, renamed, or has its test body changed.

# .gate/check_flake_freeze.py
from __future__ import annotations

import json
import subprocess
from pathlib import Path


def diff_paths(base: str) -> list[str]:
    raw = subprocess.check_output(
        ["git", "diff", "--name-only", base, "--", "tests"],
        text=True,
    )
    return [line.strip() for line in raw.splitlines() if line.strip()]


def touched_nodeids(base: str) -> set[str]:
    raw = subprocess.check_output(
        ["git", "diff", "-U0", base, "--", "tests"],
        text=True,
    )
    found: set[str] = set()
    current_file = ""
    for line in raw.splitlines():
        if line.startswith("+++ b/"):
            current_file = line[6:]
        if line.startswith("@@") or line.startswith("+") or line.startswith("-"):
            if "def test_" in line and current_file:
                name = line.split("def ", 1)[-1].split("(", 1)[0]
                found.add(f"{current_file}::{name}")
    return found


def check(base: str, freeze_path: Path) -> None:
    frozen = set(json.loads(freeze_path.read_text())["nodeids"])
    hit = frozen.intersection(touched_nodeids(base))
    if hit:
        raise SystemExit("frozen node ids edited in patch: " + ", ".join(sorted(hit)))

Status stays open until a human changes it. There is no expiry field in this design. Expiry invites a second agent pass that “refreshes” the freeze. If you need a calendar, keep it in the issue tracker, not in a file the model can patch.

5. One runner, two cheap roles

Wire the pieces so a laptop and a remote job execute the same commands.

set -euo pipefail
BASE="${BASE:-origin/main}"
python -m gate.lock_fixtures --out .gate/fixtures.lock.json
python -m gate.check_fixtures --lock .gate/fixtures.lock.json
python -m gate.check_oracle --base-oracle .gate/oracle.json --allowed .gate/allowed_drift.json
pytest .gate/test_relations.py -q
python -m gate.check_flake_freeze --base "$BASE" --freeze .gate/flake_freeze.json

The first four commands are CPU and disk. They do not need a GPU. A free server option is enough to run capture and relations off the laptop so the gate is not an honor system. A free model can propose extra relations from oracle.json previews. Those proposals belong in .gate/relation_candidates.md. They do not belong in CI until a reviewer copies a function into relations.py.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Candidate text should look like a rejected idea, not like a merged test.

## candidate: discount-code concatenation
Claim: quote(code_a) + quote(code_b) == quote(code_a+code_b)
Reject unless discounts are proven additive. Most carts are not.

That split matters. Free model access is useful as a generator of hypotheses about the captured I/O. It is not an oracle. The captured hashes and the freeze file remain human-owned.

Decision table for the patch job

Signal	Meaning	Action
Oracle hash match, relations pass, freeze untouched	Behavior pin held	Review the diff as a normal change
Oracle hash miss, row in allowed_drift.json	Intentional behavior change	Require the drift note to name the case id
Oracle hash miss, no drift row	Unexplained I/O change	Fail the job
Fixture hash miss	Input corpus edited	Fail unless the lock file is updated in a human commit
Relation fail	Patch broke a stated law	Fail; do not skip the relation
Frozen node id in git diff	Agent touched a flake	Fail; restore the test body
New tests only, none of the above	Circular evidence	Do not treat as a pass of this gate

The last row is the one most agent workflows skip. New coverage is welcome. It does not replace the pin.

Limitations

Characterization freezes bugs as well as behavior. If compute_quote is wrong on empty-cart today, the oracle will demand that wrongness tomorrow. That is the point of a pin, and also the cost. Schedule a human edit of oracle.json when you actually fix the bug. Do not let the agent refresh the oracle in the same diff that changes the function.

Relations are only as strong as the sentence you wrote. Idempotence does not apply to ticket reservation. Monotonicity does not apply to a function that applies a max-discount cap in a non-monotone way. A relation that does not hold on main will fail forever. Capture it as a non-goal, or do not add it.

The freeze parser above is a heuristic on unified diffs. It will miss renamed files that git presents poorly, and it will miss parametrized node ids that do not appear as def test_. If your suite is heavily parametrized, store the full pytest node id from --collect-only instead of parsing def lines.

This workflow assumes a merge base, a tests/ tree, and a review path that can reject a diff. It does not replace threat review, load review, or a typed contract with an external vendor.

Who should not use this

Skip it if the module is greenfield and has no behavior worth pinning. Skip it if no person will own flake_freeze.json. Skip it for security-sensitive patches where I/O pins on happy-path fixtures would create a false pass. Skip it if the team’s only CI signal is “pytest is green” and no one will read allowed_drift.json.

Do not use a free model as an unattended author of relations.py or as a janitor for flaky tests. Proposal text in relation_candidates.md is the ceiling.

A minimal first cut

Pick one pure function. Capture ten cases. Lock the fixtures it reads. Add one relation you can defend in review. Freeze every currently flaky node id in that area. Run the five commands on the merge base, then on an agent diff.

If the oracle snapshot already disagrees with comments in the function, stop. Fix the comments, or fix the capture, before you invite a model to edit the file. The gate is the old behavior. Everything after that is a hypothesis about a patch.