A Replay Oracle for Agent-Generated Diffs

A green CI job after an agent rewrite is not proof that behavior held. It is proof that whatever tests still exist did not fail. Those are different claims.

The useful signal is observational equivalence against a pinned replay corpus. Hash the outputs on the unpatched tree. Apply the agent diff in a second worktree. Replay the same inputs. Fail closed on digest drift, on denylist edits, and on lost quarantine identities.

This article proposes that harness. It is a method, not a production incident report. Treat the code as a labeled, unexecuted example you can adapt.

The failure mode the suite cannot see

Agent patches optimize for the tests they can read. A rewrite that deletes an assertion, widens a matcher, or retries a flake will still go green. Coverage can even rise. None of that tells you whether yesterday's inputs still produce yesterday's outputs.

A replay oracle answers a narrower question. Given this frozen bag of inputs, did the observable results change? If they did, the patch needs a human-written behavior note, not a silent merge.

Three files own that question. The agent must not write them.

1. corpus/inputs.jsonl — pinned requests, filenames, or CLI argv records.
2. oracle/digests.json — SHA-256 of canonical stdout, stderr, and exit code per input id.
3. oracle/quarantine.json — test identities that are skipped, never deleted, with a human expiry.

Where generation stops and scoring starts

Candidate patches have to come from somewhere. A separate generate tree is enough. MonkeyCode's free model access and free server option can sit on that generate side: produce a diff, copy it into an isolated worktree, and refuse to mount the oracle directory there.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The scoring machine only needs a checkout, a corpus, and a deterministic runner. It does not need the model. Mixing the two trees is the defect this workflow exists to prevent.

Artifact: a two-worktree replay gate

The layout is intentional. Oracle files live beside the repo, not inside the agent's usual test glob.

repo/
  src/
  tests/                 # agent may propose edits; humans review
  oracle/
    digests.json         # human-owned
    quarantine.json      # human-owned
    denylist.txt         # paths the agent cannot touch
    noise.txt            # optional regexes for non-functional bytes
  corpus/
    inputs.jsonl
  tools/
    replay_gate.py
    metamorphic.py

Numbered procedure:

1. Create a clean worktree at the merge base. Record digests only when the oracle is empty and only after a human reviews the corpus.
2. Apply the agent patch to a second worktree. Reject the patch if git diff --name-only intersects oracle/denylist.txt.
3. Replay corpus/inputs.jsonl in the patched tree. Canonicalize output. Hash it.
4. Compare hashes to oracle/digests.json. A missing id or a changed digest is a failed gate, not a prompt to refresh goldens.
5. Cross-check oracle/quarantine.json against the patched test index. A vanished identity is a failed gate.
6. Run a small metamorphic set that lives in tools/, not in tests/.

Canonicalization rules

Hashing raw bytes will false-fail on clocks, absolute paths, and map iteration. Canonicalize first. Then hash.

Proposed rules, all explicit:

• Parse JSON when the payload is JSON. Dump with sorted keys and no extra whitespace.
• Replace the repo root and /tmp prefixes with stable tokens.
• Drop lines matching a human-maintained oracle/noise.txt list. Keep that list short.
• Prefix the body with the numeric exit code before hashing.
• Invoke the binary with SOURCE_DATE_EPOCH=0 and PYTHONHASHSEED=0 when those knobs exist.

If you cannot canonicalize a subsystem, do not put it in the corpus. A noisy digest is worse than an honest gap. Gaps are visible. Noise trains the team to ignore the gate.

Example harness (proposed)

#!/usr/bin/env python3
"""Proposed replay gate. Unexecuted example. Adapt before use."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
ORACLE = ROOT / "oracle"
CORPUS = ROOT / "corpus" / "inputs.jsonl"


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def canonicalize(stdout: bytes, stderr: bytes, code: int) -> bytes:
    body = stdout.replace(str(ROOT).encode(), b"$ROOT")
    # Keep stderr in the digest. Silent stderr changes are still changes.
    return b"%d\n" % code + b"---stdout---\n" + body + b"\n---stderr---\n" + stderr


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows


def run_case(worktree: Path, argv: list[str]) -> bytes:
    proc = subprocess.run(
        argv,
        cwd=worktree,
        capture_output=True,
        env={"SOURCE_DATE_EPOCH": "0", "PYTHONHASHSEED": "0"},
    )
    return canonicalize(proc.stdout, proc.stderr, proc.returncode)


def changed_denylist(patched: Path, denylist: list[str]) -> list[str]:
    diff = subprocess.check_output(
        ["git", "diff", "--name-only"], cwd=patched, text=True
    )
    names = [n for n in diff.splitlines() if n]
    hits = []
    for name in names:
        for rule in denylist:
            if name == rule or name.startswith(rule.rstrip("/") + "/"):
                hits.append(name)
    return hits


def main() -> int:
    if len(sys.argv) != 3:
        print("usage: replay_gate.py <base-worktree> <patched-worktree>")
        return 2
    patched = Path(sys.argv[2]).resolve()
    denylist = [
        d for d in (ORACLE / "denylist.txt").read_text().splitlines()
        if d and not d.startswith("#")
    ]
    hits = changed_denylist(patched, denylist)
    if hits:
        print("denylist write:")
        print("\n".join(hits))
        return 1

    expected = json.loads((ORACLE / "digests.json").read_text())
    quarantine = json.loads((ORACLE / "quarantine.json").read_text())
    observed: dict[str, bytes] = {}
    for row in load_jsonl(CORPUS):
        observed[row["id"]] = run_case(patched, row["argv"])

    failures: list[str] = []
    for case_id, digest in expected.items():
        got = observed.get(case_id)
        if got is None:
            failures.append(f"missing:{case_id}")
            continue
        if sha256_bytes(got) != digest:
            failures.append(f"drift:{case_id}")

    index = subprocess.check_output(
        ["python", "-m", "pytest", "--collect-only", "-q"],
        cwd=patched,
        text=True,
    )
    for ident, meta in quarantine.items():
        if ident not in index:
            failures.append(f"quarantine_missing:{ident}")
        if not meta.get("expires"):
            failures.append(f"quarantine_no_expiry:{ident}")

    if failures:
        print("replay_gate failed")
        print("\n".join(failures))
        return 1
    print("replay_gate ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Seed files stay boring on purpose. Boring is reviewable.

{"id": "fmt-001", "argv": ["python", "-m", "app.fmt", "corpus/sample.json"]}
{"id": "fmt-002", "argv": ["python", "-m", "app.fmt", "corpus/empty.json"]}

oracle/
oracle/digests.json
oracle/quarantine.json
oracle/denylist.txt
oracle/noise.txt
corpus/
tools/replay_gate.py
tools/metamorphic.py

Wire the gate as a required check that does not share a writable filesystem with the generate job.

git worktree add /tmp/base HEAD
git worktree add /tmp/patched HEAD
git -C /tmp/patched apply /tmp/agent.patch
python tools/replay_gate.py /tmp/base /tmp/patched

The generate job may run on a free server. The command above should not. Copy the patch file across. Do not mount oracle/ into the generate workspace.

Metamorphic checks live beside the oracle

Classic unit tests are easy to tautologize. Metamorphic relations are harder, because they do not hard-code a single expected literal. They relate two runs.

Keep a handful in tools/metamorphic.py, outside the agent's default glob. Name the identity in one sentence per function. If you cannot name it, it is not a relation yet.

"""Proposed metamorphic checks. Unexecuted example."""

def relation_roundtrip(run):
    """Formatter identity: fmt(fmt(x)) == fmt(x)."""
    src = open("corpus/roundtrip.json", "rb").read()
    once = run(["python", "-m", "app.fmt", "-"], input=src)
    twice = run(["python", "-m", "app.fmt", "-"], input=once.stdout)
    assert once.returncode == 0
    assert once.stdout == twice.stdout


def relation_key_order(run):
    """Object sum is permutation-insensitive on keys."""
    a = run(["python", "-m", "app.sum_json", "corpus/obj_a.json"])
    b = run(["python", "-m", "app.sum_json", "corpus/obj_a_permuted.json"])
    assert a.stdout == b.stdout

These are still not proofs. They catch a class of helpful agent edits that preserve one golden file and break an algebraic identity. Run them after the digest compare. A digest match plus a broken relation is a real finding: the corpus was too thin, not too strict.

Decision table

Observation	Gate	Human next step
Digests match, denylist clean, quarantine intact	Pass	Review the diff for non-functional risk
Digest drifts on one corpus id	Fail	Require a corpus amendment PR, not a test rewrite
Agent edits oracle/ or tools/replay_gate.py	Fail	Drop the patch
Quarantine identity missing	Fail	Restore the test; do not accept skip-by-deletion
Quarantine expiry in the past	Hold the pipeline	Human re-triages the flake
Output cannot be canonicalized	Hold	Remove the case from the corpus until it is deterministic

A held case is not success. Leave the check non-green until a human either pins a new digest or removes the row. Do not collapse an unusable case into a passing job.

Quarantine is an identity check, not a flake policy

Flakes still exist. The oracle does not diagnose them. It records a stable test identity, a reason string, and an expiry date that only a human may extend.

{
  "tests/test_net.py::test_retry_window": {
    "reason": "depends on wall clock in CI",
    "expires": "2026-10-06"
  }
}

The gate fails if that node disappears from collection. Deleting the test, renaming it, or wrapping it in an always-true skip all look the same from this file: the identity is gone. That is the point. An agent that silences a flake by removal should not get a quieter suite as a reward.

Expiry in the past fails the pipeline, not the patch. Someone has to re-open the flake. The agent does not get a vote on that calendar.

Limitations

This oracle measures observational equivalence on the corpus you actually pinned. It does not measure correctness, security, or performance. A patch can keep every digest and still introduce a vulnerability on an input you never recorded.

It assumes a deterministic command boundary. GUIs, live network calls, wall-clock schedulers, and unseeded concurrency will churn hashes. Do not widen oracle/noise.txt until the digest is stable. That file is how a team accidentally freezes a bug.

Quarantine will not tell you why a test flaps. It will tell you the agent deleted the node you were ignoring. Corpus growth is a human process. An agent that adds corpus rows can hide a behavior change by adding the new output as expected. Deny writes to corpus/ in the same denylist as oracle/.

The harness also says nothing about patch quality when the corpus is a toy. Two JSON files and a formatter identity will not protect a payment path. Size the corpus to the risk, then stop claiming the gate covers what it does not execute.

Who should not use this

Do not adopt the harness if the product has no batch-reproducible entrypoint. Do not adopt it as a substitute for review on auth, crypto, or schema migrations. Do not adopt it if the only available tests are end-to-end runs against shared staging. Those runs are not a corpus.

Teams that already let an agent update snapshots automatically will fight this design. That is expected. Snapshot update is the opposite of a replay oracle. If the workflow you want is "regenerate goldens until green," this method has no value for you.

Close

Generate the diff on an isolated machine. Score it with files that tree cannot write. If the digests move, the patch is a behavior change. Name it that way in the PR, or reject it.

If you already generate patches on a free server, keep the replay harness off that disk. Start with a tiny corpus and a strict denylist. Expand only after a digest mismatch has forced one real human decision.