Replay the Parent Surface. Do Not Score Agent Refactors on Editable Tests.

A green unit job is not a merge signal for an agent refactor. The patch can rewrite the assertions that would have failed. Score the refactor against a recording of the public surface taken from the parent revision, an add-only inventory of test asserts, and a flake quarantine that does not live on the branch the agent can edit.

Unit tests that sit in the same worktree as the patch are evidence of local consistency, not of preserved behavior. That distinction matters when the author is a model with a search loop. The loop can satisfy a suite by editing the suite.

This workflow treats three objects as the scoring surface. None of them is the job color on tests/.

The three scoring objects

1. Parent cassettes. Record argv, stdin, env, and the fields you actually guarantee. Replay them on a binary built from HEAD and a binary built from the merge base.
2. Assert inventory. Count assertion nodes with the AST, not with git diff of whole files. Refuse silent deletions.
3. Off-branch flake quarantine. If a test is too noisy to score, list it on a protected ref the agent cannot push. Do not freeze it in the patch.

The rest of this article is a proposed runner, not a production study. Commands are labeled. Swap the CLI shape for HTTP if your public surface is a server.

1. Record the public surface from the parent revision

Build the parent first. Record against that binary only. A cassette captured from the patch is not a characterization test. It is a snapshot of the new behavior.

# proposed: two worktrees, one writable by the agent, one not
git worktree add /tmp/parent MERGE_BASE --detach
git worktree add /tmp/patch HEAD

( cd /tmp/parent && make cli )
( cd /tmp/patch  && make cli )

mkdir -p /tmp/oracle/cassettes
./scripts/record_cassettes.py \
  --bin /tmp/parent/cli \
  --out /tmp/oracle/cassettes \
  --cases ./oracle/cases.yaml

Keep oracle/cases.yaml small and boring. Each case names the surface, not an internal function. Example:

# oracle/cases.yaml — proposed case list, human-owned
cases:
  - id: empty-cart-rejects
    argv: ["order", "create", "--cart", "-"]
    stdin: "{\"items\":[]}"
    env: {NO_COLOR: "1", TZ: "UTC"}
    compare: [exit_code, stdout.json, stderr.sha256]
  - id: idempotent-put
    argv: ["order", "put", "--id", "ord_1", "--json", "-"]
    stdin: "{\"items\":[{\"sku\":\"a\",\"qty\":1}]}"
    env: {NO_COLOR: "1", TZ: "UTC"}
    compare: [exit_code, stdout.json.pointer:/order/id, stdout.json.pointer:/order/total]

Timestamps, request ids, and wall-clock fields do not belong in compare. If you need them for debugging, store them as noise, not as oracles.

2. Replay both binaries against the same bytes

The scoring question is narrow. Did the patch binary disagree with the parent binary on a compared field? Intentional contract changes go through an allowlist of JSON pointers, not through a deleted test.

# scripts/replay_cassettes.py — proposed runner
from __future__ import annotations

import hashlib, json, os, subprocess, sys
from pathlib import Path
from typing import Any

import yaml

POINTER_SEP = "/"


def load_yaml(path: Path) -> dict[str, Any]:
    return yaml.safe_load(path.read_text()) or {}


def pointer_get(doc: Any, pointer: str) -> Any:
    if pointer in ("", "/"):
        return doc
    cur = doc
    for part in pointer.lstrip("/").split(POINTER_SEP):
        part = part.replace("~1", "/").replace("~0", "~")
        if isinstance(cur, list):
            cur = cur[int(part)]
        else:
            cur = cur[part]
    return cur


def run_case(bin_path: Path, case: dict[str, Any]) -> dict[str, Any]:
    env = os.environ.copy()
    env.update({k: str(v) for k, v in (case.get("env") or {}).items()})
    proc = subprocess.run(
        [str(bin_path), *case["argv"]],
        input=case.get("stdin", "").encode(),
        env=env,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        check=False,
    )
    out = proc.stdout.decode("utf-8", errors="replace")
    err = proc.stderr.decode("utf-8", errors="replace")
    payload: dict[str, Any] = {
        "exit_code": proc.returncode,
        "stdout.raw": out,
        "stderr.sha256": hashlib.sha256(err.encode()).hexdigest(),
    }
    try:
        payload["stdout.json"] = json.loads(out) if out.strip() else None
    except json.JSONDecodeError:
        payload["stdout.json"] = None
    return payload


def field(payload: dict[str, Any], spec: str) -> Any:
    if spec.startswith("stdout.json.pointer:"):
        if payload["stdout.json"] is None:
            return "<unparseable>"
        return pointer_get(payload["stdout.json"], spec.split(":", 1)[1])
    return payload.get(spec, "<missing>")


def main() -> int:
    cases = load_yaml(Path("oracle/cases.yaml"))["cases"]
    allow = load_yaml(Path("oracle/allowlist.yaml")).get("allow", [])
    allowed = {(a["id"], a["field"]) for a in allow}
    parent, patch = Path(sys.argv[1]), Path(sys.argv[2])
    failures = []
    for case in cases:
        a, b = run_case(parent, case), run_case(patch, case)
        for spec in case["compare"]:
            if (case["id"], spec) in allowed:
                continue
            if field(a, spec) != field(b, spec):
                failures.append(
                    {
                        "id": case["id"],
                        "field": spec,
                        "parent": field(a, spec),
                        "patch": field(b, spec),
                    }
                )
    Path("replay-report.json").write_text(json.dumps(failures, indent=2))
    print(f"cassette_mismatches={len(failures)}")
    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it as a blocking job, not as a comment bot.

python scripts/replay_cassettes.py /tmp/parent/cli /tmp/patch/cli

A mismatch with no allowlist row is a failed patch. An allowlist row without a reviewer-signed change note is also a failed patch. Put the note next to the pointer, in the same file the CI reads.

# oracle/allowlist.yaml — proposed, CODEOWNERS-protected
allow:
  - id: empty-cart-rejects
    field: stdout.json.pointer:/error/message
    reason: "copy change only; error code stays CART_EMPTY"
    reviewer: "human"

3. Diff the assert inventory, not the test file list

Deleting tests/test_cart.py is obvious. Replacing four asserts with one tautology is not. Count assertion nodes on both trees. Fail if the patch count drops, unless the deleted nodes are named in the allowlist.

# scripts/count_asserts.py — proposed inventory lock
from __future__ import annotations

import ast, json, sys
from pathlib import Path

ASSERT_TYPES = (ast.Assert, ast.Raise)
PYTEST_FAIL = {"pytest.fail", "pytest.xfail"}


def is_pytest_fail(node: ast.AST) -> bool:
    if not isinstance(node, ast.Call):
        return False
    name = ast.unparse(node.func) if hasattr(ast, "unparse") else ""
    return name in PYTEST_FAIL


def count_file(path: Path) -> int:
    tree = ast.parse(path.read_text(), filename=str(path))
    n = 0
    for node in ast.walk(tree):
        if isinstance(node, ast.Assert):
            n += 1
        elif isinstance(node, ast.Call) and is_pytest_fail(node):
            n += 1
        elif isinstance(node, ast.Call):
            func = ast.unparse(node.func) if hasattr(ast, "unparse") else ""
            if func.startswith("self.assert") or func.endswith("assertEqual"):
                n += 1
    return n


def count_tree(root: Path) -> dict[str, int]:
    out: dict[str, int] = {}
    for path in sorted(root.rglob("test_*.py")):
        rel = str(path.relative_to(root))
        out[rel] = count_file(path)
    return out


def main() -> int:
    parent, patch = Path(sys.argv[1]), Path(sys.argv[2])
    a, b = count_tree(parent), count_tree(patch)
    deleted = sorted(set(a) - set(b))
    dropped = {
        k: {"parent": a[k], "patch": b.get(k, 0)}
        for k in sorted(set(a) | set(b))
        if b.get(k, 0) < a.get(k, 0)
    }
    report = {
        "parent_total": sum(a.values()),
        "patch_total": sum(b.values()),
        "deleted_files": deleted,
        "dropped_files": dropped,
    }
    Path("assert-inventory.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))
    if deleted or report["patch_total"] < report["parent_total"]:
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Pair it with a name-level check so a file rename does not look like a deletion:

# proposed: copies plus AST, not either alone
git -C /tmp/patch diff --diff-filter=D --name-only MERGE_BASE -- 'tests/**'
python scripts/count_asserts.py /tmp/parent /tmp/patch

Added tests are allowed. They do not raise the score. They are extra documentation. The score is parent cassettes plus the surviving assert count.

4. Quarantine flakes on a ref the agent cannot write

A freeze file committed on the agent branch is a merge lever. The model can add names to it. Put the list on oracle/main, or in a protected Actions variable, and check it out as a second source.

# on ref oracle/main — humans only
quarantine:
  - id: tests/test_retry.py::test_eventual
    until: "2026-10-05"
    reason: "clock skew vs fake broker"
    owner: "platform"

# proposed CI fragment
git fetch origin oracle/main:oracle/main
git show oracle/main:quarantine.yaml > /tmp/quarantine.yaml
python scripts/check_quarantine.py \
  --now 2026-09-21 \
  --file /tmp/quarantine.yaml \
  --junit patch-junit.xml

The checker has three hard rules. Expired rows fail the job, they do not silently return to the scoring set. Rows that name a test absent from both trees fail the job. The patch worktree is not a valid place to create, extend, or delete rows.

Property candidates follow the same split. A seeded property that never hits its predicate is not a freeze candidate. It is an incomplete case. Keep it off oracle/main until a human has seen a hit on the parent binary.

# proposed: seed is part of the oracle, not a pytest default
# labeled unexecuted example
def test_order_total_non_negative(order_strategy, seed=20260921):
    ...

Decision table

Signal	Patch-editable?	Merge effect
Unit job on tests/	Yes	Advisory only
Cassette mismatch vs parent	No, recording is off-branch	Block
Allowlisted pointer with reviewer note	File is CODEOWNERS-protected	Allow that field only
Assert count drop or deleted test_*.py	Visible in AST/inventory	Block
New tests added by the agent	Yes	No score credit
Flake listed on oracle/main and not expired	No	Exclude from score, do not skip cassettes
Expired quarantine row	No	Block until a human removes or extends it
Property with zero hits on parent	N/A	Do not promote to oracle

One green lane does not override a cassette mismatch. The table is the gate. The job graph is just I/O.

Generating extra cases without letting them score

Cassette authoring is slow if every row starts as a human transcript. A cheap generation loop can propose extra argv and stdin shapes from --help, an OpenAPI file, or existing cases. Those proposals stay out of oracle/main until a parent replay produces a stable compared field.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are one place to draft candidate cases and run the dual-binary replay off the merge queue. The merge gate still has to read artifacts from a ref the agent cannot write. Generated rows that only pass on the patch binary are discarded.

Do not paste generator output into cases.yaml in the same PR that changes the CLI. That collapses the oracle into the patch again.

Limitations

Characterization freezes parent bugs. If the parent accepted a bad cart, the replay will demand that the patch accept it too, until a human adds an allowlist row and a contract test that describes the new reject. That is the point of the allowlist. It is also the cost.

Dual builds are slower than a single pytest job. Non-deterministic fields that leak into compare will page you. JSON-pointer allowlists can be rubber-stamped. AST assert counts miss custom helper wrappers unless you extend count_file. None of this inspects concurrency, authorization, or data loss outside the recorded surface.

A free generation loop will happily propose tautologies. Seed and hit-count on the parent binary before promotion, or the oracle becomes another editable suite.

Who should not use this

Skip the workflow if there is no stable public surface yet. Skip it if you cannot protect a second ref or a CODEOWNERS path. Skip it for one-off spikes where the CLI contract is the experiment. Skip it if the only observable is a GUI screenshot with no byte-stable encoding.

Teams that already run consumer-driven contract suites can reuse those fixtures as cassettes. They still need the inventory lock and the off-branch quarantine. The missing piece is usually not more unit tests. It is a recording the patch cannot edit.

If you already isolate oracles this way, a free server is one option to host the replay so merge runners only read replay-report.json and assert-inventory.json. The scoring rules stay in your protected CI.