Require Parent Failures: Dual-Checkout Evidence for Agent Patches

A green suite on the agent's tip is not merge evidence. The gate needs a second checkout: the parent commit. New tests that already pass there do not constrain the patch. Characterization hashes captured before the edit tell you what must stay put. Flakes belong in a freeze record with a class and an expiry, not in an unconditional skip.

This article is a proposed workflow, not a claim about a production fleet. Examples below are labeled as such. They are meant to be copied, run, and discarded if they do not fit the repo.

Why tip-green is a weak signal

Agent patches often arrive with extra tests. Those tests can be useful. They can also be tautological wrappers around the new code, or they can encode the post-patch behavior so both parent and tip stay green.

A test that passes on the parent does not detect the change. A test that fails on both checkouts is broken, not protective. Only a test that is red on parent and green on tip is evidence that the patch altered an observable the author intended to alter.

Existing tests still matter. They are the characterization surface. If an agent rewrite changes a hash that nobody allowlisted, the suite is reporting a behavior drift, even when the new assertions look tidy.

Three artifacts, one gate

Keep three files outside the agent's write set. The runner, not the model, owns them.

1. oracle/characterization.json — content hashes of parent outputs for frozen fixtures.
2. oracle/properties.py — predicates that never import production helpers the agent is allowed to edit.
3. oracle/flake_freeze.yaml — skips that expire, classified by failure class, never by test name fashion.

The patch may add tests under tests/. The merge job dual-checkouts parent and tip, then classifies every new node. Unclassified green is not a pass.

Decision table

Observation	Gate action
New test fails on parent, passes on tip	Accept as patch evidence
New test passes on parent and tip	Reject; not a constraint
New test fails on parent and tip	Reject; broken or mis-targeted
Existing characterization hash changed, not allowlisted	Reject
Property module import graph touches patched files	Reject
Flake freeze expired or missing class	Unskip; re-run; do not merge
Agent diff includes oracle/	Reject

The table is the policy. Test names are not.

Workflow

Use git worktrees so both trees stay on disk. Do not stash and bounce HEAD; that races with local edits.

1. Snapshot the parent characterization

Run this on a clean parent checkout. Treat the JSON as an oracle, not as a golden file the agent may rewrite.

# proposal: commands, not a recorded production run
git fetch origin
PARENT=$(git rev-parse HEAD^)
git worktree add /tmp/parent-wt "$PARENT"
git worktree add /tmp/tip-wt HEAD

python tools/char_snapshot.py --cwd /tmp/parent-wt --out /tmp/parent-char.json
python tools/char_snapshot.py --cwd /tmp/tip-wt --out /tmp/tip-char.json
python tools/char_diff.py /tmp/parent-char.json /tmp/tip-char.json --allowlist oracle/allow_drift.txt

char_snapshot.py should hash fixture outputs, not pytest node ids. Node ids move when files are renamed. Hashes of canonical bytes do not.

# proposal: tools/char_snapshot.py
from __future__ import annotations

import hashlib, json, sys
from pathlib import Path

def digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def snapshot(root: Path) -> dict[str, str]:
    out = {}
    for p in sorted((root / "fixtures" / "locked").glob("**/*")):
        if p.is_file():
            rel = str(p.relative_to(root))
            out[rel] = digest(p)
    return out

if __name__ == "__main__":
    root = Path(sys.argv[sys.argv.index("--cwd") + 1])
    payload = snapshot(root)
    Path(sys.argv[sys.argv.index("--out") + 1]).write_text(json.dumps(payload, indent=2) + "\n")

If the agent also rewrites locked fixtures, the characterization job will fail closed. That is the point.

2. Dual-run new tests only

Collect node ids that exist on tip and not on parent. Run that set twice. Same seed. Same fixture digest. Different trees.

# proposal: tools/dual_gate.py
from __future__ import annotations

import json, subprocess, sys
from pathlib import Path

def nodes(cwd: str) -> set[str]:
    p = subprocess.run(
        ["pytest", "--collect-only", "-q"],
        cwd=cwd, check=True, capture_output=True, text=True,
    )
    return {line.strip() for line in p.stdout.splitlines() if "::" in line}

def run(cwd: str, select: list[str], seed: str) -> dict[str, str]:
    cmd = ["pytest", "-q", f"--randomly-seed={seed}", *select]
    p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
    # Map node -> passed|failed|error from junit or pytest-json-report in a real gate.
    return {"exit": str(p.returncode), "stderr_tail": p.stderr[-800:]}

def main() -> int:
    parent, tip, seed = sys.argv[1], sys.argv[2], sys.argv[3]
    added = sorted(nodes(tip) - nodes(parent))
    if not added:
        print("no new tests; relying on characterization + properties only")
        return 0
    parent_res = run(parent, added, seed)
    tip_res = run(tip, added, seed)
    Path("/tmp/dual_gate.json").write_text(json.dumps({"added": added, "parent": parent_res, "tip": tip_res}, indent=2))
    if parent_res["exit"] == "0":
        print("reject: new tests already pass on parent")
        return 2
    if tip_res["exit"] != "0":
        print("reject: new tests fail on tip")
        return 3
    print("accept: parent-red, tip-green")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Wire --randomly-seed or an equivalent so order noise does not masquerade as a parent failure. If the project has no shuffle plugin, pin PYTHONHASHSEED and a fixture RNG explicitly.

3. Keep properties off the patch surface

Property checks should import public call shapes, not the modules the agent just rewrote. A cheap static check is enough for a first gate.

# proposal: tools/oracle_imports.py
from __future__ import annotations

import ast, sys
from pathlib import Path

BANNED_PREFIXES = ("app.", "src.")  # adjust to the production package

def imports_of(path: Path) -> set[str]:
    tree = ast.parse(path.read_text())
    found: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom) and node.module:
            found.add(node.module)
        elif isinstance(node, ast.Import):
            found.update(a.name for a in node.names)
    return found

def main() -> int:
    bad = []
    for p in Path("oracle").glob("**/*.py"):
        for mod in imports_of(p):
            if mod.startswith(BANNED_PREFIXES):
                bad.append(f"{p}: {mod}")
    if bad:
        print("reject: oracle imports production modules\n" + "\n".join(bad))
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A property that calls the same helper the patch inlined is not independent. Move the predicate to a pure function of inputs and outputs. Keep generators in oracle/gen/ with a frozen seed list.

4. Freeze flakes by class, not by skip

An open-ended pytest.mark.skip is how suites rot. Require a record. Refuse unknown classes.

# proposal: oracle/flake_freeze.yaml
- id: tests/test_ledger.py::test_concurrent_post
  class: order          # order | timing | net | rand | fs
  seed: "2147483647"
  fixture_digest: "sha256:8f1c…"
  last_seen: "2026-09-18"
  expires: "2026-10-02"
  owner: "payments"
  note: "two postings swapped under default pytest order"

# proposal: tools/flake_gate.py
from __future__ import annotations

import datetime as dt, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    raise SystemExit("install pyyaml for this proposal script")

ALLOWED = {"order", "timing", "net", "rand", "fs"}

def main() -> int:
    today = dt.date.fromisoformat("2026-09-20")  # inject from CI clock
    rows = yaml.safe_load(Path("oracle/flake_freeze.yaml").read_text()) or []
    errors = []
    for row in rows:
        if row.get("class") not in ALLOWED:
            errors.append(f"unknown class: {row}")
        if dt.date.fromisoformat(row["expires"]) <= today:
            errors.append(f"expired freeze: {row['id']}")
        if not row.get("seed") or not row.get("fixture_digest"):
            errors.append(f"freeze missing seed or digest: {row['id']}")
    if errors:
        print("reject flake freeze\n" + "\n".join(errors))
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

On expiry the job unskips the node. If it is still red, classify again. If it is green across three consecutive dual-runs, delete the freeze. Do not convert it into a permanent ignore.

5. Optional hunk coverage, after the dual run passes

Parent-failing tests can still miss a silent hunk. If you already emit coverage.json on the tip run, intersect executed lines with the unified diff. Reject hunks that no parent-red test executed.

This step is optional because coverage of a line is not correctness of a line. Use it as a net, not as a proof.

Where a free runner helps

The dual checkout doubles wall time. Local laptops hide that cost until the suite grows. A spare server that can check out two worktrees and run pytest is enough. Model access is only needed if you later ask a model to propose freeze classifications; the gate itself should stay deterministic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that second checkout without inventing a new CI product. The policy above does not depend on a vendor. If the dual-run is cheaper on your existing runners, keep it there.

Limitations

The parent-fail rule assumes the parent suite is already meaningful. If main is red, dual-checkout classification collapses. Fix main first.

Tests that need live network, wall-clock timers, or GPU nondeterminism will look like parent failures when they are flakes. Those belong in flake_freeze.yaml, or they do not belong in the merge gate.

Characterization hashes do not encode meaning. Two different wrong outputs can hash differently and still both be wrong. Properties have to carry the meaning. Hashes only detect drift.

Do not use this workflow on generated UI screenshots without a perceptual comparator. Byte hashes on images thrash. Do not use it as a substitute for code review of security-sensitive diffs. A parent-red test can still assert the wrong contract.

Teams of one, on a repo with no fixtures and no agent, will pay setup cost for little signal. Skip the gate until an automated patcher is in the loop.

What to implement first

Start with worktrees and the added-node classifier. Add characterization hashes next. Import-graph isolation for oracle/ is a small static check; put it in before freeze records. Freeze records last, because they are how you prevent the other two from being silenced.

The core conclusion does not change with tooling. If a new test is already green on parent, it is documentation of the old system, not evidence about the patch. Split the suite across two checkouts until that distinction is mechanical.