Park Agent Diffs in a Replay Worktree Before CI Starts

An agent-authored patch should not be allowed to start your paid CI. Local green is not a ticket. A hermetic replay against frozen fixtures, written to a report your workflow can reject, is the ticket.

You already know the failure. A model emits a plausible diff. You run two tests on your laptop. You open the pull request. GitHub Actions then burns minutes on a change that never would have survived a frozen fixture pack. The patch is not evil. It is just unreplayed.

This article gives you a small gate for that gap. You park the diff in a throwaway worktree, run only the fixtures the diff claims to touch, and emit replay_report.json. CI reads the report before it schedules anything expensive. If the report is missing, stale, or hashed against the wrong tree, the workflow stops. No merge-queue lore. No second lane. Just a report that must exist before CI starts.

The failure you are gating

Agent patches fail in boring ways. They edit the test instead of the code. They miss the path the fixture encodes. They rename a helper and leave the caller on the old symbol. None of that needs a GPU. It needs a replay you can hash.

Paid CI is the wrong place to discover those misses. Runners are shared. Caches lie. Secrets are in scope. You want the cheap, secret-free check to happen first, on a worktree that dies when the script exits.

Treat the model as an untrusted patch source. You would not merge a zip file from the internet because a unit test passed on someone's laptop. Do not treat an agent diff as stronger evidence than that zip file.

What the report must contain

Keep the contract tiny. If a field does not help you reject a run, drop it.

{
  "schema": "replay-report/v1",
  "git_head": "4f2c91a0e1b7d3aa",
  "patch_sha256": "b7c1…",
  "worktree": "/tmp/replay-4f2c91a0",
  "fixtures": [
    {"id": "cart.total.v3", "path": "fixtures/cart/total.json", "sha256": "aa19…"},
    {"id": "cart.tax.v1", "path": "fixtures/cart/tax.json", "sha256": "0c44…"}
  ],
  "commands": [
    {"cmd": "python -m pytest tests/test_cart.py -q", "exit": 0}
  ],
  "result": "pass",
  "created_at": "2026-09-14T12:04:11Z"
}

git_head pins the base. patch_sha256 pins the bytes you applied. Each fixture hash pins the input the test actually saw. CI will recompute those three and compare. If any drift, the job fails closed.

Do not put model prompts, API keys, or hostnames in this file. The report is an admission ticket, not a chat log.

1. Freeze a fixture manifest

Start with the files your tests already read. You do not need a new framework. You need hashes.

# tools/hash_fixtures.sh
set -euo pipefail
manifest="fixtures/manifest.json"
python - <<'PY'
import hashlib, json, pathlib, sys
rows = []
root = pathlib.Path("fixtures")
for path in sorted(root.rglob("*.json")):
    if path.name == "manifest.json":
        continue
    digest = hashlib.sha256(path.read_bytes()).hexdigest()
    rows.append({"id": path.stem, "path": str(path), "sha256": digest})
pathlib.Path("fixtures/manifest.json").write_text(json.dumps({"fixtures": rows}, indent=2) + "\n")
print(f"wrote {len(rows)} fixture hashes", file=sys.stderr)
PY

Commit fixtures/manifest.json. When someone edits a fixture, the manifest changes, and every old replay report becomes invalid. That is the point. Replay is not a vibe. It is a hash match.

If a test reads a path that is not in the manifest, fail the replay. Unmapped inputs are how silent skips sneak in. You do not need a long policy document. One missing hash is enough to stop the run.

2. Apply the patch in a disposable worktree

Never replay on your dirty checkout. You will mix agent bytes with leftover edits and then trust the result. Use a worktree, apply the patch, run the listed tests, write the report, then remove the worktree.

# tools/replay_agent_patch.sh
set -euo pipefail
patch_file="${1:?usage: replay_agent_patch.sh <patch>}"
head_sha="$(git rev-parse --short=16 HEAD)"
work="/tmp/replay-${head_sha}-$$"
report="replay_report.json"

git worktree add --detach "$work" HEAD
cleanup() { git worktree remove --force "$work" 2>/dev/null || rm -rf "$work"; }
trap cleanup EXIT

patch_sha="$(python - <<PY
import hashlib, pathlib
print(hashlib.sha256(pathlib.Path("$patch_file").read_bytes()).hexdigest())
PY
)"

(cd "$work" && git apply --check "$patch_file" && git apply "$patch_file")

# Fail closed if the patch touches files with no fixture mapping.
python tools/check_diff_to_fixtures.py "$work" "$patch_file" fixtures/manifest.json

set +e
(cd "$work" && python -m pytest tests/test_cart.py -q)
exit_code=$?
set -e

python tools/write_replay_report.py \
  --head "$head_sha" \
  --patch-sha "$patch_sha" \
  --worktree "$work" \
  --manifest fixtures/manifest.json \
  --cmd "python -m pytest tests/test_cart.py -q" \
  --exit "$exit_code" \
  --out "$report"

test "$exit_code" -eq 0

The script is boring on purpose. Boring scripts get copied. Clever wrappers get skipped when someone is in a hurry.

Label the mapping checker as a proposal if your repo does not yet have path rules. A minimal version is enough to start:

# tools/check_diff_to_fixtures.py
"""Fail if the patch touches src/cart but no fixtures/cart/* hash exists."""
import pathlib, json, subprocess, sys

work, patch, manifest_path = sys.argv[1], sys.argv[2], sys.argv[3]
manifest = json.loads(pathlib.Path(manifest_path).read_text())
mapped_dirs = {pathlib.Path(row["path"]).parent.as_posix() for row in manifest["fixtures"]}

diff = subprocess.check_output(["git", "apply", "--numstat", patch], text=True)
changed = []
for line in diff.splitlines():
    parts = line.split()
    if len(parts) >= 3:
        changed.append(parts[-1])

for path in changed:
    if not path.startswith("src/"):
        continue
    expected = "fixtures/" + "/".join(pathlib.Path(path).parts[1:-1])
    hits = [d for d in mapped_dirs if d.startswith(expected) or expected.startswith(d)]
    if not hits:
        raise SystemExit(f"unmapped path {path}; no fixture under {expected}")

You can replace the src/ prefix with whatever layout you use. The rule is the same: a changed production path must name a hashed fixture directory, or the replay never writes result: pass.

3. Make CI refuse work without the report

The workflow does one thing first. It verifies the report. Only then does it run the suite you actually pay for.

# .github/workflows/require-replay.yml
name: require-replay
on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  replay-ticket:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1
      - name: Require replay_report.json
        run: |
          test -f replay_report.json
          python tools/verify_replay_report.py replay_report.json
      - name: Paid tests (only after replay)
        run: python -m pytest -q

verify_replay_report.py should recompute, not trust:

# tools/verify_replay_report.py
import hashlib, json, pathlib, subprocess, sys

report = json.loads(pathlib.Path(sys.argv[1]).read_text())
if report.get("schema") != "replay-report/v1":
    raise SystemExit("unknown replay schema")
if report.get("result") != "pass":
    raise SystemExit("replay did not pass")

head = subprocess.check_output(["git", "rev-parse", "--short=16", "HEAD"], text=True).strip()
if report["git_head"] != head:
    raise SystemExit(f"report head {report['git_head']} != {head}")

manifest = json.loads(pathlib.Path("fixtures/manifest.json").read_text())
wanted = {row["path"]: row["sha256"] for row in manifest["fixtures"]}
for row in report["fixtures"]:
    on_disk = hashlib.sha256(pathlib.Path(row["path"]).read_bytes()).hexdigest()
    if wanted.get(row["path"]) != row["sha256"] or on_disk != row["sha256"]:
        raise SystemExit(f"fixture drift: {row['path']}")
print("replay report ok")

Commit the report with the pull request, or upload it as a workflow artifact from a pre-job you control. Either way, the paid job must not start on a missing file. A skipped step is not a pass.

4. Decide what belongs in the replay

Not every test earns a place in the worktree. Use a table, not a meeting.

Kind of check	In the replay?	Why
Pure unit test with a JSON fixture	Yes	Hashable, secret-free, fast
Contract test against a recorded HTTP cassette	Yes, if the cassette is in the manifest	Replay needs the same bytes
Live staging call	No	Not hermetic; do not hash the network
Job that needs production secrets	No	The worktree should not see those secrets
UI screenshot diff	Only if the baseline image is hashed	Otherwise the report cannot fail closed
Flaky integration suite	No	A replay that needs retries is not a ticket

If a check cannot be hashed, it does not belong in replay_report.json. Put it behind the report, not inside it.

Where a spare server actually helps

The harness above runs on a laptop. Many agent patches still waste the next hop: you regenerate the diff, forget to replay, and push anyway. A small always-on box that only runs replay_agent_patch.sh removes that skip. You drop a patch on the box. You get a report or a rejection. You do not get a GitHub Actions bill for the misses.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a place to host that loop without standing up your own runner, MonkeyCode's free model access and free server option can hold the replay harness and regenerate a patch when the report comes back result: fail. Keep your schema. Do not outsource the hash check. The product is a spare environment, not a replacement for verify_replay_report.py.

Use the model only after a failed replay, with the report as the prompt context: which fixture hash failed, which command exited nonzero, which path was unmapped. A regenerated patch that cannot produce a new report still does not start CI. That is the whole policy.

Limitations

This gate does not review design. A replay can pass on a patch that deletes a feature you still need. Humans still read the diff.

It does not replace branch protection. If someone can force-push and rewrite replay_report.json without running the script, you have a process hole, not a tooling hole. Protect the workflow file. Require the verify job.

It does not make non-deterministic tests honest. If pytest talks to the clock, the network, or a shared database, the report will flap and you will start ignoring it. Ignore the report once, and the gate is gone. Keep replay on hashed inputs only.

It also does not prove the patch is the patch GitHub will merge. Rebase, squash, and late review comments all change bytes. After those events, run the script again. An old report on a new head is a failed verify, which is the correct outcome.

Who should not use this

Skip this if your pull requests never include model-authored hunks and your laptop tests already match CI. You would be hashing ceremony.

Skip it if your only tests are live end-to-end runs against shared staging. You cannot freeze those inputs, so you cannot write a report worth verifying. Fix the test shape first.

Skip it if you cannot keep secrets out of the worktree. A replay box that mounts production credentials is a new incident, not a cheaper CI.

If you do use it, start with one package and two fixtures. Expand the manifest when a missed path burns you, not before. The merge path does not care which laptop produced the patch. It cares that the replay hash still matches.