Replay the Patch on a Worktree the Agent Never Mounted

I do not treat a green test from the agent's working tree as evidence. I treat it as a rumor.

The model sat in a dirty directory. Maybe a leftover fixture. Maybe a pytest cache. Maybe a dependency it installed and never wrote down. Then it shouted that everything passed. Should I merge that? No. I replay the same patch on a second worktree built from the same commit SHA. The agent never mounts that tree. If the replay is red, the job is red. Full stop.

This is not a vibes piece about whether AI is "replacing" review. It is a from-zero workflow you can run this afternoon. Each stage has a command and a verification step. If a stage fails, you stop. You do not negotiate with the model.

Why the agent's tree lies

A working tree is a memory. Caches, untracked files, pip install leftovers, and exported env vars all count as memory. The agent can make that memory look like a proof.

Did it prove the patch? Or did it prove that this directory, right now, happens to be green?

Those are different claims. I only accept the second claim after I destroy the first environment and rebuild it from a SHA I chose.

I am not claiming I measured a fleet of production agents. The script below is a proposed gate. Label it that way. Run it before you trust it.

What you will have at the end

A disposable worktree, a frozen test command, a unified diff, and a JSON result file. Nothing merges until the replay process exits 0. That is the whole contract.

You need git, a test runner, and a patch file. I use pytest in the examples. Swap the lockfile command if your repo is not Python. Do not let the model pick the command. That is the point.

Stage 0 — Pin the base SHA

Start from a clean index on the branch you actually intend to merge into. Not on the agent's scratch branch. Yours.

# proposed gate — run in your repo, not in the agent's cwd
git checkout main
git pull --ff-only
git rev-parse HEAD > .agent/BASE_SHA
git status --porcelain > .agent/status.before
test ! -s .agent/status.before

Verify: .agent/BASE_SHA is 40 hex characters. status.before is empty. If status is not empty, you are already in a dirty tree. Stop. You would be measuring noise.

Why freeze the SHA first? Because "latest main" is not a value. It moves while the agent is talking.

Stage 1 — Lock the test command the model cannot edit

Create .agent/TEST_CMD yourself. One line. No comments the model can "clarify."

mkdir -p .agent
printf '%s\n' 'python -m pytest -q --maxfail=1 tests' > .agent/TEST_CMD
chmod 444 .agent/TEST_CMD

Verify:

test "$(wc -l < .agent/TEST_CMD)" -eq 1
test ! -w .agent/TEST_CMD

If the agent later proposes pytest -k test_happy_path, that is not a test run. That is a narrowing. Reject it. The lockfile is the command. There is no second command.

Stage 2 — Take a patch, not a story

The agent may write a recap. Throw the recap away. You want a unified diff against the SHA from Stage 0. Nothing else is merge input.

BASE=$(cat .agent/BASE_SHA)
# the agent (or you) must produce this file; do not accept a directory dump
test -f .agent/agent.patch
git apply --check --recount .agent/agent.patch

--check only asks "does this parse as a patch?" It does not ask "is this true?" That is later.

Verify: git apply --check exits 0. head -1 .agent/agent.patch looks like a diff header, not markdown. If the file starts with "Sure, here is what I changed," you do not have a patch. You have a chat log.

Stage 3 — Build a worktree the model cannot see

This is the gate. A second checkout. Same SHA. Different path. No shared dist/, no shared .pytest_cache, no shared venv unless you create it inside the new tree on purpose.

BASE=$(cat .agent/BASE_SHA)
REPLAY="/tmp/replay-$BASE"
git worktree add --detach "$REPLAY" "$BASE"

Copy only the patch and the lockfile into that tree. Do not rsync your dirty laptop state "to save time." That is how the rumor comes back.

mkdir -p "$REPLAY/.agent"
cp .agent/agent.patch .agent/TEST_CMD "$REPLAY/.agent/"

Verify:

test "$(git -C "$REPLAY" rev-parse HEAD)" = "$(cat .agent/BASE_SHA)"
git -C "$REPLAY" status --porcelain
# must print nothing except the files you just copied

If HEAD does not match, you replayed the wrong history. Kill the job. Do not "fix it up."

Stage 4 — Apply the patch inside the replay tree

cd "$REPLAY"
git apply --index --recount --whitespace=nowarn .agent/agent.patch
git diff --cached --stat > .agent/replay.stat

I keep --index so the replay tree's index matches the patch. I do not git commit yet. A commit is a decision. This stage is still a measurement.

Verify: git diff --cached --name-only is non-empty and does not include .agent/TEST_CMD. If the patch touches the lockfile, the agent is rewriting the exam. Fail the job.

Optional budget, still a proposal:

# fail if the patch is a rewrite pretending to be a fix
files=$(git diff --cached --name-only | wc -l)
test "$files" -le 20

Pick your own number. The number is not the insight. The cap is.

Stage 5 — Install from the replay tree, not from memory

If the repo has a lockfile, use it. If the agent "helpfully" ran pip install cool-lib on your laptop, that install does not exist here. Good.

cd "$REPLAY"
python -m venv .venv
. .venv/bin/activate
if test -f requirements.lock; then
  pip install -r requirements.lock
elif test -f requirements.txt; then
  pip install -r requirements.txt
fi

Verify: pip freeze | sha256sum > .agent/freeze.sha. Keep that hash next to the result JSON. If a later replay of the same SHA plus the same patch produces a different freeze hash, your install is not deterministic. That is a repo problem, not an agent problem. Fix the lockfile before you blame the model.

Stage 6 — Run the locked command and write a receipt

cd "$REPLAY"
. .venv/bin/activate
cmd=$(cat .agent/TEST_CMD)
start=$(date +%s)
set +e
$cmd > .agent/pytest.out 2> .agent/pytest.err
code=$?
set -e
end=$(date +%s)

Write a tiny JSON receipt. Not a blog post. A receipt.

python - <<'PY'
import json, hashlib, pathlib, os, time
base = pathlib.Path(".agent")
out = (base/"pytest.out").read_bytes()
err = (base/"pytest.err").read_bytes()
receipt = {
    "base_sha": pathlib.Path(".agent/BASE_SHA").read_text().strip() if pathlib.Path(".agent/BASE_SHA").exists() else open("/dev/null"),
    "exit_code": int(os.environ.get("REPLAY_CODE", "1")),
    "duration_s": int(os.environ.get("REPLAY_DUR", "0")),
    "stdout_sha256": hashlib.sha256(out).hexdigest(),
    "stderr_sha256": hashlib.sha256(err).hexdigest(),
}
# BASE_SHA lives in the original tree; pass it in
PY

The snippet above is incomplete on purpose if you paste it blindly. Here is the runnable version I actually want in the replay tree:

export REPLAY_CODE="$code"
export REPLAY_DUR="$((end-start))"
python - <<'PY'
import hashlib, json, os, pathlib
base = pathlib.Path(".agent")
def sha(p):
    return hashlib.sha256(p.read_bytes()).hexdigest()
receipt = {
    "exit_code": int(os.environ["REPLAY_CODE"]),
    "duration_s": int(os.environ["REPLAY_DUR"]),
    "stdout_sha256": sha(base / "pytest.out"),
    "stderr_sha256": sha(base / "pytest.err"),
    "test_cmd": (base / "TEST_CMD").read_text().strip(),
}
(base / "replay.json").write_text(json.dumps(receipt, indent=2) + "\n")
raise SystemExit(0 if receipt["exit_code"] == 0 else 1)
PY

Verify: .agent/replay.json exists. exit_code is 0. test_cmd equals the lockfile. If the duration is many times your known suite time, you did not get a proof. You got a hang. Fail it. I do not publish a magic threshold here because I did not benchmark your repo.

Stage 7 — Copy evidence back, then destroy the tree

cp "$REPLAY/.agent/replay.json" .agent/replay.json
cp "$REPLAY/.agent/replay.stat" .agent/replay.stat
git worktree remove --force "$REPLAY"

Verify: git worktree list no longer contains /tmp/replay-. The original working tree still matches BASE_SHA plus your own edits, not the agent's leftovers. Now, and only now, you may git apply the patch onto a real branch.

If Stage 6 failed, you still copy the JSON. Red evidence is useful. A missing file is not.

Where a remote box actually helps

I want the generator and the replay on different filesystems. My laptop is the source of truth. The model should not sit in it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a box that is not your checkout, MonkeyCode's free model access and free server option is one place to run the generator while this replay script stays on a tree you control. I am a writer on that outreach, not a benchmark lab. I am not going to invent model names, quotas, or hardware claims. The useful part is the split: generate somewhere disposable, replay somewhere you can delete with git worktree remove.

Decision table

Observation	Action
Agent tree is green, replay tree is red	Reject the patch. The agent used hidden state.
Both red	The patch is wrong, or the locked command is wrong. Fix tests yourself.
Replay green, agent tree red	Trust the replay. The agent tree was dirty in the other direction.
Replay green, freeze hash drifted	Stop. Your install is not pinned.
Patch edits TEST_CMD, CI, or lockfiles	Reject. The exam changed.
No tests in the repo	Do not use this gate. You have nothing to replay.

Limitations, said plainly

This does not catch a flaky test that is green twice by luck. It does not replace reading the diff. It does not prove performance, security, or product intent. Two worktrees cost disk and a few minutes. If your suite needs network, secrets, or a running cluster, a bare worktree will lie in a new way. You will need a dedicated CI job, not /tmp.

Who should not use this? Anyone without an automated test command they are willing to freeze. Anyone who needs the agent to mutate a live tree in place. Anyone hoping a second checkout will replace code review. It will not.

What I actually trust

I trust a SHA I wrote down. I trust a test command I made unwritable. I trust a worktree the model never mounted. I trust a JSON file with an exit code.

I do not trust a paragraph that says "all tests passed."

Replay it. Then delete the tree. If you cannot delete the tree, you never isolated the job.