Replay Yesterday's Bugs to Test Today's Agent-Patch Gate

Agent coding tools promoted every developer to reviewer. Nobody tested the reviewer. Your pre-merge gate—property checks, fixture contracts, a flaky-test freezer—is software too. It can be blind in exactly the places your code was broken before. This article gives you a cheap way to estimate the gate's recall: replay historical bugs in isolated worktrees and measure how many the gate still rejects.

Recall is more useful than coverage here. Coverage says which lines ran. Recall says which known bugs the pipeline would catch. Synthetic injection tells you the gate can fail. Replaying real bugs tells you it fails on failures your project actually had. You need both signals. This article is about the second one.

Build a bug corpus from closed fixes

Your history is already a bug database. Use it.

1. List fix commits from the last few months. A coarse query is enough:

git log --grep='fix' --oneline -50

1. For each fix, create a patch that reverts only the implementation, not the test that proves the bug is gone. Starting from the fixed commit:

git revert --no-commit <fix-commit>
git checkout <fix-commit> -- tests/test_the_bug.py
git diff > corpus/<bug-name>/source-revert.patch
git revert --abort

1. Store one patch per bug:

corpus/
  2026-04-cart-cursor/
    source-revert.patch
  2026-06-queue-race/
    source-revert.patch

If the fix touched multiple test files, restore all of them before saving the patch. The resulting worktree diff should be "broken source plus the regression test that should fail."

Run the replay loop

Now replay each bug through the exact command your merge process calls. The script isolates every replay in a throwaway worktree. No contamination between runs.

#!/usr/bin/env bash
set -euo pipefail

CORPUS=${1:?usage: gate-recall.sh corpus base-commit gate-cmd}
BASE_COMMIT=${2:?base}
GATE_CMD=${3:?gate}

total=0; caught=0; missed=0; bogus=0

for patch in "$CORPUS"/*/source-revert.patch; do
  bug_dir=$(dirname "$patch")
  bug_name=$(basename "$bug_dir")
  wt=/tmp/gate-recall-$bug_name
  total=$((total + 1))

  git worktree add -q "$wt" "$BASE_COMMIT"
  (cd "$wt" && git apply "$patch")

  set +e
  timeout 900 bash -c "cd '$wt' && $GATE_CMD" >"$wt/gate.log" 2>&1
  status=$?
  set -e

  if [[ $status -eq 0 ]]; then
    echo "MISSED  $bug_name"
    missed=$((missed + 1))
  elif [[ $status -eq 124 ]]; then
    echo "BOGUS  $bug_name (timeout)"
    bogus=$((bogus + 1))
  else
    echo "CAUGHT  $bug_name"
    caught=$((caught + 1))
  fi

  git worktree remove --force "$wt"
done

echo "caught=$caught missed=$missed bogus=$bogus total=$total"
awk -v c="$caught" -v d="$((total - bogus))" 'BEGIN { printf "recall=%.1f%%", (100 * c / d) }'
echo

A useful GATE_CMD for agent patches looks like this:

GATE_CMD='pytest -x -q tests/unit && pytest -q tests/property && python -m fixture_contract --check && python freeze_flaky.py --max-runs 3'

Property checks and fixture contracts catch the reverted behavior. The flaky freeze matters because a test that fails intermittently can rerun itself into a false green and turn a CAUGHT into a MISSED.

Read the matrix like a reviewer

Sample output:

CAUGHT  bug-042-cart-cursor
MISSED  bug-117-queue-race
BOGUS   bug-131-timeout
recall=66.7%

If a known bug is missed, you have found the exact invariant your gate is blind to. Trace the original fix for that bug. Add the missing invariant as a property check or a fixture assertion. Re-run the replay until that bug moves into CAUGHT. You are now testing the gate itself, not just the patch.

BOGUS is a gate crash, not a test failure. A timeout usually means the suite is too slow for the fresh worktree, the bug is in an integration area that needs a seeded service, or the timeout is too small. Increase the timeout or narrow the corpus before interpreting recall.

Why free resources are enough

A replay matrix is embarrassingly parallel, but every worktree burns CPU. A laptop can limp through a few; ten fixes need something with a little more isolation. MonkeyCode's free server option can host the replay loop as a short-lived background job, and its free model access can draft a first property-check skeleton from a bug report. Treat that draft as a starting point, review it like any generated code, and run it through the same replay loop before trusting it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

• A replay corpus is only as good as your fix history. Greenfield projects with fewer than a handful of real regressions should use synthetic bug injection instead.
• Revert patches drift. Pin BASE_COMMIT close to the fix date; replaying a six-month-old revert against today's main branch will produce noise, not recall.
• The gate command must be hermetic. If it needs hidden environment variables or a live service, BOGUS results will drown the true signal.
• The free model access is not an oracle. A weak property check still passes a reverted bug; the replay loop will show you the miss only after you run it.
• Do not use this as a replacement for human review. Use it to decide where review attention goes next.

Who should not use this: a repo with no bug history, a suite that cannot execute in a fresh worktree, or a merge process that is not yet scripted into one command. Fix those gaps first, then replay yesterday's bugs.

Verdict

Your gate is software. Replaying yesterday's bugs is the cheapest regression test that software can have. One suite checks the patch. The second lane checks the gate that reviews the patch. In an agent-first world, that second lane is no longer optional.