Run the Patch Twice: A Paired-Run Matrix to Prove an Agent's Tests Actually Observe the Change

The short version: a green test run on an agent's patch is one data point, not evidence. Run the identical command against the patched tree and a reconstructed base tree, under the same fixture fingerprint, and only trust the quadrants where the two runs disagree.

A single green run cannot distinguish three very different situations: the tests genuinely exercise the new behavior, the tests execute the new code but assert nothing about it, or the run passed because a fixture default hid the branch entirely. The paired run separates them cheaply, and it needs no special tooling.

Why one green run is ambiguous

Agent patches usually arrive with tests attached. That is good, but it moves the failure mode from "no tests" to "tests that cannot fail".

Three mechanisms produce false confidence:

1. Inert observation. The new test calls the changed function, the call succeeds, but the assertion reads a value the patch never affected.
2. Fixture masking. A default in a fixture (a zero timeout, an empty list, a mock that returns early) makes the changed branch unreachable in the test process.
3. Instability. The pass depends on ordering, time, or a shared resource, so one sample cannot separate signal from noise.

The paired run attacks all three at once. If the tests never observe the change, the base tree passes too, and the pair looks identical. If a fixture hides the branch, it hides it on both sides. If the test is unstable, the two runs will not agree when you repeat them.

The verdict matrix

Record the exit status of the same test command on both trees, then classify:

Patched tree	Base tree	Verdict	Action
PASS	FAIL	Discriminating	Proceed — the tests changed behavior relative to base
PASS	PASS	Inert or masked	Do not merge on this evidence; find the assertion that should have failed
FAIL	PASS	Regression	Block; the patch broke behavior the base had
FAIL	FAIL	Uninformative	Fix the base or the environment first; nothing can be concluded

PASS/FAIL is the only row that carries real information, and it is exactly the row a single-run gate cannot see.

Step 1: pin both trees to hashes

Reconstruct the base from the commit the agent actually started from, not from your current main.

BASE=$(git merge-base HEAD origin/main)   # or the commit recorded when the agent session opened
git worktree add ../patch-base "$BASE"
git rev-parse HEAD > reports/patched.sha
git -C ../patch-base rev-parse HEAD > reports/base.sha

Two hashes in the report matter more than two dates. If you cannot name the base commit, you cannot run this matrix, and you should treat the patch run as unverified.

Step 2: fingerprint the fixtures

The comparison is void if the two runs do not share fixtures. Hash them before either run.

# sketch — adapt paths to your repo; not executed as written here
import hashlib, pathlib

def fingerprint(root="tests/fixtures"):
    h = hashlib.sha256()
    for p in sorted(pathlib.Path(root).rglob("*")):
        if p.is_file():
            h.update(str(p.relative_to(root)).encode())
            h.update(p.read_bytes())
    return h.hexdigest()

Write that digest into both report files. If the digests differ, the matrix is invalid, no matter what the exit codes say.

Step 3: write properties that can see the diff

A property check is only useful if it fails when the changed branch regresses. Derive the property from the diff, not from the existing test file. Three shapes cover most agent patches:

1. Round trip — decode(encode(x)) == x for inputs that cross the changed branch.
2. Idempotence — applying the patched operation twice equals applying it once.
3. Refusal — invalid input raises the specific error the patch is supposed to raise, not a generic one.

# sketch: exercise the changed function, do not assert around it
for case in cases_crossing_the_changed_branch:
    out = apply_behavior(case)
    assert round_trip(out) == out          # property 1
    assert apply_behavior(out) == out      # property 2

If a property passes on the base tree as well, it is not testing the patch. Move it to the regression set and find another one.

Step 4: the paired-run script

#!/usr/bin/env bash
set -uo pipefail
mkdir -p reports

run() { # $1 = worktree, $2 = label
  ( cd "$1" && pytest -q --junitxml="$OLDPWD/reports/$2.xml" >/dev/null 2>&1 )
  echo "$?"
}

patched=$(run "$PWD" patched)
base=$(run ../patch-base base)
printf 'patched=%s base=%s\n' "$patched" "$base"

Exit codes collapse into the matrix with a four-line mapping. Keep the JUnit XML files: they tell you which tests passed on both sides, which is the difference between "inert suite" and "inert single test".

Step 5: repeat once, and only classify on agreement

Run the pair twice. Two identical verdicts are a result; any disagreement is unstable, and an unstable test gets no verdict at all. Record a quarantine expiry date alongside it so the exclusion cannot silently become permanent — the paired run decides what enters quarantine, not how long it stays.

Where a free model endpoint changes the economics

This gate costs two runs per patch, four with the repetition. When every agent iteration is metered per call, the rational move is to sample one run and call it evidence, which is how inert suites survive review.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access makes the second run affordable inside the loop instead of a manual step someone skips. The free server option matters for a different reason: both trees must live on one machine with one fixture fingerprint, and if your agent's workspace is remote, running patched locally against base remotely recreates the exact cross-environment comparison this matrix is designed to eliminate. Put both worktrees on the same host and keep fixtures inside the repository so the fingerprint travels with the tree. If you want to try it, the free server option is enough for a small repo — but the matrix works the same on any host you already pay for.

Limitations, and who should not use this

• Detection is not correctness. PASS/FAIL proves the test is sensitive to the patch. It does not prove the patch is right.
• Runtime doubles. On a suite that already takes an hour, 2x to 4x may exceed your budget. Map changed paths to test files first and run the subset.
• Broken baselines are uninformative. If the base tree does not build, every comparison lands in FAIL/FAIL. Fix the base before trusting anything.
• Clock, network, and third-party state. Neither run is stable, so the matrix will mostly return unstable. Record the traffic and replay it; a second run will not help.
• Exploratory spikes. If you are throwing the patch away tomorrow, skip the gate.
• Large monorepos with path-dependent build caches. Worktree parity is harder than it looks; verify that both trees resolve the same lockfile before you trust a PASS.

Pre-merge checklist

1. Base commit hash recorded, worktree created from it.
2. Fixture fingerprint identical on both trees.
3. Same command, same environment variables, same seed on both sides.
4. Exit codes mapped to a quadrant; PASS/FAIL or regression found.
5. Paired run repeated; no disagreement.
6. Inert assertions rewritten or the patch held back.