Red-Green or It Didn't Happen: Auditing the Tests Inside Agent-Generated PRs

Most agent-generated PRs arrive with tests attached. That reads like reassurance; it is closer to a second claim. A test hunk asserts three things at once — that the behaviour is new, that this particular change is what makes the test pass, and that nothing outside the diff was needed to get there. None of the three is visible in the diff itself.

This post is about checking claims two and three mechanically: run the agent's tests against the base commit and against the head commit, and demand a specific kind of failure at the first step. Two git worktrees and a test runner you already have are enough.

Why the diff cannot settle it

When you read a test hunk next to a source hunk, your brain fills in the causal link. That link is exactly what agents are good at faking by accident. Three failure shapes recur:

• The tautology. The test imports the module and asserts it is not None. Green on base, green on head, zero information.
• The broken red. The test would fail on base — but because a fixture or a helper import does not exist there, not because the behaviour is missing. Looks like proof, is not.
• The unrelated pin. The test genuinely fails on base and passes on head, but it pins a neighbour of the changed line: an off-by-one boundary that the mutation <= → < would not disturb.

A gate that catches the first two is cheap. Catching the third costs one extra command.

The two-worktree harness

The idea is old (it is how you would verify a bug report by hand) and the automation is short. Take the merge-base, materialise it, copy only the PR's test files into it, and run them.

#!/usr/bin/env bash
# gate.sh <base-ref> <head-ref>
# Requires: git >= 2.5 (worktree). Test command comes from $TEST_CMD.
set -uo pipefail

BASE="${1:?base ref}"
HEAD="${2:?head ref}"
TEST_CMD="${TEST_CMD:-python -m pytest -q}"
MB="$(git merge-base "$BASE" "$HEAD")"
ROOT="$(git rev-parse --show-toplevel)"
WORK="$(mktemp -d)"

git -C "$ROOT" worktree add -q --detach "$WORK/base" "$MB"   || exit 2
git -C "$ROOT" worktree add -q --detach "$WORK/head" "$HEAD" || exit 2
trap 'git -C "$ROOT" worktree remove --force "$WORK/base" "$WORK/head"' EXIT

mapfile -t TESTS < <(git -C "$ROOT" diff --name-only --diff-filter=AM "$MB" "$HEAD" -- 'tests/')
for f in "${TESTS[@]}"; do
  mkdir -p "$WORK/base/$(dirname "$f")"
  git -C "$ROOT" show "$HEAD:$f" > "$WORK/base/$f"
done

run () {  # $1 = worktree, $2 = test path
  ( cd "$1" && $TEST_CMD "$2" ) > "$WORK/last.log" 2>&1
  echo $?
}

for f in "${TESTS[@]}"; do
  base_rc=$(run "$WORK/base" "$f")
  head_rc=$(run "$WORK/head" "$f")
  kind='assert-red'
  if [ "$base_rc" -ne 0 ] && grep -qE 'ERROR collecting|ModuleNotFoundError|ImportError' "$WORK/last.log"; then
    kind='broken-red'
  fi
  printf '%s\tbase=%s\thead=%s\t%s\n' "$f" "$base_rc" "$head_rc" "$kind"
done

Run it as TEST_CMD='python -m pytest -q' ./gate.sh origin/main HEAD. Output is a small table you can paste into the review thread, which matters more than it sounds: it turns an opinion into an artefact the author can argue with.

Separate useful red from broken red

The kind line above is the whole trick, and it is deliberately crude. pytest reports a missing module as a collection error, not a failure; a test that cannot start on base has proven nothing. Tune the matcher to your runner — go test compile errors and jest module-resolution errors both need their own pattern — and treat an unmatched red as suspicious rather than as evidence.

A useful red has a readable assertion message. If the base run dies with a stack trace in conftest.py, you learned about fixtures, not about retry semantics.

Mutation probe: does the assertion touch the line?

Red on base plus green on head still permits a test that pins something adjacent. One deliberate mutation of the source hunk settles it. Pick the smallest plausible edit — a comparison boundary, a default value, a null guard.

# mutation_probe.sh <head-ref> <file> <sed-expr> <test-path>
git worktree add -q --detach /tmp/probe "$1"
sed -i "$3" "/tmp/probe/$2"
( cd /tmp/probe && ${TEST_CMD:-python -m pytest -q} "$4" )
echo "exit=$? (non-zero means the test pinned the mutated line)"
git worktree remove --force /tmp/probe

Concrete case: the PR adds src/retry.py with if attempts <= max_retries: and a test file. Probe with s/<=/</ on that file and the PR's test as the target. Non-zero exit: the boundary is covered. Zero exit: the test exercises the function but not the branch that changed, which is a request-for-change, not a merge.

Two mutations are usually enough. More than three and you are re-implementing mutation testing on a single PR, which is not worth the review latency.

Reading the results

base run	head run	reading	action
assert-red	green	change is pinned	read the assertion text, then probe the boundary
broken-red	green	unproven — fixture gap	treat as no test; ask for the missing helper in the same PR
green	green	test constrains nothing new	revert the test-only hunks, or request the missing case
assert-red	red	incomplete or flaky	re-run three times before blaming the agent
no test hunk	—	untested behaviour	hold behind a flag or send back

Two rows deserve emphasis. Green/green is the common case in agent PRs that add "coverage" — the test proves the module imports, not that retries work. And the dead red row is where honest reviewers get fooled: a test that fails on base because it cannot import is the most convincing-looking useless test there is.

Where a free tier actually helps

Running the gate twice per PR multiplies compute, and mutation candidate generation is a task where being wrong is cheap and being fast matters. That is the narrow slot where MonkeyCode's free model access and free server option fit this workflow: use a model to propose two or three plausible single-line mutations for a hunk, then let the shell probe above decide which ones actually move the test. The model never gets a vote on mergeability; it only proposes edits that the harness either kills or confirms.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability claims are the operator's, not measurements: free model access and a free server option, with the free tier described by the operator as a 10-million-token allowance. I have not benchmarked throughput, model selection, or uptime, and quotas change — check the project page before you build a process around it. For the harness itself, a free server is most useful as a disposable runner: the two worktrees are throwaway state, and a machine you do not mind thrashing is the right place for them.

Then the model becomes useful for the review comment itself — summarising which mutation was killed and which survived — not for the verdict.

Limits, and who should skip this

• Hermetic suites only. If your test command needs a database, a container, or a network fixture, two worktrees double that cost, and the gate will be the slowest thing you run today.
• Test-file-only copying breaks. A PR that adds a conftest.py change or a new shared factory needs those files copied too, or the base run fails for the wrong reason. Widen the -- 'tests/' pathspec deliberately, not reflexively.
• It checks pinning, not correctness. A test can pin the wrong behaviour perfectly. The harness tells you the assertion is load-bearing; a human still has to agree with what it asserts.
• It misses edited tests. If the agent modified an existing test to match its own new behaviour, that hunk is an AM diff like any other and will pass. Review modified test lines by hand.
• Skip it for single-file scripts, generated code, and repos with suites longer than about ten minutes per file — the review latency outweighs the signal.

What the gate buys you

It buys a defensible answer to one question: did this PR change behaviour, and does it know it did? That is a smaller question than "is this correct", and it is the one you can automate without a model's opinion anywhere in the path. If you keep one habit from this, keep the ordering — base run first, head run second, mutation third, and only then read the source hunk with any confidence.

If you try the harness, start with a single PR where you already suspect the test is decorative; the first green/green row is usually convincing enough to keep the script.