The Test That Can't Fail: Reading the Test Diff of an Agent-Generated PR

Agent-generated PRs fail review in a specific way. The production code looks reasonable, the CI is green, and the new tests prove nothing — because they mock the one function the PR changed.

Strip the tests out and the diff is usually small. That is the tell. When 200 lines of test text surround 15 lines of behavior, the volume is doing rhetorical work, not verification work.

This post is about reviewing that test diff. Not the architecture, not the naming, not whether the agent's approach is elegant. Just one question: if the behavior broke tomorrow, would any of these new tests turn red?

Why the test diff is the least trustworthy part

Production code and test code in an agent PR are written by the same model, in the same session, from the same misunderstanding. If the agent believes a function returns a normalized path, both the implementation and the assertion will encode that belief. Reading them together feels like corroboration. It is actually one claim repeated twice.

So calibrate trust by executability, not by volume:

• Lines that execute in production — a new branch, a changed constant, a new call site. You can test these.
• Lines that describe behavior — tests, docstrings, comments, PR-body summaries, changelog entries, assert True placeholders, eval scripts that were never run. These are claims.

The failure mode is double counting. The agent asserts X in the PR body, asserts X in a test name, and you count three pieces of evidence where there is one.

A 15-minute diff autopsy

Do this before reading the implementation in detail. Order matters.

1. Split the diff by kind. Separate production files from test files from docs/lockfiles.
2. Write down one sentence describing the behavior the production hunk claims to add.
3. Find the assertion that would fail if that behavior were absent. Not the test that passes when it is present — the one that fails when it is missing.
4. Break the production line on purpose in a scratch copy and rerun only the new tests.
5. Revert anything you could not make fail. A test that survives a broken implementation is not a test.

# 1. what actually changed, by kind
git diff --stat "${BASE:-origin/main}"...HEAD -- ':!tests' ':!*_test.*'
git diff --name-only "${BASE:-origin/main}"...HEAD -- 'tests/*' '*_test.*'

# 2. did the new tests ever pass on their own?
python -m pytest -q $(git diff --name-only "$BASE"...HEAD -- 'tests/*' | tr '\n' ' ')

Step 4 is the one people skip, because it is repetitive. Make it scriptable.

The artifact: a mutation smoke test for the review thread

This is a blunt, regex-level mutation loop. It takes the production files touched by the PR, mutates numeric literals and booleans, and reruns only the new test files. Any mutation that leaves the suite green is a survivor — a behavior the new tests cannot detect.

This script has not been executed by the article author. Treat the paths as placeholders and run it in a throwaway worktree, never in your working tree.

# mutate_smoke.py — survivors are untested behavior, not proof of a bug.
import pathlib, re, shutil, subprocess, sys, tempfile

TARGETS = sys.argv[1:6]          # production files from the diff
NEW_TESTS = sys.argv[6:]         # only the test files the PR added/changed

PATTERNS = [
    (r"\b(\d+)\b", lambda m: str(int(m.group(1)) + 1)),  # n -> n+1
    (r"\bTrue\b",  lambda m: "False"),
    (r"\bFalse\b", lambda m: "True"),
]

def sites(src):
    for pattern, replace in PATTERNS:
        for match in re.finditer(pattern, src):
            yield match.start(), match.end(), replace(match)

survivors, killed = [], 0

with tempfile.TemporaryDirectory() as tmp:
    work = pathlib.Path(tmp) / "repo"
    shutil.copytree(".", work, ignore=shutil.ignore_patterns(".git", "__pycache__"))

    for target in TARGETS:
        original = (work / target).read_text()
        for start, end, replacement in sites(original):
            mutated = original[:start] + replacement + original[end:]
            (work / target).write_text(mutated)
            run = subprocess.run(
                ["python", "-m", "pytest", "-q", *NEW_TESTS],
                cwd=work, capture_output=True, text=True,
            )
            if run.returncode == 0:
                survivors.append(f"SURVIVED {target}:{start} -> {replacement!r}")
            else:
                killed += 1
            (work / target).write_text(original)

print(f"killed={killed} survived={len(survivors)}")
print("\n".join(survivors) or "every mutation was caught")

Read the output like a reviewer, not a metric. A surviving +1 on a retry count is a real gap. A surviving digit inside a string literal is noise — the regex does not parse Python. Cap the run at roughly 20 mutations per PR; beyond that, the loop stops being something a human will actually do.

What to trust, what to revert, what to test

Diff element	Default action	Evidence that flips the default
New production branch	Test it, then trust	Mutation survives → revert or add a test
Retry / timeout / limit constant	Revert	A test that fails when the value is out of range
Test that mocks the function under review	Revert	Rewritten to exercise a real boundary
New dependency pin	Revert	Lockfile diff plus a linked upstream changelog
Prompt string	Treat as data, never as policy	Server-side validation exists independent of it
Tool/schema registration	Test a denied call	Executor denies by default
PR-body "verified" claim	Ignore	A copy-pasteable command with its output
Changelog / doc edits	Collapse to one paragraph	Claim traceable to a code line

Two rules keep the table honest. First, a revert is not a rejection — it is a request for the smallest diff that carries its own evidence. Second, never merge because the tests are green; merge because you broke the code and the tests noticed.

Where a free re-run fits

Steps 1–4 are free for pure code. They stop being free when the mutation loop calls a model.

If the PR touches a model call, the survivor check needs a live endpoint, and a review pass can quietly become a metered task. That is the specific gap MonkeyCode covers in this workflow: per the operator it provides free model access and a free server option, which makes the reproduce-and-re-run half of review cheap enough to do per PR instead of per quarter.

The operator states the free tier currently includes a 10M-token allowance. Confirm the current terms on the project page before you plan a review cadence around it — allowances change, and this article should not be your source of record. MonkeyCode is presented as an open-source project; check the repository's own documentation for what is actually released.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three constraints apply, and they are not negotiable:

• A re-run only counts as evidence if someone else can repeat it. Paste the command and the raw output into the review thread. Otherwise it is an anecdote with a green checkmark.
• Bound the re-run. Use a scratch credential and a scratch server, never the production key, for the review loop.
• A free endpoint is not an allowlist. Cheapness is not a security control, and it is not a reason to merge.

Limitations and who should skip this

• Cost scales as mutants × test runtime. On a fat config module you will generate survivor spam and learn nothing. Scope to files the PR touched.
• It detects tests that cannot fail — not behavior that is missing. A feature with no test at all produces zero survivors. The autopsy steps still matter more.
• Regex mutation is deliberately crude. It edits string literals and comments. Review the list; do not paste the count into a dashboard.
• Skip it if your CI already runs mutation testing, if the PR is docs-only or a pure dependency bump, or if the changed tests require a live third-party service you cannot isolate.
• Skip it if you are the only reviewer and there is no second person to hand the survivor list to. The artifact is a communication device. Without a reader, it is just latency.

The point is not to make agents write better tests. It is to make the tests stop being decoration. Break the code, watch for red, and revert whatever stayed green.

If you want to try the re-run half of this loop, MonkeyCode's free model access and free server option are aimed at exactly this unglamorous work — the review discipline still has to come from you.