Green Tests, Empty Assertions: Reviewing Agent PRs That Write Their Own Tests

A 47-file agent PR reached my review queue with 38 new tests and a coverage jump from 61% to 74%. Every test passed. I reverted the production changes, kept the tests, and ran the suite again. It was still green.

That single experiment is the review heuristic I now run first. When the same agent writes the fix and the test, the test is not independent evidence. It is a second copy of the same assumption in a different syntax. The review question is not 'do the tests pass' but 'which line would have to be wrong for this test to fail'.

Why agent-written tests pass for the wrong reasons

Five patterns account for most of what I see in generated suites.

• Truthiness instead of value. assert result where result is a dict, a list, or a model response object. Almost nothing can make this fail except None.
• The mock asserts the mock. The test patches the function under review, then asserts the patch was called with the arguments the test itself passed in. Delete the production code and it still passes.
• pytest.raises(Exception). Too wide to be evidence. It passes for a KeyError caused by a typo, not only for the domain error the PR documents.
• Expected values copied from the implementation. The agent writes assert total == 1187, then you find BASE_FEE = 1187 in the diff. The test encodes the constant, not the rule.
• Snapshots of new output. A recorded output is a regression detector, not a correctness check. If the snapshot was generated by the same buggy function, the test freezes the bug.

None of these are AI-specific. Agents just produce them at volume, inside the same PR, with high coverage numbers attached.

The two-minute check: revert the fix, keep the tests

Before reading a single assertion, spend two minutes on a throwaway worktree.

git fetch origin pull/1234/head:pr-1234
git worktree add ../review-pr-1234 pr-1234
cd ../review-pr-1234
git log --oneline origin/main..HEAD        # identify the production commits
git revert --no-commit <fix-sha>           # production only, tests stay
pytest -q tests/test_billing.py            # the tests the PR added
git reset --hard && git worktree remove --force ../review-pr-1234

Read the result honestly:

1. Green after reverting the fix -> the new tests do not constrain the new behavior. Revert them with the fix, or rewrite them until they go red.
2. Red, but on an unrelated test -> the PR changed shared state. That is your real review target.
3. Red on the intended tests -> the tests are doing work. Now review them on quality.

The third case is rarer than the coverage number suggests.

What to trust, what to revert, what to test

Change in the PR	Default verdict	Evidence needed before merge
assert x is not None on a new path	Revert	A case that actually produces None
Test patches the function it tests	Revert	One test through the real call path
pytest.raises(Exception)	Rewrite	A named exception type
Fixture mocked to the shape of the new code	Test	A contract check against the real schema
Recorded snapshot of new output	Trust for regressions only	A human read of the snapshot diff
time.sleep before an assertion	Revert	A deterministic wait or clock injection
Assertion values matching a new constant	Rewrite	Hand-written literals from the spec

The table is not a lint rule. It is a triage order: take the cheap reverts first, then spend real time on the rows marked test.

Artifact: a mutation probe scoped to the PR diff

Full mutation testing on a large suite is too slow for review. You do not need it. You need mutation coverage of the lines this PR touched.

The sketch below reads the unified diff, mutates one operator per added source line, and runs only the tests the PR added. A surviving mutant means no new test pins that line.

# probe_pr_tests.py - reference sketch, adapt paths before use.
# Run inside a throwaway git worktree: it rewrites source files in place.
import re, sys, subprocess, pathlib

SRC = {'.py', '.ts', '.js', '.go'}
MUTATIONS = [('==', '!='), ('<=', '<'), ('>=', '>'), ('+', '-'), ('and', 'or')]

def added_lines(diff_text):
    files, current, lineno = {}, None, 0
    for line in diff_text.splitlines():
        if line.startswith('+++ b/'):
            path = line[6:]
            current = path if pathlib.Path(path).suffix in SRC else None
        elif current and line.startswith('@@'):
            lineno = int(re.search(r'\+(\d+)', line).group(1))
        elif current and line.startswith('+'):
            files.setdefault(current, []).append(lineno)
            lineno += 1
        elif current and not line.startswith('-'):
            lineno += 1
    return files

def tests_green(test_targets):
    r = subprocess.run(['python', '-m', 'pytest', '-q', '-x', *test_targets],
                       capture_output=True, text=True)
    return r.returncode == 0

def main(rev_range, test_targets):
    diff = subprocess.run(['git', 'diff', '-U0', rev_range],
                          capture_output=True, text=True, check=True).stdout
    survivors = []
    for path, lines in added_lines(diff).items():
        original = pathlib.Path(path).read_text()
        rows = original.splitlines()
        for n in lines:
            source = rows[n - 1]
            for old, new in MUTATIONS:
                if old not in source:
                    continue
                rows[n - 1] = source.replace(old, new, 1)
                pathlib.Path(path).write_text('\n'.join(rows) + '\n')
                if tests_green(test_targets):
                    survivors.append((path, n, source.strip(), rows[n - 1].strip()))
                rows[n - 1] = source
        pathlib.Path(path).write_text(original)
    for path, n, before, after in survivors:
        print(f'UNPINNED {path}:{n}  {before}  ->  {after}')

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2:])

Run it from the PR worktree:

cd ../review-pr-1234
python probe_pr_tests.py origin/main tests/test_billing.py
# UNPINNED src/billing.py:88  if retries <= MAX:   ->  if retries < MAX:
# UNPINNED src/billing.py:91  fee = base + surcharge  ->  fee = base - surcharge

Every printed line is a line the PR's own tests do not constrain. Two outcomes matter. If the mutation changes documented behavior, you found a missing test and a review comment with a reproduction attached. If it only changes an implementation detail nobody promised, note it and move on; mutation survival is not automatically a defect.

Triaging that output is mostly cheap summarisation work: group survivors by file, decide which are behavioral, and turn each into one review comment. This is where MonkeyCode's free model access and free server option are relevant - the free tier is somewhere to run the variant generation and survivor triage without spending metered tokens on a review chore, and the free server option keeps the probe off your laptop while the suite runs. Both availability claims are operator-supplied; check the current terms and limits yourself before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not run this on a shared CI runner against main. Run it on a throwaway worktree, against review-sized test targets, and treat the mutation list as a heuristic, not a statement of intent.

Limitations and who should not use this

• Small or untested repos. If the PR is the only test coverage, revert-first review tells you little. Read the function instead.
• Flaky suites. A flaky test makes a mutant look pinned or unpinned at random. Fix flakes before mutation probes.
• Large monorepos with slow suites. Scoping by changed lines keeps this viable, but one test file that takes four minutes still makes the sweep expensive.
• Behavior that lives outside the diff. Schema changes, config defaults, and feature flags rarely appear as added source lines.
• Teams without a revert budget. The two-minute check assumes you can safely check out a PR branch locally.

None of this replaces reading the diff. It tells you where reading pays off.

Checklist before approval

1. Revert the production change; confirm the new tests fail.
2. Grep the new tests for raises(Exception), sleep(, and is not None.
3. For every mock, ask what breaks if the mock is wrong.
4. Run the diff-scoped mutation probe on the tests the PR added.
5. Split the PR into: revert now, rewrite assertions, trust as regression coverage.
6. Only then read the production diff line by line.

A green suite authored by the same agent that wrote the fix is a hypothesis, not a verdict. Review the failure modes first; the coverage number will still be there when you are done.

If you want a place to run the probe and the triage without metered-token pressure, start on the free tier and keep the merge judgment for yourself.