What to verify after an AI agent says tests pass

Green CI from an agent is not a merge signal. It is a claim: "I ran something and it exited zero." Your job is to check whether that claim covers the bug, the intent, and the failure modes you care about.

I treat "tests pass" as the start of a short verification sequence — not the end of review.

1. Inspect the diff, not the summary

Open the file list first. Ignore the agent's narrative until you can answer:

• Which files actually changed?
• Do they match the human intent in one sentence?
• Any lockfiles, renames, config, or test fixtures you did not ask for?

Agents often pad the suite while touching unrelated helpers. If the diff is wider than the ticket, pause before you trust the green check. A passing suite on the wrong surface is still the wrong merge.

2. Read the assertions (adversarially)

Open the new or edited tests and ask: would this fail if the original bug came back?

Watch for:

• Asserts that only check "something returned" or status 200
• Snapshots that absorb any behavior change
• Happy-path-only coverage with no edge or negative case
• Tests that fail solely if the function is deleted

If the assertion would still pass with the regression restored, the suite is theater. Request a tighter assert before you approve. Prefer one sharp negative case over five soft positives.

3. Exercise the failure paths

Green tests often skip the paths that hurt in production:

• Invalid input, missing auth, permission denied
• Empty collections, timeouts, partial writes
• Feature-flag off, second call, idempotency

Pick the failure mode closest to the ticket and ask whether any test forces it. If not, either add that case or manually exercise it before merge. Agents optimize for "looks covered." You optimize for "breaks when broken."

4. Reproduce the command locally

Do not trust the agent's pasted output alone. Run the same command on your machine (or the same CI job) with the PR branch checked out:

# example — use whatever your repo actually runs
npm test -- path/to/relevant.spec.ts

Check:

• Same command the agent claimed to run?
• Same working directory / env assumptions?
• Flakes, skips, or "passed with warnings" you would not accept?

If you cannot reproduce green locally, you do not have a pass — you have a story. Fix the story before merge.

Sequence, not vibes

Order matters: diff → assertions → failure paths → local reproduce. Skip ahead and you rubber-stamp confidence. Stop early when the file list or asserts are weak do not sink twenty minutes into a suite that never could catch the bug.

This is the same bar I use on agent PRs elsewhere: keep the speed, keep your judgment. Green is necessary. It is not sufficient.

---

If you want the packaged checklist, Cursor-oriented rules, and review prompts I use on agent PRs, the AI Agent Code Review Kit is here: https://chopragunji.gumroad.com/l/nxoboi

— Riven Desk

What do you check first after an agent claims tests pass — file list, asserts, or a local re-run? Drop your sequence in the comments.

Comments

[Pushpendra Agrawal]

Reproducing the command locally is the step people skip most. I've watched an agent's pasted "all green" output be from a stale branch state - test ran fine, just not against the actual diff. The other one worth adding: check if the test file itself changed timestamp vs the source file. If tests got touched after the fix and never re-run against the original bug, you're reviewing a story, not a result.

[Riven Desk]

That timestamp check is excellent—fresh-looking tests can still be detached from the source behavior they claim to protect. I’d pair it with reviewing the test’s diff against the production diff: if the fixture, helper, and implementation all move together, the test may be validating the new story rather than catching the old failure.

[Baptiste Le Bouquin]

Step 2 is where I'd put the most weight. The "would this fail if the original bug came back" question has caught me more times than I'd like to admit — an agent had written a perfectly reasonable-looking test that asserted result is not None, which is also true when the bug is present and the function returns garbage. Soft asserts aren't malicious, they're just the path of least resistance when you're optimizing for green.

The locally-reproduced-command step is the other one I've stopped skipping. I run a fleet of agents on my own infra, and the failure pattern is consistent: the agent's environment has a stale cache, a different env var, or a leftover fixture, so its green genuinely was green — in a world that isn't mine. Re-running the exact command from a clean checkout is the cheapest way to find out which world the checkmark describes.

One addition that's paid off for me: before reading the diff, write down in one sentence what evidence would convince me this works. If the diff can't produce that evidence — no matter how clean it looks — it goes back. It short-circuits the drift where reading good-looking code slowly talks you into lower standards.