AI-generated PRs have a tell: they're too clean. Perfect formatting, thorough comments, confident summaries. And that's exactly why they need a different review: the polish hides the same failure modes every time.
Reviewing AI-generated PRs: a checklist · NotVibesCoded
AI-generated pull requests look confident and complete. That confidence is the hazard. A 10-minute review checklist that catches the specific ways…
notvibescoded.com
The 10-minute checklist
Minutes 0–2: What did it touch that it shouldn't have?
Open the file list first, not the diff. Agents are notorious for scope creep: reformatting untouched files, "fixing" adjacent code, adding dependencies for one helper function. If the PR touches files outside the task, that's your first question, not your first approval.
Minutes 2–4: The tests it wrote for itself
Agents write tests that pass, not tests that protect. Check:
- Do the tests assert behavior or implementation? A test that mirrors the code line-by-line will pass forever and catch nothing.
- Are the edge cases real? Empty inputs, nulls, permission failures, timeouts, or just the happy path three times?
- Would the test fail if the code were wrong? Mutate the logic mentally. If the test still passes, it's decoration.
Minutes 4–6: Error handling and the unhappy path
This is where AI code most consistently underperforms. Look for:
- Bare
except:/ empty catch blocks - Errors swallowed and logged but never surfaced
- Retries without backoff or limits
- User-facing errors that leak stack traces or internals
Ask: "what happens when this fails?" If you can't answer from reading the code, neither can the on-call engineer.
Minutes 6–8: Security basics
Agents don't think adversarially. Scan for:
- SQL/query construction with string interpolation
- Missing authorization checks (authentication ≠ authorization)
- Secrets or tokens in code, logs, or error messages
- Mass assignment / over-permissive input binding
You don't need a full audit, just the OWASP top-10 instincts applied for two minutes.
Minutes 8–10: Would you have written it this way?
Not style, judgment. Did it pick the simple approach or the clever one? Did it add an abstraction for a single use? Did it duplicate something that exists three files over? Agents optimize for completing the task, not for the codebase's future. That's your job, and it's the part that takes the longest to learn and the least time to check once you know the smell.
What to automate
Everything above that's mechanical belongs in CI, not in your 10 minutes:
- Scope check: fail if files outside the ticket's area change without explanation
- Test mutation score on AI-generated PRs (higher bar than human PRs, not lower)
- Secret scanning, obviously
- Lint rules for the error-handling patterns your team keeps seeing
Check the description against the diff
The checklist above reviews the code. Spend one of your ten minutes reviewing the story the PR tells about the code, because AI-written descriptions are fluent, confident, and occasionally fictional.
- "Added tests for X": open the test file. Count the assertions. Agents regularly describe the tests they intended to write.
- "No behavior change" on a refactor: spot-check one call site of anything that moved. Refactors that rename are safe; refactors that rename and adjust a condition are behavior changes wearing a costume.
- "Handles the error case": find the error case in the diff. If the handling is a log line, the description and the code disagree, and the code is the one that ships.
This isn't paranoia; it's calibration. The description is generated by the same process that generated the code, with the same confidence and the same blind spots. Trust it exactly as much as you'd trust a summary written by someone who skimmed.
When the approach itself is wrong
The hardest AI PRs to review are the ones where every line is locally fine and the whole thing is wrong. The agent added a utility that already exists two folders over. It solved the problem in the controller when your architecture solves it in the service layer. It introduced a second state-management pattern next to the one the codebase standardized on.
Line-by-line review waves these through, because each line is defensible. The defense is the file list from minutes 0–2 and one question: "If a new hire had written this, where would I tell them this code belongs?" If the answer is "not here," the review is a redirect, not a line edit. Ask for the approach change early and specifically, naming the existing pattern to follow. Agents take structural feedback well; it's the feedback they almost never get, because reviewers default to commenting on lines.
A related tell lives in the commit history. One giant commit named "implement feature" is normal for agents and useless for review; a sequence of small commits, each doing one thing, is reviewable. You can't rewrite the agent's habits, but you can set the bar in your context file: small commits, messages that say why, no unrelated changes in the same commit. Review quality is partly a function of what the reviewer is handed, and shaping the input is cheaper than straining over a bad one.
The mindset shift
Reviewing human code, you ask "is this correct?" Reviewing AI code, ask "what did it not consider?" The agent did the typing. Your review is the engineering.
This checklist is published on NotVibesCoded, alongside field guides on context files for coding agents, database migrations with AI, and shipping AI-built software without the slop.
Top comments (0)