A pull request touched retry logic in a checkout service. The AI review came back clean-ish: a naming suggestion, a request to add docstrings, and one line saying the retry "should probably check for duplicates."
That line is the whole problem. It reads like a finding. It isn't one, it's a hedge wearing the format of a finding, sitting at the same visual weight as a docstring suggestion.
I started running every AI review output, mine and my team's, through five questions before trusting it. Not because AI reviews are bad at spotting patterns, but because they're very good at producing something that looks like a review while quietly failing to check the thing that mattered.
What's actually at risk in this change?
A generic review prompt treats every changed line as equally worth a comment. It has no way to know that one function touches the payment path and another is a logging tweak, so attention gets spread evenly across both. You end up with a review that reads thorough and misses the one thing that mattered, because comment count isn't risk coverage.
The fix isn't a smarter model, it's giving the model risk context before the review starts: which parts of this codebase are business-critical, which have caused incidents before. Asking it to infer risk from the diff alone doesn't work.
Would this pass or fail under the failure mode it's meant to prevent?
"Looks good, handles errors appropriately" isn't verification, it's a vibe. It doesn't say what failure was checked. The useful version forces specificity: does this retry logic behave correctly under a duplicate request? Does this auth check hold if the token is expired but not yet revoked?
"Review this for bugs" invites a general assessment. "Would this retry logic double-charge a user under a duplicate request with the same idempotency key?" invites an actual check. The difference in output quality between those two prompts is not subtle.
Is this finding a real risk, or a style preference wearing risk's clothes?
Undifferentiated output puts "this could cause a production incident" next to "consider renaming this variable" at the same font size. Nothing forces a distinction, so blocking issues and nits compete for the same five seconds of reviewer attention.
This is fixable structurally, not by asking the model to "be more careful": require every finding to carry a severity tag plus a one-line justification for that tag. A finding that can't justify its severity in a sentence usually doesn't deserve a high one.
What did this review not check, and why?
Almost no AI review states its own scope. It reviews the diff and goes quiet about everything outside it the caller of the function, the config flag that determines whether this path even runs, test coverage for the new branch. Silence reads as "nothing else to worry about," which is a much bigger claim than "I didn't check."
A review that says "this covers the diff only; it did not check the calling context in the payment service" gives you something honest to act on instead of false confidence.
If this recommendation is wrong, what would that look like?
This is the one that catches hallucinated fixes before they ship. A confidently wrong suggestion and a correct one sound identical the only way to tell them apart is to ask what evidence would prove the suggestion wrong, then go check it.
In practice, that means a second pass: feed the first review's findings back and ask specifically what might be a false positive and why.
Re-running it
Applying all five to that checkout PR changes the outcome completely. With risk context supplied, a specific failure scenario stated (duplicate request, same idempotency key), severity tags with justification, an explicit scope statement, and a validation pass on the top finding the duplicate-charge risk becomes the one clearly flagged blocking issue, and everything else drops below it.
Diff alone
↓
Undifferentiated comments
↓
Real risk buried, scope unclear, confidence unverified
vs.
Diff + risk context + specific failure framing
↓
Severity-tagged findings with stated scope
↓
Second-pass confidence check
↓
One clear blocking issue, ranked correctly
Where this doesn't pay off
Running all five questions on a one-line config change is more overhead than the change deserves. The value scales with what's actually at stake, and even a review that answers all five well is still probabilistic reasoning, not proof. It should raise the floor of what a human catches, not replace judgment on genuinely ambiguous logic.
I ended up building this into an actual workflow instead of asking the five questions by hand every time, risk classification, context collection, failure-mode framing, severity scoring, scope declaration, validation pass, all templated per repo. I put that structure together as a documented system, https://medium.com/@nagatomopedro05/five-questions-your-code-review-should-always-answer-66be919bb200 for anyone who wants the reusable version instead of rebuilding it PR by PR.
What's the failure mode you've actually seen slip past an AI review, not a hypothetical, something that shipped? I'm curious whether it traces back to missing risk context, or to a confidently-wrong suggestion nobody double-checked.
Top comments (0)