AI Code Review Fails on False Positives, Not Missed Bugs — Here's the Fix

The first AI code reviewer most teams build gets abandoned within a month. Not because it misses bugs — because it finds too many things that aren't bugs, and reviewers stop reading its comments. Once a human learns that half the findings are noise, they skim past all of them, including the real ones. That's worse than not running review at all: it trains the team to ignore a channel that occasionally carries something true.

The instinct is to fix this by tightening the prompt — "only report high-confidence issues," "be conservative," "don't nitpick." It helps a little and plateaus fast, because the underlying problem isn't confidence calibration in a single pass. It's that a single pass has no way to check its own work.

The fix: make the agent argue with itself

Split review into two passes instead of one, run by two independent agent calls that don't share context:

1. Find. One agent reads the diff and reports every candidate issue it can — correctness bugs, security issues, missing tests. Cast a wide net on purpose; recall matters more than precision here.
2. Verify. For each candidate, a second agent gets only the finding and the relevant code — not the first agent's reasoning — and tries to construct a concrete input or scenario where the claimed bug actually fires. If it can't, the finding gets dropped.

Only findings that survive verification post as comments. Everything else disappears silently — the human reviewer never sees the discarded 60-70% that a single-pass agent would have surfaced with equal confidence.

This works for a structural reason, not a prompting trick: the finder and the verifier are optimizing for different things. A finder that's told to catch bugs will pattern-match on things that look like bugs — a missing null check, an unusual control flow — because that's what "catching bugs" rewards. A verifier whose only job is "prove this specific claim or kill it" has no incentive to hedge. Forcing it to name a concrete failing input is what actually filters — "this could theoretically be a problem" doesn't survive that bar, but "call this with an empty array and it throws" does.

What changes if you skip step 2

Teams that ship single-pass review usually notice the same three symptoms, in order:

• Week 1: reviewers are impressed, the agent catches a real bug on day two.
• Week 2: reviewers start seeing findings that are technically true but don't matter (a style-adjacent "issue" in dead code, a "missing" test for a function that's trivially covered elsewhere).
• Week 3+: PR comments from the bot get a reflexive resolve-without-reading. The one genuine catch in twenty gets lost with the rest.

The verify pass doesn't just reduce volume — it changes what kind of thing survives. A finder-only agent's output is ranked by how plausible each issue sounds. A verified agent's output is ranked by whether each issue is demonstrated. Reviewers can tell the difference within a few PRs, and that's what determines whether they keep reading.

Making it concrete

If you're building this yourself:

• Give the finder agent a narrow contract: what it hunts (correctness, security, missing test coverage) and what it explicitly ignores (style, formatting, naming preference). A finder without ignore rules will happily report a hundred nits alongside the two real bugs.
• Pass the verifier only the finding and the code it references — not the finder's chain of reasoning. If the verifier sees the finder's justification, it tends to agree with it rather than independently checking it.
• Require the verifier to produce a concrete failure scenario, not a confidence score. "I'm 80% sure this is a bug" is not verification; "these inputs cause this crash" is.
• Log the confirmed rate over time (findings that survive verify ÷ total findings). If it creeps toward 100%, your finder has probably started under-reporting to please the verifier — widen its net back out.

None of this requires a specific tool. It's a control-flow pattern: two calls, one gate, log the drop rate. The version worth building is the one where a reviewer trusts every comment enough to act on it without re-deriving the bug themselves — that's the only number that actually matters.

Full playbook, including how to wire this into a PR pipeline and what to track once it's running: https://agentkitworks.com/use-cases/automate-code-review