I Killed My AI Code Reviewer's False Positives by Making It Argue With Itself

The failure mode of AI code review isn't that it misses bugs. It's that it reports ten "issues" and eight of them are wrong, so you stop reading before you get to the two that matter. A single LLM pass over a diff is fundamentally a confirming process; you ask it "what's wrong with this?" and it pattern-matches against everything that superficially resembles a problem, with no built-in mechanism to check its own claims before handing them to you. The more thorough you ask it to be, the more false positives you get, because "be thorough" and "only report real things" pull in opposite directions for a model that has no adversary pushing back on its first draft.

The setup I use for PR review in my own tooling attacks this directly: instead of one model reviewing a diff, it's two roles with opposed incentives, followed by a third pass that adjudicates between them. A hostile finder whose only job is to break the artifact, a skeptical validator whose only job is to disprove the finder, and a final pass that keeps only what survives both. It's a Find → Validate → Adjudicate pipeline, and the interesting part isn't the LLM prompting; it's the pipeline architecture, and specifically the failure modes that show up once you actually try to build the validator honestly instead of just asking it to "double-check."

Why one model reviewing its own output doesn't work

The obvious first idea is: have the model review the diff, then ask it "are you sure?" This mostly doesn't help, because the model that generated a finding has no incentive structure pushing it to abandon that finding; it wrote it because it looked plausible, and asking the same context "are you sure" tends to get you a confirmation, not a reconsideration. You need something closer to actual adversarial process: a different framing, ideally a role that's rewarded for being right about the opposite thing.

So the design splits review into two subagents with genuinely opposed system prompts, not two calls to the same prompt:

The finder is told, explicitly, at the top of its instructions:

You are a hostile systems engineer. Your job is to BREAK this artifact, not validate it. You succeed by finding real flaws, not by confirming the artifact works.

Do NOT comment on what the artifact does well. Do NOT say "overall this looks good." Every output must be a finding.

The validator, dispatched separately with the finder's raw output, gets the mirror image:

You are a defense attorney for this artifact. For each finding given to you, try to DISPROVE it. You succeed by showing findings are wrong, not by confirming them.

Your default stance is that each finding is a false positive. Only mark survives when you cannot disprove it after actively trying.

Neither of these is a jailbreak trick or a clever prompt hack; they're just honest role assignment. The finder's only success condition is producing findings; it has zero incentive to soften anything. The validator's only success condition is knocking findings down; it has zero incentive to rubber-stamp. Running them as two separate subagent dispatches (not two turns of one conversation) means the validator never sees the finder's reasoning or any framing that would nudge it toward agreement; it only sees the finding as a bare claim it has to independently stand up or knock down.

The three lenses that keep the finder from just listing everything

A hostile reviewer with no scope constraint will report anything; style nits, hypotheticals, things that are technically true but don't matter. The finder is scoped to exactly three lenses, applied in order:

1. Hidden Assumptions - what does this artifact assume that isn't enforced? (type contracts, caller behavior, ordering guarantees)
2. Failure Scenarios - how does this actually break? (concurrency, partial failure, timeout, retry storms, data shape variance)
3. Blast Radius - if this fails, what else breaks? (downstream consumers, shared state, rollback safety)

And a calibration target: 3-10 findings. Under 3 means the pass wasn't thorough. Over 10 means it's reporting noise. This range matters more than it looks; without an explicit ceiling, an LLM asked to "find issues" will happily generate twenty marginal ones, because more output looks more thorough to the model even though it's strictly worse for the human reading the report. The ceiling forces prioritization at generation time instead of hoping filtering catches it all downstream.

The bug in "just ask it to double-check": unverifiable disproof

Here's the failure mode that's actually worth stopping on, because it's the one that would have quietly broken the whole pipeline's credibility. Early versions of the validator's instructions just said "try to disprove using disproof strategies: check the type system, check surrounding code, check whether it's realistic." That's reasonable; until the validator hits a finding about a cross-system interaction it can't actually verify, and does what any language model asked to be helpful will do: it assumes.

Concretely: a finder reports "this handler doesn't re-check auth before acting on this row." A shallow validator response might be "the upstream service already removes stale rows before this handler runs, so this is fine"; which sounds like a disproof, reads like a disproof, and is actually just a plausible-sounding guess about a system the validator never read. If the validator is scored (implicitly, by the pipeline downstream) on filtering findings, it has every incentive to produce confident-sounding disproofs, and a claim about another service it can't see is exactly the kind of thing that feels verifiable without actually being verified.

The fix is a named discipline in the validator's instructions, explicit enough to close the loophole rather than just discourage it:

The disproof strategies say "read source to verify." When the evidence you would need lives in another system, service, or repo you cannot read from here, you have NOT verified... you have assumed. A disproof that rests on an unverifiable cross-system guarantee ("the upstream service removes the row before this handler runs", "the other repo's types already match", "the gateway authenticates upstream") does NOT count as grounded.

Rule: if you cannot reach the evidence that would settle a finding, keep it survives and record the gap in evidence ("disproof would require confirming X in other system, unreachable from this artifact"). Reserve disproved for findings you ruled out with evidence you actually read.

This is the single most important sentence in the whole pipeline design, in my opinion: absence of evidence you can reach is not evidence of absence, and a validator has to be told that explicitly or it will paper over the gap with a plausible-sounding guess. The rule also caps confidence: >85 on a disproved verdict requires evidence the validator actually read, not an assumption, however reasonable that assumption sounds. Without this rule, a removed auth check justified only by "well presumably the caller already checked this" would get quietly disproved and vanish from the report; which is precisely the case where a false negative is most expensive, because it's a security-relevant finding disappearing on the strength of an assumption nobody actually confirmed.

Adjudication is deterministic code, not a third LLM opinion

The most counter-intuitive part of the design, to me, is that the third phase, deciding what makes the final report, isn't a language model call at all. It's plain filtering logic run in the orchestrating context:

1. Drop findings where verdict = disproved.
2. Drop findings where verdict = survives but confidence < 70.
3. Deduplicate overlapping findings, keeping the higher-confidence version.
4. Rank survivors by severity × confidence.

You could imagine a third "judge" agent that reads both the finding and the disproof attempt and makes a final call. That would just be reintroducing the exact problem the first two stages exist to avoid; another LLM call whose judgment you now have to trust and verify, at exactly the point where the pipeline is trying to converge on a trustworthy answer. A deterministic filter over structured output has no persuadability, no mood, and produces the same answer given the same two JSON arrays every time. The LLM's job is generating and disproving claims; the LLM's job is explicitly not deciding which claims to trust; that's a mechanical decision made once the claims already carry a verdict and a confidence score.

There's a verification step built into this phase too, worth calling out because it catches a specific way validators degrade over time: if the filtering step drops zero findings, meaning the validator agreed with literally everything the finder said, the pipeline doesn't just accept that. It re-dispatches the validator once with an explicit instruction to apply the false-positive rules more aggressively. A validator that never disagrees isn't validating; it's rubber-stamping, and a Find→Validate pipeline that lets that pass silently has quietly degraded back into the single-pass "list everything" problem it was built to solve; just with extra latency and an illusion of rigor.

Why the schema is boring on purpose

Both the finder and validator are constrained to return nothing but a JSON array; no prose, no preamble, no "here's my analysis." The finder's schema:

{
  "lens": "Hidden Assumptions | Failure Scenarios | Blast Radius",
  "location": "path/to/file.ext:LINE",
  "claim": "One-sentence statement of the flaw",
  "evidence": "the diff snippet or referenced code",
  "severity": "High | Medium | Low"
}

The validator's output preserves those fields and adds exactly two:

{
  "verdict": "survives | disproved",
  "evidence": "Specific file/line or rule citation supporting the verdict",
  "confidence": 0
}

And verdict is constrained to exactly two literal strings, "survives" or "disproved", with an explicit warning against synonyms like "valid", "confirmed", or "refuted", because the downstream filtering step does an exact string match on disproved to drop findings. This looks like a pedantic detail, but it's actually load-bearing in a way that's easy to miss: a model that writes "verdict": "false_positive" instead of "disproved" doesn't cause an error; it causes a silent filter miss. The finding just leaks through into the final report as if it had survived scrutiny, with no exception thrown anywhere. If you're building a pipeline where a later deterministic step keys off a model's categorical output, the failure mode to design against isn't "the model refuses" or "the model errors"; it's "the model paraphrases the value you needed to be exact," and the only real defense is naming the exact literal in the prompt and treating any deviation as a contract violation.

What generalizes past code review

The specific domain here is PR review, but the architecture is a general answer to "how do I get an LLM pipeline to stop reporting things that aren't true":

• Split generation and verification into separately-framed roles, not two turns of the same conversation. A model doesn't naturally argue with its own prior output; a differently-incentivized role will.
• Give the generating role a scope (lenses) and a calibration target (a count range), or it will optimize for looking thorough instead of being accurate.
• Name the specific way your verifier will cheat, and forbid it explicitly. "Try to disprove this" sounds sufficient until you watch a validator confidently disprove a claim about a system it never read; the unverifiable-disproof rule only exists because that failure mode actually showed up.
• Keep the final decision in deterministic code, not a third model call. Adjudication should be a fixed function of the structured verdicts, not one more opinion to weigh.
• Treat categorical model output as a contract, and defend the exact literal, not just the general meaning. If a downstream step does exact-match filtering, a model saying "false_positive" instead of "disproved" is a silent bug, not a close-enough answer.

None of these five ideas are specific to code review; they apply to any pipeline where you're using an LLM to make a claim and want another pass (LLM or otherwise) to actually be capable of catching it when the claim is wrong.