The failure mode of AI code review isn't that it misses bugs. It's that it reports ten "issues" and eight of them are wrong, so you stop reading before you get to the two that matter. A single LLM pass over a diff is fundamentally a confirming process; you ask it "what's wrong with this?" and it pattern-matches against everything that superficially resembles a problem, with no built-in mechanism to check its own claims before handing them to you. The more thorough you ask it to be, the more false positives you get, because "be thorough" and "only report real things" pull in opposite directions for a model that has no adversary pushing back on its first draft.
The setup I use for PR review in my own tooling attacks this directly: instead of one model reviewing a diff, it's two roles with opposed incentives, followed by a third pass that adjudicates between them. A hostile finder whose only job is to break the artifact, a skeptical validator whose only job is to disprove the finder, and a final pass that keeps only what survives both. It's a Find → Validate → Adjudicate pipeline, and the interesting part isn't the LLM prompting; it's the pipeline architecture, and specifically the failure modes that show up once you actually try to build the validator honestly instead of just asking it to "double-check."
Why one model reviewing its own output doesn't work
The obvious first idea is: have the model review the diff, then ask it "are you sure?" This mostly doesn't help, because the model that generated a finding has no incentive structure pushing it to abandon that finding; it wrote it because it looked plausible, and asking the same context "are you sure" tends to get you a confirmation, not a reconsideration. You need something closer to actual adversarial process: a different framing, ideally a role that's rewarded for being right about the opposite thing.
So the design splits review into two subagents with genuinely opposed system prompts, not two calls to the same prompt:
The finder is told, explicitly, at the top of its instructions:
You are a hostile systems engineer. Your job is to BREAK this artifact, not validate it. You succeed by finding real flaws, not by confirming the artifact works.
Do NOT comment on what the artifact does well. Do NOT say "overall this looks good." Every output must be a finding.
The validator, dispatched separately with the finder's raw output, gets the mirror image:
You are a defense attorney for this artifact. For each finding given to you, try to DISPROVE it. You succeed by showing findings are wrong, not by confirming them.
Your default stance is that each finding is a false positive. Only mark
surviveswhen you cannot disprove it after actively trying.
Neither of these is a jailbreak trick or a clever prompt hack; they're just honest role assignment. The finder's only success condition is producing findings; it has zero incentive to soften anything. The validator's only success condition is knocking findings down; it has zero incentive to rubber-stamp. Running them as two separate subagent dispatches (not two turns of one conversation) means the validator never sees the finder's reasoning or any framing that would nudge it toward agreement; it only sees the finding as a bare claim it has to independently stand up or knock down.
The three lenses that keep the finder from just listing everything
A hostile reviewer with no scope constraint will report anything; style nits, hypotheticals, things that are technically true but don't matter. The finder is scoped to exactly three lenses, applied in order:
- Hidden Assumptions - what does this artifact assume that isn't enforced? (type contracts, caller behavior, ordering guarantees)
- Failure Scenarios - how does this actually break? (concurrency, partial failure, timeout, retry storms, data shape variance)
- Blast Radius - if this fails, what else breaks? (downstream consumers, shared state, rollback safety)
And a calibration target: 3-10 findings. Under 3 means the pass wasn't thorough. Over 10 means it's reporting noise. This range matters more than it looks; without an explicit ceiling, an LLM asked to "find issues" will happily generate twenty marginal ones, because more output looks more thorough to the model even though it's strictly worse for the human reading the report. The ceiling forces prioritization at generation time instead of hoping filtering catches it all downstream.
The bug in "just ask it to double-check": unverifiable disproof
Here's the failure mode that's actually worth stopping on, because it's the one that would have quietly broken the whole pipeline's credibility. Early versions of the validator's instructions just said "try to disprove using disproof strategies: check the type system, check surrounding code, check whether it's realistic." That's reasonable; until the validator hits a finding about a cross-system interaction it can't actually verify, and does what any language model asked to be helpful will do: it assumes.
Concretely: a finder reports "this handler doesn't re-check auth before acting on this row." A shallow validator response might be "the upstream service already removes stale rows before this handler runs, so this is fine"; which sounds like a disproof, reads like a disproof, and is actually just a plausible-sounding guess about a system the validator never read. If the validator is scored (implicitly, by the pipeline downstream) on filtering findings, it has every incentive to produce confident-sounding disproofs, and a claim about another service it can't see is exactly the kind of thing that feels verifiable without actually being verified.
The fix is a named discipline in the validator's instructions, explicit enough to close the loophole rather than just discourage it:
The disproof strategies say "read source to verify." When the evidence you would need lives in another system, service, or repo you cannot read from here, you have NOT verified... you have assumed. A disproof that rests on an unverifiable cross-system guarantee ("the upstream service removes the row before this handler runs", "the other repo's types already match", "the gateway authenticates upstream") does NOT count as grounded.
Rule: if you cannot reach the evidence that would settle a finding, keep it
survivesand record the gap inevidence("disproof would require confirming X in other system, unreachable from this artifact"). Reservedisprovedfor findings you ruled out with evidence you actually read.
This is the single most important sentence in the whole pipeline design, in my opinion: absence of evidence you can reach is not evidence of absence, and a validator has to be told that explicitly or it will paper over the gap with a plausible-sounding guess. The rule also caps confidence: >85 on a disproved verdict requires evidence the validator actually read, not an assumption, however reasonable that assumption sounds. Without this rule, a removed auth check justified only by "well presumably the caller already checked this" would get quietly disproved and vanish from the report; which is precisely the case where a false negative is most expensive, because it's a security-relevant finding disappearing on the strength of an assumption nobody actually confirmed.
Adjudication is deterministic code, not a third LLM opinion
The most counter-intuitive part of the design, to me, is that the third phase, deciding what makes the final report, isn't a language model call at all. It's plain filtering logic run in the orchestrating context:
- Drop findings where
verdict = disproved. - Drop findings where
verdict = survivesbutconfidence < 70. - Deduplicate overlapping findings, keeping the higher-confidence version.
- Rank survivors by severity × confidence.
You could imagine a third "judge" agent that reads both the finding and the disproof attempt and makes a final call. That would just be reintroducing the exact problem the first two stages exist to avoid; another LLM call whose judgment you now have to trust and verify, at exactly the point where the pipeline is trying to converge on a trustworthy answer. A deterministic filter over structured output has no persuadability, no mood, and produces the same answer given the same two JSON arrays every time. The LLM's job is generating and disproving claims; the LLM's job is explicitly not deciding which claims to trust; that's a mechanical decision made once the claims already carry a verdict and a confidence score.
There's a verification step built into this phase too, worth calling out because it catches a specific way validators degrade over time: if the filtering step drops zero findings, meaning the validator agreed with literally everything the finder said, the pipeline doesn't just accept that. It re-dispatches the validator once with an explicit instruction to apply the false-positive rules more aggressively. A validator that never disagrees isn't validating; it's rubber-stamping, and a Find→Validate pipeline that lets that pass silently has quietly degraded back into the single-pass "list everything" problem it was built to solve; just with extra latency and an illusion of rigor.
Why the schema is boring on purpose
Both the finder and validator are constrained to return nothing but a JSON array; no prose, no preamble, no "here's my analysis." The finder's schema:
{
"lens": "Hidden Assumptions | Failure Scenarios | Blast Radius",
"location": "path/to/file.ext:LINE",
"claim": "One-sentence statement of the flaw",
"evidence": "the diff snippet or referenced code",
"severity": "High | Medium | Low"
}
The validator's output preserves those fields and adds exactly two:
{
"verdict": "survives | disproved",
"evidence": "Specific file/line or rule citation supporting the verdict",
"confidence": 0
}
And verdict is constrained to exactly two literal strings, "survives" or "disproved", with an explicit warning against synonyms like "valid", "confirmed", or "refuted", because the downstream filtering step does an exact string match on disproved to drop findings. This looks like a pedantic detail, but it's actually load-bearing in a way that's easy to miss: a model that writes "verdict": "false_positive" instead of "disproved" doesn't cause an error; it causes a silent filter miss. The finding just leaks through into the final report as if it had survived scrutiny, with no exception thrown anywhere. If you're building a pipeline where a later deterministic step keys off a model's categorical output, the failure mode to design against isn't "the model refuses" or "the model errors"; it's "the model paraphrases the value you needed to be exact," and the only real defense is naming the exact literal in the prompt and treating any deviation as a contract violation.
What generalizes past code review
The specific domain here is PR review, but the architecture is a general answer to "how do I get an LLM pipeline to stop reporting things that aren't true":
- Split generation and verification into separately-framed roles, not two turns of the same conversation. A model doesn't naturally argue with its own prior output; a differently-incentivized role will.
- Give the generating role a scope (lenses) and a calibration target (a count range), or it will optimize for looking thorough instead of being accurate.
- Name the specific way your verifier will cheat, and forbid it explicitly. "Try to disprove this" sounds sufficient until you watch a validator confidently disprove a claim about a system it never read; the unverifiable-disproof rule only exists because that failure mode actually showed up.
- Keep the final decision in deterministic code, not a third model call. Adjudication should be a fixed function of the structured verdicts, not one more opinion to weigh.
-
Treat categorical model output as a contract, and defend the exact literal, not just the general meaning. If a downstream step does exact-match filtering, a model saying
"false_positive"instead of"disproved"is a silent bug, not a close-enough answer.
None of these five ideas are specific to code review; they apply to any pipeline where you're using an LLM to make a claim and want another pass (LLM or otherwise) to actually be capable of catching it when the claim is wrong.
Top comments (5)
The
confidencefield is a seconddisprovedchannel, and there is no evidence rule sitting on it.You closed the categorical loophole. A validator can no longer write
disprovedon a guess about a system it never read. But step 2 of the adjudicator drops anysurvivesfinding under 70, and nothing in the validator's instructions constrains what it writes into that number. A validator that cannot honestly disprove a finding can mark itsurviveswith confidence 62, and the deterministic filter deletes it one step later, silently, on the strength of exactly the unread assumption the previous section spent its whole length forbidding. Your>85 on disproved requires read evidencecap guards one of the two paths that delete a finding.So the case you say the rule rescues, the removed auth check justified by "presumably the caller already checked this," comes back. It just dies at step 2 instead of step 1.
Also
confidenceis undefined on thesurvivesbranch. Confidence in what? That the finding is real, or that the disproof attempt was thorough? You gave the validator a success condition of knocking findings down. A role scored on disproving reports low numbers about the things it failed to disprove. The incentive you engineered on purpose leaks through the scalar into the filter, where nothing is waiting to catch it.Second thing, and I think it is the bigger one. The unverifiable-disproof rule is one-sided. The validator must keep a finding when the settling evidence is unreachable. The finder is under no symmetric constraint. Its only success condition is producing findings, so the cheapest way to make one stick is to aim it where neither subagent can look: "this handler assumes upstream dedupe," "this depends on the gateway authenticating." Those claims are unfalsifiable from inside the artifact, which means the pipeline is structurally obligated to promote them. Unfalsifiable outranks confirmed.
The fix I would reach for makes reachability a property of the claim itself. Right now it is a property of how hard the validator tried. Your schema's
locationandevidencesay where the flaw lives; nothing says what would refute it. Require the finder to name the artifact whose contents would settle its own claim. A claim that cannot name its refutation condition is a hypothesis. Then unreachable evidence routes to a third verdict,unresolved, instead of inheritingsurvivesby default. Two verdicts force every finding into real or refuted, and there is a genuine third state: not checkable from here. Collapsing it intosurvivesis what makes an ungrounded finding indistinguishable from a confirmed one in the final report, and severity × confidence will cheerfully sort the ungrounded one to the top.Smaller last point. The determinism holds at the filter. The pipeline does not inherit it. Your zero-drop re-dispatch conditions the validator's prompt on the validator's own prior output, and it pushes one way: apply the false-positive rules more aggressively. Same two JSON arrays in, same report out, granted. Which two arrays you get is now a function of the first pass, weighted toward dropping. What re-dispatches a validator that disproved nine of ten? Given a role whose stated default stance is that every finding is a false positive, that looks like the likelier degeneracy.
Thank you for the insightful comment! These are all correct. Let me take them in order.
On confidence as a second deletion channel: You're right, and there's no real defense here. The grounding rule closes the disproved path for unverifiable claims but leaves survives, confidence: 62 fully open. The auth-check example from the article just routes through step 2 instead of step 1. The confidence field also has an undefined semantic on the survives branch; I described it in the context of disproved ("reserve >85 when you actually read the evidence") but never specified what it means when the validator failed to disprove. "How confident am I my disproof is correct" and "how confident am I the finding is real" are different questions, and a role incentivized to knock things down will push that number low on everything it couldn't disprove. That leaks exactly through the filter I described as deterministic.
On asymmetric reachability: This is the structural issue. The grounding discipline constrains the validator's disproof attempts; the finder is under no symmetric constraint, so the cheapest way to produce a surviving finding is to aim it at a cross-system assumption no one can check. Those claims are unfalsifiable from inside the artifact, which means the pipeline is structurally obligated to promote them; and with the confidence scalar uninstructed on survives, they may sort to the top by severity × confidence.
Your fix is the right one. Require the finder to name its refutation condition, what artifact or content would settle this claim, and introduce a third verdict (unresolved) for claims where that evidence is unreachable. That makes "not checkable from here" a first-class state instead of inheriting survives by default, and it applies the reachability constraint to both agents symmetrically rather than just the one doing the checking.
On re-dispatch directionality: Correct. The zero-drop re-dispatch only fires in one direction. A validator that disproves 9 of 10 has no circuit-breaker, and given the role's stated default stance ("every finding is a false positive"), that's probably the more common degeneracy in practice. I don't have a clean fix for this one yet beyond auditing the confidence distribution; a validator over-confident in a large set of disproved verdicts is a signal worth checking before accepting the report.
The unresolved verdict and the confidence-on-survives semantic are the two changes I'd actually make to the pipeline description. The asymmetric reachability issue you named is the one that would have quietly invalidated the "find cases that can't be disproved" use case the whole design is meant to support.
The remaining failure mode is the over-disproving validator, and it stays slippery because the current circuit never tests any verdict against a surface outside the two roles. A confidence histogram can work as an alarm. It is still a distributional smell test. It will tell you a validator behaved unusually across a batch, and it will never tell you that finding 7 was killed by a bad citation or a misread line.
The cleaner escape is the dual of the
unresolvedverdict you just accepted. If a finding whose settling evidence is unreachable cannot be promoted, hold the other role to the same standard: a disproof whose cited evidence is unreachable, or which cites nothing an external checker could re-fetch and re-check, isunresolvedtoo. That is what makes the grounding rule bite. Grounding stops being a prose requirement and becomes a replay contract.Once every disproof carries re-fetchable content, the audit moves from aggregate suspicion to item-level falsification: sample the disproved set, re-run the cited evidence deterministically with no LLM anywhere in that path, and compare what the citation actually contains against the verdict it was used to justify, at which point a wrong disproof stops being a statistical anomaly and shows up as a specific mismatch, the cited config saying the opposite of what was claimed, or the referenced test never exercising the path at all. The circuit-breaker can then fire symmetrically. There is finally an object for it to break on.
On
confidence, defining asurvivessemantic leaves the cheapest lever in place. Delete the field. A scalar can be moved by either role without spending any work, and if ranking consumes it, it is the easiest channel in the system to game. Something re-derivable from the transcript would carry the same weight honestly: how many distinct disproof attempts were actually executed against the finding, and what each attempt cited. Thensurvivesgets ordered by how hard the finding was attacked. That ordering is a fact about the run, recomputable by anyone reading the log, rather than a feeling a role reported about itself.Rough edge worth conceding: the replay contract only reaches evidence the runner can actually fetch. For the cross-system assumptions that started this thread it will keep returning
unresolved, which is the honest answer and probably also an uncomfortable fraction of the report.Agreed on both, and these are sharper than the previous round.
The replay contract is the real fix for grounding; not a tighter prose rule on top of an honor system, but a structural requirement that every disproof carry evidence in a form an external checker can re-fetch and verify independently. A validator that writes "the type system handles this" now has to cite src/types/foo.ts:42 and include the actual content, at which point a wrong disproof stops being a distributional smell and shows up as a specific mismatch: the cited line says the opposite, or the referenced test never exercises the path. That also resolves the asymmetry cleanly, a disproof with unreachable or uncited evidence routes to unresolved for the same reason a finding with unreachable settling evidence does. Not as a prose rule, but as a schema constraint. Grounding is finally bilateral.
On confidence: yeah, delete it. Attempt count and citation list derived from the transcript carry the same information honestly. You get the ordering you actually want, survives findings ranked by how hard they were attacked, without handing either role a scalar it can move by feel. And the over-disproving validator problem becomes detectable at item level: nine single-attempt uncited disproofs look nothing like nine well-cited multi-attempt ones. The circuit-breaker has an object.
The uncomfortable concession on unresolved: cross-system assumptions will stay there, and they'll probably be a real fraction of any interesting report. That's the honest answer and the right tradeoff. A system that forces unfalsifiable findings to wear their status is more useful even when it's less satisfying.
One implementation gap worth naming before someone builds the replay step: "re-run the cited evidence deterministically" means different things depending on what the evidence is. A file path + line range is trivial to re-fetch and diff. "The type system guarantees this" requires actually running the type checker against the cited types. "This test exercises the path" requires running the test and tracing execution. Those aren't the same level of infrastructure, and a real implementation of the replay contract probably needs to separate them; citation re-fetch is cheap and stateless, type/runtime verification is a build step. Both are auditable, but collapsing them into "evidence" hides that difference.
Evidence isn't flat, agreed, and the axis I'd tier it on is replay cost to whoever's checking. A file-and-line citation replays with a read: cheapest, no environment, anyone holding the tree confirms or kills it. A type-checker claim needs the toolchain loaded. A test-execution claim needs a runnable env that's actually deterministic, which is where "evidence" quietly stops meaning one thing.
The part I'd push on is that the tier isn't only infra bookkeeping. It's a second quality signal on the disproof. If a finding could have been settled by a cheap read and the validator instead reaches for "the tests fail," that's a smell: it picked the least independently-checkable evidence it had. So label each disproof with its tier and let cheap-and-still-valid outrank expensive-and-opaque. A disproof that hides in the top tier is doing work nobody downstream can audit.