I run a review pass on top of AI-generated findings — code review comments, "this is a bug" claims, that kind of thing — where a second pass of agents tries to knock each finding down before it's shown to me. The idea is simple: spawn a few independent skeptics, each told to argue the finding is wrong, and only keep findings that survive a majority vote. It catches the plausible-but-wrong stuff that a single confident agent produces constantly — the finding that reads perfectly, cites real line numbers, and describes a bug that doesn't actually happen under any input.
It worked, for a while. Going from zero verification to three-vote majority verification killed a huge chunk of false positives — the kind where the "bug" is a scenario that can't actually occur, or a misread of what the code does. That was the easy win. Then I tried to push further: five verifiers, seven verifiers, assuming more votes would keep buying more precision. It didn't. Past three-ish, adding another identical verifier barely moved the false-positive rate at all, it just cost more tokens per finding.
Why more of the same vote doesn't help
The verifiers weren't independent in any way that mattered. They were the same prompt — "try to refute this claim, default to refuted if uncertain" — run three, five, seven times. Same framing, same blind spots, same tendency to accept a finding that sounds rigorous because it uses the right vocabulary (specific line numbers, a named function, a plausible-sounding failure mode) even when the underlying claim is wrong. If verifier #1 gets fooled by a finding that name-drops the right symbols, verifiers #2 through #7 get fooled by the exact same thing, for the exact same reason. You're not sampling independent opinions. You're re-rolling the same biased die and calling the agreement "confidence."
That's the actual failure mode adversarial verification runs into: it catches errors the verifier prompt is capable of noticing, and it's blind to whatever that prompt is structurally bad at noticing, no matter how many times you ask it. Majority vote among correlated failures doesn't average out — it just confirms the shared blind spot with more apparent authority. A finding that fools one refuter-shaped prompt has a good chance of fooling every refuter-shaped prompt.
What actually moved the number
Diversity of vantage point, not count. Instead of N identical "try to refute this" agents, I split verification across a small number of genuinely different lenses — each one told to evaluate the same finding from a different angle:
Lens: correctness — does this claim hold given the actual code path, ignoring style?
Lens: reproducibility — can you construct concrete inputs that trigger the described
failure? If you can't build a repro, say so explicitly.
Lens: security/impact — even if technically true, does this matter? What's the
actual blast radius if left unfixed?
Three verifiers, three different failure modes to catch, instead of three chances for the same failure mode to slip through. A finding that survives "does this reproduce concretely" catches a different class of false positive than one that survives "is this actually a correctness issue" — a plausible-sounding bug with no constructible repro dies at the reproducibility lens even if the correctness lens (reading the code in isolation, the way the original finder did) would have waved it through.
The mechanical change is small — same number of agent calls, same voting structure — but the framing is different: I stopped asking "how many times can I ask basically the same question" and started asking "how many structurally different ways can this claim fail, and do I have a verifier assigned to each one." For code-review findings specifically, correctness / reproducibility / actual-impact turned out to be enough to catch most of what identical-prompt voting was missing, without a bigger verifier fleet.
The generalizable part
This isn't specific to code review. Any pipeline that adds a "second pass to catch the first pass's mistakes" runs into the same ceiling if the second pass is structurally identical to itself, just repeated. Three identical judges is one judge with extra confidence, not three opinions. If you're seeing a verification step plateau — precision stops climbing no matter how many more votes you throw at it — the fix usually isn't more votes. It's asking what specific way the underlying claim could be wrong that your current verifier prompt structurally can't see, writing a verifier whose whole job is to look for exactly that, and only then deciding whether you actually need more voters or just needed different ones.
Top comments (0)