There's a point where running more AI reviewers stops helping and starts quietly lying to you, and I think most teams hit it without noticing. It shows up the moment your review loop routes a surge of AI-generated PRs to several AI reviewers so no single one is the single point of failure. That instinct is right. The math underneath it is not what you think it is.
This post walks the evidence: a 2026 Amazon paper proving that aggregated AI judges are not independent votes, why that matters specifically when the code being reviewed was also AI-written, and a boring, reproducible way to check whether your review budget is actually buying information or just spending tokens.
The paper: correlated judges flip the vote
The anchor is Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models (Balasubramanian, Podkopaev, Kasiviswanathan, January 2026). The setup is label aggregation, which is exactly what a code-review loop does when it asks several reviewers for a verdict and combines them.
Classical aggregation, weighted majority vote and Dawid-Skene, assumes every annotator is conditionally independent given the true label. For LLM judges that assumption is violated, because the reviewers share data, architectures, prompts, and failure modes. Ignore the dependence and you get miscalibrated posteriors and, in the paper's own words, "even confidently incorrect predictions."
The sharpest result is the finite-K example: conditional-independence methods can flip the Bayes label even while matching per-annotator accuracy on the margin. That is not a marginals error. Your N reviewers can each look correct in isolation and still steer the aggregate to the wrong answer, because they are wrong together.
The same paper shows the fix direction: model the dependence explicitly (their Ising formulation makes the Bayes log-odds quadratic in votes when judges share a coupling) and you recover the right posterior. The lesson is not "stop aggregating." It's that aggregation is only worth anything when the inputs carry independent signal.
Why this bites hardest on AI-generated code
Everything above is true for code review in general. It gets worse when the code under review was itself generated by an LLM, for two reasons.
First, the reviewer and the generator often share the same prior. If Claude writes the code and a Claude-family reviewer looks at it, the reviewer is checking the code against the same learned distribution that produced it. Both teams of parameters converge on the same confident wrongness. You are not getting a second opinion; you are getting the same opinion restated. My collaborator wrote this up from the reviewer-evals side as "your AI reviewer is judging its own output", and the Ising paper is the formal skeleton under that intuition: shared prior is exactly the coupling term the independence assumption ignores.
Second, generation is cheap and correlated. When the team faces the volume problem, the temptation is to scale review by adding more AI reviewers, because that is the same cheap unit as generation. But you cannot spend your way out of an independence problem by adding more copies of the dependent signal. Five same-model reviewers catching the same bug is five flags and one piece of information.
The two lanes that actually add information
There are exactly two ways to add genuine independence to a review loop, and both cost more than another API call.
The first is cross-model. A reviewer on a different architecture with different training draws from a different distribution, so inspecting the same patch genuinely adds a second sample. That is real signal, but it is expensive, so you do not spend it on every PR. You spend it where the risk-to-flip-cost ratio is highest: security-sensitive diffs, migrations, anything where a missed critical bug is expensive.
The second is non-model verification, and it is the strongest witness you have because it encodes human intent rather than a learned prior. A test suite the reviewer did not write, a build that actually compiles and runs, static analysis that reasons about dataflow rather than prose rather than just reading the patch. Execution is close to unconditionally independent of any LLM judge, which is why it is the reference signal your human reviewers lean on. The reviewed AI code that gets merged is increasingly whatever passes this lane, not whatever five reviewers agreed on.
A three-lane benchmark before you add headcount
Before you scale AI review headcount, run a boring two-week measurement. Take a fixed PR slice, log which AI review verdicts a human flipped, and bucket the flips by who produced them:
- same model as the code, another review call on the same vendor,
- a different model or vendor on the same patch,
- a test, build, or static check that caught something none of the AI reviewers did.
If the third bucket has a meaningful hit rate, which it usually does, your budget should follow the lanes that find real bugs, not the lanes that feel busy. A process that routes everything through the same model in five wrappers is spending tokens to manufacture the appearance of balance. Evaluating AI review tools on a harness that measures real flipped findings against a human baseline is the only way to see which of your lanes is earning its cost.
The math is on the side of independence
More reviewers is not more verification. The Amazon paper's separation result is the blunt version: even as the number of judges grows, independence-assuming methods keep a nonvanishing excess risk under latent factors. Adding reviewers without breaking the correlation does not converge you to the truth, it converges you to the shared prior, confidently.
So when a team tells me they solved the AI-code volume problem by running three AI reviewers on every PR, my first question is whether any of the three draws from a different distribution, and my second is how many flips each of them produced against a human baseline. If the answers are "all the same model" and "we don't track flips," the review loop is reporting activity, not information.
Top comments (1)
Correlated judges is the right frame, and the case you name is worse than the paper's setup. When the code under review was also model-written, the reviewers share not just a prior but a prior that already matches the artifact. You are not sampling five opinions, you are sampling one opinion five times against something drawn from the same distribution.
The practical consequence is that adding reviewers raises confidence without raising information, which is the most expensive failure in review because it feels like rigour.
Any cheap independent signal buys more than a sixth reviewer: a test that actually runs, a different model family, or one human reading the diff.