A week ago I wrote that an AI reviewer reporting 96% precision had a measurement nobody ran: recall. The reaction was mostly "sure, someone picked a flattering metric." I want to make a wider claim this time. The precision-only score is a symptom of a structural problem in how everybody evals code review tools, and it has almost nothing to do with the models and everything to do with what the judge costs.
The artifact that made this click is jevals, a Show HN that landed over the weekend. It replaces the LLM-as-judge with typed, calibrated yes/no decision models. The README runs the comparison: on identical traces, LangChain measured GPT and Claude judges with 92x to 913x the score variance of the Jev decision model. A judge that changes its verdict between runs of the same input is not a test suite, it is noise. And the fix is not a smarter frontier model, it is a judge cheap enough to run on every trace.
The judge is the expensive part
Nobody decides to subsample out of laziness. The judge is the cost center of the whole pipeline, and the metric libraries most people pull from make it obvious.
Take Ragas, where most teams got their metric vocabulary. Faithfulness is two LLM calls per sample. Answer relevancy is three calls plus an embedding. Context precision is one call per retrieved chunk. After you add few-shot examples, JSON parsing, and retries when the JSON comes back malformed, a four-metric run on a single sample works out to six to eleven round trips. Each one is frontier-model tokens, generated token by token, and seconds of latency.
At that price you sample 1% of traffic, run it nightly, and the numbers never get near the request path. You cannot run this gate on a live PR because the reply would arrive after lunch. So your offline evaluation covers a corner of the dataset while your production behavior is a different thing measured by a different, or absent, process.
jevals argues this trade is not fundamental. If the judge's decisions are typed questions, yes/no, pick-one, rubric, you do not need a paragraph of reasoning plus a JSON blob to get the single label you actually keep. Jev packs every eval for a trace into one request: a few thousandths of a cent, p50 latency around 244ms through the Vercel gateway. The README shows one trace, eight evals, one request, 1388 tokens, six hundredths of a cent, 0.33 seconds.
That changes what you can promise. At that marginal cost you eval 100% of traces, and you can run the same gated check inside the agent loop against the same question definitions you used offline. One eval class, three uses: offline metric, production monitor, in-loop gate. The gate enforces exactly what you measured. That is the gap between "our tool scored well on a held-out sample" and "every merged change was cross-checked by the same rubric we validated offline."
Correlated judges and the precision trap
This connects straight back to my recall post. The reason a tool quotes precision and not recall is usually not deception. Recall is hard to measure because it needs a labeled ground truth of issues that exist, including ones nobody flagged. Precision is easy because you only look at what the tool flagged. An expensive judge makes the hard measurement harder, so the cheap metric wins.
The other trap is the correlated judge problem. When the model that produced a suggestion is also the judge scoring it, you have one model's opinion measured N times, not an independent verdict. Five reviewers built on the same family and judged by that same family will agree with each other and all be wrong together. That is why a recall measure or any cross-lineage check matters more than the count of reviewers.
Cheap, typed, calibrated judges matter because they are a different kind of signal from the model being evaluated. A decision model returning a calibrated probability for "is this claim supported by the evidence" is not the reviewer patting itself on the back, it is a separate measurement. That separation is the property worth caring about, more than any single score improving.
Here is what I want teams evaluating reviewers to actually change. When a vendor quotes a headline accuracy, ask how many traces that number came from and whether the judge was the vendor's own model. When you pick between reviewer tools, look at whether the harness scores with a judge independent of the thing being reviewed, and whether the reported metric can even detect a missed bug. A reviewer that only reports precision is telling you the recall side was not measured, and the cost of an LLM judge is a good default explanation for why. Kodus, CodeRabbit, and the rest put out accuracy numbers the same way, and the same scrutiny applies. If the metric came from the vendor's own model judging its own output on a subsample, treat it as one model's opinion, not a verified rate.
Independent verification is not consensus
The correlated-judge problem has a subtle version that keeps slipping past teams. Averaging several LLM judges is not a second opinion. If they share a training lineage, you have measured one belief N times and the average does not make it more true. This is the failure mode in the Amazon dependence-aware label aggregation work on correlated LLM judges. The fix is not more judges from the same family. It is either a non-LLM decision layer, which is what jevals does, or labeled synthetic ground truth where the answer is known in advance instead of argued about.
Everyone has gotten good at building agents that call tools and write code. The weak layer is the one that decides whether what an agent did was correct. Keep paying frontier-model prices per verdict and you keep subsampling, and you keep letting the model grade its own homework when you sample. A cheap judge decouples those two and makes recall-style metrics a routine output instead of a luxury.
I would love to see any reviewer tool run the easy measurement: a fixed, labeled PR slice with known defects, scored by a judge from a different lineage than the reviewer, with the recall number published beside the precision number. That piece of data is worth more than a dozen scorecards. If the judge is cheap and deterministic, running it on the whole slice is no longer a cost problem, it is just a decision. The tools that ship that measurement are the ones I will trust to review my PRs. The rest are a model grading its own homework on a sample it got to pick.
Top comments (0)