The 96% accuracy headline and the half it hides
A team at Ericsson, working with Blekinge Institute of Technology, shipped something most AI code review papers don't: a real industrial evaluation. Their multi-agent review solution combines specialized agents across four dimensions, readability, maintainability, reliability, and performance, plus project-specific context, and they ran it on actual code commits at the case company. The abstract reports 96% accuracy on correctly identified issues, with roughly 69% of the correct flags rated important, split into 33% severe-must-fix and 36% important-should-fix. The full writeup is on arXiv.
That is a precision claim, and a reasonably strong one. But the headline number reads better than the method it stands on, and the gap is a property of how most AI code review evaluations get built, not a mistake this one paper made.
Precision is the easy half. Recall is the one nobody measures
Every review evaluation I have seen in this space reports the same shape: of the issues the tool flagged, a high percentage were real. That is precision. The number nobody reports is recall, which is the fraction of everything that was actually wrong that the tool managed to catch.
To measure recall you need a reference set of bugs you already know are there, real or planted, and you need to check whether the tool found them. The Ericsson study does not do that. It generated reviews over several commits, collected more than 200 issues, and had the company's developers validate those issues for correctness and importance. Nothing outside the flagged set was audited, so the design cannot tell you what was missed. If the agent surfaced two genuine issues and missed forty, precision sits at 96% and the review still looks like a win.
This gets more dangerous as agents generate more of the code. When a coding agent writes a large share of a change, the review layer is the last filter, so its misses are what reaches production. A reviewer that is precise but blind to most of what is wrong hands you confidence in the wrong direction. It is the same quieter-versus-correct tension behind the wait-versus-read analysis I wrote for reducing PR review time. Reviewers can be fast and precise without catching the bugs that matter, and the two things feel identical from a dashboard.
There is a real, industry-scale illustration of why recall is the harder problem. This survey of test-driven LLM approaches reviews 87 records and lands on the point that test passing alone does not establish behavioral equivalence, effective feedback, or process adherence. The same logic applies to a reviewer that flags "correct" issues: a flag matching an agreed label tells you nothing about equivalence, about whether unrelated defects were missed, or about whether the flagged item is the one that will bite later. Aggregate accuracy can conceal very different outcomes across tasks and denominators.
Why does the field default to precision? Because recall is expensive. Precision is cheap to measure: collect flags, show them to people, count the ones they agree with. Recall requires a ground-truth set of bugs that predates the tool, which means either injecting defects deliberately, which changes the shape of the code and the review, or mining real historical defects, which is laborious and rarely cuts cleanly across commits. So papers and vendors take the cheap measurement, and the cheap measurement systematically flatters. A tool that flags almost nothing gets a high precision, because the few things it does flag were probably flagged for a reason. The missing-ninety-percent case never shows up.
Who labels "correct," and why that is a judge problem
The second trap in the Ericsson evaluation is the ground truth. The issues were validated by the developers of the case company. A flag the developers agree on counts as accurate. That is consensus, not ground truth, and consensus among people who are guessing about severity has its own failure mode.
I wrote a dev.to post on the mechanics of this in the context of people asking a model to review code it wrote itself: five flakes from the same model are five readings of one opinion, not five independent signals (five AI reviewers, one model's opinion). The correlated-judge problem is not limited to self-review. When the developers who validate the labels are the same developers who wrote or own the code under review, the "accuracy" label is a shared prior scored against itself. A known code smell the team already talks about gets flagged as accurate because it is recognizable. The tool is rewarded for agreeing with the existing mental model, not for finding what the mental model missed.
That is a genuine limitation, and it does not sink the study. The Ericsson result is still useful evidence that a context-aware multi-agent reviewer produces recognizable, defensible findings in the field, which is more than most papers offer. The point is that "recognizable to the people who built it" is not the same as "predicts the defect that ships." The distinction is exactly what a test-driven evaluation-independence argument is after: keep label availability separate from label validity, and treat independent verification as its own property rather than an aggregate that hides it.
A cheap way to actually measure recall
You do not need a formal benchmark to get a first recall number. A small, cheap protocol gets most of the signal. Pick a handful of recent real changes that contain a bug that was found later, a fixed issue that shipped and then got patched. These are the planted bugs, except they are real ones with known fixes.
Run the AI reviewer over that exact pre-fix diff and count how many of the known bugs it flags. Do not show it the fix or the postmortem. Then count. Recall is simply the number of known bugs the reviewer caught divided by the total known bugs you verified. A corpus of fifteen or twenty such changes is enough to see whether the tool is catching a useful fraction or a rounding error.
When you do this, keep three things fixed so the result is reproducible, which matters more than which tool you test. Fix the exact version of the reviewer. Fix the context carry-over, because most review quality lives or dies on whether the tool sees the whole repo or just the diff, and the harness moves scores more than the model does. And fix the judge: for this test, the judge is the list of known bugs you pre-verified, not the tool's own verdicts. Run against a known-defect corpus once before you trust a reviewer on real code.
What to check before you trust an AI review tool
The practical takeaway holds for any vendor demo or industrial writeup in this space. When a tool reports that most of its flags are correct, that is a precision claim, and the easy one. Before acting on it, ask the measurement questions the headline skips.
- Will they even admit recall is a question? A tool that is quiet on how much it misses is either not measuring it or does not want you to look. Ask directly, and do the known-defect run if the answer is vague.
- Who wrote the labels it was scored against? If the ground truth is the same developers whose code was reviewed, you are measuring agreement with a prior, not correctness against reality.
- Does a high accuracy number change your release behavior? A precise-but-blind reviewer waves through. For the code that matters, rely on the layer that was actually verified, and treat a recall run as a regular part of tool selection rather than a one-off.
The 96% number is real and it is a legitimate industrial result. It just covers the easy half of the sentence. The half nobody shouted is the one you need: what did the reviewer look for, and what did it never see. That is the number worth measuring before you let an AI reviewer be the last gate.
Top comments (0)