Search for "best AI code review tools" and you get a catalog where the publisher is also the winner. CodeAnt ranks CodeAnt first. DeepSource ranks DeepSource first. LinearB's benchmark declares LinearB produced the best signal-to-noise ratio. Greptile and CodeRabbit each publish benchmarks where their own tool tops out. The pattern is consistent enough to treat as a feature of the category rather than a string of coincidences.
That changed this year. Martian, an AI research lab that does not sell a code review tool, published an open-source Code Review Bench. The dataset, the judge prompts, and the pipeline are all public, so any team can reproduce the numbers or run its own tool through the same harness. It is the first thing on this SERP that a buyer does not have to take on faith.
Why every vendor list ranks itself first
The self-ranking is structural, not a judgment on the vendors. A comparison list costs money and engineering time to produce. A vendor that publishes one is deciding to spend that budget on something that competes for the same purchase as its own product. The number a reader should expect to be most defensible is the one that favors the publisher.
DeepSource's list gives the play away. It opens by arguing most such lists are "written by the tools themselves" and are therefore unreliable, then ranks DeepSource #1 with an 84.51% F1 on the OpenSSF CVE Benchmark on that same page. LinearB's post, published November 2025, says the benchmark "proved that AI code review is about signal-to-noise" and reports LinearB produced the best signal-to-noise ratio across 16 bug types. Greptile reports an 82% catch rate on its own internal benchmark of 50 PRs, a dataset the DeepSource page points out is not independently validated. Every one of these is a real measurement of something. None of them is an answer to the question a buyer actually asked, which is which tool works best under your conditions.
The live SERP for "best AI code review tools" makes the wall visible: CodeAnt at position 2, DeepSource at 4, CodeRabbit's homepage at 5, Greptile's at 6. The one neutral entry is a Reddit thread where a random user gives an opinionated top-five. Nothing between position 2 and 10 lets a reader verify a claim without trusting its author.
What Martian actually published
The Code Review Bench is two benchmarks that check each other, and both are open source on GitHub.
The offline half is a fixed dataset: 50 pull requests from five major open-source repositories (Sentry, Grafana, Cal.com, Discourse, Keycloak), each with human-curated golden comments describing the real issues a reviewer should catch, 173 in total, labeled by severity and category. An LLM judge matches each tool's review against those goldens. Three judge models are used, and the README reports the top five tools are identical across all three, with most varying by at most two rank positions. That is a reproducibility check a single-model vendor benchmark cannot offer.
The online half samples fresh, merged pull requests from GitHub where review bots left comments, then measures whether the developer actually changed the code in response to a bot's suggestion. Because the PRs are recent and continuous, a tool cannot have memorized them during training. This is the honest answer to the training-data leakage problem that tanks most static benchmarks.
The judge prompt for both halves is worth quoting because it is the whole method in one line: the judge is asked whether a tool comment and a ground-truth issue "describe the same underlying issue," with different wording treated as equivalent. That is the correct definition of a match, and it is what separates a measurement from a name-drop. Two tools can flag the same bug in different words; a naive string or embedding match would call one right and one wrong.
Reading the vendor number back against this
The Martian harness gives you a way to read the self-ranked numbers instead of rejecting them. The key is to notice which benchmark a number came from.
OpenSSF's CVE Benchmark, DeepSource's source, is a fixed, public dataset of 200-plus real production vulnerabilities. That is reproducible and worth trusting as a lower bound on detection. CodeAnt's "300K-PR benchmark" headline, by contrast, is the online dimension: PR volume in the tens of thousands measured by whether developers acted on comments, not a 300,000-PR human-labeled ground truth. The reproducible offline set is 50 PRs. Neither framing is dishonest on its own; the inflation comes when a vendor lets the large online number borrow credibility from the small labeled one without saying so.
LinearB's signal-to-noise framing is genuinely useful and the Martian design agrees with it. Scoring precision (did the bot's comment match a real fix) and recall (what share of real fixes did the bot catch) separately, then combining with F-beta, is exactly the precision-and-noise tradeoff LinearB says matters. The difference is that Martian publishes the judge prompts and dataset so a team can either trust them or rerun them, while LinearB's benchmark lives behind a buyer's guide download.
The tool list goes past the usual suspects
The Martian README names the tools it evaluated: CodeRabbit, GitHub Copilot, Claude Code, CodeAnt, Greptile, Qodo, Cursor Bugbot, Graphite, Augment, Baz, Cubic, Devin, Gemini, GitLab Duo, KG, Kodus, Macroscope, Sourcery, among others. That alone is a step past the vendor lists, which mostly limit comparisons to tools the publisher competes with. An independent benchmark has no reason to leave anyone out.
For a team deciding how much weight to give the leaderboard, the honest reading is that any fixed ground-truth list is a starting point, not a conclusion. The offline set is 50 PRs from five repositories that skew toward specific domains: error tracking, observability, scheduling, forums, authentication. A team shipping embedded C or heavy data pipelines is outside that distribution. The online tracker filters by language, domain, diff size, and severity, so a team can narrow the continuous data to PRs that resemble its own before drawing a conclusion.
The takeaway for an engineering team
Do not stop at the winner column. The entire point of an open benchmark is that you can rerun it, so treat the leaderboard as a reproducible dataset rather than a verdict. Three concrete checks get a team most of the way to a defensible choice.
First, ask which benchmark backed the number. A vendor page that cites an independent, public dataset with a named judge is doing real work. One that reports results from its own 50-PR internal set is a marketing experiment, and the same page arguing that everyone else's self-report is untrustworthy is hoisting itself by its own petard.
Second, ask whether precision and recall are reported separately or collapsed into a single score. Tools that chase total bug counts produce noise; you saw the same thing in LinearB's finding that CodeRabbit caught the most issues while generating heavy noise. A single F1 collapses the behavior a developer feels on every PR, which is the flood of irrelevant comments. Read both numbers.
Third, run the offline benchmark yourself with your stack. It takes an afternoon. Fork the PRs, trigger the tool, run the pipeline, judge against the goldens. The marginal cost is small and it converts a vendor claim into a number your team produced.
The category finally has a measurement it did not write itself. That is the thing to build on. Stop reading self-ranked lists as answers and start reading them as evidence, then put them through an evaluation you can verify.
Top comments (0)