"Can I trust Copilot review or CodeRabbit to actually catch bugs, or am I just getting a rubber stamp?" is one of the questions we keep seeing asked, on Hacker News and Stack Overflow both. We went looking for a clean, independent answer. We didn't find one — we found competing vendor-adjacent benchmarks that disagree with each other, which turns out to be the more useful finding.
The numbers depend on who's publishing them
One independent 2026 benchmark run across 309 pull requests put GitHub Copilot's review bot at 54% bug detection, CodeRabbit at 44%, and Greptile (not one of the 11 tools we track, but relevant here) well ahead at 82%. CodeRabbit's own blog, meanwhile, claims it tops a different benchmark entirely. Both can be technically true — the discrepancy comes down to each using its own evaluation harness and PR corpus, and the source of the first number is itself a competing code-review vendor's resource page. Neither is a number we'd repeat as settled fact.
One data point we found corroborated across independent searches, not tied to a single vendor's messaging, is more interesting than the headline detection rate: on the OpenSSF CVE Benchmark (200+ real, validated vulnerabilities in open-source JS/TS projects), CodeRabbit showed very high precision but caught only a fraction of the actual vulnerabilities present — it stays quiet unless it's confident, rather than flagging borderline cases. That's a specific, describable failure mode: not "wrong," but "silent" on exactly the cases you'd most want flagged.
The pattern matches something we already wrote about
We covered this same shape of problem yesterday: Wiz's report on a Snowflake GitHub Actions vulnerability, where GitHub Copilot Autofix reviewed the merged pull request containing the flaw and marked it clean — not because it wrote the vulnerable code (an earlier headline claim GitHub disputed and Wiz itself corrected), but because it missed a real, exploitable issue during review. A clean pass read as "nothing obvious," and a real bug shipped anyway.
That's the same "high precision, low recall" pattern the OpenSSF numbers describe in the abstract, showing up as an actual incident.
What this means practically
- A green check from an AI review bot is a first pass, not an audit. Treat it the way you'd treat a linter that only flags what it's sure about — useful, but silence isn't proof of absence.
- The loudest complaint in developer threads isn't missed bugs, it's noise: nitpicks flagged as "major," PRs cluttered with comments developers admit to dismissing without reading closely. A bot that's too noisy trains you to stop reading it, which is its own failure mode.
- If you're reviewing across multiple worktrees or parallel AI-agent branches, the practical fix isn't a better bot — it's reviewing one diff fully before moving to the next, rather than skimming several bot-approved PRs in a row.
- For anything security-sensitive, a bot's "clean" result is not a substitute for a human reading the diff, especially around auth, input handling, or anything touching CI/CD scripts (the exact category the Snowflake incident fell into).
Two things worth flagging while we're here
Our own best-for/code-review page currently lists Amazon Q Developer at "$19/user/mo" without noting that AWS closed Q Developer to new individual signups on 2026-05-15 (existing subscribers are supported through 2027-04-30; Kiro is AWS's named replacement). It also lists Cody as "Free" — Sourcegraph discontinued Cody's individual Free/Pro tiers back in mid-2025 and it's enterprise-only now, roughly $59/user/month and up on a contract, not a self-serve free tier. Both are corrections to file against the source page, not against anything in this post.
Disclosure: we have no confirmed affiliate or commission relationship with GitHub, Sourcegraph/Cody, or Amazon Q.
Full picks and pricing for code review specifically: https://devtoolsreview.com/best-for/best-ai-coding-tools-code-review/
Top comments (0)