We planted 23 known bugs in a production-realistic codebase and ran eight AI code review tools against it.
The best tool caught 14 of them. The worst caught 7. The average detection rate across all eight tools was 60.3%. Which means the average AI code review tool missed roughly 40% of real, exploitable, production-impacting bugs that a human reviewer would be expected to catch.
Before the pitchforks come out, this isn't an indictment of AI code review. These tools catch genuine issues, improve code quality, and save reviewer time. But the marketing claims of "catches bugs before they ship" need calibration against what they actually catch and what they miss.
We published this because nobody else is testing these tools on realistic bugs. The existing comparisons test detection of toy examples or run tools against their own cherry-picked test cases. We wanted to know: if you're a senior engineer who would catch a race condition in review, does the AI tool catch it too?
The answer, for 40% of realistic bugs, is no.
The test setup
We created a deliberately buggy version of a mid-sized Python/FastAPI application, approximately 15,000 lines of code across 34 modules. The application handles user authentication, payment processing, order management, and a notification system. It's representative of the kind of B2B SaaS backend that most of these tools are marketed for.
We planted 23 bugs across eight categories. Every bug was a real bug pattern we've encountered in production systems during our code review and technical due diligence work. None were synthetic puzzles. All would have real consequences if deployed.
The categories and planted bugs:
SQL injection vulnerabilities: 3 bugs. One in a search endpoint using string concatenation. One in a reporting query with dynamic column selection. One in an admin tool with parameterised query syntax that was assembled incorrectly.
Authentication and authorisation bypasses: 3 bugs. One IDOR (Insecure Direct Object Reference) where user A could access user B's data by modifying the resource ID. One missing authentication check on an internal API endpoint. One JWT validation that checked expiry but not signature.
Race conditions: 3 bugs. One TOCTOU in payment processing where balance check and deduction weren't atomic. One in a coupon redemption where concurrent requests could redeem beyond the limit. One in a task queue where concurrent workers could pick up the same job.
Memory and resource leaks: 3 bugs. One database connection not returned to pool on exception path. One file handle left open in an error branch. One Redis connection created per request without pooling.
Off-by-one and boundary errors: 3 bugs. One pagination that skipped the last page. One date range that excluded the end date. One array index that would crash on empty input.
Input validation failures: 3 bugs. One path traversal in a file upload endpoint. One XXE vulnerability in XML parsing. One ReDoS-susceptible regex pattern.
Logic errors: 3 bugs. One pricing calculation that applied discount after tax instead of before. One notification that was sent on failed status instead of success. One sorting that reversed the intended order.
Data handling violations: 2 bugs. One endpoint that logged PII in plaintext. One that returned deleted user data in API responses.
The eight tools tested
We ran eight tools that represent the current market spectrum: GitHub Copilot code review, Cursor's built-in review, CodeRabbit, Qodo (formerly CodiumAI), Amazon CodeGuru, Sourcery, DeepSource, and a custom Claude-based review pipeline we built for client work.
Each tool was given the same codebase in the same state. We submitted the buggy code as a PR against the clean baseline and captured every issue each tool flagged.
The detection matrix
What the data reveals
The category breakdown is more interesting than the aggregate numbers.
SQL injection and input validation were the best-detected categories across all tools. Most tools caught the obvious string concatenation injection and the path traversal. These are well-represented in training data and well-documented as vulnerability patterns. If your concern is common OWASP-style vulnerabilities, most AI review tools provide decent coverage.
Race conditions were catastrophically underdetected. Only two tools caught any race conditions at all. The TOCTOU in payment processing, arguably the highest-impact bug in the entire test set, was caught by only two of eight tools. Race conditions require understanding concurrent execution flows, which most AI review tools don't reason about effectively. They analyse the code path sequentially and don't model what happens when two threads execute the same function simultaneously.
This matters because race conditions in payment processing, inventory management, and resource allocation are among the most costly production bugs. They're also among the hardest for human reviewers to catch, which means they're the category where AI assistance would be most valuable and they're the category where AI assistance is currently weakest.
Logic errors were nearly invisible to most tools. The pricing calculation bug, discount applied after tax instead of before, was caught by only three tools. This is because detecting logic errors requires understanding business intent, not just code structure. The code was syntactically correct, followed good patterns, and would pass any static analysis. It just didn't do what the business required. No tool can catch this without understanding what the code is supposed to do, which requires context that most review tools don't have.
Authentication bypasses had moderate detection but the patterns that were caught were the obvious ones. The missing auth check on an endpoint was caught by most tools. The subtle JWT signature validation issue, where the code checked expiry but not signature, was caught by only two. The IDOR was caught by three. The pattern: obvious security omissions get caught, subtle security logic errors get missed.
The false positive problem
Detection rate is only half the story. False positive rate determines whether developers actually use the tool or start ignoring its output.
CodeRabbit had the highest detection rate among commercial tools but also the highest false positive count. Sourcery had the worst ratio, more false positives than true positives. The custom Claude pipeline had the best balance of high detection with low false positives, largely because we tuned the review prompt specifically for this codebase's patterns and explicitly instructed the model to flag only issues with concrete impact.
False positives above 40% are adoption killers. When developers see AI review comments that are wrong or irrelevant more than once in every three comments, they stop reading the comments. The tool becomes noise that the team learns to ignore, which is worse than not having the tool at all because it creates a false sense of security.
What this means for teams using AI code review
AI code review is valuable. It catches real bugs, enforces consistency, and frees human reviewers from mechanical checks. But it is not a replacement for human review on the bug categories that matter most, race conditions, logic errors, subtle security vulnerabilities, and business rule violations.
The effective deployment pattern: AI review handles the first pass. It catches the common vulnerabilities, the style issues, the obvious errors. Human reviewers then focus their attention on the categories AI consistently misses, concurrency, business logic, architectural decisions, and the subtle security patterns that require understanding intent rather than just structure.
Tool selection is only half the battle. The other half is workflow integration, where in your PR cycle does AI review sit, what does it escalate to human reviewers, and how do you prevent the team from treating AI review as sufficient review? The AI code review enterprise guide covers the full workflow design, including the organisational changes that make AI review additive rather than a false safety net.
Published by Dextra Labs, AI Consulting and Enterprise Agent Development
Top comments (0)