I build correctness-critical software for a living (hospital integrations: HL7, DICOM, FHIR, under EU medical-device rules). When I started using AI agents to review code, the question I cared about was not "can it find bugs?" but "can I act on what it says without re-checking everything myself?"
So I built a small benchmark and measured it.
The setup
18 pull requests, each a real git repository with a main and a feature branch:
-
14 with seeded production bugs of the kind that sail through a green test suite: an idempotency check-then-act race that double-charges a card, an invoice download with no ownership check, a page count that drops the last partial page,
forEach(async ...)that never awaits, anexcept Exception: return 0that silently zeroes bank amounts (plus the test that was weakened to hide it), SQL built with an f-string, a naive vs aware datetime comparison that crashes every password reset, a "simplification" that breaks a caller outside the diff, a cross-tenant cache leak, an open redirect through an unanchored regex, a rounding split that no longer sums to the total, a connection leak on the error path, an inclusive date range filtered with<, andoverride || defaulttreating0as unset. - 4 clean PRs, two of them deliberately tempting to flag (retrying a POST that receivers deduplicate; a shared index in a concurrency helper that is actually safe).
- Every bug was confirmed at runtime. Every feature branch passes its own tests.
- 8 of the cases were written by a separate agent that never saw the review skill I was testing, so I could not teach to the test.
Two reviewers, same model, same instructions ("review this PR"), each on a fresh copy of every repo:
- A prompt-only reviewer: the most-installed code-review skill on the marketplace I sell on. A solid checklist: bugs, security, performance, maintainability, severity buckets.
- Proven Code Review, the skill I wrote. It maps every caller of the changed code (including files the PR never touched), tries to refute each suspicion before reporting it, and must prove every Critical and High finding with a failing test or command it actually runs.
A third agent judged every pair of reviews blind (anonymised as X and Y, randomised per case) against a written answer key.
Result 1: detection is a tie
Both reviewers found 15/15 seeded bugs, on Claude Opus and on Claude Haiku, and both gave 18/18 correct verdicts.
I did not expect that. Current models find this class of bug no matter which skill wraps them. If someone tells you their review prompt "finds more bugs", ask for the benchmark.
Result 2: trust is not a tie
| Opus: prompt-only | Opus: proven | Haiku: prompt-only | Haiku: proven | |
|---|---|---|---|---|
| Real findings backed by a run | 43% | 78% (19/19 Critical+High) | 14% | 76% |
| Precision (real / reported) | 98% | 100% | 74% | 94% |
| Nits to read past | 18 | 9 | 25 | 5 |
| False positives | 1 | 0 | 0 | 1 |
On the fast, cheap model the gap is about 5x on both noise and proof. That is the setting most people actually run reviews in, at scale, in CI.
What "proof" looks like in practice: on the open-redirect PR, the review did not say "the regex might be too permissive". It wrote a test, ran it, and pasted four URLs that land the user on evil.test after login, plus a second bypass (https://app.example.com//evil.test) that was not even in my answer key.
Where my skill was wrong
On Haiku it reported one false positive (it called deliberate, already-tested cent rounding a Critical under-billing bug) and inflated one severity. Version 1.1 adds two rules: behaviour that a test explicitly asserts is intended unless the PR or spec says otherwise, and when torn between two severities, pick the lower one.
Limits
18 cases is small. They are short, mostly single-module PRs in Python and JavaScript. The judge is a model scoring against an answer key. Everything is public so you can check or extend it: cases, answer keys, the raw 72 reviews, the blind scores and the scripts are in the repo.
Try it
- Benchmark and raw data: https://github.com/Ces107/agent-forge/tree/main/bench/code-review
- The skill is free. In Claude Code:
/plugin install agent-forge --marketplace Ces107/agent-forge. Any SKILL.md agent:npx skills add Ces107/agent-forge. Also on Agensi: https://agensi.io/skills/proven-code-review-every-serious-bug-comes-with-a-failing-test
The thing I would take away even if you never use my skill: stop grading review tools on what they find and start grading them on what you can act on without re-checking. Proof is cheap now. Ask for it.
Top comments (4)
Two findings here are bigger than the benchmark itself.
"Detection is a tie" is the most important sentence in the post. The model finds the bugs; the skill decides how much of your attention they cost. People buy the wrong axis when they shop for "better" review prompts.
The "refute before reporting" step is the real mechanism. A finding that needs verification costs minutes; a false positive costs you the next review too, because you stop reading carefully. On the cheap model that noise gap (25 nits to 5) is what decides whether the review runs in CI or gets turned off in a month.
One caution from the same data: the "prove it with a failing test" bar is exactly right for this bug set, but it silently filters findings that cannot be proven cheaply — races you cannot reproduce in the sandbox, auth gaps in code you cannot run against real data. The 0 false positives may partly be the proof bar choosing what gets reported. Still the right tradeoff; just know the blind spot.
You're right about the blind spot for hard-to-repro bugs like race conditions or auth gaps, but I'd happily trade that coverage if it means developers actually read and trust 100% of the findings that do make it through.
This matches what I've seen building an AI PR reviewer on the side: finding the bug is the easy part, and false confidence is the expensive part. One noisy 'possible race condition' with no repro and people stop reading the rest. I ended up adding a plain non-AI pre-check and a human verify step before anything gets surfaced. Love that you included the tempting-but-clean PRs. That's the test most benchmarks skip.
yes, humans make errors, AI make errors. Learning from mistakes makes us add better defense layers. It reminds me of the 'Swiss Cheese model' in aviation: no single check (or AI model) is perfect, but stacking deterministic pre-checks, AI analysis, and human verification aligns the slices so critical errors don't slip through 😀😀