I benchmarked AI code-review skills on 18 seeded-bug PRs. Detection was a tie. Trust wasn't.

César Pereiro — Tue, 20 Oct 2026 11:41:29 +0000

I build correctness-critical software for a living (hospital integrations: HL7, DICOM, FHIR, under EU medical-device rules). When I started using AI agents to review code, the question I cared about was not "can it find bugs?" but "can I act on what it says without re-checking everything myself?"

So I built a small benchmark and measured it.

The setup

18 pull requests, each a real git repository with a main and a feature branch:

14 with seeded production bugs of the kind that sail through a green test suite: an idempotency check-then-act race that double-charges a card, an invoice download with no ownership check, a page count that drops the last partial page, forEach(async ...) that never awaits, an except Exception: return 0 that silently zeroes bank amounts (plus the test that was weakened to hide it), SQL built with an f-string, a naive vs aware datetime comparison that crashes every password reset, a "simplification" that breaks a caller outside the diff, a cross-tenant cache leak, an open redirect through an unanchored regex, a rounding split that no longer sums to the total, a connection leak on the error path, an inclusive date range filtered with <, and override || default treating 0 as unset.
4 clean PRs, two of them deliberately tempting to flag (retrying a POST that receivers deduplicate; a shared index in a concurrency helper that is actually safe).
Every bug was confirmed at runtime. Every feature branch passes its own tests.
8 of the cases were written by a separate agent that never saw the review skill I was testing, so I could not teach to the test.

Two reviewers, same model, same instructions ("review this PR"), each on a fresh copy of every repo:

A prompt-only reviewer: the most-installed code-review skill on the marketplace I sell on. A solid checklist: bugs, security, performance, maintainability, severity buckets.
Proven Code Review, the skill I wrote. It maps every caller of the changed code (including files the PR never touched), tries to refute each suspicion before reporting it, and must prove every Critical and High finding with a failing test or command it actually runs.

A third agent judged every pair of reviews blind (anonymised as X and Y, randomised per case) against a written answer key.

Result 1: detection is a tie

Both reviewers found 15/15 seeded bugs, on Claude Opus and on Claude Haiku, and both gave 18/18 correct verdicts.

I did not expect that. Current models find this class of bug no matter which skill wraps them. If someone tells you their review prompt "finds more bugs", ask for the benchmark.

Result 2: trust is not a tie

	Opus: prompt-only	Opus: proven	Haiku: prompt-only	Haiku: proven
Real findings backed by a run	43%	78% (19/19 Critical+High)	14%	76%
Precision (real / reported)	98%	100%	74%	94%
Nits to read past	18	9	25	5
False positives	1	0	0	1

On the fast, cheap model the gap is about 5x on both noise and proof. That is the setting most people actually run reviews in, at scale, in CI.

What "proof" looks like in practice: on the open-redirect PR, the review did not say "the regex might be too permissive". It wrote a test, ran it, and pasted four URLs that land the user on evil.test after login, plus a second bypass (https://app.example.com//evil.test) that was not even in my answer key.

Where my skill was wrong

On Haiku it reported one false positive (it called deliberate, already-tested cent rounding a Critical under-billing bug) and inflated one severity. Version 1.1 adds two rules: behaviour that a test explicitly asserts is intended unless the PR or spec says otherwise, and when torn between two severities, pick the lower one.

Limits

18 cases is small. They are short, mostly single-module PRs in Python and JavaScript. The judge is a model scoring against an answer key. Everything is public so you can check or extend it: cases, answer keys, the raw 72 reviews, the blind scores and the scripts are in the repo.

Try it

Benchmark and raw data: https://github.com/Ces107/agent-forge/tree/main/bench/code-review
The skill is free. In Claude Code: /plugin install agent-forge --marketplace Ces107/agent-forge. Any SKILL.md agent: npx skills add Ces107/agent-forge. Also on Agensi: https://agensi.io/skills/proven-code-review-every-serious-bug-comes-with-a-failing-test

The thing I would take away even if you never use my skill: stop grading review tools on what they find and start grading them on what you can act on without re-checking. Proof is cheap now. Ask for it.