DEV Community

César Pereiro
César Pereiro

Posted on

I benchmarked AI code-review skills on 18 seeded-bug PRs. Detection was a tie. Trust wasn't.

#ai #codereview #testing #claude

I build correctness-critical software for a living (hospital integrations: HL7, DICOM, FHIR, under EU medical-device rules). When I started using AI agents to review code, the question I cared about was not "can it find bugs?" but "can I act on what it says without re-checking everything myself?"

So I built a small benchmark and measured it.

The setup

18 pull requests, each a real git repository with a main and a feature branch:

  • 14 with seeded production bugs of the kind that sail through a green test suite: an idempotency check-then-act race that double-charges a card, an invoice download with no ownership check, a page count that drops the last partial page, forEach(async ...) that never awaits, an except Exception: return 0 that silently zeroes bank amounts (plus the test that was weakened to hide it), SQL built with an f-string, a naive vs aware datetime comparison that crashes every password reset, a "simplification" that breaks a caller outside the diff, a cross-tenant cache leak, an open redirect through an unanchored regex, a rounding split that no longer sums to the total, a connection leak on the error path, an inclusive date range filtered with <, and override || default treating 0 as unset.
  • 4 clean PRs, two of them deliberately tempting to flag (retrying a POST that receivers deduplicate; a shared index in a concurrency helper that is actually safe).
  • Every bug was confirmed at runtime. Every feature branch passes its own tests.
  • 8 of the cases were written by a separate agent that never saw the review skill I was testing, so I could not teach to the test.

Two reviewers, same model, same instructions ("review this PR"), each on a fresh copy of every repo:

  1. A prompt-only reviewer: the most-installed code-review skill on the marketplace I sell on. A solid checklist: bugs, security, performance, maintainability, severity buckets.
  2. Proven Code Review, the skill I wrote. It maps every caller of the changed code (including files the PR never touched), tries to refute each suspicion before reporting it, and must prove every Critical and High finding with a failing test or command it actually runs.

A third agent judged every pair of reviews blind (anonymised as X and Y, randomised per case) against a written answer key.

Result 1: detection is a tie

Both reviewers found 15/15 seeded bugs, on Claude Opus and on Claude Haiku, and both gave 18/18 correct verdicts.

I did not expect that. Current models find this class of bug no matter which skill wraps them. If someone tells you their review prompt "finds more bugs", ask for the benchmark.

Result 2: trust is not a tie

Same bugs found, far less noise, far more proof

Opus: prompt-only Opus: proven Haiku: prompt-only Haiku: proven
Real findings backed by a run 43% 78% (19/19 Critical+High) 14% 76%
Precision (real / reported) 98% 100% 74% 94%
Nits to read past 18 9 25 5
False positives 1 0 0 1

On the fast, cheap model the gap is about 5x on both noise and proof. That is the setting most people actually run reviews in, at scale, in CI.

What "proof" looks like in practice: on the open-redirect PR, the review did not say "the regex might be too permissive". It wrote a test, ran it, and pasted four URLs that land the user on evil.test after login, plus a second bypass (https://app.example.com//evil.test) that was not even in my answer key.

Where my skill was wrong

On Haiku it reported one false positive (it called deliberate, already-tested cent rounding a Critical under-billing bug) and inflated one severity. Version 1.1 adds two rules: behaviour that a test explicitly asserts is intended unless the PR or spec says otherwise, and when torn between two severities, pick the lower one.

Limits

18 cases is small. They are short, mostly single-module PRs in Python and JavaScript. The judge is a model scoring against an answer key. Everything is public so you can check or extend it: cases, answer keys, the raw 72 reviews, the blind scores and the scripts are in the repo.

Try it

The thing I would take away even if you never use my skill: stop grading review tools on what they find and start grading them on what you can act on without re-checking. Proof is cheap now. Ask for it.

Top comments (0)