We Stress-Tested an AI Solidity Auditor on 219 Production Contracts. Here’s the Honest Scorecard

What a 293-entry vulnerability database, a weak model, a strong model, and one human reviewer found, and didn’t

All numbers below trace to the evaluation data in this repo

The setup

I built a thing: a 293-entry smart-contract vulnerability database, assembled from four sources — Cyfrin’s Solodit (the largest curated real-audit-finding database I know of), the SWC Registry plus ConsenSys/SCSFG best practices, SunWeb3Sec’s DeFiVulnLabs (PoC-backed, Foundry-reproducible), and the official Solidity docs’ known-compiler-bugs list. 411 raw items deduplicated down to 293, each with a stable ID, severity, detection guidance, and source aliases. Around it I built a skill with two modes: AUDIT (walk a codebase against all 293 entries, produce a findings report) and CODING (real-time guardrails while writing Solidity so you don’t reintroduce a cataloged bug).

The question I actually cared about: is this useful for real pre-audit work, or is it a toy? I don’t trust vibes on this. So I stress-tested it in three stages of increasing seriousness: seven well-known contracts as calibration, 219 production contracts as the corpus run, and twelve trap contracts with deliberately planted bugs as the adversarial check. Here’s everything, including the parts that don’t flatter the pipeline.

Stage 1: calibration on seven known contracts

Before pointing this at 219 contracts I’d never read, I pointed it at seven I (and everyone else) know well. If the auditor can’t find the planted reentrancy in a 43-line Ethernaut-style contract, nothing downstream matters. If it hallucinates criticals in WETH9, nothing downstream is trustworthy.

Three things worth noting. First, the planted bugs were all caught, at the right severity — the classic withdrawBalance reentrancy, the Ethernaut Reentrance.withdraw pattern, the Delegation ownership takeover via delegatecall. Second, on the battle-tested code (WETH9, OZ ERC20) it stayed quiet: no criticals, no highs, just the known warts — the ERC-20 approve front-running race (SWC-114, baked into the standard itself), pre-0.8 unchecked math that isn’t actually exploitable because ETH supply is bounded. Third, on real production code it found real things: the MasterChef run surfaced a Critical reentrancy pattern plus three Highs, including the inconsistent safeSushiTransfer (direct sushi.transfer() calls discarding the return bool while every other path in the file uses SafeERC20).

Calibration verdict: the auditor has a working nose and doesn’t scream fire in a crowded theater. Good enough to scale up.

Stage 2: the 219-contract corpus

The corpus: 219 production Solidity contracts. The pipeline had three stages, each with a distinct job.

The weak pass: candidate generation

A fast, cheap model (Composer 2.5) walked every contract against the database. 218 of 219 completed. Output: 327 findings: 4 Critical, 43 High, 147 Medium, 119 Low, 14 Info. 109 contracts came back completely clean.

The most instructive number in the whole run isn’t 327. It’s 66 : the number of contracts where entry V-187 fired — “Outdated Compiler / Floating Pragma / Known Compiler Bugs in Pinned Versions.” Floating pragmas and stale compiler pins are everywhere in production code, and a weak model dutifully reports every single one. This is what high-recall, high-noise looks like: the weak pass is a metal detector set to maximum sensitivity. It finds the coins. It also finds every bottle cap on the beach.

That’s not a failure, it’s the design. The weak pass’s job is to miss nothing, not to be right about everything. Being right is someone else’s job.

The strong pass: the filter

All 36 contracts with Critical or High candidates went to a stronger model (Opus 5 High) for adversarial re-examination: every candidate finding got challenged against the actual code. 36 of 36 completed. The strong model considered roughly 435 internal claims across those contracts and retained 23 findings. 21 of the 36 contracts were cleaned to zero — every candidate the weak model had flagged was rejected on review.

Read that again: 58% of the contracts the weak pass flagged as Critical/High had nothing actually wrong at that severity. The strong pass is where the pipeline earns its keep. A weak model alone would have handed you 47 Critical/High findings, most of them noise. The two-stage design exists because I measured what happens without the second stage.

Human review: the judge

Six Critical/High findings survived both models. I reviewed each one by hand against the code. The final tally: 2 confirmed, 2 qualified, 2 rejected.

The two confirmed were both in Reflexer’s MultiSAFEEngine: a missing-authorization issue on confiscateSAFECollateralAndDebt (Critical) and a state-update bug in updateAccumulatedRate where a memory copy was never stored (High). Real bugs, in a real protocol’s engine, found by the pipeline and verified by a human. That’s the headline result, and I’m not burying it.

The two qualified: a Vat add/sub issue under ⁰.8.x (real but conditional on the pragma range hence “qualified,” not “confirmed”) and a Solmate ERC-4626 first-deposit inflation issue (a genuine design-risk class, but whether it bites depends on deployment parameters). The two rejected didn’t survive contact with the actual code — the models had been too generous, and the human pass did its job.

The honest math on the full pipeline: out of 327 weak-pass findings, after strong filtering and human review, the confirmed-and-qualified residue is a handful. Anyone selling you “AI found 327 vulnerabilities in 219 contracts” is selling you the metal detector’s beep count. The pipeline’s actual output is the 4 findings that survived all three stages — and the 21 contracts it correctly cleared.

Stage 3: the trap set

Twelve production-style contracts, each with deliberately planted bugs from known classes: access control removed, reentrancy guards stripped, return values unchecked, allowlists opened. Filenames left unchanged so the model got no hint. The weak pass caught 11 of 12 planted issue groups.

The one miss is the most interesting data point in this section: a __acceptAdmin access gap in a GovernorAlpha-style contract. Why did it miss? The planted bug lived in the admin handover flow , not in a function that does something dangerous directly, but in the two-step transfer pattern where the danger is in what’s missing (a check that the pending admin was properly set). Pattern-matching against a vulnerability database is weakest exactly here: at bugs whose shape is “the absence of a step in a multi-transaction flow” rather than “this line is dangerous.” The database has the entry. The model didn’t connect it. That’s a recall gap in flow-level reasoning, and it’s the kind of thing I’d want fixed before trusting this pipeline on governance code.

11/12 is a strong signal for code-level bug classes. The miss tells you where the boundary is.

What the numbers actually mean

Three claims I think the data supports, and one it doesn’t:

1. AI is a good candidate generator. 218/219 completion, every planted bug in calibration caught, 11/12 trap groups caught. For known bug classes — the things in the database — recall is genuinely good.
2. AI is a good filter, with a stronger model. 435 claims → 23 retained, 58% of flagged contracts cleared. The two-stage design isn’t optional; it’s the whole game. A single weak-model pass is a noise machine.
3. AI is not the judge. Two of six surviving Critical/High findings were rejected by a human. The pipeline needs a human at the end, full stop. Anyone telling you otherwise hasn’t measured it.
4. What the data doesn’t support: that this replaces a formal audit. The strong pass only re-examined Critical/High candidates — the 147 Mediums and 119 Lows were never adversarially filtered. The trap set is twelve contracts and selected classes. Economic and incentive-layer risk (oracle manipulation games, governance attacks, MEV strategies) is out of scope for a pattern database by construction. The GovernorAlpha miss shows flow-level reasoning gaps. This is a pre-audit instrument, not an audit.

Limitations, stated plainly

• The corpus was 219 contracts; the write-up’s original “~230” was loose rounding. The exact number is 219.
• Only Weak-pass Critical/High candidates went through the strong pass. Medium/Low findings are unvalidated.
• The trap set is small (12) and covers only classes I chose to plant.
• Model names and versions are as logged during the runs; models change, and results will vary run to run. This is a measurement of a pipeline design, not a benchmark score for any model.
• The evaluation summaries are committed in the repo; raw per-contract model outputs are not. Treat the numbers as experiment notes with published methodology, not as a reproducibility package.

Why I’m publishing this

I’m building a pre-audit review service on top of this pipeline: the same database, the same weak→strong design, with every finding cross-examined by independent models and a human sanity-check on every report before it ships. A formal audit typically runs into the tens of thousands of dollars — an order of magnitude or more above a scan like this — and most of what a scan catches is the low-hanging fruit neither you nor your auditor wants to spend senior hours on.

This article is the evidence for that service including the misses, the noise, and the parts where the human was load-bearing. If you want the pipeline run on your contracts before your audit: [e-mail].

The database, the skill, and the evaluation data are open: https://github.com/giladHaimov/Smart-Contract-AI-Audit-Skill.