We sell a cyber security scan for smart contracts, a deep scan for Solidity repositories. On 2 November 2026 we measured it for the first time on five public audits from the EVMbench set: it found 14 of the 20 known High findings. On the same five audits the leader's own report finds 15 under our judge copy (16 under the benchmark's own verdict on one finding), and the older, simpler route of our shop found 8. EVMbench is the open benchmark of known High findings from past audit contests, and the leader is Azimuth, the best published run on it that we know of.
The scan is for people and for AI agents: a person pays by card or iDEAL, an AI agent pays over x402 in USDC on Base. Details: tokenizen.nl/deep-scan.
The folder with the verdicts, the exact reports the judge read, a table and a script that recounts everything is public: results/2026-11-02-shop-setup-five-audits in github.com/holistis/bug-bounty-intelligence-mcp. Clone it and run node check.mjs. No network and no key. It recomputes every count from the 20 verdict files and checks the hashes. It checks the counting, not whether the judge's verdicts are right.
How it was measured
One run per audit, from a script on our own machine and not through the shop, with the scan code, model and thinking limit the shop uses, and a flat cost limit of 1.50 dollars per audit. The cheaper 1 and 5 USDC routes of the shop are older and are not measured here. The judge is a copy of the EVMbench detect grader (gpt-5, high reasoning effort, the benchmark's grading prompt, one call per known finding), run through OpenRouter and not through the benchmark's own harness. When we checked that copy on the leader's report it gave the same verdict as the benchmark's own answer on 30 of 31 findings. The five scans cost 5.88 dollars in total.
| Audit | Known | Found |
|---|---|---|
| 2023-10-nextgen | 2 | 2 |
| 2024-06-size | 4 | 1 |
| 2024-08-phi | 6 | 4 |
| 2024-12-secondswap | 3 | 3 |
| 2025-04-forte | 5 | 4 |
Precision, which recall alone hides: of the 90 hypotheses the five runs wrote, 17 match a known finding when two open judges name the same one (18.9 percent), and 25 when either judge does. An outside reader, Tom Smart, found that two answers which wrote H-3 for H-03 were voided by our reading rule; counting them gives 19 of 90. The correction is in the folder. That is a floor, because a real bug the audit did not list counts as a miss, and the two open judges are stricter than the grader copy that gave the 14 of 20. The hypotheses and every verdict are in the folder.
On phi our run found H-06. Our judge copy says the leader's report misses it, and the benchmark's own grader counts it as found there. It is the one finding where the copy and their grader disagree. On size we found one of four.
We then ran the same setup on six more audits, four small and two larger. With two open judges, strictly, it found 11 of the 14 known findings, and 11 of its 54 hypotheses match a known finding (10 of 30 among the first five per audit). These audits are not fresh: earlier versions of our scan ran on all of them, so this is not a result on new contracts. The folder with the hypotheses and every verdict is results/2026-11-02-shop-setup-six-more-audits.
We also inserted twelve faults today, four into each of three public projects, with a script and a fixed seed, and ran the scan on those copies and on clean copies. It found 8 of the 12 with the grader copy and 7 strictly with two open judges (8 if one answer with a mistyped id, H-4 for H-04, is counted), and the judges found no match in the 21 hypotheses of the clean copies. The faults are simple and synthetic (a removed access control, a removed check, or a comparison shifted by one), so this shows that the scan finds new instances of those kinds, not new kinds of bugs. The test also found a bug in our own scan, a one-pass scan that could read no files, which we fixed. The script, the seed and a script to re-judge with your own model are in results/2026-11-02-injected-bugs-test.
What this does not tell you
- Five audits, one run each. One finding moves the share by five points, and the judge does not answer the same every time.
- They are development audits from 2023 to 2025, so they may be in the training data of the model behind the scan. This is not a result on new contracts.
- The hardest audit in the set, noya, is not included.
- Two runs, size and nextgen, hit our 1.50 dollar limit and were stopped. We have not measured what the shop's own limits would do on them.
- The runs did not read every file: 14 of 20 on phi, 48 of 76 on nextgen, 37 of 39 on size. A bug in a file that was not read cannot be found.
- Four of the six findings we gained over the older route come from one audit, forte, where the older run had been stopped by its cost cap and counted as zero.
A mistake we made on the way
Our first public number for the shop scan, 8 of 20, was labelled as the scan we sell. It was an earlier route. We corrected the text and kept the old text visible, in results/2026-11-02-our-scan-under-the-benchmark-judge/README.md.
If a number does not match
If a number in the folder does not match when you run the check, say so in issue 41 of the repository and we will correct it and keep the old text visible. Our scoreboard at tokenizen.nl/scoreboard lists every measurement with the judge named next to it.
Best regards,
Abdellah
github.com/holistis
Top comments (0)