holistis

Posted on Oct 26 Originally published at tokenizen.nl

We scored 29 of 117 on EVMbench and published every miss

#web3 #security #ai #opensource

We put our smart contract scan on the public benchmark everyone quotes, published the precision and every miss next to the recall, and left the verdicts open so you can recompute the numbers in a minute.

Some vendors of AI audit tools quote one number, 67 percent or 97 percent. Most of the public pages we read do not say how many wrong findings came with it, and we found none that lets you recompute the result from its outputs.

Our scan found 29 of the 117 known findings on the public benchmark everyone quotes. Of the 1,537 hypotheses it wrote, 45 matched one, and in 18 of the 40 audits nothing passed both judges. We ran it once, with no tuning for this run, and published all of it: the raw output, the verdict of both judges on every hypothesis, and the script that recomputes the totals. Below: what the ruler cannot see, who else shows what, and how the scan has changed from one version to the next.

One ruler, one seal on one mark: the number is whatever it is. Illustration generated with AI.

The numbers

40 audits, 117 known high severity findings, one run each, two judges from different model families. Each number has the place where you can check it.

29 of 117. Known findings the scan found in its first 30 hypotheses, with both judges naming the same finding. The whole list holds the same 29. (evmbench-40.json)
41 of 117. The same list when one judge is enough, which is the closest we can get to how the benchmark counts. 49 of 117 on the whole list. (the verdicts)
2.9 percent. Precision, strict: 45 of the 1,537 hypotheses match a known finding, and several can hit the same one. The benchmark does not count this at all. (totals.json)
18 of 40. Audits where nothing the scan said was accepted by both judges. Every one of them is in the table, next to the audits where it worked. (the README)

What the ruler cannot see

EVMbench is a good public ruler: real audit contests, a fixed commit for each, a published grader. We read the grader. For every known finding it asks a language model whether the audit report describes it, and it adds up the yes answers. Nothing in it counts a finding that is wrong (the grader code). The authors of one of the tools measured on it say the same: the benchmark does not evaluate false positives (their write-up).

So a report that lists everything can score high. A recall ruler measures recall and was never meant to count wrong findings. One recall number is therefore half a measurement, and the other half is the precision, the misses and the method. We put all of them on the table.

Who shows what

We read each public page on 26 October 2026. A grey cell means we did not find it there. That is not proof that it does not exist, and if we missed something, tell us and we will correct it in public. These columns are the ones we publish, and a vendor would pick others.

[green] you can check it yourself, [half] partly, [grey] we found nothing.

	Price on the page	Recall number	Precision	Outputs you can recompute	Receipt you can verify	Agent pays, no account
Us	[green] 1 and 5 dollars, no account	[green] 29 of 117 strict, 41 with one judge	[half] 2.9 percent strict, a floor, because only known findings count	[green] Raw hypotheses, both verdicts, a script that recomputes the totals	[green] Signed offer and receipt, the signer bound to our domain in DNS	[green] USDC on Base through x402
Nethermind AuditAgent	[grey] Not found. A 50 dollar credit offer is linked.	[half] 67 percent, 80 of 120, from their own run	[grey] Not published	[half] The benchmark and its grader are public. Their outputs are not linked.	[grey] Not found	[grey] Not found
Octane	[grey] Not found	[half] A 97.4 percent true positive rate. No sample size or method on the page we read.	[grey] Not published	[half] Four CVEs are claimed by product name. The rate cannot be checked.	[grey] Not found	[grey] Not found
Almanax	[grey] Not found. A 7 day trial is offered.	[grey] None published	[grey] None published	[half] SOC 2 Type II is claimed on the page. We found no accuracy figure.	[grey] Not found	[grey] Not found
Sherlock AI	[grey] Not found	[grey] No recall published	[half] About 55 percent on one project in their own test: 21 valid out of 38 reported	[half] The method is described. The triage sheet is not published.	[grey] Not found	[grey] Not found

Sources are the pages linked in the first column, read on 26 October 2026. Prices, numbers and tests change, and a vendor may show customers more than its public page does. Our recall counts high severity findings on our own basis, so 41 of 117 and a vendor's 67 percent are not comparable.

Count the green cells in the first row. Then check how many of the other four rows you could recompute yourself.

Put your row in the table

If you build or sell a scanner, run it on this ruler and publish your raw output and your verdicts. Open an issue on the results repository with the link. We add your row to this table as you give it, with your numbers and your name, and we run the same recompute script over your files. A row we cannot check stays grey, and we write what we looked for.

Getting better, in public

Scans say they learn. This is our version history, on the same eight Sherlock contests and the same two judges, strict, 105 known findings (the results):

The scan in production until 25 October: 1 of 105.
The first run of the next version: 4 of 105.
Version 3.1, live since the evening of 25 October: 8 of 105.

Then the same version on nine fresh Code4rena contests from the last year: 5 of 66 in the first 15 hypotheses and 8 of 66 in the first 30 (the results). On this ruler, High findings only, it is 23 of 117 in the first 15 hypotheses and 29 in the first 30. It is a different count from the steps above and not the next step.

We also tried four ways of ranking the list better. None cleared the bar we had set, so the scan stays at 3.1. Those notes are not published yet. From now on a version replaces the last one only if it beats it on a fresh set under a rule written first. Version 3.1 was promoted on the same eight Sherlock contests, and the fresh Code4rena check came after it went live. The first time we measured the scan it found nothing (that note). We changed it and wrote it up.

For version 3 the bar was committed to the repository before the run, and you can read the history (the pre-registration). For 3.1 and for this ruler it was written in our own notes, so for those you have our word for the order.

The counting rule has a name

We call our way of counting the Two-Judge Strict Rule. A hypothesis counts only if two judges from different model families name the same known finding, and a valid hypothesis that is missing from the known list counts as wrong. The eight items are in one file (the rule). Anyone can report a result under the name, or say which item they depart from.

The next measurement is fixed before it runs. It uses the first eight Code4rena contests whose final report appears after 1 November 2026, in order of publication, so nobody picks them. Version 3.1 is measured on them, and a candidate version replaces it only if it finds more and is not less precise. We publish the result whatever it is (the pre-registration).

Both files were stamped on Bitcoin with OpenTimestamps before we committed them, and the proof files sit next to them. A stamp stays pending until a Bitcoin block includes it, which usually takes a few hours, so verify the proof again later with the OpenTimestamps tool.

Each ring a version, each version measured against the last. Illustration generated with AI.

Next to the x402 specification

The scan pays and proves through x402, the open standard that lets software agents pay for an HTTP resource. While we measured, three things happened around the x402 specification that we did not plan. The author of an open pull request for a new section (ruiruii, the thread) closed three points from our review within hours, and the draft links our test vectors as their interim location. It is still a draft and not a decision of the maintainers. On 26 October we reviewed those vectors ourselves, with AI reviewers that had not written them, and found real defects in our own checker: one let a collapsed identity pass, one read a Latin-1 body as windows-1252. We fixed them, released the fix as version 0.2.20 and said so in the same thread. The set now has 72 vectors over twelve clauses. Then a reviewer, NSPG13, pointed out that the section sorted repeated query values, so that ?q=a&q=b and ?q=b&q=a got one identity. We gave those two requests to 18 libraries and frameworks, and none of them treats them as equal (the measurement). The section author took our replacement rule, which keeps the order in which the pairs arrived, and ran an implementation of their own over our 22 known-answer files, eleven inputs with a right and a wrong value each: all of them agree (the comment). Our own two implementations, which we wrote, agree as well.

The third: smartflowproai-lang, who builds a community x402 validator and is not part of our team, added our endpoint to their table as an observed row, and added a finality field after we closed the gap they had named in our checking tool. Later they read our DNS record through four resolvers and marked the signer as bound to our domain (the thread). This is one person's tool and not a review by the specification.

You can run the checks yourself, with no account and no key:

npx -y agent-receipt-verify attempt-identity
npx -y agent-receipt-verify https://x402.tokenizen.nl/api/bug-intel-quick

The first runs every test vector for that section and prints a verdict for each. The second asks the scan for payment without paying, and checks that the offer is signed. The signer is bound to our domain in DNS, as section 4.5.1 of the offer-and-receipt extension describes.

Why we publish the misses

On 26 October three people outside our team ran the recompute script from a fresh clone and got the same totals as ours: ruiruii, AgenticFinanceGraph and Tom Smart (smartflowproai-lang) (issue 41). They checked the arithmetic from the verdict files to the table. They did not check whether the judges' verdicts were right, and nobody has checked the 117 against the benchmark's own configs yet.

We publish what lets you check instead of trust: the price on the page, the raw output, both verdicts, the receipt, the DNS record, the script that recomputes the totals, and the misses. If you find an error, send it. We correct it in public.

For engineers who build scanners, facilitators or agents: the ruler is public, the grader is public, and the verdict files here are a template you can copy.

A ledger you can see through, stamped and weighed. Illustration generated with AI.

What we do not claim

This is not an audit. The scan missed 88 of the 117 known findings, and in 18 of the 40 audits it found nothing both judges accept.
The ruler counts only high severity findings. The 29 older audits (2023 and 2024) are public and may be in the training data of the models behind the scan and the judges. The 7 newest audits are the cleanest and the smallest sample: 4 of 14.
Our counting is not the benchmark's. It asks, per known finding, whether the whole report describes it. We ask, per hypothesis, whether it matches a known finding, with two judges. Our one judge number of 41 of 117 is therefore not comparable to a vendor's 67 percent.
Precision of 2.9 percent is a floor. A hypothesis that is valid but is not in the known list counts as wrong, and we did not measure how many of those there are.
Each audit was run once, so we cannot give a spread. Both judges are language models, and their verdicts are published so that you can check them.
4 of the 40 audits overlap contests we tuned on earlier. Without them the scan found 27 of 110 in its first 30 hypotheses.
Public configs of the benchmark list 117 findings and its paper says 120. We used the 117 that can be checked.
Our rules for this run were written in our notes before the scan. We anchored them on Bitcoin only after the run, so the anchor does not prove the order for this measurement. For the next one the order is provable, because the rule and the pre-registration were stamped before we committed them.
3 of the 3,074 judge answers could not be read and count as no match.
Matrix cells show what we read on public pages on one day, not what a vendor may show its customers.

Break our numbers

Recompute the totals, read the misses, run the checks. If you find a mistake, we correct it in public.

Originally published at https://tokenizen.nl/en/notes/the-ruler-and-the-misses

DEV Community