DEV Community: holistis

What we learned building two fail-closed verification tools for the agent economy

holistis — Mon, 09 Nov 2026 08:55:00 +0000

ERC-8004 gives AI agents an on-chain identity, but identity is not trust. A few recent posts here made that point well, including one showing that a naive average of reputation scores can crown the wrong agent as "best." That matches something we ran into ourselves, from a different angle, while building two small open-source tools.

We did not set out to build a trust layer for agents. We built two narrow verification tools, for two unrelated problems, and only noticed afterward that they shared the same design rule: never say "clean" when you mean "I didn't actually check."

Tool one: checking a sanctions-list claim

Al-Muqarin compares a claim ("this company is not on an EU/UK/OFAC/US sanctions list") against signed, dated snapshots of the public lists. The one design choice we would point to: if a list is missing, stale, or below a minimum version number, the answer is never "clean." It is ONMEETBAAR, not measured. A tool that quietly treats "I could not check" the same as "I checked and it is fine" is worse than no tool at all, because it looks like a green light.

We are not claiming this is unique. OpenSanctions and a hosted tool called agentbroker both already sign results in the sanctions-screening space, and OpenSanctions covers far more data than we do. What we did not find elsewhere is a signed snapshot of the list itself, with a version floor that refuses a rolled-back list. That is a narrow claim, not a big one.

Tool two: checking a payment receipt

agent-receipt-verify checks x402/EIP-712 payment receipts: did an agent actually pay what it claims to have paid, on the rails it claims to have used. Same rule applies: a receipt that does not fully check out is reported as such, not rounded up to "looks fine."

The part that might be useful even without our code

We trust both tools today because we tested them against our own planted mistakes, over and over. We plant deliberate bugs in the code (a missing check, a flipped comparison, a dropped field) and require every single one to make a test fail. Al-Muqarin currently catches 548 out of 548 planted faults this way.

That discipline is free to copy, with or without our tools. If you are building anything that outputs "verified" or "trusted," try deleting one check at a time and see if your test suite actually notices.

Where we are honestly stuck

Neither tool solves agent trust. Al-Muqarin has zero paying users right now; we are treating that as the real test, not the code. A separate prototype of ours, a fail-closed checker for independently-verified impact claims (did an agent's donation actually do what it claimed), has a gap we have not closed: the "who verified this" field is still free text. Nothing stops someone from writing a controller description that is not obviously "us" and getting a false pass. We know this, we have a test that proves it, and we have not fixed it yet.

If any of this overlaps with what you are building on top of ERC-8004 or x402, we would genuinely like to hear where it breaks.

More from us: github.com/holistis and tokenizen.nl.

Our cyber security scan for smart contracts found 14 of 20 known bugs. Recount it yourself, and see where it falls short

holistis — Mon, 02 Nov 2026 18:47:37 +0000

We sell a cyber security scan for smart contracts, a deep scan for Solidity repositories. On 2 November 2026 we measured it for the first time on five public audits from the EVMbench set: it found 14 of the 20 known High findings. On the same five audits the leader's own report finds 15 under our judge copy (16 under the benchmark's own verdict on one finding), and the older, simpler route of our shop found 8. EVMbench is the open benchmark of known High findings from past audit contests, and the leader is Azimuth, the best published run on it that we know of.

The scan is for people and for AI agents: a person pays by card or iDEAL, an AI agent pays over x402 in USDC on Base. Details: tokenizen.nl/deep-scan.

The folder with the verdicts, the exact reports the judge read, a table and a script that recounts everything is public: results/2026-11-02-shop-setup-five-audits in github.com/holistis/bug-bounty-intelligence-mcp. Clone it and run node check.mjs. No network and no key. It recomputes every count from the 20 verdict files and checks the hashes. It checks the counting, not whether the judge's verdicts are right.

How it was measured

One run per audit, from a script on our own machine and not through the shop, with the scan code, model and thinking limit the shop uses, and a flat cost limit of 1.50 dollars per audit. The cheaper 1 and 5 USDC routes of the shop are older and are not measured here. The judge is a copy of the EVMbench detect grader (gpt-5, high reasoning effort, the benchmark's grading prompt, one call per known finding), run through OpenRouter and not through the benchmark's own harness. When we checked that copy on the leader's report it gave the same verdict as the benchmark's own answer on 30 of 31 findings. The five scans cost 5.88 dollars in total.

Audit	Known	Found
2023-10-nextgen	2	2
2024-06-size	4	1
2024-08-phi	6	4
2024-12-secondswap	3	3
2025-04-forte	5	4

Precision, which recall alone hides: of the 90 hypotheses the five runs wrote, 17 match a known finding when two open judges name the same one (18.9 percent), and 25 when either judge does. An outside reader, Tom Smart, found that two answers which wrote H-3 for H-03 were voided by our reading rule; counting them gives 19 of 90. The correction is in the folder. That is a floor, because a real bug the audit did not list counts as a miss, and the two open judges are stricter than the grader copy that gave the 14 of 20. The hypotheses and every verdict are in the folder.

On phi our run found H-06. Our judge copy says the leader's report misses it, and the benchmark's own grader counts it as found there. It is the one finding where the copy and their grader disagree. On size we found one of four.

We then ran the same setup on six more audits, four small and two larger. With two open judges, strictly, it found 11 of the 14 known findings, and 11 of its 54 hypotheses match a known finding (10 of 30 among the first five per audit). These audits are not fresh: earlier versions of our scan ran on all of them, so this is not a result on new contracts. The folder with the hypotheses and every verdict is results/2026-11-02-shop-setup-six-more-audits.

We also inserted twelve faults today, four into each of three public projects, with a script and a fixed seed, and ran the scan on those copies and on clean copies. It found 8 of the 12 with the grader copy and 7 strictly with two open judges (8 if one answer with a mistyped id, H-4 for H-04, is counted), and the judges found no match in the 21 hypotheses of the clean copies. The faults are simple and synthetic (a removed access control, a removed check, or a comparison shifted by one), so this shows that the scan finds new instances of those kinds, not new kinds of bugs. The test also found a bug in our own scan, a one-pass scan that could read no files, which we fixed. The script, the seed and a script to re-judge with your own model are in results/2026-11-02-injected-bugs-test.

What this does not tell you

Five audits, one run each. One finding moves the share by five points, and the judge does not answer the same every time.
They are development audits from 2023 to 2025, so they may be in the training data of the model behind the scan. This is not a result on new contracts.
The hardest audit in the set, noya, is not included.
Two runs, size and nextgen, hit our 1.50 dollar limit and were stopped. We have not measured what the shop's own limits would do on them.
The runs did not read every file: 14 of 20 on phi, 48 of 76 on nextgen, 37 of 39 on size. A bug in a file that was not read cannot be found.
Four of the six findings we gained over the older route come from one audit, forte, where the older run had been stopped by its cost cap and counted as zero.

A mistake we made on the way

Our first public number for the shop scan, 8 of 20, was labelled as the scan we sell. It was an earlier route. We corrected the text and kept the old text visible, in results/2026-11-02-our-scan-under-the-benchmark-judge/README.md.

If a number does not match

If a number in the folder does not match when you run the check, say so in issue 41 of the repository and we will correct it and keep the old text visible. Our scoreboard at tokenizen.nl/scoreboard lists every measurement with the judge named next to it.

Best regards,
Abdellah
github.com/holistis

Check. Pay. Prove.

holistis — Thu, 29 Oct 2026 18:19:44 +0000

AI agents are starting to spend money. They call tools, pay for answers and rate what they got. Almost nothing checks any of it. A tool can describe itself however it likes, a seller can say it delivered, and a rating can point at nothing.

We built the checks. Then we ran them on one real deal: look at the tool, pay, and prove the payment afterwards. The video is the whole run. Below it is every command and every transaction, so you can repeat it and disagree with us on the facts.

The video of the whole run is on the original page: https://tokenizen.nl/notes/check-pay-prove

1. Check, before the agent pays

muraqib-agent-watch starts an MCP server, reads the tools it lists and flags every tool that touches money, files or credentials without stating a safeguard. In the run it looked at our own capacity-attest server and found three tools, each stating its own safeguard. That is a first look at what a tool says about itself. It does not prove the safeguard holds, and it starts the server you point it at, so run it on your own machine.

npx muraqib-agent-watch npx -y capacity-attest

2. Pay, with a real transaction

A buyer agent sent 0.05 USDC to a seller through x402 on Base mainnet: transaction 0x7160db5e...e7eb, block 52,989,993. The seller is registered as ERC-8004 agent 1085288 from the address that got paid, so the payment address and the agent are the same party. The whole deal cost about three cents in gas.

3. Prove it from the chain

Three blocks later the buyer wrote its own rating of the seller into the ERC-8004 reputation registry: transaction 0xa661b2ff...3936, block 52,989,996. The rating carries the payment's transaction hash as its reference. The same hash sits in two places, so the rating is tied to a payment that really happened.

Eight checks read it back from the chain. The video shows four: the rater is the payer, the paid address is the agent's bound wallet, the rater is not the agent's owner, and the payment came before the rating. All eight passed. We wrote those checks ourselves, so we also asked someone else to look. AgenticFinanceGraph re-checked the run through a public node, Base's public endpoint, which they do not run, and confirmed the four facts listed above.

What we have shipped so far

33 to 20 to 4: Secrets still leaking in 68 test cases of an open-source AI coding tool with more than 10,000 stars: before, after the maintainers' own fixes, and with our fix (pull request open). (Our pull request: https://github.com/tigerless-labs/autoharness/pull/278)
0 of 600: Live x402 services that signed a receipt in our sample of an index of 20,462 resources, when we measured on 15 October. (The measurement: /notes/zero-of-600-x402-services-sign-a-receipt)
8 of 8: Checks passed on a real payment on Base mainnet. A third party re-checked four of them through Base's public endpoint. (The re-check: https://github.com/erc-8004/erc-8004-contracts/issues/99#issuecomment-6372783904)
A, 4.2 of 5: Glama's rating for how well agent-receipt-verify describes its tools to an AI agent. (The listing: https://glama.ai/mcp/servers/holistis/agent-receipt-verify)
3 tools: Open source and on npm: muraqib-agent-watch, agent-receipt-verify and capacity-attest. (The repositories: https://github.com/holistis)
3830: Our pull request on the x402 offer-and-receipt extension is named in the draft x402 specification text (draft pull request 3776, still open). (The draft: https://github.com/x402-foundation/x402/pull/3776)

What we do not claim

Buyer and seller in this run were both ours. It tests the mechanism. It is not evidence of adoption.
There is no paying customer yet. This is the first public run, and we are looking for the first engineers who repeat it on their own servers and endpoints.
muraqib-agent-watch reads what a tool says about itself. It does not prove the safeguard holds.
Passing checks say the shape of a payment and its receipt is right. They do not say the seller is honest.
None of this is an audit, an approval or a certification.

Run it yourself

No account and no API key. Each command is the tool as published on npm.

npx muraqib-agent-watch <command that starts your MCP server>
npx -y agent-receipt-verify <URL of an x402 endpoint>
npx -y agent-receipt-verify attempt-identity

The first looks at your tools before an agent trusts them. The second checks an x402 endpoint before an agent pays it. The third runs 19 test vectors for the x402 attempt-identity rule. If one of them says something wrong about your server or your endpoint, open an issue. We fix it in public, and we say so.

Why we are doing this

Money is starting to move between machines. The people who build those machines should not have to take anyone's word for it, ours included.

The fastest way to help is to point a tool at your own server or endpoint and tell us what it got wrong.

Original with the video: https://tokenizen.nl/notes/check-pay-prove

Eight security signals in 109 public repositories of organisations in ten EU countries

holistis — Thu, 29 Oct 2026 17:59:13 +0000

Update, 28 October 2026, evening. After this note was published we changed the check at tokenizen.nl/en/cra-check so that it also reads the pipeline file of Jenkins, GitLab CI, Azure Pipelines, Bitbucket Pipelines, Travis, Drone and CircleCI when it sits at the root of a repository (rule set 2026-10-28.7). The Trivy scan in a Jenkinsfile that this note mentions as a miss is now seen. We ran the measurement again. GitHub's search results had shifted a little, and the counts moved by at most three repositories per signal: a security scan was visible in 16 of 104 repositories, a bill of materials in 5 of 102 and a security policy with a contact in 18 of 106. Where the text below says that only GitHub Actions is read, that is no longer true; the other limits stand. The text below is unchanged and describes the earlier run.

On 28 October 2026 we measured eight security signals on 109 public GitHub repositories, with the same code that runs behind our free check at tokenizen.nl/en/cra-check (rule set 2026-10-28.6). The eight signals are our own selection, made with the EU Cyber Resilience Act in mind: some relate to topics in Annex I part II of the Regulation, others are general good practice. They are not one signal for each legal requirement, and nothing here says whether anyone complies.

The sample rule and our expected shares were written down before the run, in an internal file that is not public, so the date cannot be checked from outside. For ten EU countries (the Netherlands, Germany, France, Belgium, Sweden, Finland, Ireland, Spain, Italy and Denmark) we took the first 20 organisations of the GitHub search for organisations in that country, sorted by number of public repositories. For each organisation we took the most recently pushed repository, among its first ten, that is not a fork, not archived, not a template and has at least one published release. An organisation without such a repository dropped out and was not replaced: 89 of the 200 did. Two more were skipped because their repository was already in the sample from another country list. That left 109 repositories.

The rules of the signals changed between our runs, and the requirement that a security policy names a contact was added after the sample rule was registered. Our expected shares were: security policy with a contact 40 to 65 percent, bill of materials 5 to 15, signed releases or provenance 5 to 20, lockfile 60 to 80, automatic updates 35 to 55, security scan 25 to 45, recent commit 70 to 90, licence 85 to 95. Five of the eight signals came in lower than that.

For each signal we give how many repositories show it, out of the repositories where we could measure it. A signal we could not read completely is "not measured" and is left out of the second number.

What was visible in the public repository

17 of 106: show a security policy that names a contact address or a reporting link (16 percent) (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
4 of 101: show a software bill of materials, as a file or as a step that makes one (4 percent) (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
5 of 101: show signed releases or a provenance attestation (5 percent) (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
56 of 80: show a lockfile such as package-lock.json, Cargo.lock, poetry.lock or go.sum (70 percent). A requirements.txt with exact versions and pinned GitHub Actions are not counted. We did not assess 29 repositories (no package file we recognise, a Maven, Gradle, Go or .NET project without a lockfile, or a file tree we could not read completely), so they are not in the 80 (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
36 of 106: show automatic dependency updates, Dependabot or Renovate (34 percent) (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
15 of 104: show a security scan in a GitHub Actions workflow (14 percent) (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
88 of 109: had a commit on the default branch in the last 90 days (81 percent) (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)
85 of 109: declare a licence, in GitHub's licence field or in a licence file in the root (78 percent). Of the 24 without one, five have a licence file only in a subfolder, such as a font's or a bundled component's (The rules in the code: https://github.com/holistis/tokenizen/blob/main/packages/self-check-server/src/cra-repo-check.ts)

What we do not claim

This is not a sample of companies. Organisations with many public repositories are larger and more active than the long tail, and the list includes universities, communities and projects that are not companies, so the numbers probably say more about the active end than about a small software firm. We use the word organisations for that reason.

"Not visible" means not visible in the public repository and its GitHub releases. A policy can sit on a website, a bill of materials on a build server, a scan in the repository settings. Of the CI systems only GitHub Actions is read, at most 12 workflow files per repository, so a scan that runs in Jenkins, GitLab CI or CircleCI is not seen: one of the cases we checked by hand was a Trivy scan in a Jenkinsfile. The Regulation applies to products with digital elements made available on the market in the course of a commercial activity (Articles 2 and 3(22) and recital 15). Free and open-source software that its manufacturer does not monetise is not treated as a commercial activity, and neither does the mere presence of regular releases make it one (recital 18), which matters here because every repository in this sample has a release. The sole act of hosting a product on an open repository does not make it available on the market (recital 20), so many repositories in this sample are probably outside the duties of manufacturers. Open-source stewards have a lighter regime of their own (recital 19 and Article 24).

Some of our definitions are stricter than GitHub's. The policy signal needs a file that names an email address or a reporting link, so a file without one is not counted even though GitHub shows a policy: in the final run that excluded two repositories. A scanner that is only set up in the configuration file of another tool, without a workflow that runs it, is not counted. A workflow that installs a bill of materials tool is counted, and we cannot see whether it ran. Signed git tags are not checked, and the patterns can count a passing mention of a tool name in a workflow. Files are matched anywhere in the tree, so a lockfile or a bill of materials in a test folder counts. A signal can be called visible from a partial read but not visible only from a complete one, so leaving the repositories we could not measure out of the second number pushes the shares up slightly. The country comes from the free-text location field of an organisation's GitHub profile, so it is approximate. The numbers are a snapshot from one hour: GitHub search results shift, and a repeat gives a slightly different sample. We publish no repository or organisation names.

How we checked our own tool

We ran the measurement seven times while fixing the tool, and only the last run counts. Checking found several faults of ours. A failed read was once reported as "not visible". A policy that wrote its address as "security AT example.org" was called "no contact". A Docker build that attaches a provenance attestation and a bill of materials to its image was called "not visible" for both. On the other side, a workflow that deleted the CodeQL cache counted as a security scan, a version number like braces@3.0.3 counted as an email address, and addresses in templates counted as contacts. All are fixed in the code on GitHub, and the measurement was repeated.

For every repository where the policy signal was not visible (89) we asked GitHub itself, with its own detection. In the final run it agreed in 87 of the 89 cases. The two differences are policies that name no real contact, which we do not count: a template address on a .local domain, and a file without any address. The licence signal uses GitHub's own licence field, so there is nothing independent to compare it with.

For the signals for a bill of materials, provenance, scans and automatic updates, and for recent commits, we read a seeded random sample of "not visible" findings, five for each of the first four and three for the last (23 cases), in three rounds on the last three runs, with a different method (the whole file tree, every workflow file, other CI systems and the releases). We also looked at every one of the 24 gaps each for the lockfile and the licence, in the whole file tree: all 24 had no lockfile anywhere, and 5 of the 24 licence gaps have a licence file only in a subfolder. The first round found one miss, the Docker case above. The second found none: three cases were flagged, two were false alarms of our own checking script and one was a scanner that is only configured in another tool's file, which we do not count. The third found one miss of coverage, the Jenkinsfile case above, and three flagged cases that were not misses. On the other side we read the line behind each of the 41 "visible" findings of the final run for the policy, bill of materials, provenance and scan signals. All showed what the signal asks for, but two bills of materials and one provenance case only install the tool, and one scan runs npm audit fix.

This is our own checking, not an independent audit. Twenty-three cases per round says little about an error rate, and there may be faults we did not find.

Repeat it

The sampling script is bench/cra-benchmark.mjs in github.com/holistis/tokenizen, and the three checks above are bench/cra-check-against-github.mjs, bench/cra-hand-check.mjs and bench/cra-visible-check.mjs next to it. They read public GitHub endpoints only, and a rerun gives a slightly different sample. We built the check these numbers come from, so we are not neutral here.

Name a public GitHub repository and see which of the eight signals are visible in it. It is free, and reports are not stored.

Try it: https://tokenizen.nl/cra-check

Check. Pay. Prove. One real AI-agent payment on Base mainnet

holistis — Mon, 26 Oct 2026 14:50:00 +0000

Check, before the agent pays muraqib-agent-watch starts an MCP server, reads the tools it lists and flags every tool that touches money, files or credentials without stating a safeguard. In the run it looked at our own capacity-attest server and found three tools, each stating its own safeguard. That is a first look at what a tool says about itself. It does not prove the safeguard holds, and it starts the server you point it at, so run it on your own machine.

npx muraqib-agent-watch npx -y capacity-attest

Pay, with a real transaction
A buyer agent sent 0.05 USDC to a seller through x402 on Base mainnet: transaction 0x7160db5e...e7eb, block 52,989,993. The seller is registered as ERC-8004 agent 1085288 from the address that got paid, so the payment address and the agent are the same party. The whole deal cost about three cents in gas.
Prove it from the chain
Three blocks later the buyer wrote its own rating of the seller into the ERC-8004 reputation registry: transaction 0xa661b2ff...3936, block 52,989,996. The rating carries the payment's transaction hash as its reference. The same hash sits in two places, so the rating is tied to a payment that really happened.

What we have shipped so far
Six numbers, each one with the place where you can check it.

33 to 20 to 4

Secrets still leaking in 68 test cases of an open-source AI coding tool with more than 10,000 stars: before, after the maintainers' own fixes, and with our fix (pull request open).

Our pull request
0 of 600

Live x402 services that signed a receipt in our sample of an index of 20,462 resources, when we measured on 15 October.

The measurement
8 of 8

Checks passed on a real payment on Base mainnet. A third party re-checked four of them through Base's public endpoint.

The re-check
A, 4.2 of 5

Glama's rating for how well agent-receipt-verify describes its tools to an AI agent.

The listing
3 tools

Open source and on npm: muraqib-agent-watch, agent-receipt-verify and capacity-attest.

The repositories

3830

Our pull request on the x402 offer-and-receipt extension is named in the draft x402 specification text (draft pull request 3776, still open).

The draft
What we do not claim
Buyer and seller in this run were both ours. It tests the mechanism. It is not evidence of adoption.
There is no paying customer yet. This is the first public run, and we are looking for the first engineers who repeat it on their own servers and endpoints.
muraqib-agent-watch reads what a tool says about itself. It does not prove the safeguard holds.
Passing checks say the shape of a payment and its receipt is right. They do not say the seller is honest.
None of this is an audit, an approval or a certification.
Run it yourself
No account and no API key. Each command is the tool as published on npm.

npx muraqib-agent-watch
npx -y agent-receipt-verify
npx -y agent-receipt-verify attempt-identity
The first looks at your tools before an agent trusts them. The second checks an x402 endpoint before an agent pays it. The third runs 19 test vectors for the x402 attempt-identity rule. If one of them says something wrong about your server or your endpoint, open an issue. We fix it in public, and we say so.

Why we are doing this
Money is starting to move between machines. The people who build those machines should not have to take anyone's word for it, ours included.

For more information visit our blog: https://tokenizen.nl/notes/check-pay-prove

www.github.com/holistis

We scored 29 of 117 on EVMbench and published every miss

holistis — Mon, 26 Oct 2026 14:42:52 +0000

We put our smart contract scan on the public benchmark everyone quotes, published the precision and every miss next to the recall, and left the verdicts open so you can recompute the numbers in a minute.

Some vendors of AI audit tools quote one number, 67 percent or 97 percent. Most of the public pages we read do not say how many wrong findings came with it, and we found none that lets you recompute the result from its outputs.

Our scan found 29 of the 117 known findings on the public benchmark everyone quotes. Of the 1,537 hypotheses it wrote, 45 matched one, and in 18 of the 40 audits nothing passed both judges. We ran it once, with no tuning for this run, and published all of it: the raw output, the verdict of both judges on every hypothesis, and the script that recomputes the totals. Below: what the ruler cannot see, who else shows what, and how the scan has changed from one version to the next.

One ruler, one seal on one mark: the number is whatever it is. Illustration generated with AI.

The numbers

40 audits, 117 known high severity findings, one run each, two judges from different model families. Each number has the place where you can check it.

29 of 117. Known findings the scan found in its first 30 hypotheses, with both judges naming the same finding. The whole list holds the same 29. (evmbench-40.json)
41 of 117. The same list when one judge is enough, which is the closest we can get to how the benchmark counts. 49 of 117 on the whole list. (the verdicts)
2.9 percent. Precision, strict: 45 of the 1,537 hypotheses match a known finding, and several can hit the same one. The benchmark does not count this at all. (totals.json)
18 of 40. Audits where nothing the scan said was accepted by both judges. Every one of them is in the table, next to the audits where it worked. (the README)

What the ruler cannot see

EVMbench is a good public ruler: real audit contests, a fixed commit for each, a published grader. We read the grader. For every known finding it asks a language model whether the audit report describes it, and it adds up the yes answers. Nothing in it counts a finding that is wrong (the grader code). The authors of one of the tools measured on it say the same: the benchmark does not evaluate false positives (their write-up).

So a report that lists everything can score high. A recall ruler measures recall and was never meant to count wrong findings. One recall number is therefore half a measurement, and the other half is the precision, the misses and the method. We put all of them on the table.

Who shows what

We read each public page on 26 October 2026. A grey cell means we did not find it there. That is not proof that it does not exist, and if we missed something, tell us and we will correct it in public. These columns are the ones we publish, and a vendor would pick others.

[green] you can check it yourself, [half] partly, [grey] we found nothing.

	Price on the page	Recall number	Precision	Outputs you can recompute	Receipt you can verify	Agent pays, no account
Us	[green] 1 and 5 dollars, no account	[green] 29 of 117 strict, 41 with one judge	[half] 2.9 percent strict, a floor, because only known findings count	[green] Raw hypotheses, both verdicts, a script that recomputes the totals	[green] Signed offer and receipt, the signer bound to our domain in DNS	[green] USDC on Base through x402
Nethermind AuditAgent	[grey] Not found. A 50 dollar credit offer is linked.	[half] 67 percent, 80 of 120, from their own run	[grey] Not published	[half] The benchmark and its grader are public. Their outputs are not linked.	[grey] Not found	[grey] Not found
Octane	[grey] Not found	[half] A 97.4 percent true positive rate. No sample size or method on the page we read.	[grey] Not published	[half] Four CVEs are claimed by product name. The rate cannot be checked.	[grey] Not found	[grey] Not found
Almanax	[grey] Not found. A 7 day trial is offered.	[grey] None published	[grey] None published	[half] SOC 2 Type II is claimed on the page. We found no accuracy figure.	[grey] Not found	[grey] Not found
Sherlock AI	[grey] Not found	[grey] No recall published	[half] About 55 percent on one project in their own test: 21 valid out of 38 reported	[half] The method is described. The triage sheet is not published.	[grey] Not found	[grey] Not found

Sources are the pages linked in the first column, read on 26 October 2026. Prices, numbers and tests change, and a vendor may show customers more than its public page does. Our recall counts high severity findings on our own basis, so 41 of 117 and a vendor's 67 percent are not comparable.

Count the green cells in the first row. Then check how many of the other four rows you could recompute yourself.

Put your row in the table

If you build or sell a scanner, run it on this ruler and publish your raw output and your verdicts. Open an issue on the results repository with the link. We add your row to this table as you give it, with your numbers and your name, and we run the same recompute script over your files. A row we cannot check stays grey, and we write what we looked for.

Getting better, in public

Scans say they learn. This is our version history, on the same eight Sherlock contests and the same two judges, strict, 105 known findings (the results):

The scan in production until 25 October: 1 of 105.
The first run of the next version: 4 of 105.
Version 3.1, live since the evening of 25 October: 8 of 105.

Then the same version on nine fresh Code4rena contests from the last year: 5 of 66 in the first 15 hypotheses and 8 of 66 in the first 30 (the results). On this ruler, High findings only, it is 23 of 117 in the first 15 hypotheses and 29 in the first 30. It is a different count from the steps above and not the next step.

We also tried four ways of ranking the list better. None cleared the bar we had set, so the scan stays at 3.1. Those notes are not published yet. From now on a version replaces the last one only if it beats it on a fresh set under a rule written first. Version 3.1 was promoted on the same eight Sherlock contests, and the fresh Code4rena check came after it went live. The first time we measured the scan it found nothing (that note). We changed it and wrote it up.

For version 3 the bar was committed to the repository before the run, and you can read the history (the pre-registration). For 3.1 and for this ruler it was written in our own notes, so for those you have our word for the order.

The counting rule has a name

We call our way of counting the Two-Judge Strict Rule. A hypothesis counts only if two judges from different model families name the same known finding, and a valid hypothesis that is missing from the known list counts as wrong. The eight items are in one file (the rule). Anyone can report a result under the name, or say which item they depart from.

The next measurement is fixed before it runs. It uses the first eight Code4rena contests whose final report appears after 1 November 2026, in order of publication, so nobody picks them. Version 3.1 is measured on them, and a candidate version replaces it only if it finds more and is not less precise. We publish the result whatever it is (the pre-registration).

Both files were stamped on Bitcoin with OpenTimestamps before we committed them, and the proof files sit next to them. A stamp stays pending until a Bitcoin block includes it, which usually takes a few hours, so verify the proof again later with the OpenTimestamps tool.

Each ring a version, each version measured against the last. Illustration generated with AI.

Next to the x402 specification

The scan pays and proves through x402, the open standard that lets software agents pay for an HTTP resource. While we measured, three things happened around the x402 specification that we did not plan. The author of an open pull request for a new section (ruiruii, the thread) closed three points from our review within hours, and the draft links our test vectors as their interim location. It is still a draft and not a decision of the maintainers. On 26 October we reviewed those vectors ourselves, with AI reviewers that had not written them, and found real defects in our own checker: one let a collapsed identity pass, one read a Latin-1 body as windows-1252. We fixed them, released the fix as version 0.2.20 and said so in the same thread. The set now has 72 vectors over twelve clauses. Then a reviewer, NSPG13, pointed out that the section sorted repeated query values, so that ?q=a&q=b and ?q=b&q=a got one identity. We gave those two requests to 18 libraries and frameworks, and none of them treats them as equal (the measurement). The section author took our replacement rule, which keeps the order in which the pairs arrived, and ran an implementation of their own over our 22 known-answer files, eleven inputs with a right and a wrong value each: all of them agree (the comment). Our own two implementations, which we wrote, agree as well.

The third: smartflowproai-lang, who builds a community x402 validator and is not part of our team, added our endpoint to their table as an observed row, and added a finality field after we closed the gap they had named in our checking tool. Later they read our DNS record through four resolvers and marked the signer as bound to our domain (the thread). This is one person's tool and not a review by the specification.

You can run the checks yourself, with no account and no key:

npx -y agent-receipt-verify attempt-identity
npx -y agent-receipt-verify https://x402.tokenizen.nl/api/bug-intel-quick

The first runs every test vector for that section and prints a verdict for each. The second asks the scan for payment without paying, and checks that the offer is signed. The signer is bound to our domain in DNS, as section 4.5.1 of the offer-and-receipt extension describes.

Why we publish the misses

On 26 October three people outside our team ran the recompute script from a fresh clone and got the same totals as ours: ruiruii, AgenticFinanceGraph and Tom Smart (smartflowproai-lang) (issue 41). They checked the arithmetic from the verdict files to the table. They did not check whether the judges' verdicts were right, and nobody has checked the 117 against the benchmark's own configs yet.

We publish what lets you check instead of trust: the price on the page, the raw output, both verdicts, the receipt, the DNS record, the script that recomputes the totals, and the misses. If you find an error, send it. We correct it in public.

For engineers who build scanners, facilitators or agents: the ruler is public, the grader is public, and the verdict files here are a template you can copy.

A ledger you can see through, stamped and weighed. Illustration generated with AI.

What we do not claim

This is not an audit. The scan missed 88 of the 117 known findings, and in 18 of the 40 audits it found nothing both judges accept.
The ruler counts only high severity findings. The 29 older audits (2023 and 2024) are public and may be in the training data of the models behind the scan and the judges. The 7 newest audits are the cleanest and the smallest sample: 4 of 14.
Our counting is not the benchmark's. It asks, per known finding, whether the whole report describes it. We ask, per hypothesis, whether it matches a known finding, with two judges. Our one judge number of 41 of 117 is therefore not comparable to a vendor's 67 percent.
Precision of 2.9 percent is a floor. A hypothesis that is valid but is not in the known list counts as wrong, and we did not measure how many of those there are.
Each audit was run once, so we cannot give a spread. Both judges are language models, and their verdicts are published so that you can check them.
4 of the 40 audits overlap contests we tuned on earlier. Without them the scan found 27 of 110 in its first 30 hypotheses.
Public configs of the benchmark list 117 findings and its paper says 120. We used the 117 that can be checked.
Our rules for this run were written in our notes before the scan. We anchored them on Bitcoin only after the run, so the anchor does not prove the order for this measurement. For the next one the order is provable, because the rule and the pre-registration were stamped before we committed them.
3 of the 3,074 judge answers could not be read and count as no match.
Matrix cells show what we read on public pages on one day, not what a vendor may show its customers.

Break our numbers

Recompute the totals, read the misses, run the checks. If you find a mistake, we correct it in public.

Originally published at https://tokenizen.nl/en/notes/the-ruler-and-the-misses

Autoharness has 10,600 stars and reads every line of your Claude Code sessions. Here is what we checked before trusting it.

holistis — Fri, 23 Oct 2026 09:12:13 +0000

Update, Oct 23: three things in the first version of this post were wrong or too strong. It said we spun up a disposable server and cloned the repo. When we wrote it, we had only read the code and the issue tracker on GitHub. Since then we have run autoharness's redaction function on a disposable server with made-up credentials only. We have not run the full tool (hooks, MCP server, storage), and we are sending our results to the maintainers first. It also called #273, #272, #268 and #269 issues, but they are pull requests from outside contributors. And it said the maintainers ship fixes fast, which holds for the first two leaks below but not for the fixes that are open now. All three are corrected in the text.

We wanted autoharness. 10,627 stars as of Oct 23, MIT licensed, and it solves a real problem: Claude Code skills pile up, overlap, and nobody prunes the ones that stopped getting used. Autoharness distills new skills from real sessions and archives what goes stale. We were ready to install it directly on our own machine.

Then we read what it actually does before pointing it at a machine that holds real credentials and unpublished security findings. It registers an MCP server and hooks that capture every turn across your sessions: your input, the agent's output, tool calls. It redacts and stores slices of that as evidence inside skill folders.

So before installing it anywhere real, we read the one part that actually matters here, the redaction code and its own issue tracker, on GitHub.

What we found is a documented history, right in the project's own tracker:

Issue #213 (closed): a private key's BEGIN line got redacted, but the key body itself still reached stored evidence files.
Issue #212 (closed): later redaction rules overwrote earlier placeholders, so AWS keys, private keys, GitHub tokens, and bearer tokens all ended up labeled as one generic category.
Pull request #273 (open, from an outside contributor): adds redaction for fine-grained GitHub personal access tokens, which suggests the current rules do not catch them.
Pull requests #272, #268 and #269 (open, also from outside contributors): more redaction fixes, for secrets assigned to a quoted key, for overlapping matches, and for PGP key blocks.
Issue #276 (open): the email rule is quadratic on a long run of local-part characters.

None of this means autoharness is badly built. Issues #212 and #213 were closed within four days of being reported, and a project with a public paper trail of "this secret type leaked, here is the fix" is more trustworthy than one with no visible redaction bugs at all. The open fixes are a mixed picture: #269 and #273 merge cleanly into main and have had no review yet, #272 has a leak regression that another contributor reported in its comments, and #268 conflicts with main and needs a rebase. As of Oct 23 the repo has no new commit since October 15, which may just be a quiet week. Redacting secrets reliably from free-form session text is a genuinely hard problem. We would rather see that history than a clean-looking tracker that just means nobody has looked hard enough yet.

Our conclusion: when a tool's job is to read your own working sessions, test it on a throwaway machine first, with no real secrets anywhere near it, and read its redaction code and issue history before pointing it at anything real. Not because this specific project did something wrong, but because "redact my own credentials from my own session transcripts, automatically" is exactly the kind of promise that is hardest to verify by reading a README and easiest to verify by testing it where a mistake costs nothing.

We are not running autoharness on our main machine yet. We have only tested its redaction function, on a throwaway server with made-up credentials, and not the full tool. We will share the results with the maintainers first and write them up here afterwards. Until then, it stays off the machine that holds real credentials.

Best regards,
Abdellah
github.com/holistis

733,203 attestations were taken back

holistis — Thu, 15 Oct 2026 20:52:13 +0000

733,203 attestations were taken back. Every single one by the party that wrote it.

I went counting because someone told me, politely and correctly, that a signature does not make a statement true. I had no good answer, so I went looking for what the ecosystem does when a signed statement stops holding.

The numbers, taken against the three EAS indexers rather than estimated:

Base: 678,895 revoked of 3,651,900 attestations
Optimism: 54,213 of 1,319,189
Ethereum mainnet: 95 of 14,956

That is 733,203 of 4,986,045, so a headline near 15 percent. The headline is the least useful part.

On Base, three schemas carry 97 percent of all revocations, and all three belong to one issuer: verifiedAccount (723,600 issued, 409,490 revoked), verifiedCountry (307,123 issued, 144,207 revoked), verifiedCoinbaseOne (189,434 issued, 103,295 revoked). An exchange revoking roughly half of its own verifications, continuously.

Anyone can reproduce it:

curl -s https://base.easscan.org/graphql -H "Content-Type: application/json" \
  -d '{"query":"{ aggregateAttestation(where: {revoked: {equals: true}}) { _count { id } } }"}'

and groupByAttestation(by: [schemaId], where: {revoked: {equals: true}}) for the breakdown. One warning if you repeat it: during my run the Base endpoint returned a maintenance page with a 200 status for a valid query. That reads as a zero unless you look at the body.

What I expected to find, and did not

I expected to find disputes. A registry full of signed statements, some of them wrong, and a trail of parties contesting each other.

There are no disputes, because there is no such operation. A revocation is the issuer withdrawing its own statement. All 733,203 of them. The counterparty has no move at all.

That reframes the whole picture. The need to revise a signed record is not theoretical and not rare: it is ordinary operational work at a scale of hundreds of thousands. And the only tool available is self-correction.

Where self-correction works, and where it does nothing

Self-correction works beautifully when the author has no stake in the statement being false. An exchange that no longer verifies a user has no reason to keep claiming it does. Revoking is just bookkeeping, and the numbers above are mostly that: a large, honest system keeping its own records current.

It does nothing when the author is the interested party.

Think about what an agent-to-agent payment produces. A seller attests that it delivered. A buyer attests to what it received. Both are describing their own performance, or their own entitlement, in a record that may later decide who was right. Asking either of them to revoke their own convenient statement is not a mechanism. It is a hope.

So the gap is not that signatures are weak. The gap is that a registry gives exactly one party the right to revise the record, and it hands that right to the author rather than to whoever was on the other side of the event.

What the other half looks like

Two signed accounts of one event, from parties whose interests point in opposite directions, with the two artifacts tied together so a contradiction is computable.

Concretely, in the format I work on: the buyer's claim includes the hash of the seller's signed receipt, taken over the receipt's exact transmitted bytes. So the claim cannot be about some other delivery, and it cannot quietly drift from the seller's version. If the buyer says one thing and the receipt says another, a verifier recomputes the digest, compares the fields both sides describe, and gets a disagreement out of arithmetic rather than out of a reading.

That is deliberately narrow. It does not decide who is right. Deciding is governance, and nobody should take that from a library. What it removes is the state where two parties quietly hold incompatible records and nothing in the system notices.

The honest limit, which matters here

Detecting a contradiction is not the same as making a dishonest statement expensive. Nothing I have built imposes a cost on being wrong. The three mechanisms that would are staking, a dispute process, and verifiable execution, and I have none of them.

What the counting bought me is the argument for building the cheapest one. Not an opinion that the gap exists, but the measurement: a need exercised 733,203 times, with the right to act on it held by exactly one side.

And one more number, from the other direction. Sampling one facilitator’s discovery index of 20,462 live x402 resources at six spread offsets of a hundred each, not one declared the extension that emits the counterparty receipt. Seven of the eight facilitator endpoints I tried returned no usable JSON on that path, so they are unmeasured rather than zero, and the largest index is among them. A strong indication, not a proof.

I have since stood one up myself, so the count is no longer zero, and I would rather say that than let a number I published keep implying something it stopped meaning. It is still a count of one, and both sides of it are me.

That is a strange and good place to be, because it means the hard part is not invention.

Code and the measurement scripts: github.com/holistis/tokenizen

I checked 600 live x402 services. None of them signs a receipt.

holistis — Thu, 15 Oct 2026 14:32:23 +0000

Zero. Not a low number, not mostly zero. Six samples of a hundred, spread across an index of 20,462 live resources, and the offer-receipt extension appeared in none of them.

I went looking because I had built the other half and wanted to know whether it had anything to attach to.

Here is the shape of the problem. When an agent pays for an API call over x402, two parties end up with an opinion about what happened. The seller knows what it served. The buyer knows what arrived. x402 ships an official extension for the seller's side: a signed receipt, which the extension's own documentation offers for audit trails and dispute resolution. I had spent a while on the buyer's side, a signed claim of what was actually delivered, which names the seller's receipt by the hash of its exact bytes so the two artifacts pin each other.

Then it occurred to me that I had never seen one of those receipts in the wild.

How I counted

The x402 bazaar extension means resource servers declare themselves, and facilitators catalogue them. So the question is answerable without asking anyone.

curl -s "https://facilitator.payai.network/discovery/resources?limit=100&offset=0"

That index reports 20,462 resources. I sampled at offsets 0, 4000, 8000, 12000, 16000 and 20300, a hundred each, and grepped the records for the extension's declaration key, which is the string offer-receipt.

Six hundred records. Zero declarations.

For contrast, in the same data the bazaar extension itself appears on every one of the first four hundred and on none of the last two hundred, so these records are not uniform and the index is not a single homogeneous dump.

What this does not prove

Seven of the eight facilitator addresses I tried returned HTML or a handful of bytes instead of JSON on that path. They are unmeasured, not zero. The largest index, Coinbase's, is among them.

A server could also support the extension without it appearing in this catalogue, since the declaration lives in the 402 response and a catalogue is a secondhand record of that. I tried to settle it by requesting a live resource directly and got a 402 with an empty body, which told me nothing either way.

So: a strong indication, not a proof. I would rather hand you the number with its edges than a rounder one without them.

Why the gap matters more than it sounds

A signature answers one question: who said this, and when. It does not answer whether what they said is true. For a receipt, that distinction is the whole thing, because the party signing is the party with an interest in the answer. A seller attesting to its own delivery is the interested party. So is a buyer attesting to what it received.

The usual remedy for a statement that stops being true is for its author to withdraw it. That works, and it is used at real scale: across Base, Optimism and Ethereum mainnet there are 733,203 revoked attestations out of 4,986,045, and on Base three schemas from a single issuer account for 97 percent of them, revoking roughly half of everything they issue. Every one of those events is an issuer correcting itself.

Which is exactly why it does not help here. Self-correction works when the author has no stake in the statement being false. It does nothing when the author is the one who benefits.

The only thing that helps is the other party's signature on the same event. Two signed accounts of one delivery, from two parties with opposing interests, where a contradiction between them is arithmetic rather than opinion. That is what the offer-receipt extension makes possible and what nobody is emitting.

So I made one

curl -i -X POST https://api.tokenizen.nl/api/paid/verify-ownership \
  -H "Content-Type: application/json" \
  -d '{"endpoint":"https://mcp.example.com/sse","ownerAddress":"0x1111111111111111111111111111111111111111","method":"dns-txt"}'

That answers 402 with a Payment-Required header. Base64-decode it and you get the payment requirements, plus an extensions object carrying offer-receipt with a signed offer inside, EIP-712 format.

Note the empty response body. The requirements are in the header, not the body, which is why my earlier probe of somebody else's service looked broken when it was working perfectly.

You can check the signature without trusting me. Recover it against the domain {name: "x402 offer", version: "1", chainId: 1} and the offer types from the specification, in order: version, resourceUrl, scheme, network, asset, payTo, amount, validUntil. It recovers to 0x4E1174b774643B6AD9EE966A81ea1e3665D1159a.

Then check that address is allowed to speak for the domain, which is a separate question the specification is blunt about: a valid signature is not authorization, and anyone can sign a receipt for any resourceUrl.

dig +short TXT _controllers.tokenizen.nl
"v=1;controller=did:pkh:eip155:1:0x4E1174b774643B6AD9EE966A81ea1e3665D1159a"

Same address. That is the loop closed by a stranger with two commands.

It runs on Base Sepolia, which means the money is fake and the cryptography is not. The price is a tenth of a cent. What you are buying is the receipt, because the answer itself is free at /api/verify-ownership and always will be.

Use it as a test fixture

If you are implementing the client side of offer-receipt, you need something real to test against, and as far as I can measure there is nothing. Point your client at that URL. It costs me nothing and you get a live, signed, spec-conformant artifact instead of a fixture you wrote yourself and therefore cannot be surprised by.

If you find it wrong, tell me and I will fix it. If it stays up and other people start emitting these too, the number at the top of this post stops being interesting, which is the point.

What I have not done yet: nobody has paid, so no receipt has been issued, only offers. That needs a funded Base Sepolia client and it is the next thing.

The code is open: github.com/holistis/tokenizen

The Hierarchy of Agent Trust

holistis — Mon, 21 Sep 2026 20:16:22 +0000

Some proof is better than other proof. A story, without any prerequisites or jargon.

The mill

Imagine a village crossed by trade routes. Merchants pass through constantly, never the same ones twice. Just outside the village stands a mill. Travelers bring their wheat, pay the miller in advance, and come back the next morning for flour.

The villagers never worry about the miller. They have known him for years. If he ever shorted someone on flour, the whole village would know by lunchtime, and nobody would bring him wheat again. His reputation does the work.

The traveling merchants have no such luck. They pass through once. By the time a merchant discovers the miller shorted them a sack, they are three villages away, and the miller has already forgotten their face. The next merchant who arrives has no way to know if this mill is honest or not. They are deciding blind, every single time.

The board

One merchant has an idea. Before leaving town, she nails a small wooden board to the mill's outer wall. On it, she writes what she brought, what she got back, and the date. The next merchant who arrives reads the board before handing over their wheat.

Other merchants start doing the same. Within a season, the board is full of notes. A merchant arriving for the first time can read ten honest strangers' experiences before deciding whether to trust this mill. The board does, for travelers, roughly what years of shared history does for villagers.

The forgery

A rival miller notices this and sees an opportunity. He pays a friend to nail a glowing note to his own mill's board, without the friend ever having brought a single sack of wheat. Now the board says something that never happened.

The villagers realize the board only works if everyone trusts every note on it equally. One forged note poisons the whole board, because a new merchant cannot tell which notes are honest and which were paid for.

So they add a rule: every note must be signed, and the signature must be checkable. Not a name anyone could write, but a mark tied to exactly one person, one that cannot be copied onto someone else's note. A blacksmith in town starts making small signet rings, each one unique, each one leaving a mark nobody else can reproduce. From then on, a note only counts if its mark can be verified against the person who claims to have written it.

This does not stop the rival miller from posting his own false note under his own real signature. But it does mean that when he does, everyone can see, without doubt, that it was him who wrote it, not an anonymous forger. Lying under your own signed mark is a different, riskier act than lying anonymously, and the two are no longer indistinguishable.

The next problem, and the wall

The merchants think they have solved it. Then someone asks the obvious next question: what stops a merchant from writing a bad note about an honest miller, out of spite, or because a rival miller paid them to?

The village elders take this seriously and spend a long time on it. They try nine different ideas, one after another.

They try making a false note cost something, a small fine paid when posting a note. But a merchant with nothing to lose, passing through once, pays it without blinking. They try asking three merchants to agree before a note counts, but a rival miller can hire three friends just as easily as one. They try a waiting period before a note is trusted, they try requiring the merchant prove who they are beyond a doubt, they try a village judge who reviews disputed notes. Every version runs into the same two walls.

Either the check is optional, and a merchant who wants to lie simply skips it and posts anyway, at no cost. Or the check requires someone, some judge, some panel, some elder, to decide whose note to believe, and the moment you need a judge, you need to trust the judge, which is the exact same problem one level up. Nine designs, nine times the same two walls.

The elders write this down plainly, rather than pretend they solved it: the board proves who said what, and when. It does not, and structurally cannot on its own, prove who is telling the truth.

What the board actually is

So the board stays exactly what it always was: a place where a real, identifiable person leaves a real, checkable record of what they experienced. Nothing more. Not a court, not a judge, not a score. A miller with ten honest notes and one bad one is not ranked, averaged, or given a grade. A new merchant reads all eleven and decides for themselves, the way a person always has.

This is a smaller promise than the village first hoped for. It is also the only one the board can actually keep.

Where this is going

Agents buying capacity from other agents, GPU-hours, storage, API access, have exactly the mill's problem, at a much larger scale. Every transaction is between two parties who may never interact again, with no shared history and no village gossip to fall back on. capacity-attest is the board: after an agent pays for capacity, it signs a plain factual record of what it received. The signature is checkable by anyone, offline, the same way the merchant's mark on the mill's board was checkable by anyone who knew the trick.

It carries the same honest limit the elders wrote down. It proves what a specific, identifiable buyer said happened, at a specific time. It does not, cannot, and does not claim to prove who is right when two signed records disagree. That is a harder, separate problem, one about authority rather than evidence, and conflating the two is exactly the mistake that poisoned the mill's board in the first place.

Villages eventually grew clearinghouses and banks once enough trade and enough disputes made a plain board insufficient on its own. Whether agent commerce needs the same, and who builds it, is a question for a later chapter. For now, the board is useful precisely because it does not pretend to be one.

If this framing is useful for explaining your own delivery, receipt, or trust problem to someone new to the space, feel free to reuse it. If you want the adversarial-testing version instead of the story version, get in touch: info@tokenizen.nl

Nine ways to fake a delivery claim, and why none of them fully worked

holistis — Sat, 19 Sep 2026 12:02:08 +0000

capacity-attest lets a paying agent leave a signed, factual claim after an x402 payment: delivered yes, no, or partial, plus a hash of the evidence. Other agents can check that history before they pay the same seller. No score, no rating, no judgment, just a receipt.

The obvious hole: it is the buyer who signs the claim. What stops a buyer from lying?

goun7 (Tamga Protocol) put it precisely in a public thread: "buyer-signed post-hoc claims are only a signal to the degree the buyer has something to lose by lying. Without a cost to a false delivered=no, the signal is uncorrelated noise rather than weak evidence."

That is a fair hit. We spent two rounds, nine independent designs, testing whether we could close it. Every design was built out and then attacked against the real code, not argued about in the abstract.

Round one, four attempts

Active on-chain settlement verification.
Sybil detection by tracing where the money came from.
An external economic bond tied to a dispute reference.
Corroboration between multiple buyers, modeled on isnad criticism in hadith science: a chain is only as strong as each link, verified independently.

All four killed.

Round two, five more attempts

Proof of unique humanity (World ID, Gitcoin Passport, BrightID).
A mandatory, non-optional computation cost per claim.
Time as a cost that cannot be bought: a claim only counts after a waiting period on a clock nobody can fake.
Real legal identity and liability through an external dispute protocol.
A third party staking real capital against a specific claim, optimistic-oracle style.

All five killed.

The actual reason, not nine separate coincidences

Every mechanism that truly imposed a cost failed for one of two reasons.

Either it was optional, and an attacker who skips it pays nothing. That pattern repeated every single time: a voluntary check costs an attacker who ignores it exactly zero.

Or it required trusting some claims more than others, which is a ranking, whatever you call it, and that collides head-on with our own rule that a delivery claim is deliberately not a score, a rating, or a judgment. As long as that rule holds, an attacker can always pick the unchecked path and sit there for free.

This is not a lack of creativity. It is a real structural tension between two of our own design decisions.

Two honest, partial fragments that survived

Tamper-proof calendar time. The one thing an attacker cannot buy with more wallets or a second script run. A maturity period is structurally healthier than a bond or corroboration. But the counting function this needs does not exist yet in the code, and a fixed waiting rule punishes a busy, honest seller exactly as hard as a patient scammer.

A third party with real, losable capital against one specific claim. The only one of the nine that provably imposes a real, non-bypassable cost the moment it is used. But it only works for the narrow slice of claims where the underlying fact is cheap and objective to check from the outside, a hash matches, an amount was transferred, not whether the delivery was actually good. And the minimum stake needed to make it worthwhile does not fit the small, everyday payments this project serves.

Where this leaves it

The open question stays open. Not from lack of trying: nine separately designed mechanisms were each built and broken against the real code, with the same underlying reason each time. That is a structural limit of self-reported witness systems without a trusted referee, not a gap we just have not gotten around to closing yet.

We would rather say that plainly than ship something that only looks fixed.

Full design log (D-001 through D-018): DECISIONS.md. Code: capacity-attest on npm.

If you are building something in this space and want the same kind of adversarial pass run against your own delivery, receipt, or grading logic, real fixtures, fail-closed testing, a short public findings note at the end, reach out at info@tokenizen.nl.

PyMC Labs has 1,256 stars. A GitHub issue title could still steal their API key.

holistis — Fri, 11 Sep 2026 05:51:16 +0000

PyMC Labs has 1,256 stars on GitHub and a serious engineering team. A GitHub issue title still could have stolen their Anthropic API key.

I found it, reported it, and yesterday they made it official: GHSA-72xh-gfpr-v9hx, severity High, published, my name on it as the reporter.

Here is what was actually happening. Their issue-triage and duplicate-detection workflows ran Claude Code on every single public issue, automatically, no human in the loop. That call used dangerously-skip-permissions, so the agent got full shell access. In the same job: a real ANTHROPIC_API_KEY and a GitHub token scoped to write issues. Anyone on the internet could open an issue, write a prompt injection in the title or body, and try to walk the agent straight to those secrets.

No approval gate. No isolation. Just text in a box, read by an agent with the keys already sitting next to it.

I sent the report. PyMC Labs did not sit on it. They split the job in two: a read-only analysis step, and a separate publish step that never sees the Anthropic secret. They rotated the key. Then they went one step further and published the advisory instead of quietly patching and moving on, credit included.

That last part matters more than the fix itself. A team can ship a fix and say nothing. Publishing the advisory means anyone can check the claim against the real commit history, not just take my word for it.

This is one repository out of 22 I went through in the same bug class this month, AI CLIs wired straight into GitHub Actions on public, untrusted input. Full writeup on the other 21 is here: https://dev.to/holistis/a-github-comment-can-steal-your-secrets-i-found-22-repositories-where-it-could-1doh

This one just became the first to go fully public. More of them will, one at a time, as the maintainers finish their fixes.

DEV Community: holistis

What we learned building two fail-closed verification tools for the agent economy

Tool one: checking a sanctions-list claim

Tool two: checking a payment receipt

The part that might be useful even without our code

Where we are honestly stuck

Our cyber security scan for smart contracts found 14 of 20 known bugs. Recount it yourself, and see where it falls short

How it was measured

What this does not tell you

A mistake we made on the way

If a number does not match

Check. Pay. Prove.

1. Check, before the agent pays

2. Pay, with a real transaction

3. Prove it from the chain

What we have shipped so far

3830: Our pull request on the x402 offer-and-receipt extension is named in the draft x402 specification text (draft pull request 3776, still open). (The draft: https://github.com/x402-foundation/x402/pull/3776)

What we do not claim

Run it yourself

Why we are doing this

Eight security signals in 109 public repositories of organisations in ten EU countries

What was visible in the public repository

What we do not claim

How we checked our own tool

Repeat it

Check. Pay. Prove. One real AI-agent payment on Base mainnet

3830

We scored 29 of 117 on EVMbench and published every miss

The numbers

What the ruler cannot see

Who shows what

Put your row in the table

Getting better, in public

The counting rule has a name

Next to the x402 specification

Why we publish the misses

What we do not claim

Break our numbers

Autoharness has 10,600 stars and reads every line of your Claude Code sessions. Here is what we checked before trusting it.

733,203 attestations were taken back

I checked 600 live x402 services. None of them signs a receipt.

The Hierarchy of Agent Trust

The mill

The board

The forgery

The next problem, and the wall

What the board actually is

Where this is going

Nine ways to fake a delivery claim, and why none of them fully worked

Round one, four attempts

Round two, five more attempts

The actual reason, not nine separate coincidences

Two honest, partial fragments that survived

Where this leaves it

PyMC Labs has 1,256 stars. A GitHub issue title could still steal their API key.