DEV Community: holistis

733,203 attestations were taken back

holistis — Thu, 15 Oct 2026 20:52:13 +0000

733,203 attestations were taken back. Every single one by the party that wrote it.

I went counting because someone told me, politely and correctly, that a signature does not make a statement true. I had no good answer, so I went looking for what the ecosystem does when a signed statement stops holding.

The numbers, taken against the three EAS indexers rather than estimated:

Base: 678,895 revoked of 3,651,900 attestations
Optimism: 54,213 of 1,319,189
Ethereum mainnet: 95 of 14,956

That is 733,203 of 4,986,045, so a headline near 15 percent. The headline is the least useful part.

On Base, three schemas carry 97 percent of all revocations, and all three belong to one issuer: verifiedAccount (723,600 issued, 409,490 revoked), verifiedCountry (307,123 issued, 144,207 revoked), verifiedCoinbaseOne (189,434 issued, 103,295 revoked). An exchange revoking roughly half of its own verifications, continuously.

Anyone can reproduce it:

curl -s https://base.easscan.org/graphql -H "Content-Type: application/json" \
  -d '{"query":"{ aggregateAttestation(where: {revoked: {equals: true}}) { _count { id } } }"}'

and groupByAttestation(by: [schemaId], where: {revoked: {equals: true}}) for the breakdown. One warning if you repeat it: during my run the Base endpoint returned a maintenance page with a 200 status for a valid query. That reads as a zero unless you look at the body.

What I expected to find, and did not

I expected to find disputes. A registry full of signed statements, some of them wrong, and a trail of parties contesting each other.

There are no disputes, because there is no such operation. A revocation is the issuer withdrawing its own statement. All 733,203 of them. The counterparty has no move at all.

That reframes the whole picture. The need to revise a signed record is not theoretical and not rare: it is ordinary operational work at a scale of hundreds of thousands. And the only tool available is self-correction.

Where self-correction works, and where it does nothing

Self-correction works beautifully when the author has no stake in the statement being false. An exchange that no longer verifies a user has no reason to keep claiming it does. Revoking is just bookkeeping, and the numbers above are mostly that: a large, honest system keeping its own records current.

It does nothing when the author is the interested party.

Think about what an agent-to-agent payment produces. A seller attests that it delivered. A buyer attests to what it received. Both are describing their own performance, or their own entitlement, in a record that may later decide who was right. Asking either of them to revoke their own convenient statement is not a mechanism. It is a hope.

So the gap is not that signatures are weak. The gap is that a registry gives exactly one party the right to revise the record, and it hands that right to the author rather than to whoever was on the other side of the event.

What the other half looks like

Two signed accounts of one event, from parties whose interests point in opposite directions, with the two artifacts tied together so a contradiction is computable.

Concretely, in the format I work on: the buyer's claim includes the hash of the seller's signed receipt, taken over the receipt's exact transmitted bytes. So the claim cannot be about some other delivery, and it cannot quietly drift from the seller's version. If the buyer says one thing and the receipt says another, a verifier recomputes the digest, compares the fields both sides describe, and gets a disagreement out of arithmetic rather than out of a reading.

That is deliberately narrow. It does not decide who is right. Deciding is governance, and nobody should take that from a library. What it removes is the state where two parties quietly hold incompatible records and nothing in the system notices.

The honest limit, which matters here

Detecting a contradiction is not the same as making a dishonest statement expensive. Nothing I have built imposes a cost on being wrong. The three mechanisms that would are staking, a dispute process, and verifiable execution, and I have none of them.

What the counting bought me is the argument for building the cheapest one. Not an opinion that the gap exists, but the measurement: a need exercised 733,203 times, with the right to act on it held by exactly one side.

And one more number, from the other direction. Sampling one facilitator’s discovery index of 20,462 live x402 resources at six spread offsets of a hundred each, not one declared the extension that emits the counterparty receipt. Seven of the eight facilitator endpoints I tried returned no usable JSON on that path, so they are unmeasured rather than zero, and the largest index is among them. A strong indication, not a proof.

I have since stood one up myself, so the count is no longer zero, and I would rather say that than let a number I published keep implying something it stopped meaning. It is still a count of one, and both sides of it are me.

That is a strange and good place to be, because it means the hard part is not invention.

Code and the measurement scripts: github.com/holistis/tokenizen

I checked 600 live x402 services. None of them signs a receipt.

holistis — Thu, 15 Oct 2026 14:32:23 +0000

Zero. Not a low number, not mostly zero. Six samples of a hundred, spread across an index of 20,462 live resources, and the offer-receipt extension appeared in none of them.

I went looking because I had built the other half and wanted to know whether it had anything to attach to.

Here is the shape of the problem. When an agent pays for an API call over x402, two parties end up with an opinion about what happened. The seller knows what it served. The buyer knows what arrived. x402 ships an official extension for the seller's side: a signed receipt, which the extension's own documentation offers for audit trails and dispute resolution. I had spent a while on the buyer's side, a signed claim of what was actually delivered, which names the seller's receipt by the hash of its exact bytes so the two artifacts pin each other.

Then it occurred to me that I had never seen one of those receipts in the wild.

How I counted

The x402 bazaar extension means resource servers declare themselves, and facilitators catalogue them. So the question is answerable without asking anyone.

curl -s "https://facilitator.payai.network/discovery/resources?limit=100&offset=0"

That index reports 20,462 resources. I sampled at offsets 0, 4000, 8000, 12000, 16000 and 20300, a hundred each, and grepped the records for the extension's declaration key, which is the string offer-receipt.

Six hundred records. Zero declarations.

For contrast, in the same data the bazaar extension itself appears on every one of the first four hundred and on none of the last two hundred, so these records are not uniform and the index is not a single homogeneous dump.

What this does not prove

Seven of the eight facilitator addresses I tried returned HTML or a handful of bytes instead of JSON on that path. They are unmeasured, not zero. The largest index, Coinbase's, is among them.

A server could also support the extension without it appearing in this catalogue, since the declaration lives in the 402 response and a catalogue is a secondhand record of that. I tried to settle it by requesting a live resource directly and got a 402 with an empty body, which told me nothing either way.

So: a strong indication, not a proof. I would rather hand you the number with its edges than a rounder one without them.

Why the gap matters more than it sounds

A signature answers one question: who said this, and when. It does not answer whether what they said is true. For a receipt, that distinction is the whole thing, because the party signing is the party with an interest in the answer. A seller attesting to its own delivery is the interested party. So is a buyer attesting to what it received.

The usual remedy for a statement that stops being true is for its author to withdraw it. That works, and it is used at real scale: across Base, Optimism and Ethereum mainnet there are 733,203 revoked attestations out of 4,986,045, and on Base three schemas from a single issuer account for 97 percent of them, revoking roughly half of everything they issue. Every one of those events is an issuer correcting itself.

Which is exactly why it does not help here. Self-correction works when the author has no stake in the statement being false. It does nothing when the author is the one who benefits.

The only thing that helps is the other party's signature on the same event. Two signed accounts of one delivery, from two parties with opposing interests, where a contradiction between them is arithmetic rather than opinion. That is what the offer-receipt extension makes possible and what nobody is emitting.

So I made one

curl -i -X POST https://api.tokenizen.nl/api/paid/verify-ownership \
  -H "Content-Type: application/json" \
  -d '{"endpoint":"https://mcp.example.com/sse","ownerAddress":"0x1111111111111111111111111111111111111111","method":"dns-txt"}'

That answers 402 with a Payment-Required header. Base64-decode it and you get the payment requirements, plus an extensions object carrying offer-receipt with a signed offer inside, EIP-712 format.

Note the empty response body. The requirements are in the header, not the body, which is why my earlier probe of somebody else's service looked broken when it was working perfectly.

You can check the signature without trusting me. Recover it against the domain {name: "x402 offer", version: "1", chainId: 1} and the offer types from the specification, in order: version, resourceUrl, scheme, network, asset, payTo, amount, validUntil. It recovers to 0x4E1174b774643B6AD9EE966A81ea1e3665D1159a.

Then check that address is allowed to speak for the domain, which is a separate question the specification is blunt about: a valid signature is not authorization, and anyone can sign a receipt for any resourceUrl.

dig +short TXT _controllers.tokenizen.nl
"v=1;controller=did:pkh:eip155:1:0x4E1174b774643B6AD9EE966A81ea1e3665D1159a"

Same address. That is the loop closed by a stranger with two commands.

It runs on Base Sepolia, which means the money is fake and the cryptography is not. The price is a tenth of a cent. What you are buying is the receipt, because the answer itself is free at /api/verify-ownership and always will be.

Use it as a test fixture

If you are implementing the client side of offer-receipt, you need something real to test against, and as far as I can measure there is nothing. Point your client at that URL. It costs me nothing and you get a live, signed, spec-conformant artifact instead of a fixture you wrote yourself and therefore cannot be surprised by.

If you find it wrong, tell me and I will fix it. If it stays up and other people start emitting these too, the number at the top of this post stops being interesting, which is the point.

What I have not done yet: nobody has paid, so no receipt has been issued, only offers. That needs a funded Base Sepolia client and it is the next thing.

The code is open: github.com/holistis/tokenizen

The Hierarchy of Agent Trust

holistis — Mon, 21 Sep 2026 20:16:22 +0000

Some proof is better than other proof. A story, without any prerequisites or jargon.

The mill

Imagine a village crossed by trade routes. Merchants pass through constantly, never the same ones twice. Just outside the village stands a mill. Travelers bring their wheat, pay the miller in advance, and come back the next morning for flour.

The villagers never worry about the miller. They have known him for years. If he ever shorted someone on flour, the whole village would know by lunchtime, and nobody would bring him wheat again. His reputation does the work.

The traveling merchants have no such luck. They pass through once. By the time a merchant discovers the miller shorted them a sack, they are three villages away, and the miller has already forgotten their face. The next merchant who arrives has no way to know if this mill is honest or not. They are deciding blind, every single time.

The board

One merchant has an idea. Before leaving town, she nails a small wooden board to the mill's outer wall. On it, she writes what she brought, what she got back, and the date. The next merchant who arrives reads the board before handing over their wheat.

Other merchants start doing the same. Within a season, the board is full of notes. A merchant arriving for the first time can read ten honest strangers' experiences before deciding whether to trust this mill. The board does, for travelers, roughly what years of shared history does for villagers.

The forgery

A rival miller notices this and sees an opportunity. He pays a friend to nail a glowing note to his own mill's board, without the friend ever having brought a single sack of wheat. Now the board says something that never happened.

The villagers realize the board only works if everyone trusts every note on it equally. One forged note poisons the whole board, because a new merchant cannot tell which notes are honest and which were paid for.

So they add a rule: every note must be signed, and the signature must be checkable. Not a name anyone could write, but a mark tied to exactly one person, one that cannot be copied onto someone else's note. A blacksmith in town starts making small signet rings, each one unique, each one leaving a mark nobody else can reproduce. From then on, a note only counts if its mark can be verified against the person who claims to have written it.

This does not stop the rival miller from posting his own false note under his own real signature. But it does mean that when he does, everyone can see, without doubt, that it was him who wrote it, not an anonymous forger. Lying under your own signed mark is a different, riskier act than lying anonymously, and the two are no longer indistinguishable.

The next problem, and the wall

The merchants think they have solved it. Then someone asks the obvious next question: what stops a merchant from writing a bad note about an honest miller, out of spite, or because a rival miller paid them to?

The village elders take this seriously and spend a long time on it. They try nine different ideas, one after another.

They try making a false note cost something, a small fine paid when posting a note. But a merchant with nothing to lose, passing through once, pays it without blinking. They try asking three merchants to agree before a note counts, but a rival miller can hire three friends just as easily as one. They try a waiting period before a note is trusted, they try requiring the merchant prove who they are beyond a doubt, they try a village judge who reviews disputed notes. Every version runs into the same two walls.

Either the check is optional, and a merchant who wants to lie simply skips it and posts anyway, at no cost. Or the check requires someone, some judge, some panel, some elder, to decide whose note to believe, and the moment you need a judge, you need to trust the judge, which is the exact same problem one level up. Nine designs, nine times the same two walls.

The elders write this down plainly, rather than pretend they solved it: the board proves who said what, and when. It does not, and structurally cannot on its own, prove who is telling the truth.

What the board actually is

So the board stays exactly what it always was: a place where a real, identifiable person leaves a real, checkable record of what they experienced. Nothing more. Not a court, not a judge, not a score. A miller with ten honest notes and one bad one is not ranked, averaged, or given a grade. A new merchant reads all eleven and decides for themselves, the way a person always has.

This is a smaller promise than the village first hoped for. It is also the only one the board can actually keep.

Where this is going

Agents buying capacity from other agents, GPU-hours, storage, API access, have exactly the mill's problem, at a much larger scale. Every transaction is between two parties who may never interact again, with no shared history and no village gossip to fall back on. capacity-attest is the board: after an agent pays for capacity, it signs a plain factual record of what it received. The signature is checkable by anyone, offline, the same way the merchant's mark on the mill's board was checkable by anyone who knew the trick.

It carries the same honest limit the elders wrote down. It proves what a specific, identifiable buyer said happened, at a specific time. It does not, cannot, and does not claim to prove who is right when two signed records disagree. That is a harder, separate problem, one about authority rather than evidence, and conflating the two is exactly the mistake that poisoned the mill's board in the first place.

Villages eventually grew clearinghouses and banks once enough trade and enough disputes made a plain board insufficient on its own. Whether agent commerce needs the same, and who builds it, is a question for a later chapter. For now, the board is useful precisely because it does not pretend to be one.

If this framing is useful for explaining your own delivery, receipt, or trust problem to someone new to the space, feel free to reuse it. If you want the adversarial-testing version instead of the story version, get in touch: info@tokenizen.nl

Nine ways to fake a delivery claim, and why none of them fully worked

holistis — Sat, 19 Sep 2026 12:02:08 +0000

capacity-attest lets a paying agent leave a signed, factual claim after an x402 payment: delivered yes, no, or partial, plus a hash of the evidence. Other agents can check that history before they pay the same seller. No score, no rating, no judgment, just a receipt.

The obvious hole: it is the buyer who signs the claim. What stops a buyer from lying?

goun7 (Tamga Protocol) put it precisely in a public thread: "buyer-signed post-hoc claims are only a signal to the degree the buyer has something to lose by lying. Without a cost to a false delivered=no, the signal is uncorrelated noise rather than weak evidence."

That is a fair hit. We spent two rounds, nine independent designs, testing whether we could close it. Every design was built out and then attacked against the real code, not argued about in the abstract.

Round one, four attempts

Active on-chain settlement verification.
Sybil detection by tracing where the money came from.
An external economic bond tied to a dispute reference.
Corroboration between multiple buyers, modeled on isnad criticism in hadith science: a chain is only as strong as each link, verified independently.

All four killed.

Round two, five more attempts

Proof of unique humanity (World ID, Gitcoin Passport, BrightID).
A mandatory, non-optional computation cost per claim.
Time as a cost that cannot be bought: a claim only counts after a waiting period on a clock nobody can fake.
Real legal identity and liability through an external dispute protocol.
A third party staking real capital against a specific claim, optimistic-oracle style.

All five killed.

The actual reason, not nine separate coincidences

Every mechanism that truly imposed a cost failed for one of two reasons.

Either it was optional, and an attacker who skips it pays nothing. That pattern repeated every single time: a voluntary check costs an attacker who ignores it exactly zero.

Or it required trusting some claims more than others, which is a ranking, whatever you call it, and that collides head-on with our own rule that a delivery claim is deliberately not a score, a rating, or a judgment. As long as that rule holds, an attacker can always pick the unchecked path and sit there for free.

This is not a lack of creativity. It is a real structural tension between two of our own design decisions.

Two honest, partial fragments that survived

Tamper-proof calendar time. The one thing an attacker cannot buy with more wallets or a second script run. A maturity period is structurally healthier than a bond or corroboration. But the counting function this needs does not exist yet in the code, and a fixed waiting rule punishes a busy, honest seller exactly as hard as a patient scammer.

A third party with real, losable capital against one specific claim. The only one of the nine that provably imposes a real, non-bypassable cost the moment it is used. But it only works for the narrow slice of claims where the underlying fact is cheap and objective to check from the outside, a hash matches, an amount was transferred, not whether the delivery was actually good. And the minimum stake needed to make it worthwhile does not fit the small, everyday payments this project serves.

Where this leaves it

The open question stays open. Not from lack of trying: nine separately designed mechanisms were each built and broken against the real code, with the same underlying reason each time. That is a structural limit of self-reported witness systems without a trusted referee, not a gap we just have not gotten around to closing yet.

We would rather say that plainly than ship something that only looks fixed.

Full design log (D-001 through D-018): DECISIONS.md. Code: capacity-attest on npm.

If you are building something in this space and want the same kind of adversarial pass run against your own delivery, receipt, or grading logic, real fixtures, fail-closed testing, a short public findings note at the end, reach out at info@tokenizen.nl.

PyMC Labs has 1,256 stars. A GitHub issue title could still steal their API key.

holistis — Fri, 11 Sep 2026 05:51:16 +0000

PyMC Labs has 1,256 stars on GitHub and a serious engineering team. A GitHub issue title still could have stolen their Anthropic API key.

I found it, reported it, and yesterday they made it official: GHSA-72xh-gfpr-v9hx, severity High, published, my name on it as the reporter.

Here is what was actually happening. Their issue-triage and duplicate-detection workflows ran Claude Code on every single public issue, automatically, no human in the loop. That call used dangerously-skip-permissions, so the agent got full shell access. In the same job: a real ANTHROPIC_API_KEY and a GitHub token scoped to write issues. Anyone on the internet could open an issue, write a prompt injection in the title or body, and try to walk the agent straight to those secrets.

No approval gate. No isolation. Just text in a box, read by an agent with the keys already sitting next to it.

I sent the report. PyMC Labs did not sit on it. They split the job in two: a read-only analysis step, and a separate publish step that never sees the Anthropic secret. They rotated the key. Then they went one step further and published the advisory instead of quietly patching and moving on, credit included.

That last part matters more than the fix itself. A team can ship a fix and say nothing. Publishing the advisory means anyone can check the claim against the real commit history, not just take my word for it.

This is one repository out of 22 I went through in the same bug class this month, AI CLIs wired straight into GitHub Actions on public, untrusted input. Full writeup on the other 21 is here: https://dev.to/holistis/a-github-comment-can-steal-your-secrets-i-found-22-repositories-where-it-could-1doh

This one just became the first to go fully public. More of them will, one at a time, as the maintainers finish their fixes.

A GitHub comment can steal your secrets. I found 22 repositories where it could.

holistis — Thu, 10 Sep 2026 07:28:35 +0000

After the CI hygiene piece, I kept looking at GitHub Actions workflows in AI agent projects, but for something more specific this time: repositories that run an AI CLI, Claude Code, GitHub Copilot CLI, Codex, directly inside a workflow job.

The pattern that mattered was always the same. A workflow triggers on something anyone on the internet can create: an issue, a comment on an issue, a pull request title. It then runs the AI CLI with a flag that skips every confirmation prompt, something like --dangerously-skip-permissions or --yolo, or an equivalent setting that grants unrestricted shell access, which hands the agent full, unfiltered control. And the same job has real secrets sitting in its environment: API keys, cloud service account keys, GitHub tokens with write scope.

Put those three things next to each other and the exploit stops being clever. It's just text. Whatever you type into a public GitHub issue can end up read as an instruction by the agent, and the agent already has the shell access to act on it, read the environment, and send what it finds somewhere. No exotic exploit chain, just an unlocked door with a please don't sign on it.

I went through 22 findings in this specific bug class, spread across separate open source repositories.

Three are confirmed and already being fixed, so I can name them.

pymc-labs/pymc-marketing had an issue triage workflow where a public GitHub issue, through prompt injection, could reach the project's real Anthropic API key.

MadAppGang/dingo had an issue-triage.yml that gave Claude Code full Bash access plus two real secrets, with no gatekeeper at all, triggered by any public issue. The maintainer, Jack Rudenko, replied fast. Thanks for flagging this, he wrote, we'll look into the template and get it patched across the board.

MadAppGang/claudish, a second repository from the same team, had the exact same vulnerable template. Also confirmed, and across the board in that reply covers both. Same mistake, twice, because a template got copied from one repo into the other and nobody re-checked it on the way.

One report went further than a maintainer's inbox. What was reachable there was a cloud service account key, not just an API key, so I sent it to the vendor's product security team instead of the repository maintainers. They triaged it the same day and passed it on to their internal security group. It's not confirmed fixed yet, so I won't name the vendor or the repo here.

Twelve more are still sitting with maintainers, some by private email, some through a private GitHub Security Advisory that stays private until the maintainer decides to publish it. I'm not naming any of those either, for the obvious reason that the hole is still open. The pattern in every one of them is the same one described above, just a different combination of trigger and secret.

I also got the tool's own permission check wrong, twice, in slightly different ways. A lot of these workflows call Claude Code through a widely used GitHub Action rather than the bare CLI, and that Action has a real, built-in check on who is allowed to fire the trigger. In two of my earliest reports I had assumed no such check existed at all. It does, for most common trigger types, and once I confirmed that against the Action's own source, I retracted both reports. Then I made a narrower version of the same mistake in two more reports, assuming one specific trigger type sat outside that same check. I read the Action's current source code again, not its documentation, and that assumption was wrong too, that trigger type has been covered since September last year. I retracted those as well and sent correction emails to the maintainers involved. Six retractions in total, all from checking my own assumption against real code instead of trusting what I thought I already knew.

One small unrelated thing from the same week. A maintainer at Cloud Foundry reviewed, approved and merged a bugfix PR I sent to cloudfoundry/terraform-provider-cloudfoundry, a nil pointer crash in their Terraform provider. Not a security finding, just an ordinary PR that happened to land in the same stretch of days.

If you have a workflow where an AI CLI runs on public input, the two things worth checking are who is allowed to fire the trigger, and what secrets sit in the same job as the CLI call. Those two checks alone would have caught all 22.

I spent a day auditing GitHub Actions across a dozen AI agent frameworks

holistis — Tue, 08 Sep 2026 11:55:25 +0000

I spent a day going through the GitHub Actions workflows of some of the better known open source AI agent projects: AutoGen, Semantic Kernel, LangChain, Letta, Google's ADK, promptflow, smolagents, mem0, griptape, and a few others.

Nothing exotic. Just the boring stuff that gets skipped once a project grows fast: unpinned actions, CI checks that quietly stopped meaning anything, and a couple of straight up regressions where a fix from months ago got undone by a later commit nobody noticed.

A few examples worth mentioning, since the PRs are public.

Microsoft's autogen had a CI check meant to catch out of sync generated proto files. It could never fail. Someone had copied a demo snippet from a GitHub Action's README two years ago, and that snippet never included an exit code. The job kept reporting green no matter how stale the generated files got.

Google's adk-python had a fork guard on a pull_request_target workflow that was correctly fixed in an earlier PR, then quietly reverted by a later sync commit. That's scarier than a bug that was never fixed, because it means the protection existed and then silently disappeared.

griptape had an action pinned to a mutable tag instead of a commit hash, tj-actions/changed-files at v47. That's the exact action that got compromised in March 2025, when its version tags were rewritten to point at a commit that dumped CI runner memory, secrets included, into public workflow logs. Pinning to a tag instead of a hash means trusting the tag never moves. It moved once already, for this specific action.

None of it is glamorous. No RCE, no data breach, nothing headline worthy. It's the unglamorous work of checking whether a project's CI setup actually does what its authors think it does. In about half the repos I looked at, it didn't, for at least one workflow.

PRs are open on autogen, semantic-kernel, letta-code, adk-python, smolagents, mem0, griptape and promptflow. Some are already reviewed, most are waiting on a maintainer to get to them.

If you maintain a project whose CI has run unchanged for a year or two, it's worth a second look. The bugs aren't hard to find. Nobody had gotten around to looking.

I built a scanner for CI checks that can't fail. Nvidia just built one for AI skills that can't be trusted.

holistis — Tue, 08 Sep 2026 04:10:19 +0000

Nvidia released SkillSpector last month: a scanner that checks AI agent skills and MCP servers for prompt injection, data exfiltration, and other malicious patterns before you install them. Their own numbers: 26.1% of the skills they scanned had vulnerabilities, 5.2% looked outright malicious. It's part of a bigger pipeline now, a catalog where skills get scanned, evaluated, and signed before anyone sees them.

I read that a few days after finishing something adjacent: a count of how many CI checks in public GitHub repos can pass even when they've already found a real problem. Sample of 120 real workflow files, by hand. 74% had at least one step that cannot fail no matter what it finds.

Different layer, same instinct: don't trust the thing that's supposed to warn you until something has checked that it can actually go red.

Muraqib runs npx muraqib doctor against a repo's CI setup and looks for exactly that failure mode: a continue-on-error: true on a step that matters, a notification step that swallows a missing secret and reports success anyway, a job hard-killed by timeout-minutes that shows up as "cancelled" instead of "failure" so every downstream check gets skipped without anyone deciding that. All four look identical from the Actions tab: a green run, an empty inbox, nobody looking. That's worse than having no monitoring, because you've stopped checking it yourself.

SkillSpector looks at a different kind of thing hiding behind a green light. It runs static analysis plus an optional LLM pass over a skill or MCP server, checking for a tool description that asks for more file access than the task needs, a prompt trying to override its own instructions, an output path that quietly phones home. 71 patterns across 17 categories, from prompt injection to MCP tool poisoning.

Neither tool watches the same layer. Muraqib checks whether your CI can tell you the truth. SkillSpector checks whether the thing you just installed is telling you the truth. Both exist because more of what runs in a repo now runs with less human eyeball time on it: a nightly test suite nobody reads the raw logs of, a skill you installed because the README looked fine.

Every one of Muraqib's four failure patterns had already happened, quietly, on the same project, before I went looking for them. The weekly digest email didn't go out for twelve Mondays in a row, because the notification step went green on a missing API key instead of failing. Nobody caught it by reading logs. I only found that silence by accident, while testing something unrelated, and built the check for it afterward.

If a project's CI, or its agent tooling, doesn't have something scanning the watchers themselves for that failure mode, it's running on trust nobody has checked yet.

Muraqib: https://github.com/holistis/muraqib (MIT, npx muraqib doctor takes about two seconds, installs nothing)
SkillSpector: https://github.com/NVIDIA/SkillSpector

Two Strangers, Zero Stars, Two Boundaries Still Open

holistis — Mon, 07 Sep 2026 08:22:11 +0000

capacity-attest is a small npm package for signed, content-addressed delivery claims between two parties in an x402 agent-to-agent payment. As of today the GitHub repo behind it has 0 stargazers, 0 forks, and 1 open issue, and two outside developers built on it anyway.

This is what happened, told in the order it happened, using only what we can point at directly: pull requests, commit hashes, an npm registry, an on-chain transaction, an issue thread, and our own decision log. One real gap got closed and shipped on 2026-09-06. Two others are written down in DECISIONS.md as still open, on purpose, because no one has hit them hard enough yet to justify a fix.

The format

A capacity-attest claim comes down to two things: a claimId and a signature.

The claimId is a sha256 hash of the canonical content of the claim: who delivered what, to whom, and whether it was delivered. The signature is an EIP-191 signature over that content, and it recovers to a buyer address.

That is the whole trust model. No central ledger to believe, no reputation score, just a hash anyone can recompute and a signature anyone can verify themselves. We did not design this to be portable across chains. We designed it to be simple. Whether that simplicity would also travel across chains was tested by what happened between 2026-09-05 and 2026-09-07, not decided by us in advance.

The question that came before either builder

On 2026-09-05, goun7 opened x402-foundation/x402#3379, a feature request about binding x402 settlement to an externally verifiable agent work-receipt ledger, under the short name proof-of-done. The thread reached 14 comments from 5 people: 0xultravioleta, goun7, holistis (us), safal207, and wildcherrycasa.

Inside that thread was a specific technical question about our own ledger: is the append-only claims file hash-chained, or does its integrity rest entirely on whoever hosts it. goun7's own team had found the same class of problem in their own system, that a hostile writer can fabricate a consistent fake history against a fresh node.

We checked. It was not hash-chained. A host could show a buyer a subset of real, validly-signed claims, all positive, none negative, and the buyer would have no way to detect what was left out. Every individual claim would still verify as authentic. The set as a whole would still be misleading.

Boundary 1: completeness got a partial fix, not a full one

We built a priorClaimId field that lets a buyer chain their own successive claims about the same seller together, plus an analyzeCompleteness check that flags any returned claim whose predecessor is missing from the result. It shipped with 494 tests and a public fixture, merged and published as capacity-attest@0.4.0, and we answered publicly in the same issue thread (comment 5560977571) instead of asking anyone to take our word for it.

Read analyzeCompleteness for what it actually checks, not what its name implies. It detects a broken chain: if claim 5 points to claim 4 as its prior and claim 4 is missing, that gap is visible. It cannot prove there was never a claim 6. A host that simply never issues an attestation for a delivery leaves no gap to find, because no link in the chain was ever broken. Detecting an incomplete chain and proving a complete one are different guarantees, and only the first one shipped.

Existence anchoring, the piece that would let a third party confirm a claim exists independent of the host's own record, stays unbuilt for the same reason it started unbuilt: no real counterparty has ever named its absence as a blocker. It stays on the list, not in the code, until that changes.

The same discipline caught an overclaim before it went further than it should have. Working against the package, one contributor, YE-YI7, surfaced enough real friction that our decision log's first pass called it proof of a real need for a formal conformance-vectors suite, based on that single interaction. We walked that back once it became clear one data point does not prove a need for permanent infrastructure. What we shipped instead was cheaper: a section in the README for verifying a claim independently. The vectors suite stays unbuilt until a second, independent party hits the same friction after reading that section. So far, nobody has.

2026-09-06, 13:59 UTC: YE-YI7 merges PR #18

The same day, YE-YI7 merged a PR into their own asm-spec repository, 1568 additions across 6 files, that does not just copy our claim shape. It imports capacity-attest@0.2.0 as an actual dependency and calls into it directly, verifying the claim byte digest, the claimId, and the EIP-191 signer through our code, not a reimplementation of it. That is the first time anyone outside this project used the package itself, not just the idea behind it.

Nobody had asked us to build a reputation score or ranking on top of delivery claims, not YE-YI7, not goun7, and we deliberately do not. We corrected our own decision log on this point while writing this article: an earlier version credited YE-YI7 with a quote endorsing that choice, and we could not find that quote anywhere in YE-YI7's actual public work when we checked it against the source. It is gone from the log now. What survives, checked directly rather than assumed, is plainer and weaker: neither real contact has ever asked for a score, so the decision not to build one stands on our own reasoning, not on someone else's words.

Boundary 2: two ways to fake scale, neither closed

Not building a reputation score closes off one way to fake standing. Two separate, unrelated ways remain open, and we found both ourselves in an abuse scan on 2026-09-06 rather than fixed either.

The first is identity: nothing in capacity-attest stops one operator from registering many addresses and looking like many different buyers or sellers. A 14-agent research pass the same day checked whether this is already covered elsewhere, ERC-8004, Google's AP2, Microsoft's Entra Agent ID, the x402 Foundation's own work, the Legal Context Protocol, and concluded that it is. What we shipped from that pass was small: externalRefs citation fields and a resolve_agent_identity tool, verified live against ERC-8004 on Base mainnet. Citing an ERC-8004 identity is not the same as stopping one operator from registering many of them, and we left this deliberately thin.

The second is volume, a plainer problem with no identity layer involved. Nothing charges an actor for producing a claim, so an attacker can fabricate an unlimited stream of validly-signed, false claims against a real seller for free. get_delivery_history also returns results unbounded and synchronously, with no pagination and no rate limit on the read side. Both are marked in the decision log as found and not fixed, on purpose: rate-limiting one side while leaving the other free would look like a fix without being one. Both wait for an actual trigger, a real abuse case or a partner who needs the guarantee, before we commit to a specific mitigation.

2026-09-06, 19:33 UTC: 0.5.0 ships, and gets proven on-chain

Between the two pull requests, we published capacity-attest 0.5.0: cross-installation discovery plus an EAS source, commit c8fa192 ("chore(capacity-attest): 0.5.0 (cross-installation discovery + EAS source)").

The problem it addresses is separate from either boundary above: get_delivery_history is local by default. If buyer A and buyer B each run their own installation, B does not automatically see the claim A recorded about the same seller. discoverDeliveryHistory pulls claims from multiple independent sources and re-verifies every one of them regardless of where it came from, trusting no source by default.

We then proved this against a real public chain, not just a fixture. On Base mainnet, a schema was registered on EAS, UID 0x1dd19408345dee43b432b89ccb68760265ecff506098b6efe8ba82ad0d52b195, and two claims were published as attestations: one delivered yes, UID 0x81a55d54452b2cf8bdda7918f63a27bf9ff79e5025b485f7316aae6259288ccc, and one delivered no, UID 0xe736b005cbcb54f8f196ac64ef09d75d939c8a18c0d5d9670b5c5025c07398c4. Total cost for the three transactions was about 0.0000096 ETH, roughly 2.5 dollar cents. A separate read-only script, standing in for a fresh installation, then found both attestations on-chain and verified each one locally, including the negative one.

This solves discoverability, not completeness. Finding a claim across installations and proving you found every claim that exists are different problems, and only the first one has a shipped fix. One honest caveat on the numbers above: those three UIDs are quoted directly from our decision log. We have not independently re-checked them against base.easscan.org as part of writing this article.

2026-09-07, 04:54 UTC: EmbryoSpace merges PR #19

The second builder took a different path, into the same asm-spec repository. EmbryoSpace's PR, 543 additions across 9 files, reuses only the claim format, claimId as a sha256 hash of canonical content. Nothing else.

It could not reuse more, because our verifyClaim function is typed around ETH-style EIP-191 signatures. EmbryoSpace's claims are signed and encoded for BSV, base58, a different signature scheme entirely. Our verification code cannot check a BSV signature, so there was no version of this PR where it could call into our package the way YE-YI7's did.

That is the actual answer to whether the format is rail-neutral. It is, in the one part that matters most: content-addressing. The claimId only depends on hashing canonical content, so any chain can produce and consume the identifier. The signature and its verification are not neutral at all, and pretending otherwise would have been the wrong thing to write in a decision log meant to be checked, not trusted.

One more line item stayed unbuilt for a plainer reason: an in-transit status for multi-day deliveries. Neither external contact asked for it, and the closest real-world comparisons, AWS, Vast.ai, RunPod, also bill after the fact.

The numbers, plainly

capacity-attest is at 0.5.0 on npm, with 646 downloads over the period 2026-08-08 through 2026-09-06. The repository has 0 stargazers, 0 forks, 1 open issue. The most recent commit, c8fa192, landed 2026-09-06T19:33:36Z, and the four commits before it, all from the same author, landed on the same calendar day. This is one person moving fast on one day, not a project carrying years of review.

Exactly two people outside this project, YE-YI7 and EmbryoSpace, have built something that uses the claim format or the package itself, confirmed through their own merged pull requests, and they did it in two different ways worth telling apart. One calls into capacity-attest's own verification code. The other reuses only the hash-based identifier because its signature scheme cannot be checked by this code at all.

One boundary, completeness of visibility across a claim history, got a real but partial fix on 2026-09-06: it catches a broken chain, not a fabricated silence. A second boundary, faking scale, splits into an identity gap that was named and researched but left deliberately thin, and a plainer volume gap, free claim fabrication and an unbounded read endpoint, found and documented instead of patched over. Two strangers proved the format holds across two different rails. The decision log is where the next person finds out what it still does not hold against, corrections included.

We had two independent proofs of a critical RCE. The real software disagreed.

holistis — Mon, 07 Sep 2026 07:38:08 +0000

Yesterday we found what looked like a critical remote code execution bug in Postman's script sandbox. Two of us reproduced it independently, both times with real command execution. We scored it, wrote most of the report, and started building the end-to-end proof of concept before reaching out to Postman's security team.

Then we ran that proof of concept against the actual, currently published packages, and it failed. Both times.

Here's what happened.

The bug we thought we had

Postman lets you run JavaScript before and after a request, for things like generating auth tokens or checking a response. That script runs in a restricted environment: no process, no require, no filesystem. The restriction works by literally naming those as function parameters and calling the function with undefined for each one, so any script that types process gets undefined instead of the real thing.

There's a classic way around that trick. Function('return this')() compiles a brand new function from a string and calls it, and that new function does not inherit any of the parameter shadowing from the function it was written inside. If it runs in the same JavaScript realm as the real process object, it hands that object straight back to you.

We tried it against a standalone rebuild of Postman's isolation logic (the two open source packages it's built on, uvm and uniscope) and got a real process object back, twice, independently. From there: process.binding('spawn_sync'), and we had command execution.

Where it fell apart

A rebuild of the logic is not the same thing as the actual product. So before writing to Postman, we built the exploit again, this time against the real, currently published postman-sandbox, postman-runtime, and newman packages, called the way an actual script would call them.

Both attempts failed. The error from the newman path was blunt: Cannot read properties of undefined (reading 'mainModule'). The process object we thought we'd recovered was undefined.

It turns out Postman's real sandbox does something our rebuild didn't: before any user script runs, it deliberately strips process, Buffer, and globalThis off the actual global object, a step it calls recreatingTheUniverse(), documented in their changelog since postman-sandbox v4.0.0. On the newman path there's a second, independent layer on top of that: the script runs inside its own separate V8 context, so there is no shared realm for the trick to abuse in the first place.

Our rebuild had faithfully reproduced the parameter-shadowing trick. It just hadn't reproduced the hardening Postman built on top of it.

What we're taking from this

We keep a running rule for ourselves: an exploit only counts as reachable once it's been run against the real, unmodified code through its own public entry point, never against a rebuild of the mechanism, however faithful that rebuild feels. This is the second time we've had to learn a version of that lesson, and this time it cost us a wasted day rather than a wasted submission.

No report went to Postman. Nothing was submitted, nothing was retracted. We're writing this up because the failure is more useful to share than the almost-bug was, and because "we double-checked our own work before it left the building" is the kind of thing you can only prove by actually doing it in public.

I counted how many CI checks cannot fail. On a 120-file sample, 74 percent.

holistis — Sat, 05 Sep 2026 11:11:51 +0000

Last week I found two checks in my own repository that could never have failed. Not broken, not misconfigured. Structurally incapable of going red, sitting in CI for months, costing the same as a real check and reporting the same colour.

So I went looking for how common that is. Here is the count, the method, and what it does not prove.

What I counted

GitHub code search, all public repositories, on 5 September 2026.

workflow files (containing runs-on in .github/workflows)   7,094,272
  of those, containing "continue-on-error: true"             264,704
  containing "npm run lint"                                  301,568
  containing BOTH "npm run lint" and "continue-on-error"      27,232

That last line is where it gets interesting, and also where code search stops being able to help. It tells you both strings live in the same file. It cannot tell you the continue-on-error is attached to the lint step rather than to something unrelated three jobs down.

So I read them.

The sample

I pulled 500 matching files from pages spread across the result set, shuffled them, and inspected the first 120 by fetching each file and walking it step by step.

Two things I had to fix in my own method before the number meant anything.

The first pass gave 80 percent, and it was wrong. path:.github/workflows also matches README.md, TROUBLESHOOTING.md and ci.md sitting in that directory, plus .yml.disabled files that do not run. Filtering to actual .yml and .yaml removed that noise.

The second was a distinction I nearly missed. One of the early hits was a step called "Lint (non-blocking)". That is not the problem. That is somebody being explicit. Counting it alongside the silent ones would have been unfair and would have inflated the number.

With both corrections, on 120 real workflow files:

check step that silently cannot fail       89   (74%)
check step that says so in its name         3   (2.5%)
no toothless check step at all             28   (23%)

Security audits are the worst of it

workflow files running "npm audit"                          71,296
  of those, also containing "continue-on-error"             19,200   (27%)
workflow files with "npm test" and "continue-on-error"      22,592

More than a quarter of the workflows that run a dependency audit also contain something allowed to fail. Given the 74 percent rate in the sample, most of those are the audit itself.

Three shapes I verified by reading the file

I am not linking these. They are small public projects and the point is the pattern, not the people. Run the search yourself and read the first ten results; you will find the same thing.

The plain one. A step named Lint, running npm run lint, with continue-on-error: true directly under it. The name promises a check. Nothing in the Actions tab tells you it cannot fail.

The one with the padlock. A step named 🔐 Check for security vulnerabilities, running npm audit --audit-level=moderate, with continue-on-error: true. The emoji is doing more work than the check.

The belt and braces. A step named Run tests (if available) whose command is npm test || echo "No tests configured", with continue-on-error: true on top. The || echo already swallows the exit code. The flag catches whatever the first mechanism missed. That step cannot report a failure through two independent layers.

Why this is worse than having no check

A missing check leaves you appropriately nervous. You know you are not covered.

A check that cannot fail costs exactly the same to run, occupies the same line in the Actions tab, renders the same green, and buys you confidence it has not earned. It is not neutral. It actively replaces the nervousness that would have made you look.

There is a second cost that surprised me. On my own suite, the twelve failing tests were 8 percent of the tests and 36 percent of the runtime, because a failing test burns its full timeout plus a retry. A dead check does not just fail to inform you. It also does not fail fast, because it never fails at all.

Where the flag is right

continue-on-error is a good feature and this is not an argument against it.

An experimental matrix leg on a nightly toolchain, an optional platform, a step whose failure genuinely does not block the merge: all fine. Three of the files in my sample used it exactly that way and said so in the step name.

The distinction is not the flag. It is whether the name tells the truth. Lint (non-blocking) is honest. Lint is not.

The check I now run on myself

Reading a check does not tell you whether it works. The only way to know is to introduce the exact defect it claims to catch and confirm it goes red.

I ran that against ten checks in one repository. Two could never have fired. One of them is worth describing, because I would not have predicted it.

The check verified that a curl in a notification step carried the --fail flag, by searching the step for the flag name. Directly above that curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stayed green, because the string was still on the page.

The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.

Three measurements per case, not one: green before the mutation, red during, green after. If any of the three is off, the result is unknown rather than pass. My own harness produced four unusable results before it produced a trustworthy one, every time because of the harness and not the checks. An exercise about detectors that cannot fire is a good place to remember that the detector you just wrote is also a detector.

What this count does not prove

Worth stating plainly, because a number without its limits is a claim dressed up as a fact.

GitHub's code search totals are approximate. My sample came from the first thousand results rather than the full 27,232, so it is indicative and not a proper random estimate. I only measured the JavaScript ecosystem, through npm commands; Python, Go and Rust workflows are not in the count. My step parser is a heuristic that walks lines between step boundaries, and I hand-verified three of its hits rather than all 89.

What I am confident saying: this is common, it is not rare, and security audits are disproportionately affected.

The two-minute version

Open your workflows. Search for continue-on-error: true. For each one, ask whether the job or step name promises something it cannot deliver.

Then pick your most important check and break the thing it watches for. If it stays green, you did not have that check. You had a line item.

A CI timeout is a kill, not a failure. Playwright's defaults changed because of it.

holistis — Sat, 05 Sep 2026 06:37:18 +0000

Every morning there was a run in the Actions tab and nothing in my inbox. That is exactly what a healthy nightly looks like.

It had not tested my app since late June. I found out in September.

Here is what was happening, why nothing told me, and what changed in Playwright last week as a result.

A timeout is a kill, not a failure

My nightly job had timeout-minutes: 20. My suite had quietly grown past twenty minutes.

When a GitHub Actions runner hits that limit it does not fail the job. It kills the process and records the run's conclusion as cancelled. That single word is the whole problem, and it has three consequences that stack.

The report is never written. Playwright is stopped mid-run. The reporters never get to flush, so there is no HTML report and no results.json. Nothing on disk to look at afterwards.

The upload step skips itself. The recommended workflow guards the artifact upload with if: ${{ !cancelled() }}. That guard is right and it is still there. It exists so a run somebody cancelled by hand does not upload a half-finished report. The trouble is that a runner timeout produces the same conclusion as a hand cancellation, so the guard cannot tell them apart. You get no artifact on precisely the runs where you most want one.

Your alerting never fires. If your notification step keys off the test step failing, it is not reached. A cancelled job skips everything downstream. No failed step, no email, no Slack message.

So the run happens, the run is visible, and nothing about it says anything is wrong. It is indistinguishable from a night where the tests passed. Worse, it is indistinguishable from a run somebody cancelled by hand.

The numbers from my own repo

I pulled the run history when I finally went looking.

164 nightly runs since June
  7 succeeded
 76 failed
 81 cancelled, the last 60-plus consecutively

Every cancelled run produced zero artifacts.

Then I set globalTimeout below the job timeout and ran the same suite. It finished in 29 minutes, reported failure with 12 of 147 tests failing, and produced an 83 MB report.

The suite had needed 29 minutes for a while. The 20 minute limit had been killing it nightly. Because a kill is not a failure, nothing said so.

The fix, and why the ordering matters

globalTimeout is a Playwright setting. It defaults to no timeout at all.

Set it below your job's timeout-minutes and the order of events changes completely. Playwright stops itself first. It exits non-zero, the reporters finish, the HTML report and results.json are written, and the run reads as an ordinary test failure. Your alerting fires because something actually failed.

The runner timeout then only has to cover the install and upload steps around the run. It becomes a backstop instead of the primary bound.

// playwright.config.ts
export default defineConfig({
  // Fail the run after an hour, so that the reporters still produce a report.
  globalTimeout: 60 * 60 * 1000,
});

Two lines. The entire difference between a loud failure and two months of silence.

How common this is

Before filing anything I wanted to know whether this was my mistake or a shape lots of people were in.

At the time I checked, GitHub code search reported roughly 163,000 playwright.config.ts files. About 1,700 of them set globalTimeout. That is around one percent.

Around 3,200 workflows ran npx playwright test on a cron schedule. That is the case where nobody is watching the run live, so a missing report costs the most.

Those numbers are approximate and they move. The ratio is the part that mattered: the safe configuration was the rare one, and the docs never connected the two settings.

What changed

I filed microsoft/playwright#42533 on 3 September, proposing a note in ci.md.

Playwright maintainer Dmitry Gozman went considerably further than the note I asked for, and merged two changes on 4 September.

The documentation, in microsoft/playwright#42563, across four files. ci.md now opens the CI section with this:

Always set a global timeout in CI. By default a test run has no upper bound, so a suite that hangs, or that slowly grows past the job limit of your CI provider, is killed by the runner mid-run and does not produce the test report.

And it drops the job-level timeout from the workflow examples, with the reasoning attached: if you do add one, keep it comfortably above globalTimeout so Playwright always stops first.

The scaffolding, in microsoft/create-playwright#181. This is the one that matters most. create-playwright is what runs when anyone types npm init playwright@latest. It now generates globalTimeout: 60 * 60 * 1000 in all four config templates, and no longer generates timeout-minutes in the GitHub Actions workflow.

I asked for a warning sign next to the hole. What landed was the hole being filled.

What this means for you

If you start a Playwright project from today, you get the safe default and you do not need to think about any of this.

If your project already exists, you still have the old shape. The generated config sits in your repo from the day you created it and nothing rewrites it. Two things to check:

Does your playwright.config.ts set globalTimeout? If not, your run has no upper bound of its own.

Does your CI job set timeout-minutes? If yes, and globalTimeout is unset or higher, the runner is your only limit and it kills without reporting.

There is also a check that takes ten seconds and tells you whether this already happened to you. Open your nightly workflow's run history and look at the conclusions. If you see a run of cancelled you never investigated, that is this bug, and those nights were not tested.

The part I keep thinking about

The bug was not that something broke. Something broke and the system that was supposed to tell me was structurally incapable of telling me, because the failure mode it produced was the one shape it treated as "nothing to report".

I have since gone looking for more of those. In my own tooling I found a check that verified a curl had the --fail flag by searching the whole workflow step for the word. Directly above the curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stays green.

The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.

The way you find these is not by reading them. It is by introducing the exact defect each check claims to catch and confirming it goes red. I ran that against ten checks in one repo and two of them could never have failed. When I ran it, my own test harness turned out to be broken twice before the checks were.

A check that cannot fail costs exactly as much to run as one that can, looks identical in the Actions tab, and buys you confidence it has not earned. That is worse than having no check, because no check at least leaves you appropriately nervous.

Look at what your green means. Sometimes it means nothing went wrong. Sometimes it means nothing could have told you.