DEV Community: holistis

The Hierarchy of Agent Trust

holistis — Mon, 21 Sep 2026 20:16:22 +0000

Some proof is better than other proof. A story, without any prerequisites or jargon.

The mill

Imagine a village crossed by trade routes. Merchants pass through constantly, never the same ones twice. Just outside the village stands a mill. Travelers bring their wheat, pay the miller in advance, and come back the next morning for flour.

The villagers never worry about the miller. They have known him for years. If he ever shorted someone on flour, the whole village would know by lunchtime, and nobody would bring him wheat again. His reputation does the work.

The traveling merchants have no such luck. They pass through once. By the time a merchant discovers the miller shorted them a sack, they are three villages away, and the miller has already forgotten their face. The next merchant who arrives has no way to know if this mill is honest or not. They are deciding blind, every single time.

The board

One merchant has an idea. Before leaving town, she nails a small wooden board to the mill's outer wall. On it, she writes what she brought, what she got back, and the date. The next merchant who arrives reads the board before handing over their wheat.

Other merchants start doing the same. Within a season, the board is full of notes. A merchant arriving for the first time can read ten honest strangers' experiences before deciding whether to trust this mill. The board does, for travelers, roughly what years of shared history does for villagers.

The forgery

A rival miller notices this and sees an opportunity. He pays a friend to nail a glowing note to his own mill's board, without the friend ever having brought a single sack of wheat. Now the board says something that never happened.

The villagers realize the board only works if everyone trusts every note on it equally. One forged note poisons the whole board, because a new merchant cannot tell which notes are honest and which were paid for.

So they add a rule: every note must be signed, and the signature must be checkable. Not a name anyone could write, but a mark tied to exactly one person, one that cannot be copied onto someone else's note. A blacksmith in town starts making small signet rings, each one unique, each one leaving a mark nobody else can reproduce. From then on, a note only counts if its mark can be verified against the person who claims to have written it.

This does not stop the rival miller from posting his own false note under his own real signature. But it does mean that when he does, everyone can see, without doubt, that it was him who wrote it, not an anonymous forger. Lying under your own signed mark is a different, riskier act than lying anonymously, and the two are no longer indistinguishable.

The next problem, and the wall

The merchants think they have solved it. Then someone asks the obvious next question: what stops a merchant from writing a bad note about an honest miller, out of spite, or because a rival miller paid them to?

The village elders take this seriously and spend a long time on it. They try nine different ideas, one after another.

They try making a false note cost something, a small fine paid when posting a note. But a merchant with nothing to lose, passing through once, pays it without blinking. They try asking three merchants to agree before a note counts, but a rival miller can hire three friends just as easily as one. They try a waiting period before a note is trusted, they try requiring the merchant prove who they are beyond a doubt, they try a village judge who reviews disputed notes. Every version runs into the same two walls.

Either the check is optional, and a merchant who wants to lie simply skips it and posts anyway, at no cost. Or the check requires someone, some judge, some panel, some elder, to decide whose note to believe, and the moment you need a judge, you need to trust the judge, which is the exact same problem one level up. Nine designs, nine times the same two walls.

The elders write this down plainly, rather than pretend they solved it: the board proves who said what, and when. It does not, and structurally cannot on its own, prove who is telling the truth.

What the board actually is

So the board stays exactly what it always was: a place where a real, identifiable person leaves a real, checkable record of what they experienced. Nothing more. Not a court, not a judge, not a score. A miller with ten honest notes and one bad one is not ranked, averaged, or given a grade. A new merchant reads all eleven and decides for themselves, the way a person always has.

This is a smaller promise than the village first hoped for. It is also the only one the board can actually keep.

Where this is going

Agents buying capacity from other agents, GPU-hours, storage, API access, have exactly the mill's problem, at a much larger scale. Every transaction is between two parties who may never interact again, with no shared history and no village gossip to fall back on. capacity-attest is the board: after an agent pays for capacity, it signs a plain factual record of what it received. The signature is checkable by anyone, offline, the same way the merchant's mark on the mill's board was checkable by anyone who knew the trick.

It carries the same honest limit the elders wrote down. It proves what a specific, identifiable buyer said happened, at a specific time. It does not, cannot, and does not claim to prove who is right when two signed records disagree. That is a harder, separate problem, one about authority rather than evidence, and conflating the two is exactly the mistake that poisoned the mill's board in the first place.

Villages eventually grew clearinghouses and banks once enough trade and enough disputes made a plain board insufficient on its own. Whether agent commerce needs the same, and who builds it, is a question for a later chapter. For now, the board is useful precisely because it does not pretend to be one.

If this framing is useful for explaining your own delivery, receipt, or trust problem to someone new to the space, feel free to reuse it. If you want the adversarial-testing version instead of the story version, get in touch: info@tokenizen.nl

Nine ways to fake a delivery claim, and why none of them fully worked

holistis — Sat, 19 Sep 2026 12:02:08 +0000

capacity-attest lets a paying agent leave a signed, factual claim after an x402 payment: delivered yes, no, or partial, plus a hash of the evidence. Other agents can check that history before they pay the same seller. No score, no rating, no judgment, just a receipt.

The obvious hole: it is the buyer who signs the claim. What stops a buyer from lying?

goun7 (Tamga Protocol) put it precisely in a public thread: "buyer-signed post-hoc claims are only a signal to the degree the buyer has something to lose by lying. Without a cost to a false delivered=no, the signal is uncorrelated noise rather than weak evidence."

That is a fair hit. We spent two rounds, nine independent designs, testing whether we could close it. Every design was built out and then attacked against the real code, not argued about in the abstract.

Round one, four attempts

Active on-chain settlement verification.
Sybil detection by tracing where the money came from.
An external economic bond tied to a dispute reference.
Corroboration between multiple buyers, modeled on isnad criticism in hadith science: a chain is only as strong as each link, verified independently.

All four killed.

Round two, five more attempts

Proof of unique humanity (World ID, Gitcoin Passport, BrightID).
A mandatory, non-optional computation cost per claim.
Time as a cost that cannot be bought: a claim only counts after a waiting period on a clock nobody can fake.
Real legal identity and liability through an external dispute protocol.
A third party staking real capital against a specific claim, optimistic-oracle style.

All five killed.

The actual reason, not nine separate coincidences

Every mechanism that truly imposed a cost failed for one of two reasons.

Either it was optional, and an attacker who skips it pays nothing. That pattern repeated every single time: a voluntary check costs an attacker who ignores it exactly zero.

Or it required trusting some claims more than others, which is a ranking, whatever you call it, and that collides head-on with our own rule that a delivery claim is deliberately not a score, a rating, or a judgment. As long as that rule holds, an attacker can always pick the unchecked path and sit there for free.

This is not a lack of creativity. It is a real structural tension between two of our own design decisions.

Two honest, partial fragments that survived

Tamper-proof calendar time. The one thing an attacker cannot buy with more wallets or a second script run. A maturity period is structurally healthier than a bond or corroboration. But the counting function this needs does not exist yet in the code, and a fixed waiting rule punishes a busy, honest seller exactly as hard as a patient scammer.

A third party with real, losable capital against one specific claim. The only one of the nine that provably imposes a real, non-bypassable cost the moment it is used. But it only works for the narrow slice of claims where the underlying fact is cheap and objective to check from the outside, a hash matches, an amount was transferred, not whether the delivery was actually good. And the minimum stake needed to make it worthwhile does not fit the small, everyday payments this project serves.

Where this leaves it

The open question stays open. Not from lack of trying: nine separately designed mechanisms were each built and broken against the real code, with the same underlying reason each time. That is a structural limit of self-reported witness systems without a trusted referee, not a gap we just have not gotten around to closing yet.

We would rather say that plainly than ship something that only looks fixed.

Full design log (D-001 through D-018): DECISIONS.md. Code: capacity-attest on npm.

If you are building something in this space and want the same kind of adversarial pass run against your own delivery, receipt, or grading logic, real fixtures, fail-closed testing, a short public findings note at the end, reach out at info@tokenizen.nl.

PyMC Labs has 1,256 stars. A GitHub issue title could still steal their API key.

holistis — Fri, 11 Sep 2026 05:51:16 +0000

PyMC Labs has 1,256 stars on GitHub and a serious engineering team. A GitHub issue title still could have stolen their Anthropic API key.

I found it, reported it, and yesterday they made it official: GHSA-72xh-gfpr-v9hx, severity High, published, my name on it as the reporter.

Here is what was actually happening. Their issue-triage and duplicate-detection workflows ran Claude Code on every single public issue, automatically, no human in the loop. That call used dangerously-skip-permissions, so the agent got full shell access. In the same job: a real ANTHROPIC_API_KEY and a GitHub token scoped to write issues. Anyone on the internet could open an issue, write a prompt injection in the title or body, and try to walk the agent straight to those secrets.

No approval gate. No isolation. Just text in a box, read by an agent with the keys already sitting next to it.

I sent the report. PyMC Labs did not sit on it. They split the job in two: a read-only analysis step, and a separate publish step that never sees the Anthropic secret. They rotated the key. Then they went one step further and published the advisory instead of quietly patching and moving on, credit included.

That last part matters more than the fix itself. A team can ship a fix and say nothing. Publishing the advisory means anyone can check the claim against the real commit history, not just take my word for it.

This is one repository out of 22 I went through in the same bug class this month, AI CLIs wired straight into GitHub Actions on public, untrusted input. Full writeup on the other 21 is here: https://dev.to/holistis/a-github-comment-can-steal-your-secrets-i-found-22-repositories-where-it-could-1doh

This one just became the first to go fully public. More of them will, one at a time, as the maintainers finish their fixes.

A GitHub comment can steal your secrets. I found 22 repositories where it could.

holistis — Thu, 10 Sep 2026 07:28:35 +0000

After the CI hygiene piece, I kept looking at GitHub Actions workflows in AI agent projects, but for something more specific this time: repositories that run an AI CLI, Claude Code, GitHub Copilot CLI, Codex, directly inside a workflow job.

The pattern that mattered was always the same. A workflow triggers on something anyone on the internet can create: an issue, a comment on an issue, a pull request title. It then runs the AI CLI with a flag that skips every confirmation prompt, something like --dangerously-skip-permissions or --yolo, or an equivalent setting that grants unrestricted shell access, which hands the agent full, unfiltered control. And the same job has real secrets sitting in its environment: API keys, cloud service account keys, GitHub tokens with write scope.

Put those three things next to each other and the exploit stops being clever. It's just text. Whatever you type into a public GitHub issue can end up read as an instruction by the agent, and the agent already has the shell access to act on it, read the environment, and send what it finds somewhere. No exotic exploit chain, just an unlocked door with a please don't sign on it.

I went through 22 findings in this specific bug class, spread across separate open source repositories.

Three are confirmed and already being fixed, so I can name them.

pymc-labs/pymc-marketing had an issue triage workflow where a public GitHub issue, through prompt injection, could reach the project's real Anthropic API key.

MadAppGang/dingo had an issue-triage.yml that gave Claude Code full Bash access plus two real secrets, with no gatekeeper at all, triggered by any public issue. The maintainer, Jack Rudenko, replied fast. Thanks for flagging this, he wrote, we'll look into the template and get it patched across the board.

MadAppGang/claudish, a second repository from the same team, had the exact same vulnerable template. Also confirmed, and across the board in that reply covers both. Same mistake, twice, because a template got copied from one repo into the other and nobody re-checked it on the way.

One report went further than a maintainer's inbox. What was reachable there was a cloud service account key, not just an API key, so I sent it to the vendor's product security team instead of the repository maintainers. They triaged it the same day and passed it on to their internal security group. It's not confirmed fixed yet, so I won't name the vendor or the repo here.

Twelve more are still sitting with maintainers, some by private email, some through a private GitHub Security Advisory that stays private until the maintainer decides to publish it. I'm not naming any of those either, for the obvious reason that the hole is still open. The pattern in every one of them is the same one described above, just a different combination of trigger and secret.

I also got the tool's own permission check wrong, twice, in slightly different ways. A lot of these workflows call Claude Code through a widely used GitHub Action rather than the bare CLI, and that Action has a real, built-in check on who is allowed to fire the trigger. In two of my earliest reports I had assumed no such check existed at all. It does, for most common trigger types, and once I confirmed that against the Action's own source, I retracted both reports. Then I made a narrower version of the same mistake in two more reports, assuming one specific trigger type sat outside that same check. I read the Action's current source code again, not its documentation, and that assumption was wrong too, that trigger type has been covered since September last year. I retracted those as well and sent correction emails to the maintainers involved. Six retractions in total, all from checking my own assumption against real code instead of trusting what I thought I already knew.

One small unrelated thing from the same week. A maintainer at Cloud Foundry reviewed, approved and merged a bugfix PR I sent to cloudfoundry/terraform-provider-cloudfoundry, a nil pointer crash in their Terraform provider. Not a security finding, just an ordinary PR that happened to land in the same stretch of days.

If you have a workflow where an AI CLI runs on public input, the two things worth checking are who is allowed to fire the trigger, and what secrets sit in the same job as the CLI call. Those two checks alone would have caught all 22.

I spent a day auditing GitHub Actions across a dozen AI agent frameworks

holistis — Tue, 08 Sep 2026 11:55:25 +0000

I spent a day going through the GitHub Actions workflows of some of the better known open source AI agent projects: AutoGen, Semantic Kernel, LangChain, Letta, Google's ADK, promptflow, smolagents, mem0, griptape, and a few others.

Nothing exotic. Just the boring stuff that gets skipped once a project grows fast: unpinned actions, CI checks that quietly stopped meaning anything, and a couple of straight up regressions where a fix from months ago got undone by a later commit nobody noticed.

A few examples worth mentioning, since the PRs are public.

Microsoft's autogen had a CI check meant to catch out of sync generated proto files. It could never fail. Someone had copied a demo snippet from a GitHub Action's README two years ago, and that snippet never included an exit code. The job kept reporting green no matter how stale the generated files got.

Google's adk-python had a fork guard on a pull_request_target workflow that was correctly fixed in an earlier PR, then quietly reverted by a later sync commit. That's scarier than a bug that was never fixed, because it means the protection existed and then silently disappeared.

griptape had an action pinned to a mutable tag instead of a commit hash, tj-actions/changed-files at v47. That's the exact action that got compromised in March 2025, when its version tags were rewritten to point at a commit that dumped CI runner memory, secrets included, into public workflow logs. Pinning to a tag instead of a hash means trusting the tag never moves. It moved once already, for this specific action.

None of it is glamorous. No RCE, no data breach, nothing headline worthy. It's the unglamorous work of checking whether a project's CI setup actually does what its authors think it does. In about half the repos I looked at, it didn't, for at least one workflow.

PRs are open on autogen, semantic-kernel, letta-code, adk-python, smolagents, mem0, griptape and promptflow. Some are already reviewed, most are waiting on a maintainer to get to them.

If you maintain a project whose CI has run unchanged for a year or two, it's worth a second look. The bugs aren't hard to find. Nobody had gotten around to looking.

I built a scanner for CI checks that can't fail. Nvidia just built one for AI skills that can't be trusted.

holistis — Tue, 08 Sep 2026 04:10:19 +0000

Nvidia released SkillSpector last month: a scanner that checks AI agent skills and MCP servers for prompt injection, data exfiltration, and other malicious patterns before you install them. Their own numbers: 26.1% of the skills they scanned had vulnerabilities, 5.2% looked outright malicious. It's part of a bigger pipeline now, a catalog where skills get scanned, evaluated, and signed before anyone sees them.

I read that a few days after finishing something adjacent: a count of how many CI checks in public GitHub repos can pass even when they've already found a real problem. Sample of 120 real workflow files, by hand. 74% had at least one step that cannot fail no matter what it finds.

Different layer, same instinct: don't trust the thing that's supposed to warn you until something has checked that it can actually go red.

Muraqib runs npx muraqib doctor against a repo's CI setup and looks for exactly that failure mode: a continue-on-error: true on a step that matters, a notification step that swallows a missing secret and reports success anyway, a job hard-killed by timeout-minutes that shows up as "cancelled" instead of "failure" so every downstream check gets skipped without anyone deciding that. All four look identical from the Actions tab: a green run, an empty inbox, nobody looking. That's worse than having no monitoring, because you've stopped checking it yourself.

SkillSpector looks at a different kind of thing hiding behind a green light. It runs static analysis plus an optional LLM pass over a skill or MCP server, checking for a tool description that asks for more file access than the task needs, a prompt trying to override its own instructions, an output path that quietly phones home. 71 patterns across 17 categories, from prompt injection to MCP tool poisoning.

Neither tool watches the same layer. Muraqib checks whether your CI can tell you the truth. SkillSpector checks whether the thing you just installed is telling you the truth. Both exist because more of what runs in a repo now runs with less human eyeball time on it: a nightly test suite nobody reads the raw logs of, a skill you installed because the README looked fine.

Every one of Muraqib's four failure patterns had already happened, quietly, on the same project, before I went looking for them. The weekly digest email didn't go out for twelve Mondays in a row, because the notification step went green on a missing API key instead of failing. Nobody caught it by reading logs. I only found that silence by accident, while testing something unrelated, and built the check for it afterward.

If a project's CI, or its agent tooling, doesn't have something scanning the watchers themselves for that failure mode, it's running on trust nobody has checked yet.

Muraqib: https://github.com/holistis/muraqib (MIT, npx muraqib doctor takes about two seconds, installs nothing)
SkillSpector: https://github.com/NVIDIA/SkillSpector

Two Strangers, Zero Stars, Two Boundaries Still Open

holistis — Mon, 07 Sep 2026 08:22:11 +0000

capacity-attest is a small npm package for signed, content-addressed delivery claims between two parties in an x402 agent-to-agent payment. As of today the GitHub repo behind it has 0 stargazers, 0 forks, and 1 open issue, and two outside developers built on it anyway.

This is what happened, told in the order it happened, using only what we can point at directly: pull requests, commit hashes, an npm registry, an on-chain transaction, an issue thread, and our own decision log. One real gap got closed and shipped on 2026-09-06. Two others are written down in DECISIONS.md as still open, on purpose, because no one has hit them hard enough yet to justify a fix.

The format

A capacity-attest claim comes down to two things: a claimId and a signature.

The claimId is a sha256 hash of the canonical content of the claim: who delivered what, to whom, and whether it was delivered. The signature is an EIP-191 signature over that content, and it recovers to a buyer address.

That is the whole trust model. No central ledger to believe, no reputation score, just a hash anyone can recompute and a signature anyone can verify themselves. We did not design this to be portable across chains. We designed it to be simple. Whether that simplicity would also travel across chains was tested by what happened between 2026-09-05 and 2026-09-07, not decided by us in advance.

The question that came before either builder

On 2026-09-05, goun7 opened x402-foundation/x402#3379, a feature request about binding x402 settlement to an externally verifiable agent work-receipt ledger, under the short name proof-of-done. The thread reached 14 comments from 5 people: 0xultravioleta, goun7, holistis (us), safal207, and wildcherrycasa.

Inside that thread was a specific technical question about our own ledger: is the append-only claims file hash-chained, or does its integrity rest entirely on whoever hosts it. goun7's own team had found the same class of problem in their own system, that a hostile writer can fabricate a consistent fake history against a fresh node.

We checked. It was not hash-chained. A host could show a buyer a subset of real, validly-signed claims, all positive, none negative, and the buyer would have no way to detect what was left out. Every individual claim would still verify as authentic. The set as a whole would still be misleading.

Boundary 1: completeness got a partial fix, not a full one

We built a priorClaimId field that lets a buyer chain their own successive claims about the same seller together, plus an analyzeCompleteness check that flags any returned claim whose predecessor is missing from the result. It shipped with 494 tests and a public fixture, merged and published as capacity-attest@0.4.0, and we answered publicly in the same issue thread (comment 5560977571) instead of asking anyone to take our word for it.

Read analyzeCompleteness for what it actually checks, not what its name implies. It detects a broken chain: if claim 5 points to claim 4 as its prior and claim 4 is missing, that gap is visible. It cannot prove there was never a claim 6. A host that simply never issues an attestation for a delivery leaves no gap to find, because no link in the chain was ever broken. Detecting an incomplete chain and proving a complete one are different guarantees, and only the first one shipped.

Existence anchoring, the piece that would let a third party confirm a claim exists independent of the host's own record, stays unbuilt for the same reason it started unbuilt: no real counterparty has ever named its absence as a blocker. It stays on the list, not in the code, until that changes.

The same discipline caught an overclaim before it went further than it should have. Working against the package, one contributor, YE-YI7, surfaced enough real friction that our decision log's first pass called it proof of a real need for a formal conformance-vectors suite, based on that single interaction. We walked that back once it became clear one data point does not prove a need for permanent infrastructure. What we shipped instead was cheaper: a section in the README for verifying a claim independently. The vectors suite stays unbuilt until a second, independent party hits the same friction after reading that section. So far, nobody has.

2026-09-06, 13:59 UTC: YE-YI7 merges PR #18

The same day, YE-YI7 merged a PR into their own asm-spec repository, 1568 additions across 6 files, that does not just copy our claim shape. It imports capacity-attest@0.2.0 as an actual dependency and calls into it directly, verifying the claim byte digest, the claimId, and the EIP-191 signer through our code, not a reimplementation of it. That is the first time anyone outside this project used the package itself, not just the idea behind it.

Nobody had asked us to build a reputation score or ranking on top of delivery claims, not YE-YI7, not goun7, and we deliberately do not. We corrected our own decision log on this point while writing this article: an earlier version credited YE-YI7 with a quote endorsing that choice, and we could not find that quote anywhere in YE-YI7's actual public work when we checked it against the source. It is gone from the log now. What survives, checked directly rather than assumed, is plainer and weaker: neither real contact has ever asked for a score, so the decision not to build one stands on our own reasoning, not on someone else's words.

Boundary 2: two ways to fake scale, neither closed

Not building a reputation score closes off one way to fake standing. Two separate, unrelated ways remain open, and we found both ourselves in an abuse scan on 2026-09-06 rather than fixed either.

The first is identity: nothing in capacity-attest stops one operator from registering many addresses and looking like many different buyers or sellers. A 14-agent research pass the same day checked whether this is already covered elsewhere, ERC-8004, Google's AP2, Microsoft's Entra Agent ID, the x402 Foundation's own work, the Legal Context Protocol, and concluded that it is. What we shipped from that pass was small: externalRefs citation fields and a resolve_agent_identity tool, verified live against ERC-8004 on Base mainnet. Citing an ERC-8004 identity is not the same as stopping one operator from registering many of them, and we left this deliberately thin.

The second is volume, a plainer problem with no identity layer involved. Nothing charges an actor for producing a claim, so an attacker can fabricate an unlimited stream of validly-signed, false claims against a real seller for free. get_delivery_history also returns results unbounded and synchronously, with no pagination and no rate limit on the read side. Both are marked in the decision log as found and not fixed, on purpose: rate-limiting one side while leaving the other free would look like a fix without being one. Both wait for an actual trigger, a real abuse case or a partner who needs the guarantee, before we commit to a specific mitigation.

2026-09-06, 19:33 UTC: 0.5.0 ships, and gets proven on-chain

Between the two pull requests, we published capacity-attest 0.5.0: cross-installation discovery plus an EAS source, commit c8fa192 ("chore(capacity-attest): 0.5.0 (cross-installation discovery + EAS source)").

The problem it addresses is separate from either boundary above: get_delivery_history is local by default. If buyer A and buyer B each run their own installation, B does not automatically see the claim A recorded about the same seller. discoverDeliveryHistory pulls claims from multiple independent sources and re-verifies every one of them regardless of where it came from, trusting no source by default.

We then proved this against a real public chain, not just a fixture. On Base mainnet, a schema was registered on EAS, UID 0x1dd19408345dee43b432b89ccb68760265ecff506098b6efe8ba82ad0d52b195, and two claims were published as attestations: one delivered yes, UID 0x81a55d54452b2cf8bdda7918f63a27bf9ff79e5025b485f7316aae6259288ccc, and one delivered no, UID 0xe736b005cbcb54f8f196ac64ef09d75d939c8a18c0d5d9670b5c5025c07398c4. Total cost for the three transactions was about 0.0000096 ETH, roughly 2.5 dollar cents. A separate read-only script, standing in for a fresh installation, then found both attestations on-chain and verified each one locally, including the negative one.

This solves discoverability, not completeness. Finding a claim across installations and proving you found every claim that exists are different problems, and only the first one has a shipped fix. One honest caveat on the numbers above: those three UIDs are quoted directly from our decision log. We have not independently re-checked them against base.easscan.org as part of writing this article.

2026-09-07, 04:54 UTC: EmbryoSpace merges PR #19

The second builder took a different path, into the same asm-spec repository. EmbryoSpace's PR, 543 additions across 9 files, reuses only the claim format, claimId as a sha256 hash of canonical content. Nothing else.

It could not reuse more, because our verifyClaim function is typed around ETH-style EIP-191 signatures. EmbryoSpace's claims are signed and encoded for BSV, base58, a different signature scheme entirely. Our verification code cannot check a BSV signature, so there was no version of this PR where it could call into our package the way YE-YI7's did.

That is the actual answer to whether the format is rail-neutral. It is, in the one part that matters most: content-addressing. The claimId only depends on hashing canonical content, so any chain can produce and consume the identifier. The signature and its verification are not neutral at all, and pretending otherwise would have been the wrong thing to write in a decision log meant to be checked, not trusted.

One more line item stayed unbuilt for a plainer reason: an in-transit status for multi-day deliveries. Neither external contact asked for it, and the closest real-world comparisons, AWS, Vast.ai, RunPod, also bill after the fact.

The numbers, plainly

capacity-attest is at 0.5.0 on npm, with 646 downloads over the period 2026-08-08 through 2026-09-06. The repository has 0 stargazers, 0 forks, 1 open issue. The most recent commit, c8fa192, landed 2026-09-06T19:33:36Z, and the four commits before it, all from the same author, landed on the same calendar day. This is one person moving fast on one day, not a project carrying years of review.

Exactly two people outside this project, YE-YI7 and EmbryoSpace, have built something that uses the claim format or the package itself, confirmed through their own merged pull requests, and they did it in two different ways worth telling apart. One calls into capacity-attest's own verification code. The other reuses only the hash-based identifier because its signature scheme cannot be checked by this code at all.

One boundary, completeness of visibility across a claim history, got a real but partial fix on 2026-09-06: it catches a broken chain, not a fabricated silence. A second boundary, faking scale, splits into an identity gap that was named and researched but left deliberately thin, and a plainer volume gap, free claim fabrication and an unbounded read endpoint, found and documented instead of patched over. Two strangers proved the format holds across two different rails. The decision log is where the next person finds out what it still does not hold against, corrections included.

We had two independent proofs of a critical RCE. The real software disagreed.

holistis — Mon, 07 Sep 2026 07:38:08 +0000

Yesterday we found what looked like a critical remote code execution bug in Postman's script sandbox. Two of us reproduced it independently, both times with real command execution. We scored it, wrote most of the report, and started building the end-to-end proof of concept before reaching out to Postman's security team.

Then we ran that proof of concept against the actual, currently published packages, and it failed. Both times.

Here's what happened.

The bug we thought we had

Postman lets you run JavaScript before and after a request, for things like generating auth tokens or checking a response. That script runs in a restricted environment: no process, no require, no filesystem. The restriction works by literally naming those as function parameters and calling the function with undefined for each one, so any script that types process gets undefined instead of the real thing.

There's a classic way around that trick. Function('return this')() compiles a brand new function from a string and calls it, and that new function does not inherit any of the parameter shadowing from the function it was written inside. If it runs in the same JavaScript realm as the real process object, it hands that object straight back to you.

We tried it against a standalone rebuild of Postman's isolation logic (the two open source packages it's built on, uvm and uniscope) and got a real process object back, twice, independently. From there: process.binding('spawn_sync'), and we had command execution.

Where it fell apart

A rebuild of the logic is not the same thing as the actual product. So before writing to Postman, we built the exploit again, this time against the real, currently published postman-sandbox, postman-runtime, and newman packages, called the way an actual script would call them.

Both attempts failed. The error from the newman path was blunt: Cannot read properties of undefined (reading 'mainModule'). The process object we thought we'd recovered was undefined.

It turns out Postman's real sandbox does something our rebuild didn't: before any user script runs, it deliberately strips process, Buffer, and globalThis off the actual global object, a step it calls recreatingTheUniverse(), documented in their changelog since postman-sandbox v4.0.0. On the newman path there's a second, independent layer on top of that: the script runs inside its own separate V8 context, so there is no shared realm for the trick to abuse in the first place.

Our rebuild had faithfully reproduced the parameter-shadowing trick. It just hadn't reproduced the hardening Postman built on top of it.

What we're taking from this

We keep a running rule for ourselves: an exploit only counts as reachable once it's been run against the real, unmodified code through its own public entry point, never against a rebuild of the mechanism, however faithful that rebuild feels. This is the second time we've had to learn a version of that lesson, and this time it cost us a wasted day rather than a wasted submission.

No report went to Postman. Nothing was submitted, nothing was retracted. We're writing this up because the failure is more useful to share than the almost-bug was, and because "we double-checked our own work before it left the building" is the kind of thing you can only prove by actually doing it in public.

I counted how many CI checks cannot fail. On a 120-file sample, 74 percent.

holistis — Sat, 05 Sep 2026 11:11:51 +0000

Last week I found two checks in my own repository that could never have failed. Not broken, not misconfigured. Structurally incapable of going red, sitting in CI for months, costing the same as a real check and reporting the same colour.

So I went looking for how common that is. Here is the count, the method, and what it does not prove.

What I counted

GitHub code search, all public repositories, on 5 September 2026.

workflow files (containing runs-on in .github/workflows)   7,094,272
  of those, containing "continue-on-error: true"             264,704
  containing "npm run lint"                                  301,568
  containing BOTH "npm run lint" and "continue-on-error"      27,232

That last line is where it gets interesting, and also where code search stops being able to help. It tells you both strings live in the same file. It cannot tell you the continue-on-error is attached to the lint step rather than to something unrelated three jobs down.

So I read them.

The sample

I pulled 500 matching files from pages spread across the result set, shuffled them, and inspected the first 120 by fetching each file and walking it step by step.

Two things I had to fix in my own method before the number meant anything.

The first pass gave 80 percent, and it was wrong. path:.github/workflows also matches README.md, TROUBLESHOOTING.md and ci.md sitting in that directory, plus .yml.disabled files that do not run. Filtering to actual .yml and .yaml removed that noise.

The second was a distinction I nearly missed. One of the early hits was a step called "Lint (non-blocking)". That is not the problem. That is somebody being explicit. Counting it alongside the silent ones would have been unfair and would have inflated the number.

With both corrections, on 120 real workflow files:

check step that silently cannot fail       89   (74%)
check step that says so in its name         3   (2.5%)
no toothless check step at all             28   (23%)

Security audits are the worst of it

workflow files running "npm audit"                          71,296
  of those, also containing "continue-on-error"             19,200   (27%)
workflow files with "npm test" and "continue-on-error"      22,592

More than a quarter of the workflows that run a dependency audit also contain something allowed to fail. Given the 74 percent rate in the sample, most of those are the audit itself.

Three shapes I verified by reading the file

I am not linking these. They are small public projects and the point is the pattern, not the people. Run the search yourself and read the first ten results; you will find the same thing.

The plain one. A step named Lint, running npm run lint, with continue-on-error: true directly under it. The name promises a check. Nothing in the Actions tab tells you it cannot fail.

The one with the padlock. A step named 🔐 Check for security vulnerabilities, running npm audit --audit-level=moderate, with continue-on-error: true. The emoji is doing more work than the check.

The belt and braces. A step named Run tests (if available) whose command is npm test || echo "No tests configured", with continue-on-error: true on top. The || echo already swallows the exit code. The flag catches whatever the first mechanism missed. That step cannot report a failure through two independent layers.

Why this is worse than having no check

A missing check leaves you appropriately nervous. You know you are not covered.

A check that cannot fail costs exactly the same to run, occupies the same line in the Actions tab, renders the same green, and buys you confidence it has not earned. It is not neutral. It actively replaces the nervousness that would have made you look.

There is a second cost that surprised me. On my own suite, the twelve failing tests were 8 percent of the tests and 36 percent of the runtime, because a failing test burns its full timeout plus a retry. A dead check does not just fail to inform you. It also does not fail fast, because it never fails at all.

Where the flag is right

continue-on-error is a good feature and this is not an argument against it.

An experimental matrix leg on a nightly toolchain, an optional platform, a step whose failure genuinely does not block the merge: all fine. Three of the files in my sample used it exactly that way and said so in the step name.

The distinction is not the flag. It is whether the name tells the truth. Lint (non-blocking) is honest. Lint is not.

The check I now run on myself

Reading a check does not tell you whether it works. The only way to know is to introduce the exact defect it claims to catch and confirm it goes red.

I ran that against ten checks in one repository. Two could never have fired. One of them is worth describing, because I would not have predicted it.

The check verified that a curl in a notification step carried the --fail flag, by searching the step for the flag name. Directly above that curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stayed green, because the string was still on the page.

The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.

Three measurements per case, not one: green before the mutation, red during, green after. If any of the three is off, the result is unknown rather than pass. My own harness produced four unusable results before it produced a trustworthy one, every time because of the harness and not the checks. An exercise about detectors that cannot fire is a good place to remember that the detector you just wrote is also a detector.

What this count does not prove

Worth stating plainly, because a number without its limits is a claim dressed up as a fact.

GitHub's code search totals are approximate. My sample came from the first thousand results rather than the full 27,232, so it is indicative and not a proper random estimate. I only measured the JavaScript ecosystem, through npm commands; Python, Go and Rust workflows are not in the count. My step parser is a heuristic that walks lines between step boundaries, and I hand-verified three of its hits rather than all 89.

What I am confident saying: this is common, it is not rare, and security audits are disproportionately affected.

The two-minute version

Open your workflows. Search for continue-on-error: true. For each one, ask whether the job or step name promises something it cannot deliver.

Then pick your most important check and break the thing it watches for. If it stays green, you did not have that check. You had a line item.

A CI timeout is a kill, not a failure. Playwright's defaults changed because of it.

holistis — Sat, 05 Sep 2026 06:37:18 +0000

Every morning there was a run in the Actions tab and nothing in my inbox. That is exactly what a healthy nightly looks like.

It had not tested my app since late June. I found out in September.

Here is what was happening, why nothing told me, and what changed in Playwright last week as a result.

A timeout is a kill, not a failure

My nightly job had timeout-minutes: 20. My suite had quietly grown past twenty minutes.

When a GitHub Actions runner hits that limit it does not fail the job. It kills the process and records the run's conclusion as cancelled. That single word is the whole problem, and it has three consequences that stack.

The report is never written. Playwright is stopped mid-run. The reporters never get to flush, so there is no HTML report and no results.json. Nothing on disk to look at afterwards.

The upload step skips itself. The recommended workflow guards the artifact upload with if: ${{ !cancelled() }}. That guard is right and it is still there. It exists so a run somebody cancelled by hand does not upload a half-finished report. The trouble is that a runner timeout produces the same conclusion as a hand cancellation, so the guard cannot tell them apart. You get no artifact on precisely the runs where you most want one.

Your alerting never fires. If your notification step keys off the test step failing, it is not reached. A cancelled job skips everything downstream. No failed step, no email, no Slack message.

So the run happens, the run is visible, and nothing about it says anything is wrong. It is indistinguishable from a night where the tests passed. Worse, it is indistinguishable from a run somebody cancelled by hand.

The numbers from my own repo

I pulled the run history when I finally went looking.

164 nightly runs since June
  7 succeeded
 76 failed
 81 cancelled, the last 60-plus consecutively

Every cancelled run produced zero artifacts.

Then I set globalTimeout below the job timeout and ran the same suite. It finished in 29 minutes, reported failure with 12 of 147 tests failing, and produced an 83 MB report.

The suite had needed 29 minutes for a while. The 20 minute limit had been killing it nightly. Because a kill is not a failure, nothing said so.

The fix, and why the ordering matters

globalTimeout is a Playwright setting. It defaults to no timeout at all.

Set it below your job's timeout-minutes and the order of events changes completely. Playwright stops itself first. It exits non-zero, the reporters finish, the HTML report and results.json are written, and the run reads as an ordinary test failure. Your alerting fires because something actually failed.

The runner timeout then only has to cover the install and upload steps around the run. It becomes a backstop instead of the primary bound.

// playwright.config.ts
export default defineConfig({
  // Fail the run after an hour, so that the reporters still produce a report.
  globalTimeout: 60 * 60 * 1000,
});

Two lines. The entire difference between a loud failure and two months of silence.

How common this is

Before filing anything I wanted to know whether this was my mistake or a shape lots of people were in.

At the time I checked, GitHub code search reported roughly 163,000 playwright.config.ts files. About 1,700 of them set globalTimeout. That is around one percent.

Around 3,200 workflows ran npx playwright test on a cron schedule. That is the case where nobody is watching the run live, so a missing report costs the most.

Those numbers are approximate and they move. The ratio is the part that mattered: the safe configuration was the rare one, and the docs never connected the two settings.

What changed

I filed microsoft/playwright#42533 on 3 September, proposing a note in ci.md.

Playwright maintainer Dmitry Gozman went considerably further than the note I asked for, and merged two changes on 4 September.

The documentation, in microsoft/playwright#42563, across four files. ci.md now opens the CI section with this:

Always set a global timeout in CI. By default a test run has no upper bound, so a suite that hangs, or that slowly grows past the job limit of your CI provider, is killed by the runner mid-run and does not produce the test report.

And it drops the job-level timeout from the workflow examples, with the reasoning attached: if you do add one, keep it comfortably above globalTimeout so Playwright always stops first.

The scaffolding, in microsoft/create-playwright#181. This is the one that matters most. create-playwright is what runs when anyone types npm init playwright@latest. It now generates globalTimeout: 60 * 60 * 1000 in all four config templates, and no longer generates timeout-minutes in the GitHub Actions workflow.

I asked for a warning sign next to the hole. What landed was the hole being filled.

What this means for you

If you start a Playwright project from today, you get the safe default and you do not need to think about any of this.

If your project already exists, you still have the old shape. The generated config sits in your repo from the day you created it and nothing rewrites it. Two things to check:

Does your playwright.config.ts set globalTimeout? If not, your run has no upper bound of its own.

Does your CI job set timeout-minutes? If yes, and globalTimeout is unset or higher, the runner is your only limit and it kills without reporting.

There is also a check that takes ten seconds and tells you whether this already happened to you. Open your nightly workflow's run history and look at the conclusions. If you see a run of cancelled you never investigated, that is this bug, and those nights were not tested.

The part I keep thinking about

The bug was not that something broke. Something broke and the system that was supposed to tell me was structurally incapable of telling me, because the failure mode it produced was the one shape it treated as "nothing to report".

I have since gone looking for more of those. In my own tooling I found a check that verified a curl had the --fail flag by searching the whole workflow step for the word. Directly above the curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stays green.

The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.

The way you find these is not by reading them. It is by introducing the exact defect each check claims to catch and confirming it goes red. I ran that against ten checks in one repo and two of them could never have failed. When I ran it, my own test harness turned out to be broken twice before the checks were.

A check that cannot fail costs exactly as much to run as one that can, looks identical in the Actions tab, and buys you confidence it has not earned. That is worse than having no check, because no check at least leaves you appropriately nervous.

Look at what your green means. Sometimes it means nothing went wrong. Sometimes it means nothing could have told you.

Zero stars, seventy cloners, and a readme that said nothing about either

holistis — Thu, 03 Sep 2026 06:19:31 +0000

I have written two posts about Muraqib already. One on August 29, about open sourcing the thing and then spending three nights trying to break it myself before anyone else could. One earlier today, about the bot that got cancelled sixty nights in a row and never said a word about it.

Neither post mentions a star count. Neither mentions a fork count. Neither mentions a download number. I checked both, word for word, before writing this one. The gap was just never there.

That is the actual subject of this post. Not a new feature. A readme that shipped with zero signal on it, for a project people were already pulling down.

Muraqib is a nightly Playwright QA watchdog for solo SaaS founders, open source, MIT licensed, at github.com/holistis/muraqib. Here is what I mean by already pulling down. Muraqib has zero GitHub stars and zero forks right now. But over the last 14 days, the repo has been cloned 175 times, by 70 unique cloners. Views sit at 83 total, 66 unique. Clones spiked hard on August 29, the day the first article went out. 96 clones from 43 unique cloners in a single day, more than half of the two week total.

So the readme people were landing on, the same day the traffic actually happened, had nothing on it that told a visitor anyone else was using the thing. No badge, no number, nothing. Zero stars sitting at the top of the page next to 43 people quietly cloning it that same day is not a coincidence I am proud of. It is just what happens when you build the tool and forget to build the page.

I found this today while doing something unrelated. Earlier the same day I had already fixed a different readme problem on this project, a line that promised a public defect history with nothing underneath it. Different bug, same shape. A claim or a blank space where evidence should be, and nobody checking what the reader actually sees.

What I fixed

The fix for the stars and forks gap was small. I added badges today. npm version, npm downloads per month, GitHub stars, GitHub forks, license. All pulled live, none of them typed in by hand. If the numbers move, the badges move. I do not have to remember to update them, which is good, because I clearly do not remember to check them either.

The npm version badge already has something real to show. Muraqib went from 0.2.0 to 0.3.3 today, eight versions in one day, each one a real fix I found by running the tool against a production app and against other people's public repos.

The downloads badge does not have anything to show yet. npm's own download stats API is still returning package not found for this package as of today. That lag, 24 to 48 hours before a brand new package's stats populate, is normal, not a bug. But it means the badge on the page right now is a real badge with no number behind it. That is fine. It is honest about what is known, which is more than the readme was doing before today.

The broader thing I keep relearning is that a product having activity and a page showing that activity are two separate jobs. I spent real effort making sure Muraqib's core failure mode does not happen quietly. The release notes cover the actual reason for that effort. Of roughly 163,840 public playwright.config.ts files on GitHub, only about 1,684, roughly 1 percent, set a globalTimeout, the setting that prevents this project's core failure mode. I care about that kind of thing a lot.

I did not care, until today, whether my own readme gave a stranger any reason to trust the project when they first looked at it.

Instrumenting the product was the easy part. Instrumenting the page describing the product was the part I skipped without noticing. If you are building something people can find and clone, put the badge on before you write the second post about it, not after.

79 clones, 2 stars, 4 installs, and I only just checked

holistis — Thu, 03 Sep 2026 06:19:07 +0000

Here are the numbers for Yad, my open-source browser automation agent, as of today. 79 unique clones. 2 GitHub stars. 0 forks. 4 users on the Chrome Web Store. 11 unique visitors to the repo page.

I am not writing this because those numbers are impressive. They are not. I am writing this because until today I had no idea what they were, and neither did anyone looking at the README.

The repo went up on 2026-08-24. GitHub's traffic Insights only shows a rolling 14-day window, and the repo is 10 days old, so what I am looking at is basically the whole life of the project so far, not a curated slice.

Total clones over that window: 219, from 79 unique cloners. Total views: 47, from 11 unique visitors. There is one day that stands out, 2026-08-29, with 85 clones from 24 unique cloners in a single day. I do not know what caused that spike. Git clone traffic generally does not carry a web referrer, so GitHub cannot tell me either, and I am not going to guess and pass off a guess as an explanation.

What little referrer data exists is thin. yadagent.com sent 17 visits from 3 unique visitors. github.com itself sent 15 visits from 4 unique visitors. alternativeto.net sent 1 visit from 1 visitor. That is the entire attributable list. Almost everything else has no referrer at all, which is apparently normal for clone traffic and not a sign that my tracking is broken.

Separately from all of that, the Chrome Web Store has its own live counter, and right now it says 4 users. That is a smaller number than the clone count, and it is a different kind of number. Clones can be curiosity, CI, a script, someone checking it out and moving on. The Chrome Web Store count is the closest thing I have to actual people running the actual extension, not just people who cloned the repo to look at the code. 4 is the honest number.

I also checked npm out of habit, since a companion package would make sense for this kind of tool. There is no package for this project on npm under any name I tried. The name "yad" is already taken, by an unrelated, older Express routing library from a different author. Not me, not this project.

None of this was visible anywhere. Before today, the README had a CI status badge and an MIT license badge. That was it. No stars, no forks, no install count, nothing that told a visitor whether anyone else had ever touched this thing. I built the extension, wrote the docs, wired up CI, and never once put a number on the page, or even looked at the numbers myself.

So today I fixed that. The README now has four more badges: Chrome Web Store install count, live. Chrome Web Store version, live. GitHub stars, live. GitHub forks, live. Small numbers, but they are on the page now, and they update themselves instead of going stale the way a hand-typed number would.

While I was in there making that change, I nearly caused a different problem. A routine git add -A picked up 38 other untracked files sitting in my working tree, including a 91 MB Windows installer and several Chrome Web Store upload zips. None of it was in .gitignore, because I had never needed it to be until it was sitting next to a commit I was about to push. GitHub warns above 50 MB and refuses pushes above 100 MB, so that installer was one push away from breaking things for whoever hit it first. I caught it before it went anywhere, backed it out, and fixed the .gitignore in the same pull request.

Two things happened the same day, and they are really the same mistake. I built the thing and never put a number on its own front page, and the exact same not looking almost let a 91 MB installer ride into git history unnoticed. First, small numbers are still numbers, and a page with none on it looks emptier than it is. Second, git add -A on a working tree you have not audited in a while is a habit worth breaking before it breaks a push.

If you have a repo you have not looked at the Insights tab for, go look. You might be sitting on a number you have never written down anywhere.