DEV Community: holistis

PyMC Labs has 1,256 stars. A GitHub issue title could still steal their API key.

holistis — Fri, 11 Sep 2026 05:51:16 +0000

PyMC Labs has 1,256 stars on GitHub and a serious engineering team. A GitHub issue title still could have stolen their Anthropic API key.

I found it, reported it, and yesterday they made it official: GHSA-72xh-gfpr-v9hx, severity High, published, my name on it as the reporter.

Here is what was actually happening. Their issue-triage and duplicate-detection workflows ran Claude Code on every single public issue, automatically, no human in the loop. That call used dangerously-skip-permissions, so the agent got full shell access. In the same job: a real ANTHROPIC_API_KEY and a GitHub token scoped to write issues. Anyone on the internet could open an issue, write a prompt injection in the title or body, and try to walk the agent straight to those secrets.

No approval gate. No isolation. Just text in a box, read by an agent with the keys already sitting next to it.

I sent the report. PyMC Labs did not sit on it. They split the job in two: a read-only analysis step, and a separate publish step that never sees the Anthropic secret. They rotated the key. Then they went one step further and published the advisory instead of quietly patching and moving on, credit included.

That last part matters more than the fix itself. A team can ship a fix and say nothing. Publishing the advisory means anyone can check the claim against the real commit history, not just take my word for it.

This is one repository out of 22 I went through in the same bug class this month, AI CLIs wired straight into GitHub Actions on public, untrusted input. Full writeup on the other 21 is here: https://dev.to/holistis/a-github-comment-can-steal-your-secrets-i-found-22-repositories-where-it-could-1doh

This one just became the first to go fully public. More of them will, one at a time, as the maintainers finish their fixes.

A GitHub comment can steal your secrets. I found 22 repositories where it could.

holistis — Thu, 10 Sep 2026 07:28:35 +0000

After the CI hygiene piece, I kept looking at GitHub Actions workflows in AI agent projects, but for something more specific this time: repositories that run an AI CLI, Claude Code, GitHub Copilot CLI, Codex, directly inside a workflow job.

The pattern that mattered was always the same. A workflow triggers on something anyone on the internet can create: an issue, a comment on an issue, a pull request title. It then runs the AI CLI with a flag that skips every confirmation prompt, something like --dangerously-skip-permissions or --yolo, or an equivalent setting that grants unrestricted shell access, which hands the agent full, unfiltered control. And the same job has real secrets sitting in its environment: API keys, cloud service account keys, GitHub tokens with write scope.

Put those three things next to each other and the exploit stops being clever. It's just text. Whatever you type into a public GitHub issue can end up read as an instruction by the agent, and the agent already has the shell access to act on it, read the environment, and send what it finds somewhere. No exotic exploit chain, just an unlocked door with a please don't sign on it.

I went through 22 findings in this specific bug class, spread across separate open source repositories.

Three are confirmed and already being fixed, so I can name them.

pymc-labs/pymc-marketing had an issue triage workflow where a public GitHub issue, through prompt injection, could reach the project's real Anthropic API key.

MadAppGang/dingo had an issue-triage.yml that gave Claude Code full Bash access plus two real secrets, with no gatekeeper at all, triggered by any public issue. The maintainer, Jack Rudenko, replied fast. Thanks for flagging this, he wrote, we'll look into the template and get it patched across the board.

MadAppGang/claudish, a second repository from the same team, had the exact same vulnerable template. Also confirmed, and across the board in that reply covers both. Same mistake, twice, because a template got copied from one repo into the other and nobody re-checked it on the way.

One report went further than a maintainer's inbox. What was reachable there was a cloud service account key, not just an API key, so I sent it to the vendor's product security team instead of the repository maintainers. They triaged it the same day and passed it on to their internal security group. It's not confirmed fixed yet, so I won't name the vendor or the repo here.

Twelve more are still sitting with maintainers, some by private email, some through a private GitHub Security Advisory that stays private until the maintainer decides to publish it. I'm not naming any of those either, for the obvious reason that the hole is still open. The pattern in every one of them is the same one described above, just a different combination of trigger and secret.

I also got the tool's own permission check wrong, twice, in slightly different ways. A lot of these workflows call Claude Code through a widely used GitHub Action rather than the bare CLI, and that Action has a real, built-in check on who is allowed to fire the trigger. In two of my earliest reports I had assumed no such check existed at all. It does, for most common trigger types, and once I confirmed that against the Action's own source, I retracted both reports. Then I made a narrower version of the same mistake in two more reports, assuming one specific trigger type sat outside that same check. I read the Action's current source code again, not its documentation, and that assumption was wrong too, that trigger type has been covered since September last year. I retracted those as well and sent correction emails to the maintainers involved. Six retractions in total, all from checking my own assumption against real code instead of trusting what I thought I already knew.

One small unrelated thing from the same week. A maintainer at Cloud Foundry reviewed, approved and merged a bugfix PR I sent to cloudfoundry/terraform-provider-cloudfoundry, a nil pointer crash in their Terraform provider. Not a security finding, just an ordinary PR that happened to land in the same stretch of days.

If you have a workflow where an AI CLI runs on public input, the two things worth checking are who is allowed to fire the trigger, and what secrets sit in the same job as the CLI call. Those two checks alone would have caught all 22.

I spent a day auditing GitHub Actions across a dozen AI agent frameworks

holistis — Tue, 08 Sep 2026 11:55:25 +0000

I spent a day going through the GitHub Actions workflows of some of the better known open source AI agent projects: AutoGen, Semantic Kernel, LangChain, Letta, Google's ADK, promptflow, smolagents, mem0, griptape, and a few others.

Nothing exotic. Just the boring stuff that gets skipped once a project grows fast: unpinned actions, CI checks that quietly stopped meaning anything, and a couple of straight up regressions where a fix from months ago got undone by a later commit nobody noticed.

A few examples worth mentioning, since the PRs are public.

Microsoft's autogen had a CI check meant to catch out of sync generated proto files. It could never fail. Someone had copied a demo snippet from a GitHub Action's README two years ago, and that snippet never included an exit code. The job kept reporting green no matter how stale the generated files got.

Google's adk-python had a fork guard on a pull_request_target workflow that was correctly fixed in an earlier PR, then quietly reverted by a later sync commit. That's scarier than a bug that was never fixed, because it means the protection existed and then silently disappeared.

griptape had an action pinned to a mutable tag instead of a commit hash, tj-actions/changed-files at v47. That's the exact action that got compromised in March 2025, when its version tags were rewritten to point at a commit that dumped CI runner memory, secrets included, into public workflow logs. Pinning to a tag instead of a hash means trusting the tag never moves. It moved once already, for this specific action.

None of it is glamorous. No RCE, no data breach, nothing headline worthy. It's the unglamorous work of checking whether a project's CI setup actually does what its authors think it does. In about half the repos I looked at, it didn't, for at least one workflow.

PRs are open on autogen, semantic-kernel, letta-code, adk-python, smolagents, mem0, griptape and promptflow. Some are already reviewed, most are waiting on a maintainer to get to them.

If you maintain a project whose CI has run unchanged for a year or two, it's worth a second look. The bugs aren't hard to find. Nobody had gotten around to looking.

I built a scanner for CI checks that can't fail. Nvidia just built one for AI skills that can't be trusted.

holistis — Tue, 08 Sep 2026 04:10:19 +0000

Nvidia released SkillSpector last month: a scanner that checks AI agent skills and MCP servers for prompt injection, data exfiltration, and other malicious patterns before you install them. Their own numbers: 26.1% of the skills they scanned had vulnerabilities, 5.2% looked outright malicious. It's part of a bigger pipeline now, a catalog where skills get scanned, evaluated, and signed before anyone sees them.

I read that a few days after finishing something adjacent: a count of how many CI checks in public GitHub repos can pass even when they've already found a real problem. Sample of 120 real workflow files, by hand. 74% had at least one step that cannot fail no matter what it finds.

Different layer, same instinct: don't trust the thing that's supposed to warn you until something has checked that it can actually go red.

Muraqib runs npx muraqib doctor against a repo's CI setup and looks for exactly that failure mode: a continue-on-error: true on a step that matters, a notification step that swallows a missing secret and reports success anyway, a job hard-killed by timeout-minutes that shows up as "cancelled" instead of "failure" so every downstream check gets skipped without anyone deciding that. All four look identical from the Actions tab: a green run, an empty inbox, nobody looking. That's worse than having no monitoring, because you've stopped checking it yourself.

SkillSpector looks at a different kind of thing hiding behind a green light. It runs static analysis plus an optional LLM pass over a skill or MCP server, checking for a tool description that asks for more file access than the task needs, a prompt trying to override its own instructions, an output path that quietly phones home. 71 patterns across 17 categories, from prompt injection to MCP tool poisoning.

Neither tool watches the same layer. Muraqib checks whether your CI can tell you the truth. SkillSpector checks whether the thing you just installed is telling you the truth. Both exist because more of what runs in a repo now runs with less human eyeball time on it: a nightly test suite nobody reads the raw logs of, a skill you installed because the README looked fine.

Every one of Muraqib's four failure patterns had already happened, quietly, on the same project, before I went looking for them. The weekly digest email didn't go out for twelve Mondays in a row, because the notification step went green on a missing API key instead of failing. Nobody caught it by reading logs. I only found that silence by accident, while testing something unrelated, and built the check for it afterward.

If a project's CI, or its agent tooling, doesn't have something scanning the watchers themselves for that failure mode, it's running on trust nobody has checked yet.

Muraqib: https://github.com/holistis/muraqib (MIT, npx muraqib doctor takes about two seconds, installs nothing)
SkillSpector: https://github.com/NVIDIA/SkillSpector

Two Strangers, Zero Stars, Two Boundaries Still Open

holistis — Mon, 07 Sep 2026 08:22:11 +0000

capacity-attest is a small npm package for signed, content-addressed delivery claims between two parties in an x402 agent-to-agent payment. As of today the GitHub repo behind it has 0 stargazers, 0 forks, and 1 open issue, and two outside developers built on it anyway.

This is what happened, told in the order it happened, using only what we can point at directly: pull requests, commit hashes, an npm registry, an on-chain transaction, an issue thread, and our own decision log. One real gap got closed and shipped on 2026-09-06. Two others are written down in DECISIONS.md as still open, on purpose, because no one has hit them hard enough yet to justify a fix.

The format

A capacity-attest claim comes down to two things: a claimId and a signature.

The claimId is a sha256 hash of the canonical content of the claim: who delivered what, to whom, and whether it was delivered. The signature is an EIP-191 signature over that content, and it recovers to a buyer address.

That is the whole trust model. No central ledger to believe, no reputation score, just a hash anyone can recompute and a signature anyone can verify themselves. We did not design this to be portable across chains. We designed it to be simple. Whether that simplicity would also travel across chains was tested by what happened between 2026-09-05 and 2026-09-07, not decided by us in advance.

The question that came before either builder

On 2026-09-05, goun7 opened x402-foundation/x402#3379, a feature request about binding x402 settlement to an externally verifiable agent work-receipt ledger, under the short name proof-of-done. The thread reached 14 comments from 5 people: 0xultravioleta, goun7, holistis (us), safal207, and wildcherrycasa.

Inside that thread was a specific technical question about our own ledger: is the append-only claims file hash-chained, or does its integrity rest entirely on whoever hosts it. goun7's own team had found the same class of problem in their own system, that a hostile writer can fabricate a consistent fake history against a fresh node.

We checked. It was not hash-chained. A host could show a buyer a subset of real, validly-signed claims, all positive, none negative, and the buyer would have no way to detect what was left out. Every individual claim would still verify as authentic. The set as a whole would still be misleading.

Boundary 1: completeness got a partial fix, not a full one

We built a priorClaimId field that lets a buyer chain their own successive claims about the same seller together, plus an analyzeCompleteness check that flags any returned claim whose predecessor is missing from the result. It shipped with 494 tests and a public fixture, merged and published as capacity-attest@0.4.0, and we answered publicly in the same issue thread (comment 5560977571) instead of asking anyone to take our word for it.

Read analyzeCompleteness for what it actually checks, not what its name implies. It detects a broken chain: if claim 5 points to claim 4 as its prior and claim 4 is missing, that gap is visible. It cannot prove there was never a claim 6. A host that simply never issues an attestation for a delivery leaves no gap to find, because no link in the chain was ever broken. Detecting an incomplete chain and proving a complete one are different guarantees, and only the first one shipped.

Existence anchoring, the piece that would let a third party confirm a claim exists independent of the host's own record, stays unbuilt for the same reason it started unbuilt: no real counterparty has ever named its absence as a blocker. It stays on the list, not in the code, until that changes.

The same discipline caught an overclaim before it went further than it should have. Working against the package, one contributor, YE-YI7, surfaced enough real friction that our decision log's first pass called it proof of a real need for a formal conformance-vectors suite, based on that single interaction. We walked that back once it became clear one data point does not prove a need for permanent infrastructure. What we shipped instead was cheaper: a section in the README for verifying a claim independently. The vectors suite stays unbuilt until a second, independent party hits the same friction after reading that section. So far, nobody has.

2026-09-06, 13:59 UTC: YE-YI7 merges PR #18

The same day, YE-YI7 merged a PR into their own asm-spec repository, 1568 additions across 6 files, that does not just copy our claim shape. It imports capacity-attest@0.2.0 as an actual dependency and calls into it directly, verifying the claim byte digest, the claimId, and the EIP-191 signer through our code, not a reimplementation of it. That is the first time anyone outside this project used the package itself, not just the idea behind it.

Nobody had asked us to build a reputation score or ranking on top of delivery claims, not YE-YI7, not goun7, and we deliberately do not. We corrected our own decision log on this point while writing this article: an earlier version credited YE-YI7 with a quote endorsing that choice, and we could not find that quote anywhere in YE-YI7's actual public work when we checked it against the source. It is gone from the log now. What survives, checked directly rather than assumed, is plainer and weaker: neither real contact has ever asked for a score, so the decision not to build one stands on our own reasoning, not on someone else's words.

Boundary 2: two ways to fake scale, neither closed

Not building a reputation score closes off one way to fake standing. Two separate, unrelated ways remain open, and we found both ourselves in an abuse scan on 2026-09-06 rather than fixed either.

The first is identity: nothing in capacity-attest stops one operator from registering many addresses and looking like many different buyers or sellers. A 14-agent research pass the same day checked whether this is already covered elsewhere, ERC-8004, Google's AP2, Microsoft's Entra Agent ID, the x402 Foundation's own work, the Legal Context Protocol, and concluded that it is. What we shipped from that pass was small: externalRefs citation fields and a resolve_agent_identity tool, verified live against ERC-8004 on Base mainnet. Citing an ERC-8004 identity is not the same as stopping one operator from registering many of them, and we left this deliberately thin.

The second is volume, a plainer problem with no identity layer involved. Nothing charges an actor for producing a claim, so an attacker can fabricate an unlimited stream of validly-signed, false claims against a real seller for free. get_delivery_history also returns results unbounded and synchronously, with no pagination and no rate limit on the read side. Both are marked in the decision log as found and not fixed, on purpose: rate-limiting one side while leaving the other free would look like a fix without being one. Both wait for an actual trigger, a real abuse case or a partner who needs the guarantee, before we commit to a specific mitigation.

2026-09-06, 19:33 UTC: 0.5.0 ships, and gets proven on-chain

Between the two pull requests, we published capacity-attest 0.5.0: cross-installation discovery plus an EAS source, commit c8fa192 ("chore(capacity-attest): 0.5.0 (cross-installation discovery + EAS source)").

The problem it addresses is separate from either boundary above: get_delivery_history is local by default. If buyer A and buyer B each run their own installation, B does not automatically see the claim A recorded about the same seller. discoverDeliveryHistory pulls claims from multiple independent sources and re-verifies every one of them regardless of where it came from, trusting no source by default.

We then proved this against a real public chain, not just a fixture. On Base mainnet, a schema was registered on EAS, UID 0x1dd19408345dee43b432b89ccb68760265ecff506098b6efe8ba82ad0d52b195, and two claims were published as attestations: one delivered yes, UID 0x81a55d54452b2cf8bdda7918f63a27bf9ff79e5025b485f7316aae6259288ccc, and one delivered no, UID 0xe736b005cbcb54f8f196ac64ef09d75d939c8a18c0d5d9670b5c5025c07398c4. Total cost for the three transactions was about 0.0000096 ETH, roughly 2.5 dollar cents. A separate read-only script, standing in for a fresh installation, then found both attestations on-chain and verified each one locally, including the negative one.

This solves discoverability, not completeness. Finding a claim across installations and proving you found every claim that exists are different problems, and only the first one has a shipped fix. One honest caveat on the numbers above: those three UIDs are quoted directly from our decision log. We have not independently re-checked them against base.easscan.org as part of writing this article.

2026-09-07, 04:54 UTC: EmbryoSpace merges PR #19

The second builder took a different path, into the same asm-spec repository. EmbryoSpace's PR, 543 additions across 9 files, reuses only the claim format, claimId as a sha256 hash of canonical content. Nothing else.

It could not reuse more, because our verifyClaim function is typed around ETH-style EIP-191 signatures. EmbryoSpace's claims are signed and encoded for BSV, base58, a different signature scheme entirely. Our verification code cannot check a BSV signature, so there was no version of this PR where it could call into our package the way YE-YI7's did.

That is the actual answer to whether the format is rail-neutral. It is, in the one part that matters most: content-addressing. The claimId only depends on hashing canonical content, so any chain can produce and consume the identifier. The signature and its verification are not neutral at all, and pretending otherwise would have been the wrong thing to write in a decision log meant to be checked, not trusted.

One more line item stayed unbuilt for a plainer reason: an in-transit status for multi-day deliveries. Neither external contact asked for it, and the closest real-world comparisons, AWS, Vast.ai, RunPod, also bill after the fact.

The numbers, plainly

capacity-attest is at 0.5.0 on npm, with 646 downloads over the period 2026-08-08 through 2026-09-06. The repository has 0 stargazers, 0 forks, 1 open issue. The most recent commit, c8fa192, landed 2026-09-06T19:33:36Z, and the four commits before it, all from the same author, landed on the same calendar day. This is one person moving fast on one day, not a project carrying years of review.

Exactly two people outside this project, YE-YI7 and EmbryoSpace, have built something that uses the claim format or the package itself, confirmed through their own merged pull requests, and they did it in two different ways worth telling apart. One calls into capacity-attest's own verification code. The other reuses only the hash-based identifier because its signature scheme cannot be checked by this code at all.

One boundary, completeness of visibility across a claim history, got a real but partial fix on 2026-09-06: it catches a broken chain, not a fabricated silence. A second boundary, faking scale, splits into an identity gap that was named and researched but left deliberately thin, and a plainer volume gap, free claim fabrication and an unbounded read endpoint, found and documented instead of patched over. Two strangers proved the format holds across two different rails. The decision log is where the next person finds out what it still does not hold against, corrections included.

We had two independent proofs of a critical RCE. The real software disagreed.

holistis — Mon, 07 Sep 2026 07:38:08 +0000

Yesterday we found what looked like a critical remote code execution bug in Postman's script sandbox. Two of us reproduced it independently, both times with real command execution. We scored it, wrote most of the report, and started building the end-to-end proof of concept before reaching out to Postman's security team.

Then we ran that proof of concept against the actual, currently published packages, and it failed. Both times.

Here's what happened.

The bug we thought we had

Postman lets you run JavaScript before and after a request, for things like generating auth tokens or checking a response. That script runs in a restricted environment: no process, no require, no filesystem. The restriction works by literally naming those as function parameters and calling the function with undefined for each one, so any script that types process gets undefined instead of the real thing.

There's a classic way around that trick. Function('return this')() compiles a brand new function from a string and calls it, and that new function does not inherit any of the parameter shadowing from the function it was written inside. If it runs in the same JavaScript realm as the real process object, it hands that object straight back to you.

We tried it against a standalone rebuild of Postman's isolation logic (the two open source packages it's built on, uvm and uniscope) and got a real process object back, twice, independently. From there: process.binding('spawn_sync'), and we had command execution.

Where it fell apart

A rebuild of the logic is not the same thing as the actual product. So before writing to Postman, we built the exploit again, this time against the real, currently published postman-sandbox, postman-runtime, and newman packages, called the way an actual script would call them.

Both attempts failed. The error from the newman path was blunt: Cannot read properties of undefined (reading 'mainModule'). The process object we thought we'd recovered was undefined.

It turns out Postman's real sandbox does something our rebuild didn't: before any user script runs, it deliberately strips process, Buffer, and globalThis off the actual global object, a step it calls recreatingTheUniverse(), documented in their changelog since postman-sandbox v4.0.0. On the newman path there's a second, independent layer on top of that: the script runs inside its own separate V8 context, so there is no shared realm for the trick to abuse in the first place.

Our rebuild had faithfully reproduced the parameter-shadowing trick. It just hadn't reproduced the hardening Postman built on top of it.

What we're taking from this

We keep a running rule for ourselves: an exploit only counts as reachable once it's been run against the real, unmodified code through its own public entry point, never against a rebuild of the mechanism, however faithful that rebuild feels. This is the second time we've had to learn a version of that lesson, and this time it cost us a wasted day rather than a wasted submission.

No report went to Postman. Nothing was submitted, nothing was retracted. We're writing this up because the failure is more useful to share than the almost-bug was, and because "we double-checked our own work before it left the building" is the kind of thing you can only prove by actually doing it in public.

I counted how many CI checks cannot fail. On a 120-file sample, 74 percent.

holistis — Sat, 05 Sep 2026 11:11:51 +0000

Last week I found two checks in my own repository that could never have failed. Not broken, not misconfigured. Structurally incapable of going red, sitting in CI for months, costing the same as a real check and reporting the same colour.

So I went looking for how common that is. Here is the count, the method, and what it does not prove.

What I counted

GitHub code search, all public repositories, on 5 September 2026.

workflow files (containing runs-on in .github/workflows)   7,094,272
  of those, containing "continue-on-error: true"             264,704
  containing "npm run lint"                                  301,568
  containing BOTH "npm run lint" and "continue-on-error"      27,232

That last line is where it gets interesting, and also where code search stops being able to help. It tells you both strings live in the same file. It cannot tell you the continue-on-error is attached to the lint step rather than to something unrelated three jobs down.

So I read them.

The sample

I pulled 500 matching files from pages spread across the result set, shuffled them, and inspected the first 120 by fetching each file and walking it step by step.

Two things I had to fix in my own method before the number meant anything.

The first pass gave 80 percent, and it was wrong. path:.github/workflows also matches README.md, TROUBLESHOOTING.md and ci.md sitting in that directory, plus .yml.disabled files that do not run. Filtering to actual .yml and .yaml removed that noise.

The second was a distinction I nearly missed. One of the early hits was a step called "Lint (non-blocking)". That is not the problem. That is somebody being explicit. Counting it alongside the silent ones would have been unfair and would have inflated the number.

With both corrections, on 120 real workflow files:

check step that silently cannot fail       89   (74%)
check step that says so in its name         3   (2.5%)
no toothless check step at all             28   (23%)

Security audits are the worst of it

workflow files running "npm audit"                          71,296
  of those, also containing "continue-on-error"             19,200   (27%)
workflow files with "npm test" and "continue-on-error"      22,592

More than a quarter of the workflows that run a dependency audit also contain something allowed to fail. Given the 74 percent rate in the sample, most of those are the audit itself.

Three shapes I verified by reading the file

I am not linking these. They are small public projects and the point is the pattern, not the people. Run the search yourself and read the first ten results; you will find the same thing.

The plain one. A step named Lint, running npm run lint, with continue-on-error: true directly under it. The name promises a check. Nothing in the Actions tab tells you it cannot fail.

The one with the padlock. A step named 🔐 Check for security vulnerabilities, running npm audit --audit-level=moderate, with continue-on-error: true. The emoji is doing more work than the check.

The belt and braces. A step named Run tests (if available) whose command is npm test || echo "No tests configured", with continue-on-error: true on top. The || echo already swallows the exit code. The flag catches whatever the first mechanism missed. That step cannot report a failure through two independent layers.

Why this is worse than having no check

A missing check leaves you appropriately nervous. You know you are not covered.

A check that cannot fail costs exactly the same to run, occupies the same line in the Actions tab, renders the same green, and buys you confidence it has not earned. It is not neutral. It actively replaces the nervousness that would have made you look.

There is a second cost that surprised me. On my own suite, the twelve failing tests were 8 percent of the tests and 36 percent of the runtime, because a failing test burns its full timeout plus a retry. A dead check does not just fail to inform you. It also does not fail fast, because it never fails at all.

Where the flag is right

continue-on-error is a good feature and this is not an argument against it.

An experimental matrix leg on a nightly toolchain, an optional platform, a step whose failure genuinely does not block the merge: all fine. Three of the files in my sample used it exactly that way and said so in the step name.

The distinction is not the flag. It is whether the name tells the truth. Lint (non-blocking) is honest. Lint is not.

The check I now run on myself

Reading a check does not tell you whether it works. The only way to know is to introduce the exact defect it claims to catch and confirm it goes red.

I ran that against ten checks in one repository. Two could never have fired. One of them is worth describing, because I would not have predicted it.

The check verified that a curl in a notification step carried the --fail flag, by searching the step for the flag name. Directly above that curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stayed green, because the string was still on the page.

The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.

Three measurements per case, not one: green before the mutation, red during, green after. If any of the three is off, the result is unknown rather than pass. My own harness produced four unusable results before it produced a trustworthy one, every time because of the harness and not the checks. An exercise about detectors that cannot fire is a good place to remember that the detector you just wrote is also a detector.

What this count does not prove

Worth stating plainly, because a number without its limits is a claim dressed up as a fact.

GitHub's code search totals are approximate. My sample came from the first thousand results rather than the full 27,232, so it is indicative and not a proper random estimate. I only measured the JavaScript ecosystem, through npm commands; Python, Go and Rust workflows are not in the count. My step parser is a heuristic that walks lines between step boundaries, and I hand-verified three of its hits rather than all 89.

What I am confident saying: this is common, it is not rare, and security audits are disproportionately affected.

The two-minute version

Open your workflows. Search for continue-on-error: true. For each one, ask whether the job or step name promises something it cannot deliver.

Then pick your most important check and break the thing it watches for. If it stays green, you did not have that check. You had a line item.

A CI timeout is a kill, not a failure. Playwright's defaults changed because of it.

holistis — Sat, 05 Sep 2026 06:37:18 +0000

Every morning there was a run in the Actions tab and nothing in my inbox. That is exactly what a healthy nightly looks like.

It had not tested my app since late June. I found out in September.

Here is what was happening, why nothing told me, and what changed in Playwright last week as a result.

A timeout is a kill, not a failure

My nightly job had timeout-minutes: 20. My suite had quietly grown past twenty minutes.

When a GitHub Actions runner hits that limit it does not fail the job. It kills the process and records the run's conclusion as cancelled. That single word is the whole problem, and it has three consequences that stack.

The report is never written. Playwright is stopped mid-run. The reporters never get to flush, so there is no HTML report and no results.json. Nothing on disk to look at afterwards.

The upload step skips itself. The recommended workflow guards the artifact upload with if: ${{ !cancelled() }}. That guard is right and it is still there. It exists so a run somebody cancelled by hand does not upload a half-finished report. The trouble is that a runner timeout produces the same conclusion as a hand cancellation, so the guard cannot tell them apart. You get no artifact on precisely the runs where you most want one.

Your alerting never fires. If your notification step keys off the test step failing, it is not reached. A cancelled job skips everything downstream. No failed step, no email, no Slack message.

So the run happens, the run is visible, and nothing about it says anything is wrong. It is indistinguishable from a night where the tests passed. Worse, it is indistinguishable from a run somebody cancelled by hand.

The numbers from my own repo

I pulled the run history when I finally went looking.

164 nightly runs since June
  7 succeeded
 76 failed
 81 cancelled, the last 60-plus consecutively

Every cancelled run produced zero artifacts.

Then I set globalTimeout below the job timeout and ran the same suite. It finished in 29 minutes, reported failure with 12 of 147 tests failing, and produced an 83 MB report.

The suite had needed 29 minutes for a while. The 20 minute limit had been killing it nightly. Because a kill is not a failure, nothing said so.

The fix, and why the ordering matters

globalTimeout is a Playwright setting. It defaults to no timeout at all.

Set it below your job's timeout-minutes and the order of events changes completely. Playwright stops itself first. It exits non-zero, the reporters finish, the HTML report and results.json are written, and the run reads as an ordinary test failure. Your alerting fires because something actually failed.

The runner timeout then only has to cover the install and upload steps around the run. It becomes a backstop instead of the primary bound.

// playwright.config.ts
export default defineConfig({
  // Fail the run after an hour, so that the reporters still produce a report.
  globalTimeout: 60 * 60 * 1000,
});

Two lines. The entire difference between a loud failure and two months of silence.

How common this is

Before filing anything I wanted to know whether this was my mistake or a shape lots of people were in.

At the time I checked, GitHub code search reported roughly 163,000 playwright.config.ts files. About 1,700 of them set globalTimeout. That is around one percent.

Around 3,200 workflows ran npx playwright test on a cron schedule. That is the case where nobody is watching the run live, so a missing report costs the most.

Those numbers are approximate and they move. The ratio is the part that mattered: the safe configuration was the rare one, and the docs never connected the two settings.

What changed

I filed microsoft/playwright#42533 on 3 September, proposing a note in ci.md.

Playwright maintainer Dmitry Gozman went considerably further than the note I asked for, and merged two changes on 4 September.

The documentation, in microsoft/playwright#42563, across four files. ci.md now opens the CI section with this:

Always set a global timeout in CI. By default a test run has no upper bound, so a suite that hangs, or that slowly grows past the job limit of your CI provider, is killed by the runner mid-run and does not produce the test report.

And it drops the job-level timeout from the workflow examples, with the reasoning attached: if you do add one, keep it comfortably above globalTimeout so Playwright always stops first.

The scaffolding, in microsoft/create-playwright#181. This is the one that matters most. create-playwright is what runs when anyone types npm init playwright@latest. It now generates globalTimeout: 60 * 60 * 1000 in all four config templates, and no longer generates timeout-minutes in the GitHub Actions workflow.

I asked for a warning sign next to the hole. What landed was the hole being filled.

What this means for you

If you start a Playwright project from today, you get the safe default and you do not need to think about any of this.

If your project already exists, you still have the old shape. The generated config sits in your repo from the day you created it and nothing rewrites it. Two things to check:

Does your playwright.config.ts set globalTimeout? If not, your run has no upper bound of its own.

Does your CI job set timeout-minutes? If yes, and globalTimeout is unset or higher, the runner is your only limit and it kills without reporting.

There is also a check that takes ten seconds and tells you whether this already happened to you. Open your nightly workflow's run history and look at the conclusions. If you see a run of cancelled you never investigated, that is this bug, and those nights were not tested.

The part I keep thinking about

The bug was not that something broke. Something broke and the system that was supposed to tell me was structurally incapable of telling me, because the failure mode it produced was the one shape it treated as "nothing to report".

I have since gone looking for more of those. In my own tooling I found a check that verified a curl had the --fail flag by searching the whole workflow step for the word. Directly above the curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stays green.

The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.

The way you find these is not by reading them. It is by introducing the exact defect each check claims to catch and confirming it goes red. I ran that against ten checks in one repo and two of them could never have failed. When I ran it, my own test harness turned out to be broken twice before the checks were.

A check that cannot fail costs exactly as much to run as one that can, looks identical in the Actions tab, and buys you confidence it has not earned. That is worse than having no check, because no check at least leaves you appropriately nervous.

Look at what your green means. Sometimes it means nothing went wrong. Sometimes it means nothing could have told you.

Zero stars, seventy cloners, and a readme that said nothing about either

holistis — Thu, 03 Sep 2026 06:19:31 +0000

I have written two posts about Muraqib already. One on August 29, about open sourcing the thing and then spending three nights trying to break it myself before anyone else could. One earlier today, about the bot that got cancelled sixty nights in a row and never said a word about it.

Neither post mentions a star count. Neither mentions a fork count. Neither mentions a download number. I checked both, word for word, before writing this one. The gap was just never there.

That is the actual subject of this post. Not a new feature. A readme that shipped with zero signal on it, for a project people were already pulling down.

Muraqib is a nightly Playwright QA watchdog for solo SaaS founders, open source, MIT licensed, at github.com/holistis/muraqib. Here is what I mean by already pulling down. Muraqib has zero GitHub stars and zero forks right now. But over the last 14 days, the repo has been cloned 175 times, by 70 unique cloners. Views sit at 83 total, 66 unique. Clones spiked hard on August 29, the day the first article went out. 96 clones from 43 unique cloners in a single day, more than half of the two week total.

So the readme people were landing on, the same day the traffic actually happened, had nothing on it that told a visitor anyone else was using the thing. No badge, no number, nothing. Zero stars sitting at the top of the page next to 43 people quietly cloning it that same day is not a coincidence I am proud of. It is just what happens when you build the tool and forget to build the page.

I found this today while doing something unrelated. Earlier the same day I had already fixed a different readme problem on this project, a line that promised a public defect history with nothing underneath it. Different bug, same shape. A claim or a blank space where evidence should be, and nobody checking what the reader actually sees.

What I fixed

The fix for the stars and forks gap was small. I added badges today. npm version, npm downloads per month, GitHub stars, GitHub forks, license. All pulled live, none of them typed in by hand. If the numbers move, the badges move. I do not have to remember to update them, which is good, because I clearly do not remember to check them either.

The npm version badge already has something real to show. Muraqib went from 0.2.0 to 0.3.3 today, eight versions in one day, each one a real fix I found by running the tool against a production app and against other people's public repos.

The downloads badge does not have anything to show yet. npm's own download stats API is still returning package not found for this package as of today. That lag, 24 to 48 hours before a brand new package's stats populate, is normal, not a bug. But it means the badge on the page right now is a real badge with no number behind it. That is fine. It is honest about what is known, which is more than the readme was doing before today.

The broader thing I keep relearning is that a product having activity and a page showing that activity are two separate jobs. I spent real effort making sure Muraqib's core failure mode does not happen quietly. The release notes cover the actual reason for that effort. Of roughly 163,840 public playwright.config.ts files on GitHub, only about 1,684, roughly 1 percent, set a globalTimeout, the setting that prevents this project's core failure mode. I care about that kind of thing a lot.

I did not care, until today, whether my own readme gave a stranger any reason to trust the project when they first looked at it.

Instrumenting the product was the easy part. Instrumenting the page describing the product was the part I skipped without noticing. If you are building something people can find and clone, put the badge on before you write the second post about it, not after.

79 clones, 2 stars, 4 installs, and I only just checked

holistis — Thu, 03 Sep 2026 06:19:07 +0000

Here are the numbers for Yad, my open-source browser automation agent, as of today. 79 unique clones. 2 GitHub stars. 0 forks. 4 users on the Chrome Web Store. 11 unique visitors to the repo page.

I am not writing this because those numbers are impressive. They are not. I am writing this because until today I had no idea what they were, and neither did anyone looking at the README.

The repo went up on 2026-08-24. GitHub's traffic Insights only shows a rolling 14-day window, and the repo is 10 days old, so what I am looking at is basically the whole life of the project so far, not a curated slice.

Total clones over that window: 219, from 79 unique cloners. Total views: 47, from 11 unique visitors. There is one day that stands out, 2026-08-29, with 85 clones from 24 unique cloners in a single day. I do not know what caused that spike. Git clone traffic generally does not carry a web referrer, so GitHub cannot tell me either, and I am not going to guess and pass off a guess as an explanation.

What little referrer data exists is thin. yadagent.com sent 17 visits from 3 unique visitors. github.com itself sent 15 visits from 4 unique visitors. alternativeto.net sent 1 visit from 1 visitor. That is the entire attributable list. Almost everything else has no referrer at all, which is apparently normal for clone traffic and not a sign that my tracking is broken.

Separately from all of that, the Chrome Web Store has its own live counter, and right now it says 4 users. That is a smaller number than the clone count, and it is a different kind of number. Clones can be curiosity, CI, a script, someone checking it out and moving on. The Chrome Web Store count is the closest thing I have to actual people running the actual extension, not just people who cloned the repo to look at the code. 4 is the honest number.

I also checked npm out of habit, since a companion package would make sense for this kind of tool. There is no package for this project on npm under any name I tried. The name "yad" is already taken, by an unrelated, older Express routing library from a different author. Not me, not this project.

None of this was visible anywhere. Before today, the README had a CI status badge and an MIT license badge. That was it. No stars, no forks, no install count, nothing that told a visitor whether anyone else had ever touched this thing. I built the extension, wrote the docs, wired up CI, and never once put a number on the page, or even looked at the numbers myself.

So today I fixed that. The README now has four more badges: Chrome Web Store install count, live. Chrome Web Store version, live. GitHub stars, live. GitHub forks, live. Small numbers, but they are on the page now, and they update themselves instead of going stale the way a hand-typed number would.

While I was in there making that change, I nearly caused a different problem. A routine git add -A picked up 38 other untracked files sitting in my working tree, including a 91 MB Windows installer and several Chrome Web Store upload zips. None of it was in .gitignore, because I had never needed it to be until it was sitting next to a commit I was about to push. GitHub warns above 50 MB and refuses pushes above 100 MB, so that installer was one push away from breaking things for whoever hit it first. I caught it before it went anywhere, backed it out, and fixed the .gitignore in the same pull request.

Two things happened the same day, and they are really the same mistake. I built the thing and never put a number on its own front page, and the exact same not looking almost let a 91 MB installer ride into git history unnoticed. First, small numbers are still numbers, and a page with none on it looks emptier than it is. Second, git add -A on a working tree you have not audited in a while is a habit worth breaking before it breaks a push.

If you have a repo you have not looked at the Insights tab for, go look. You might be sitting on a number you have never written down anywhere.

My QA bot was cancelled 60 nights in a row and never said a word

holistis — Thu, 03 Sep 2026 04:54:40 +0000

I run a health platform in the Netherlands. About three months ago I set up a nightly job: Playwright runs my critical flows against production at 2am, and if something breaks, Claude opens a PR with a fix. One digest email on Monday. I wrote about it here before.

Yesterday I went looking at its run history for an unrelated reason.

164 nightly runs. 7 succeeded. 76 failed. And 81 were cancelled, the last 60-plus of them consecutively.

My app had not been checked since late June. I found out in September.

Nothing had told me. Not a red job, not an email, not a failed check on a PR. Every morning the Actions tab showed a run from the night before, and every morning my inbox was empty, and I read that as good news for ten weeks.

What went wrong is boring. Why it was silent is not.

The job had timeout-minutes: 20. My suite had grown to 24 spec files, run one at a time with a retry on CI, deliberately gentle on a production app. Somewhere in late June it crossed 20 minutes.

That is the boring part. Any monitoring setup outgrows a limit eventually. You expect it to complain.

Here is the part I did not know: a runner timeout is a kill, not a failure.

When GitHub Actions hits timeout-minutes, it stops the process. Playwright never finishes, so it writes no report. And GitHub records the run's conclusion as cancelled, not failure.

My workflow looked like this:

- name: Run tests
  id: tests
  run: npx playwright test || echo "tests_failed=true" >> $GITHUB_OUTPUT
  continue-on-error: true

- name: Trigger Claude fix on failure
  if: steps.tests.outputs.tests_failed == 'true'
  # ...

- name: Fail job if tests failed
  if: steps.tests.outputs.tests_failed == 'true'
  run: exit 1

Read that again with a kill in mind. The test step never gets to write tests_failed=true, because it never gets to finish at all. So the Claude dispatch is skipped. The job-failure step is skipped. No email. The run is indistinguishable from a night where everything passed, except it is also indistinguishable from a night where nothing ran.

That is worse than having no QA. With no QA you at least know you are not covered, so you check things yourself. I had stopped checking things myself, because I believed something else was.

The real bug was one level up

I could have fixed the timeout and moved on. Bigger number, done.

But sit with the shape of it for a second. Every alert in my setup hung off exactly one thing: a test run that finished and reported a result. So the entire category of "the run never reported" had no alarm attached to it, by construction. The timeout was one member of that category. There are others, and I had none of them covered either:

GitHub silently disables scheduled workflows on repositories with no pushes for 60 days. Your cron just stops. No notification.
A secret expires and the job dies during setup, before any test runs.
Someone disables a workflow to debug something and forgets to turn it back on.

All four look identical from the outside: a quiet dashboard, an empty inbox, and an app nobody is checking.

If you have a nightly check of any kind, this is the question worth asking, and it is a different question from "does it catch bugs":

What does it look like when this check stops running, and who finds out?

If the honest answer is "it looks exactly like everything being fine", the check is not finished.

The two fixes

The first one is ordering, not a bigger number.

Playwright has a globalTimeout. Set it below the job's timeout-minutes, and Playwright stops itself before the runner can kill it. It exits non-zero, the reporters still write results.json, and the whole chain behind it fires normally. A suite that got too slow now reads as a loud failure with a full report instead of silence.

// playwright.config.ts
const GLOBAL_TIMEOUT_MINUTES = Number(process.env.MURAQIB_GLOBAL_TIMEOUT_MIN ?? 35);

export default defineConfig({
  globalTimeout: GLOBAL_TIMEOUT_MINUTES * 60 * 1000,
  // ...
});

# .github/workflows/nightly.yml
jobs:
  test:
    timeout-minutes: 45   # must stay above globalTimeout

Those two numbers can drift apart later, so there is a check in CI now that fails the build if globalTimeout is missing, is not below the job timeout, or does not leave enough room for the install and upload steps around the run.

The second fix is the one that actually matters. A watchdog, running daily, asking one question: has the nightly produced a pass or a fail recently?

It alerts if the recent runs were all inconclusive, if nothing has reported inside the window, or if the workflow has no runs at all. It also flags a check that has been red for seven runs straight, because at that point it has stopped being an alert and become furniture. Everyone has learned to scroll past it, which is the same outcome as silence by a different road.

The watchdog installs nothing. No dependencies, no npm step, just the Actions API and the fetch built into Node 20. Whatever watches the watchman needs fewer moving parts than the watchman, or you have only added another thing that can go quiet.

I fed it my own run history from June onwards. It returns the alarm I needed ten weeks ago.

If you have a nightly Playwright setup

You can check yours without installing anything:

npx muraqib doctor

It reads your workflow and your Playwright config and tells you whether a slow night would fail loudly or just disappear. Two seconds. It writes nothing to your repo.

I built it because I could not be the only person whose job timeout is quietly eating their monitoring. If you run it and it comes back green, good, that is a real answer and it cost you nothing.

The whole thing is MIT licensed and runs in your own GitHub Actions on your own key. There is no service and no bill, because there is nothing to bill for.

https://github.com/holistis/muraqib

The uncomfortable part

I write software for a health platform under Dutch medical regulation. I am careful about a lot of things. And I still had a guard that had been dead for ten weeks, sitting in plain sight in my Actions tab, looking exactly like it was working.

I am fairly sure I am not unusual here. If you set up a nightly check six months ago and have not looked at it since, go look at the conclusion column, not just the green ticks. Count how many say cancelled.

I shipped a Series-A-grade security autopilot in one day. It costs €2.60/year.

holistis — Mon, 31 Aug 2026 22:14:49 +0000

I run a Dutch health AI platform alone. No co-founder, no CTO, no security team. Last Friday I shipped a security pipeline that, on paper, sits somewhere between what a 50-person scale-up and a Series-A startup deploys.

Total cost of running it: €2.60 per year in LLM tokens. Total time to build it: one day, with Claude doing the typing.

This isn't a "vibe coding wins" piece. This is the architecture, the code, what it catches, what it misses, and why I'm telling you most of your competitors are running production with a glass door and no lock on it.

The state of solo-founder security in 2026 (it's grim)

Some numbers, because opinions are cheap:

~28% of the top 100K websites have Content Security Policy headers (Mozilla Observatory, 2025)
~60% of Stripe integrations don't validate webhook signatures (Stripe's own engineering blog)
~70% of public websites run dependencies with known CVEs older than six months
A typical WordPress shop has CSP off, no rate-limiting on /wp-login.php, plugin files indexed in Google, and an admin email that is also the recovery email

The "we're too small to be a target" defense died around 2021 when AI-driven scanners like Nuclei, Burp Suite, ZAP, and (since 2024) XBOW and ZeroPath started crawling the internet 24/7. They don't pick targets. They scan everyone, score what they find, and the high-yield findings go up the pipe for follow-up.

If you're shipping production code as a solo founder, the question is no longer "will I get attacked." It's "will the attacker bother spending more than three minutes once their scanner has scored my site."

That's the bar I wanted to clear in one day.

What I have now (the six components)

                ┌──────────────────────────────────────────┐
                │  1. SecurityRadar (Friday 03:30 NL)       │
                │  → NVD + GH Advisories + npm audit + RSS  │
                │  → Haiku-classify on shortlist            │
                │  → Critical = direct email + admin UI     │
                └──────────────────────────────────────────┘
                              ↓
                ┌──────────────────────────────────────────┐
                │  2. Dependabot (weekly Saturday)          │
                │  → Auto-PRs to dev branch only            │
                │  → Major bumps ignored (no breaking)      │
                │  → Grouped to reduce noise                │
                └──────────────────────────────────────────┘
                              ↓
                ┌──────────────────────────────────────────┐
                │  3. Auto-merge bot on dev                 │
                │  → Squash-merge once CI is green          │
                │  → Major-bump PRs: skipped, await human   │
                └──────────────────────────────────────────┘
                              ↓
                ┌──────────────────────────────────────────┐
                │  4. Muraqib nightly (Playwright on dev)   │
                │  → 90 end-to-end specs against dev URL    │
                │  → Result logged in GitHub Actions        │
                └──────────────────────────────────────────┘
                              ↓
                ┌──────────────────────────────────────────┐
                │  5. dev → main promotion (daily 09:00)    │
                │  → Requires 7 consecutive green Muraqib   │
                │  → Risk-gate on protected files           │
                │  → Fast-forward merge + push              │
                └──────────────────────────────────────────┘
                              ↓
                ┌──────────────────────────────────────────┐
                │  6. Auto-rollback (post-push to main)     │
                │  → Wait 90s, poll /api/health for 15 min  │
                │  → 3 consecutive 5xx = git revert + push  │
                │  → Email alert + [skip rollback] guard    │
                └──────────────────────────────────────────┘

Six moving parts. Each is the simplest possible implementation of its job. None of them are clever. The cleverness is that they're chained.

Component 1: SecurityRadar (the eyes)

The radar runs once a week, Friday 03:30 NL. It hits four sources, all free:

npm audit --json on the local lockfile
GitHub Security Advisories REST API, scoped to packages I actually use
NIST NVD recent CVEs, last seven days, filtered to my stack keywords
RSS feeds from Stripe, Cloudflare, Anthropic, scanned for security-flavored keywords

The first three sources are deterministic. The fourth is fuzzy. To stop fuzzy from torching my budget I do static pre-filtering first, then hit the LLM only on the shortlist:

// 967 npm packages, 50+ NVD entries per week, dozens of blog posts.
// Filter to ~10 items before spending a single token.
const pkgSet = new Set(packages.map(p => p.toLowerCase()));

const nvdRelevant = nvdAdv.filter(adv => {
  const lc = `${adv.title} ${adv.description}`.toLowerCase();
  for (const pkg of pkgSet) {
    if (pkg.length > 4 && lc.includes(pkg)) return true;
  }
  return false;
});

const all = [...npmAuditAdv, ...githubAdv, ...nvdRelevant, ...vendorAdv];
const shortlist = unique(all).slice(0, 20);

Then Claude Haiku (the cheap model) classifies each item:

async function classifyAdvisory(adv: AdvisorySignal) {
  const response = await invokeLLM({
    messages: [{
      role: "user",
      content: `Assess this advisory for our stack (React/tRPC/Drizzle/Clerk/Stripe/Anthropic/Railway).

Source: ${adv.source}
Package: ${adv.packageName ?? "n/a"}
Severity: ${adv.severity}
Title: ${adv.title}

Reply ONLY with JSON:
{"relevant": true|false, "recommendation": "1-2 sentences", "action": "auto-patch"|"manual-review"|"noop"}`
    }],
    meta: { model: "claude-haiku-4-5-20251001" },
    maxTokens: 256,
  });
  return JSON.parse(extractJSON(response));
}

That's it. ~10 items per week × ~500 tokens per classification × Haiku pricing = ~€0.05 per week. Critical findings trigger an email straight to my inbox; everything else lands in the admin UI as a "proposal".

The whole file is ~480 lines. Half of that is fetching and parsing RSS, because vendor blogs don't bother with consistent feeds.

Component 5: The 7-day soak (the patience)

This is the part I'm proudest of, because it's the part most engineers would skip.

Dependabot opens a PR to dev. CI runs, my auto-merge bot squash-merges it. Now dev is one commit ahead of main. Should I push to production?

Most "auto-merge to main" setups say yes the moment CI passes. That's how you ship a bug that only surfaces in a real browser session, or a Stripe webhook that only fails when an actual invoice.payment_succeeded arrives.

My setup says: not until Muraqib has run seven consecutive nights against dev and all of them are green. Then, and only then, fast-forward dev into main.

- name: Check 7 nights Muraqib-dev-nightly green
  run: |
    RUNS=$(gh run list \
      --workflow="muraqib-dev-nightly.yml" \
      --branch=dev --limit=10 \
      --json conclusion,createdAt,status)

    SEVEN_DAYS_AGO=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)
    RECENT=$(echo "$RUNS" | jq --arg c "$SEVEN_DAYS_AGO" \
      '[.[] | select(.createdAt >= $c and .status == "completed")]')

    TOTAL=$(echo "$RECENT" | jq 'length')
    SUCCESS=$(echo "$RECENT" | jq '[.[] | select(.conclusion == "success")] | length')

    if [ "$TOTAL" -lt 7 ] || [ "$SUCCESS" -lt "$TOTAL" ]; then
      echo "Promotion delayed."
      exit 0
    fi

The promotion job also reads the diff between main and dev. If the changes touch a high-risk file (server/_core/index.ts, schema, tRPC root) AND any of the commits aren't pure security commits (deps, ci, fix(security)), promotion stops. A human pushed something risky. Humans look at it.

Seven days is arbitrary. It's the smallest number that catches "weird thing that only happens on the third Sunday of the month" without being so long that the queue piles up.

Component 6: Auto-rollback (the emergency brake)

The flight risk in this whole pipeline is: dev was green for seven nights, the merge fires, the deploy goes out, and then something breaks because production has data dev didn't. Maybe a migration deadlocks. Maybe an env var is different. Maybe the cache layer behaves differently under real load.

So the moment main gets a push, this workflow fires:

on:
  push:
    branches: [main]

jobs:
  monitor:
    if: "!contains(github.event.head_commit.message, '[skip rollback]')"
    steps:
      - name: Wait 90s for Railway deploy
        run: sleep 90

      - name: Poll /api/health for 15 minutes
        run: |
          FAILS=0
          OK=0
          DEADLINE=$(($(date +%s) + 900))
          while [ "$(date +%s)" -lt "$DEADLINE" ]; do
            S=$(curl -s -o /dev/null -w "%{http_code}" "$PROD/api/health")
            R=$(curl -s -o /dev/null -w "%{http_code}" "$PROD/")
            if [ "$S" = "200" ] && [ "$R" = "200" ]; then
              OK=$((OK + 1)); FAILS=0
              [ "$OK" -ge 3 ] && exit 0
            else
              FAILS=$((FAILS + 1)); OK=0
              if [ "$FAILS" -ge 3 ]; then
                echo "needs_rollback=true" >> $GITHUB_OUTPUT
                exit 0
              fi
            fi
            sleep 30
          done

      - name: Revert HEAD on crash
        if: steps.health.outputs.needs_rollback == 'true'
        run: |
          git revert HEAD --no-edit
          git commit --amend -m "auto-revert [skip rollback]"
          git push origin main

Three consecutive 5xx within 15 minutes = something broke. Auto-revert. Email me. The [skip rollback] marker on the revert commit prevents an infinite loop where the rollback itself triggers another monitor run.

I've now had this fire once in practice — when a non-security commit (a video-recording feature) introduced a top-level import { chromium } from "playwright" and production couldn't find the package at boot. The autopilot caught it inside a minute and reverted. I read about it in the email, not from a customer.

What the price tag actually is

Their option	Their cost	What I built
Snyk Team	€500/month	One TypeScript file
Aikido Security	€300-800/month	Six GitHub workflows
Wiz	€10K+/month	A dependabot.yml
GitHub Advanced Security	€49/user/month	Free GitHub APIs
Hiring a security engineer	€80-120K/year	Claude as on-demand CISO
My setup	€2.60/year	All of it

The €2.60 is real. Five cents of Haiku tokens per week × 52 weeks. Everything else is free: Dependabot, GitHub Actions, the NVD REST API, GitHub's Advisory API, npm audit, helmet, isomorphic-dompurify, express-rate-limit.

The price isn't the point. The point is that the time-cost dropped from "we need to hire someone, write an RFP, evaluate four vendors, sign an MSA" to "I described what I wanted, my agent wrote it, I pressed merge."

What it does NOT catch (the honest part)

Don't read this and think you have a force field. You don't. Here's what is still on your shoulders:

Zero-day exploits. Nobody catches these. By definition, they're not in any feed.
Targeted attacks. If a determined human picks your platform specifically and spends two weeks on it, autopilot will not save you. You need a pentest for that, which you should buy once a year if you're handling money or health data.
Social engineering. Your bookkeeper getting a phone call from an AI cloning your voice asking for Stripe access — no amount of helmet config saves you.
Supply-chain through-routes I don't run audits on. I cover npm. I don't cover Docker base images, OS-level packages, or my Railway runtime. That's another sprint.
Compliance certifications. SOC 2 / ISO 27001 / HIPAA / NEN 7510 require process + paperwork, not just config. The autopilot helps but doesn't get you certified.
A determined customer getting their account hacked elsewhere and reusing the password on yours. Credential stuffing is a thing. Throw in rate-limiting on /sign-in and breach-password rejection (haveibeenpwned API) for that.

So this stack is "good enough to make you a hard target." It is not "you can stop thinking about security." Anyone who tells you otherwise is selling something.

What you can copy in a weekend

If you're a solo founder or running a small team on a similar stack, here's the minimum-viable version:

Turn on Dependabot security updates (Settings → Security & analysis → Enable). Free. Five-minute setup. This alone puts you ahead of 70% of websites.
Add helmet with a real CSP to your Node server. Use the allowlist pattern in my securityRadar.ts if you're on Stripe + Clerk.
Validate your Stripe webhook signature. Yes, you probably aren't. Yes, that's how 60% of payment fraud against small SaaS happens.
Sanitize any dangerouslySetInnerHTML with DOMPurify. Especially if you have user-generated or admin-written content rendered as HTML.
Add an auto-rollback workflow (mine is 80 lines of YAML, paste and adapt the PROD_URL).
Wire your CI to gate dev → main on at least three consecutive green nights, even if you can't manage seven. Catches 90% of "works on dev fails on prod" surprises.

You don't need an LLM-driven radar to start. But once you have the rest in place, adding ~480 lines of TypeScript for the radar is half a Saturday, and it gives you €500/month of Snyk-shaped value. Worth it.

Why this matters in 2026

Five years ago this stack would have required a small ops team to build and a senior engineer to maintain. Today, the building blocks are free, the wiring is configuration, and the cognitive work — "what should I build, what should I skip, where am I exposed" — can be done in a thinking session with a sufficiently capable model.

The companies that win the next decade aren't the ones with the biggest engineering teams. They're the ones whose solo founders ship infrastructure that used to require a department, and use the time saved on what actually moves the business.

Security is plumbing. Plumbing should not be the reason your company exists. It should be the reason it survives.

Code

The full SecurityRadar lives in server/_core/autoStudio/securityRadar.ts. The four GitHub Actions workflows are in .github/workflows/. Repo is public; clone, fork, steal whatever's useful.

Built for longevityai.nl, a Dutch health AI platform I run solo. The architecture is stack-agnostic — swap React for Vue, Stripe for Mollie, Clerk for Auth0, and the spine still works.

Build it. Ship it. Tell me what you'd change.