DEV Community: Allen Brooks

How do you know a passing check still works? Make it fail on purpose.

Allen Brooks — Thu, 30 Jul 2026 10:49:15 +0000

A feature makes money. A check protects it. Those are not the same job, and the difference is why checks rot.

When you ship a feature, the market tells you fast whether it works. Someone signs up or they don't. A check has no such feedback loop. A check that has quietly stopped working looks exactly like a check that is passing, and nothing outside the build can tell you which one you have.

At corporate scale a QA team is paid to close that gap. Their whole job is the negative case: break the thing on purpose and confirm the alarm rings. At solo scale there is no such team, and the negative case is the first thing to fall off the list, because it is the work that sits furthest from revenue. Features feel like money. Checks feel like insurance. The trap is believing features are money. Features are money-making. Checks are money-protecting. Skip the second one and you don't notice for months, which is exactly the failure mode I wrote about last time.

The count I had been avoiding

In Green is not evidence I found four checks that had never done anything. An em-dash linter whose regex could not match. A security scanner that was never installed. A cost gate that was only ever red. A config file I had documented but never written. The post ended on a rule I was very pleased with: make it fail on purpose. Every gate should have a known-bad fixture, something you can point it at that must fail, because if you have never watched a check fail you do not have a check, you have a decoration that emits green.

Then I did the obvious thing I had not done. I counted how many of my own gates actually had one.

The answer was humbling. Across my repos I run several dozen distinct gates. The reusable workflow every repo calls runs nine third-party scanners: Trivy twice, Gitleaks, Checkov, Hadolint, Actionlint, Shellcheck, ZAP, and an Infracost cost gate. Not one of them has a fixture. My content pipeline runs four gates on every note: prose style, spelling, frontmatter schema, link integrity. Not one of them has a fixture. The em-dash rule, the one this whole series keeps circling back to, had none.

What did have fixtures? Only the checks I had written myself. The little scripts that scan for hardcoded hex colors, or unpinned GitHub Actions, or drift in my shared config files: every one of those has a negative test that feeds it bad input and asserts it goes red. I have a unit-testing habit, and it reaches exactly as far as the code I author. Everything I bought, and every rule I configured inside someone else's tool, I took on faith. The fixture was present wherever I already had the habit, and absent everywhere I was trusting a vendor or my own YAML.

The two places I had gone further

Twice, I had wired the negative case into CI itself, so it runs forever instead of once. One is a schema gate for my product metadata: CI points the validator at a deliberately invalid file on every run and fails the build unless the validator rejects it. The other is my design-token scanner, which re-runs its own "this hex literal must be caught" test inside every repo that calls it. Two, out of several dozen. Both, again, on code I wrote.

Has any of this ever caught a regression?

Here is the part I have to be honest about, because the whole point of the last post was honesty about what a practice actually catches. Has one of these fixtures ever caught a real regression? A gate that silently broke, and a fixture that screamed?

No. Not once.

Every real catch I can point to was something else. The em-dash breakage from last post I found by hand, poking at it, not because a fixture rang. The one genuine vulnerability my pipeline caught recently, a critical CVE in a transitive dependency that showed up the week I was writing the last post, was a live scanner doing its ordinary job, not a canary. And the closest thing I have to a fixture story runs the wrong way: my hex-color scanner once misfired, flagged a pull-request number as if it were a color, and blocked a real merge. I fixed it by adding the negative tests I should have had first. The fixture did not prevent the bug. The missing fixture allowed it, and I added the fixture afterward, as an apology.

So this is not a victory lap. I am writing down a rule I have mostly not been following.

So I built one

Then I built the first one properly, on the gate that most deserved it.

There is now a file in my repo whose only job is to be wrong. It contains a single em dash and a line telling you not to remove it. It never renders and it is not a post. And there is a step in CI that points Vale at that one file and inverts the result: if Vale reports the em dash, the step passes; if Vale comes back clean, the step fails the build and tells me the gate has gone inert. It is about fifteen lines. The em-dash rule had been silently dead once already, for the entire life of the repo, and nothing would have told me if it happened again. Now something will.

The first time I ran it, I broke the rule on purpose and watched the build go red. That is the only moment in this whole exercise that constitutes actual evidence the check works. Everything before it was faith.

A fixture is a second thing that can rot

I want to be clear-eyed about the obvious objection. A known-bad file that quietly stops being known-bad is just more green, one layer down. Adding a fixture and never watching it fail leaves you exactly where you started.

So the discipline is not "add fixtures." The discipline is: break the thing once, on purpose, watch the alarm ring with your own eyes, and then wire that watching into CI so it repeats without you. The fixture is not the point. The watching is the point. The fixture is just how you make the watching automatic, and cheap enough that you keep doing it after the day you cared.

The rule

So here is the rule, and it is going into my standards where the checklist that opens every story will ask about it. No gate ships without a fixture that proves it can fail. And the smaller, harder half, the one I keep having to relearn: a fixture you have never watched fail is just another green light.

If you run any checks at all, you almost certainly have one that has never rung. Pick it. Break the thing it is meant to catch. Watch. It takes ninety seconds, and it is the only proof you are ever going to get.

I audited 2,864 CI checks. Several had never run, and green never told me.

Allen Brooks — Thu, 30 Jul 2026 10:49:14 +0000

Someone on Hacker News asked a one-line question about the security checks I run on every PR: how often do those checks actually hold your PRs back?

I didn't know. That bothered me more than it should have, so I counted every PR check I have ever run. 2,864 of them, across nine repos.

The answer to his question turned out to be boring. What I found on the way to it did not.

The boring answer

286 of the 2,864 runs failed. About 10%.

That number is useless on its own, and it took me embarrassingly long to see why. A 10% failure rate sounds like a tax, one PR in ten held up at the door. It isn't, because the failures are not spread out. They arrive in tight three or four day clusters with months of silence between them, and every cluster sits on top of a migration. Late April, when I pulled these checks into a shared reusable workflow: eighteen cost-gate failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of nothing. A second product adopting that workflow in May, with a commit two days later called fix(ci): resolve remaining PR check failures. Skeleton repos I bootstrapped in July that lit up the same way and then went quiet.

The repos with the worst failure rates in the whole org are the ones that have only ever been in setup, with no steady state to average against. The worst sits at 57%, and it is the template that the others get cloned from.

None of that is in the numbers. I only know it because I remember those weeks. Read cold, the dates say the opposite: that my oldest and busiest repo carries the most failures and is therefore the most burdened. It carries the most failures because it is where I did the extraction. The context that told the true story from its inverse lived in my head and nowhere else in the data. That is the benign version of a problem I was about to hit in a much worse form, one where the missing context was not in my head, or anywhere I could reach.

The number that didn't fit

I had the answer to the question and could have stopped. What stopped me was a small number that didn't fit.

I have a Vale rule in my content pipeline called EmDashes. It is twelve lines long. It is set to level: error, which fails the build. It exists because I don't write with em dashes, I never have, and I would rather they never reach the site. My drafts run through Claude, which likes them, so the rule is there to strip the ones I miss. That is stated plainly on the colophon; the rule is the enforcement.

The post that started all of this shipped with fifteen em dashes in it.

Vale ran on that post. Vale passed. On that same file, in that same run, Vale reported fifteen passive-voice alerts and ten weak-word alerts, so it was finding the file, parsing the MDX, and applying my custom styles. It reported zero em dashes in a document containing fifteen of them.

Here is why. Vale's existence check compiles a tokens list into a regex with word boundaries wrapped around it:

(?i)(?m)\b(?:—|–)\b

\b asserts a boundary between a word character and a non-word character. An em dash is punctuation. There is no word character on either side of the match to anchor the boundary against, so \b can never be satisfied, so the pattern matches nothing. Ever. Vale has a nonword: true option that drops the boundaries, and the rule works correctly the moment you add it.

That rule had sat in my repo, at error level, catching absolutely nothing, since the day I wrote it. And there is no observable difference between that and a rule that works. Both emit silence. Both go green.

It was worse than one bug

I fixed the regex, shipped it, felt good about myself, and then went looking for proof that the fix actually gated anything. It didn't.

Here is the CI step that ran Vale:

vale --no-exit --output=line web/content/notes/ | tee vale-report.txt
if grep -E ' error ' vale-report.txt; then
  exit 1
fi

--no-exit forces Vale to exit 0 regardless of what it finds. That was deliberate: print the report first, decide separately. The deciding is that grep. And --output=line emits path:line:col:check:message. There is no severity field anywhere in that format. The grep could never match. The two halves cancel each other perfectly: Vale is forbidden from failing the step, and the thing meant to fail the step is searching for a word the format never prints. That job had passed every build it ever ran. It could not have done anything else.

Then, while testing the fix, Vale flagged an em dash inside a fenced code block, which markdown scoping is supposed to skip. It turns out Vale has no built-in knowledge of the .mdx extension: without an explicit [formats] mapping it treats the file as plain text, applies no markdown scoping at all, and lints your code blocks and inline code as if they were prose. My config carried a comment confidently asserting that "Vale's markdown scoping handles fenced code blocks and inline code natively." It does not, because I never told Vale what it was reading. Six of the alerts on my notes tree were phantoms from that.

So: a rule that could not match, inside a runner that could not fail, parsing a format it did not know how to read. Three independent bugs stacked on one rule, every one of them silent, the whole assembly green. I only found the second and third because I went looking for evidence that my fix to the first one worked.

Once I saw it, it was everywhere

Then I looked at the rest, and found the same disease in three more checks.

Semgrep. My own security baseline mandates it. My own post names it. It is not wired into any workflow, in any repo. A Semgrep that was never installed reports zero findings. So does a Semgrep that is running clean.

The cost gate. Failing for months in two repos, red on every Terraform PR, for the excellent reason that I never provisioned the GitHub App token it needs to read my private modules. It was not gating anything. It was just red, in the corner, next to the checks that were working.

A config file that does not exist. My post says known-false-positive suppressions for the IaC scanner go in a .checkov.yaml. Zero of my nine repos contain that file. I have never written one. I documented a system I had imagined.

I audited the tools with tools that lied

It gets worse, because the tools I used to audit the tools had the same disease.

My first pass used gh run list --limit 100. It told me my secret scanner had fired twice. It has fired thirty-five times. The window silently truncated at a hundred runs, and my busiest repo has over five thousand. My second pass filtered to pull-request events, which silently discarded every push run. A sampled window told me one mature repo had a spotless record; its real number is 27 failures out of 313. Every one of those queries returned a confident, well-formatted, wrong answer. Not one of them announced what it could not see. I was auditing my instruments with instruments that shared their failure mode.

The thing I actually learned

A check that isn't running looks exactly like a check that's passing.

That is the whole post. Green means "nothing was reported." It does not mean "nothing is wrong," and it does not even mean "something looked." The absence of an alert is perfectly compatible with the absence of a scanner, and from outside the build you cannot tell those two apart.

It is the same shape as the 10% number, pushed to its limit. There, the surface reading inverted the truth, and only what I happened to remember could set it right. Here there is nothing to remember. A green check withholds the one fact you actually need, whether a check even ran, and no amount of context brings it back.

Make it fail on purpose

Make it fail on purpose. That is the entire technique.

The only reason I know my em dash rule works now is that I fed Vale a line containing an em dash and watched it report the em dash, then removed the fix and watched it report nothing. Ninety seconds, and the before-and-after is the proof. The only reason I know my one grandfathered exemption is scoped to a single file, rather than having quietly switched the rule off everywhere, is that I temporarily broke a different post and watched the gate catch it.

Every gate you own should have a known-bad fixture: something you can point it at that must fail. If you have never watched a check fail, you do not have a check. You have a decoration that emits green.

This generalizes well past linters. Any alarm you have never heard ring is in one of two states, and you do not know which one.

The punchline

I wrote most of this up as an update to the original post. In it, I said the security stack had caught two real things in four months: a CVE in a transitive dependency, and an IAM policy attached to a user instead of a role. Thin, I said, but the premium is near zero and the downside ends the product.

The pull request carrying that sentence went red.

Trivy caught CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I had changed. It was disclosed against code I merged weeks ago, which is precisely the case that a PR-time scan exists to catch. Six minutes to bump it and go green.

So it is three, not two, and the third arrived while I was typing the sentence that said there were two.

I would take this trade again tomorrow. Near-zero premium against a downside that ends the product is not a hard call. But "we run X" is a claim, not a fact, until you have watched X fail.

What's next

The preview-environment pattern that ZAP runs against is already written up: Preview environments are the dev loop. Vercel plus a fresh Neon Postgres branch per PR, a real running app talking to a real database on every review, for almost nothing. This post was written before that one and sat in a drawer for a week. The next one is on making the invisible checks announce themselves, which is the other half of this problem.

Every pull request gets its own app and its own Postgres branch, for $0

Allen Brooks — Thu, 30 Jul 2026 10:43:56 +0000

Every place I've worked plumbed everything to one shared dev environment. One database, one deploy, everyone's half-finished work landing in the same place. The database slowly filled with test accounts named asdf and orders for negative dollars, until someone declared bankruptcy and refreshed it. Then it filled up again. I've triggered more than one of those refreshes myself.

Exactly one company I worked at had per-feature environments, and they were narrow: a preview for the GraphQL schema and nothing else. The app, the database, the auth flow, all of it still pointed at shared dev. The one thing they isolated was the one thing that was easy to isolate.

Lionshead does the opposite. Every pull request gets its own running app AND its own database. Not a mock. Not a shared dev instance with a "please don't touch" Slack message. A real Next.js deploy talking to a real Postgres database that was branched, schema and data, from the parent in under a second and will be destroyed the moment the PR closes.

The point isn't fancy infrastructure. The point is that local, preview, and production are a stack, not a substitution. Local is where I write the code. Production is where users live. Preview is the rung in between, where a change proves it actually runs before anyone trusts it. Kill that middle step and every review becomes "looks good to me" against a diff, or a push-and-pray to shared dev. The preview environment is where the dev loop closes.

Here's what happens when I open a PR.

Neon branch, created or refreshed. A copy-on-write branch named preview/pr-<number> is cut from the database's parent branch. Copy-on-write means it's not a dump-and-restore; Neon clones the storage layer by reference and only diverges on write, so the clone itself is effectively instant regardless of how much data is in the parent. The step is idempotent by branch name, so every new commit on the PR refreshes the same branch instead of piling up new ones.

Atlas applies the PR's schema. Lionshead manages Postgres schema declaratively with Atlas. Against the fresh branch, atlas schema apply reads the desired schema, diffs it against what's live, and applies the difference. The branch now has exactly the schema this PR proposes, running against real Postgres, not a SQLite stand-in or a mock that lies to you about what a foreign key does.

Vercel builds and deploys against that branch. The build runs with the per-PR DATABASE_URL injected, so anything that touches the database at build time sees this PR's branch. Then vercel deploy --prebuilt ships the artifacts with the same URL wired in at runtime. The reviewer gets a link to a live app, and that app is talking to this PR's database and no one else's.

A sticky comment carries the link. One PR comment, updated in place on every commit, holds the latest preview URL and a short per-commit history. No comment spam, one place to click.

On close, the branch is deleted. When the PR merges or closes, a cleanup job deletes the Neon branch and confirms it's actually gone. The junkyard never accumulates, because the junkyard is thrown away every time.

All of this lives in one reusable workflow, lionshead-vercel-preview-deploy.yml, that every product repo calls. There's a hard-won detail buried in it worth surfacing: it's a single job, not three, because GitHub Actions refuses to pass a step output between jobs when the value looks like a secret. A Postgres connection string contains a password, so a clean "branch job hands the URL to a deploy job" design gets the URL silently scrubbed to an empty string. The fix is unglamorous: keep the URL inside one job. Most of the good infrastructure decisions I make look like that.

What it costs

Nothing worth mentioning. Neon is on the free plan. Vercel just moved to the $20/month Team plan. That's the whole bill for this capability.

The reason it's free is the same reason it's fast. Neon bills for storage and compute, not for the number of branches, and an idle branch scales its compute to zero. A per-PR branch that nobody is actively hitting costs effectively nothing while it sits there, and it shares the parent's storage by reference until it's written to. So "a database per pull request" sounds extravagant and turns out to be a rounding error.

What it has actually caught

I'll be honest about this the way I was honest about the security tooling: most bad migrations never reach a PR, because I run migrations against a local Docker Postgres before I push. The preview branch isn't where I catch a typo'd column.

Where it earns its keep is the full-workflow bugs that are miserable to test locally. Sign-up and sign-in against Firebase, for instance, are genuinely hard to exercise on a laptop; on a real preview URL with a real database, I can watch an account get created and see exactly what we persisted, including the times we captured a wrong or incomplete value. And I can watch all of that happen without dumping a single piece of that test garbage into the shared dev database.

That's the quieter, bigger win. In every corporate project I've seen, the dev database degrades into a landfill of fake accounts and abandoned test data, until it looks nothing like production and stops being useful for real testing. Per-PR branches give the trash somewhere to live and die. The messy test data stays on the ephemeral branch that gets deleted on merge, and dev stays clean enough to keep nudging it toward looking like prod. Cleaner dev means more focused development, which means bugs that show up in testing are more likely to be real.

Why almost nobody does this

I've said before that I haven't seen this pattern at solo scale anywhere else, and I still haven't. I've seen it exactly once at corporate scale.

The reason isn't cost or difficulty. It's that the enabling primitive is rare. Plenty of databases advertise "branching," but they don't mean the same thing. PlanetScale's branching is built for schema changes; its branches don't clone your data. Supabase spins up an empty database and replays your migrations and seed script. Both are useful, neither gives you a full copy of real, production-shaped data per pull request. Neon's copy-on-write branch does, in about a second, for free, and I have not found another managed provider that does the same. Most people simply don't know that per-PR data isolation this cheap is even on the menu.

The gotchas (because there are always gotchas)

Two of these bit me last week, so they're fresh.

The 10-branch cap. The Neon free plan caps a project at ten branches. I opened several PRs in quick succession, left a few in draft, and every one of them dutifully cut a branch. I hit the ceiling, and preview deploys started failing across the board. The fix was to be stricter about which PRs earn a database: filter the workflow so draft and not-yet-ready PRs don't provision a branch until they're actually up for review. The capability was never the problem. My discipline about triggering it was.

Orphaned branches from a skipped cleanup. The nastier one. A concurrency-rule interaction was canceling the cleanup job mid-run when a late push raced a merge, so the branch-delete never executed and branches leaked, which of course marched me straight back into the 10-branch cap. The fix was giving the cleanup job its own concurrency group that never cancels in progress, plus making the delete step actually authenticate instead of failing silently while reporting green. A teardown, once it starts, has to finish. That one cost me an afternoon and a good deal of "why do I have eleven branches when I have three open PRs."

It's slow, though not the part you'd guess. Full transparency: the preview environment is by far the longest thing on the PR. The deploy job runs about five minutes, and the whole preview pipeline lands closer to ten once you count waiting on sibling checks. I pulled the step timings across recent runs to see where those five minutes actually go, and almost none of it is the database. Creating or refreshing the Neon branch takes 10 to 15 seconds on a cold create and effectively nothing when it's just refreshing an existing one; the copy-on-write clone is instant, and the seconds that do pass are spent standing up the branch's compute endpoint, not copying data. Applying the schema with Atlas adds another 10 to 15 seconds, or nothing when there's no diff. The remaining four-plus minutes are all Vercel: about 20 seconds to install the CLI, then three and a half to four minutes for the Next.js production build, then about 25 seconds to upload the artifacts. So the database work tops out around half a minute and is usually far less. Building and shipping a whole app per PR is the slow part, and it would be slow with or without the branch. Everything else on the PR finishes while the build is still going. I think that's an acceptable price for a real running app on every change, but I won't pretend it's instant.

None of these are reasons not to do it. They're the tax you pay once, learn from, and encode into the workflow so the next repo inherits the fix instead of rediscovering the bug. Which, if you've read the last post, you'll know is a lesson I keep having to learn.

What's next

The preview deploy needs secrets, a Vercel token, a Neon key, to do any of this. For a long time that meant long-lived tokens sitting in repository secrets, which is exactly the kind of thing the security post frets about. The next note in this sub-series is how that got fixed: OIDC through Doppler and GitHub Actions, and how short-lived, federated credentials killed the long-lived token entirely.

The security checks in every Lionshead PR

Allen Brooks — Thu, 30 Jul 2026 10:43:55 +0000

At enterprise scale, a breach is a bad quarter. You have a legal team to coordinate disclosure. A disaster-recovery plan you drill annually. A PR team to control the narrative. A security team to quarantine, investigate, and triage the incident. A bank account big enough to absorb regulatory fines, class-action settlements, and the customer churn that follows.

At solo scale, you have none of those. A real breach of user data will almost certainly end the product. Not "hurts the brand," not "sets us back a quarter." Ends it.

So you breach yourself first.

The security stack in every Lionshead PR is an automated self-breach. Every merge attempt is scanned by tools a mid-sized security team would run against production code weekly. If any of them find something, the merge blocks. The vulnerability never reaches production, and the version of me that would have caused an incident never gets the chance.

Here's what actually runs on every PR against every Lionshead product.

Trivy filesystem — vulnerability, secret, and license scanning across the whole repo. Runs on every PR, no path gate. Catches dependency CVEs, accidental hardcoded credentials, license conflicts. Configured via trivy.yaml at the repo root; fails on any CRITICAL or HIGH severity. Trivy is Aqua Security's open-source scanner and it's the single best free vulnerability scanner I've used.

Trivy IaC (Terraform) — the same tool, second mode, against Terraform files. Catches insecure infrastructure defaults (public buckets, permissive IAM roles, unencrypted resources). Path-gated: only runs when terraform/** files change. Same severity gate.

Gitleaks — secret scanning across the entire git history, not just the current diff. Catches API keys, tokens, private keys, connection strings that someone (probably me) committed years ago and later deleted. The secret is still in the history and still exploitable. Runs on every PR. Uses the default ruleset unless the repo drops a .gitleaks.toml override.

Checkov — a second IaC security scanner, run alongside Trivy IaC for defense in depth. Checkov and Trivy IaC use different rulesets and different heuristics; they catch different classes of problem. Path-gated on terraform/**. Skip lists for known-false-positive rules go in .checkov.yaml.

OWASP ZAP baseline — active DAST against the PR's Vercel preview URL. When a repo produces a preview deploy, ZAP hits the deployed site with a baseline security scan. Catches missing security headers, cookie configuration issues, common injection vectors. Fails on any WARN+ alert; per-rule suppressions live in the repo's .zap/rules.tsv.

Actionlint — GitHub Actions workflow linting. Not strictly security, but security-adjacent: catches syntax errors, invalid contexts, and permission-scope typos that would silently over-scope your workflow's access. Path-gated on .github/workflows/**.

Shellcheck — shell script static analysis. Catches quoting errors, command-injection patterns, and the classic "unquoted variable in an rm command" bug that has ended more careers than any single vulnerability. Path-gated on *.sh files.

Hadolint — Dockerfile linting. Catches insecure image bases, missing USER directives, and other Dockerfile-shape issues that expand blast radius on container escape. Path-gated on Dockerfile*.

Action pins check — a Lionshead-authored guardrail that validates every third-party GitHub Action referenced in workflow files is pinned to a full commit SHA, not a moving tag. Prevents a supply-chain attack against an action's tag from silently propagating into Lionshead's CI. Fails PRs that add or update actions without a SHA pin.

Infracost + OPA cost gate — not security in the CVE sense, but security in the "my card gets charged $10K/month by mistake" sense. Runs on every Terraform PR, computes the cost delta, and pauses the PR on a GitHub Environment approval gate when the delta exceeds $50/month. I covered this in more detail in the last post; mentioned here because it lives in the same workflow.

Two more layers run outside the PR gate.

Scheduled security audit — every product repo runs lionshead-security-audit.yml daily against the default branch. It runs npm audit (or the language-appropriate equivalent) plus a full Trivy filesystem scan. Findings open or update a single security:dependency tracking issue per repo; clean runs close any open issue. The point is to catch CVEs disclosed after code was merged, not just what a PR introduces. The majority of real-world CVEs are disclosed against code already in production; a PR-time scan alone misses them.

Renovate — dependency-update automation. The ci-standards Renovate preset applies to every product repo via a one-line renovate.json. Renovate opens PRs to bump dependencies, pin action SHAs, and update base container images. Its github-actions manager keeps the pinned reusable workflow SHAs current, which is how new products cloned from template-product stay on the latest ci-standards workflow without manual intervention. Weekly schedule keeps the noise bounded.

Three things are on the roadmap but not shipping yet.

Semgrep (SAST) — the security baseline mandates Semgrep OSS for static-analysis coverage as a free-and-open-source substitute for GitHub Advanced Security. Rollout across all products is in progress.

OWASP ZAP as a scheduled cron against dev — today, ZAP runs at PR time against the preview URL. The plan is to also run ZAP against the shared dev environment weekly to catch drift between preview builds and the environment that lives longer.

Pre-commit hooks — a Husky-driven local hook stack that runs the most annoying failure modes (a Gitleaks pre-commit scan, at minimum) before code even gets to the PR. Removes a full round-trip when you catch a mistake before pushing. On the shortlist to ship next.

The whole stack costs zero in licensing. Every tool above is open source. The cost is time to wire them up and ongoing time to tune false-positive suppressions. That cost is measured in hours, once, not months. What it buys is the difference between "my product survives" and "my product ends" in the scenario where I fat-finger a Terraform PR at 11pm.

That is the trade at solo scale. It happens to look like the trade at enterprise scale for exactly the same reason.

Update, 16 July 2026: what it cost, and what it caught

Someone on Hacker News asked how often these checks actually hold up my PRs. I didn't know, which bothered me, so I went and counted every PR check this stack has ever run: 2,864 of them across nine repos. 286 failed, call it 10%.

The failures aren't spread out, and that's the part I didn't expect. They arrive in tight clusters of three or four days, and every cluster sits on top of a setup event. The biggest was late April, when I pulled these checks out of Vesper and into the shared ci-standards workflow. Eighteen Infracost failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of silence. Another cluster in May when Crisper adopted the same workflow. Another in July when I bootstrapped some skeleton repos that are parked until Vesper ships. Between the clusters it's quiet for months. The repos with the worst failure rates are the ones that have only ever been in setup and have no steady state to average against.

So the cost isn't a tax on every PR. It's a bill that comes due when I migrate something, and it's mostly me misconfiguring my own plumbing rather than the scanners crying wolf.

The other surprise: tests, not security, are what actually block me. Failing tests account for roughly half of every PR check failure I've ever had. All the security tooling combined is well under a quarter.

What the security tools have caught, honestly: not much. Trivy found a real CVE in a transitive form-data dependency and I bumped it. Checkov found a real IAM policy attached to a user instead of a role. It also flagged AKIAEXAMPLEEXAMPLE00 in a .env.example, and read a Terraform format("postgresql://%s:%s@...") string as hardcoded credentials when the password was a random_password reference. Which is to say it caught the correct implementation and would have sailed straight past a bad one.

And then, while I was writing this very update, the pull request carrying it went red. Trivy flagged CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I'd changed. It was disclosed against code I merged weeks ago, and the PR-time scan is simply where I happened to meet it. Bumped to 0.7.5, green again, about six minutes of work.

So: ~~two real findings in four months~~ three real findings in four months, one of them CRITICAL, and the third arrived while I was typing the sentence that said there were only two.

That's still a thin haul if you measure it the way a security team measures things when it needs headcount approved. But that was never the trade. The premium is a few hours per migration and zero dollars, and what it covers is the scenario where I fat-finger a Terraform PR at 11pm and end the product. At that price I don't need it to catch much. I need it to catch the one. It just caught one while I was mid-sentence arguing it rarely does.

A correction and a clarification to the post above, while I'm here.

The post says the tuning cost is "measured in hours, once, not months." Hours is right. Once is wrong. It's hours per migration, and every new repo keeps rediscovering the same false positives because I've written a slightly different allowlist in each one instead of fixing template-product so the next repo inherits it. That's on me, not the tooling. For the full effect: publishing this update tripped Gitleaks on the AKIAEXAMPLEEXAMPLE00 two paragraphs up, and I fixed it by writing yet another per-repo allowlist. Which tells you exactly how much I've learned.

On ZAP: what runs at PR time today is the baseline passive scan. Full active DAST is on the roadmap, not shipping yet.

What's next

Next in the CI/CD sub-series: the preview environment pattern that ZAP runs against. Vercel plus a fresh Neon Postgres branch per PR, how the reviewer gets a real running app talking to a real database on every PR, and why that costs almost nothing.

I'm one person, and my CI does what a mid-size eng org's does. Here's why.

Allen Brooks — Thu, 30 Jul 2026 10:36:36 +0000

Corporate CI is often the red-headed stepchild of the dev team. Written six years ago by a consulting firm, held together with cron jobs whose owners left in two reorgs, touched only when it breaks. Nobody on the current team could explain what half the steps do.

Lionshead's CI runs a security suite, an Infracost cost gate, IaC scanning, secret scanning, schema-migration validation, and a preview-environment deploy before any PR can merge. Categorically the same coverage as many mid-sized engineering orgs.

I am one person.

The obvious question, from other indie founders, is why. The reusable workflows in ci-standards took months to write. Nobody is going to breach my pre-launch SaaS in month three. Why not push to main and let Vercel deploy?

Because this is the way cloud software is built. I don't know how to build it any other way, and I don't want to relearn. There's no three-person dev team here, no DevOps engineer, no IT department. Nothing about being alone changes what "shipping safely" means. It only changes who has to remember it, and if the answer is me, then the answer needs to be automation.

The shops I have seen succeed treat the CI/CD pipeline as first-class code. The developers who write the product also write the pipeline. The knowledge of how one flows into the other compounds over the years. The shops I have seen fail leave CI to consultants, or hire it out to a platform team that has never shipped their product, or worst of all, both. In my experience, no exception to that pattern has stuck. The shape of the CI pipeline is a leading indicator of how the whole engineering culture works.

Lionshead's pipeline is four areas: build, test, security, deploy. Two of those areas make choices unusual at solo scale.

Security. Trivy for vulnerabilities and IaC, Gitleaks for secrets, Checkov for policy, plus linters that keep the workflow files themselves honest. All open-source, no licensing cost. Any one of these is a one-line YAML addition and takes ten minutes to wire up. The unusual choice isn't setup. It's committing to run all of them, keep them maintained across every product repo, and enforce them as merge gates when the only person they block is me.

Preview environments. Every PR spins up its own Vercel deploy AND its own fresh Neon Postgres branch: a copy-on-write clone of prod's schema. The reviewer clicks the preview URL and gets a real running app talking to a real database. Schema migrations run against actual Postgres before merge, not against a mock or a shared dev database. When the PR closes, the branch is destroyed. This costs nearly nothing because Neon bills for storage and compute, not for branches. I have not seen this pattern at solo scale anywhere else.

The check that deserves specific mention beyond those two is Infracost. It runs on every Terraform PR and requires human approval when a change would add $50 or more to my monthly cloud spend. Fifty dollars is not the number that would ruin me. Fifty dollars is the number where I would want to look twice at the diff. Between an agent auto-completing a Terraform block wrong, and me clicking through a review too fast, I would rather burn a merge-queue slot than wake up to a $10,000-a-month resource I did not mean to create. Every solo founder running on their own cash should have this exact gate. Almost none do.

All of it runs end-to-end in about five minutes. That's not despite the number of checks. It's a function of designing the pipeline with parallel jobs and path-gated conditionals from day one, not retrofitting them on after the fact.

The line between too much CI and not enough CI is a straightforward one. It lands wherever developers no longer need to remember how to get code into a live environment, or troubleshoot when it doesn't. Below that line, you pay a tax on every ship in ceremony that used to happen automatically. Above that line, you pay a tax on every setup in ceremony that never had to happen. My CI investment sits right at the line for one person. It happens to look like the CI at a mid-sized org because the line is drawn by the same physics, not by the size of the team.

What's next

This post is the "why." The rest of the CI/CD sub-series is the how. Coming up: the specific security checks that ship with every Lionshead PR, how the Vercel + Neon-branch preview environment is provisioned, the OIDC secret flow through Doppler, and the path-gated test routing that keeps a five-product portfolio building against one shared workflow.

Every website should have a colophon

Allen Brooks — Thu, 30 Jul 2026 10:36:34 +0000

Every developer I know has the same itch. Pop the hood on a website, a SaaS app, an app on the App Store, and see how it was built. Which framework. Which hosting. Which type. The craft underneath the surface.

The colophon page is the formal scratch to that itch.

The word comes from publishing. A colophon is the small inscription on the last page of a book that tells you the typeface, the paper, the press, the binding. The book's how. Stripe Press still does this. Information Architects does this. A small corner of the web that takes its lineage from print has been doing it for years.

I had never heard the word until six months ago.

That isn't a humblebrag. Ten-plus years in corporate engineering, up through senior director, and the word "colophon" never came up. The companies I worked at shipped real software to millions of users, and not one of them had a public page that said "here is how we built this." It wasn't anyone's job. It wasn't a category.

I came across it earlier this year, looked it up, read a few examples, and built one that weekend.

Lionshead's colophon is at lionshead.digital/colophon. Stack, type, color source, build method, hosting. One screen, no decoration. The same structure now ships in every Lionshead product per a company rule. They all have colophons.

I added it because building things can be a craft, not just a job. Corporate engineering taught me to ship. Lionshead is teaching me to sign my work.

Keeping 10 repos in sync without a monorepo or a platform team

Allen Brooks — Thu, 30 Jul 2026 04:39:20 +0000

Most dev shops don't have standards. At least not codified and documented standards. They have a Confluence space with someone's 2021 attempt at a style guide buried under three years of half-finished runbooks and obsolete service docs. The rules aren't enforced because nobody reads them. The shops that do enforce standards pay for it: a platform team, a custom lint server, a quarterly review cadence, and someone whose job is to argue with developers about line length.

Lionshead has standards, and they're enforced for free. They live in code, in .claude/rules/ and a BMAD module, and the AI agents doing the keystrokes read them on every story. There's no human in the middle to forget the rule; the agent reads it and writes the code in the same breath. That's not the same trade as "humans who follow the rules better." The AI is doing the writing too; the free enforcement is a side effect of swapping the actor, not a process improvement layered over a human team. That changes the problem. The question stops being "how do we get developers to follow the rules" and starts being "how do we get the same rules into ten unrelated product repos without a monorepo or a platform team."

This post is the answer. A small CLI I call the reconciler: pull-based, hash-compared, idempotent. It ships a manifest from a central repo and lets each consumer adopt the parts it wants. Here is what it does, what it costs to build, and where it ends.

I have operated at the ten-repo scale before, and the reconciler is built for that target. Lionshead today is one product (Vesper); this is the system for where I am going, not where I am.

The shape of the problem

The familiar approaches to keeping many repos in sync all trade the same three axes: friction at edit time, friction at consumer time, and how much shared infrastructure the consumers have to swallow. Pick any two, lose the third.

Copy-paste. Zero infrastructure. The cost shows up at every fix: ten repos, ten patches, and the moment one repo skips a patch the divergence starts. Worse, copy-paste makes the canonical version invisible. No one can grep it without first deciding which repo's copy to trust. I shipped two months on copy-paste before the cost compounded; it compounds fast.

Git submodules. One source of truth, a real link between consumer and producer, and a developer experience that is genuinely awful. Submodules break IDE checkouts, break clone scripts, and force every consumer to handle a sub-checkout step that someone new will get wrong on day one. I tried submodules for two days; the cost showed up faster than copy-paste.

Monorepo. Solves drift completely. Also forces every product onto a shared CI, a shared release cadence, and a shared dependency graph. That cost is paid even by products that have nothing to do with each other. The big-company default; the wrong default for the operator who runs ten small unrelated things.

All three trade the same axes in different proportions. The reconciler trades a fourth axis: it accepts a small per-consumer install cost in exchange for keeping every consumer's repo shape, CI, release cadence, and dependency graph independent. That is the only trade I was willing to make at the ten-repo scale.

What the reconciler actually does

The CLI is published to GitHub Packages under the @lionsheaddigital scope. Consumers configure their npm token once per machine. After that, npx @lionsheaddigital/setup just works.

The CLI reads a single declarative manifest, lionshead.json, from the central ci-standards repo over plain HTTPS to raw.githubusercontent.com. The manifest lists every standards file the central repo owns: BMAD module sources, the Claude Code rule files, GitHub Actions reusable workflows, the GitHub project sync script, and a few others. Each entry carries a sha256 hash of the upstream file.

For every file declared in the manifest, the CLI walks one branch:

The heart of the loop is a hash comparison:

// packages/setup/lib/reconcile.mjs (excerpt)
function computeDelta(remoteStandards, localInstalled) {
  const remoteFiles = remoteStandards?.files ?? [];
  const localFiles = localInstalled?.files ?? [];

  const localMap = new Map(localFiles.map((f) => [f.path, f.sha256]));
  // ...

  const toDownload = [];
  for (const rf of remoteFiles) {
    const localHash = localMap.get(rf.path);
    if (localHash !== rf.sha256) {
      toDownload.push(rf);
    }
  }
  // ...
}

If a consumer's local copy of a declared file already matches the upstream hash, the CLI does nothing for that file. If the hash differs (the consumer is missing the file, or upstream has moved on, or someone hand-edited the local copy), the CLI downloads the upstream version and overwrites the local one. No prompt. No merge. No three-way confusion. Upstream wins; local edits to managed files are not allowed.

The full run on a clean consumer takes about three seconds. The full run on a clean, up-to-date consumer takes about one second and prints "Standards up to date." That second case is the one that earned the reconciler its name: it is safe to run a hundred times in a row, and the only side effect is that the operator knows the consumer is current.

The hard parts

The interesting design decisions in this kind of CLI are not the visible ones (the manifest shape, the file copy). They are the boring ones underneath.

Idempotency is the first hard part. The hash-compare and the conditional download together give you "do nothing when nothing has drifted." That looks easy until you find the case where BMAD's own installer silently overwrites a managed file with stale bundled content after the main download loop has already updated it. The reconciler now does a second verify-and-restore pass after BMAD runs; without it, a clean run would show "Standards up to date" while the on-disk SHA had quietly diverged from the manifest. That pass cost a day to discover and an afternoon to write. It is the kind of bug that only shows up at the seam between two tools that both think they own the same files.

Partial adoption is the second hard part. Not every consumer wants every standard. A static marketing site does not need the Test Architect workflow; a backend service does not need the design-tokens rule. The manifest supports per-consumer opt-outs via a small lionshead.json in the consumer repo that lists explicit skip paths. The escape hatch is necessary; without it the reconciler is a coupling mechanism dressed up as a distribution one.

Upgrade safety is the third hard part. The reconciler is itself one of the things it ships. Bootstrapping a brand-new consumer is a chicken-and-egg moment: the consumer does not yet have the CLI installed, but the CLI is what installs the CLI. The answer is npx: the CLI runs without a prior install, fetches itself from GitHub Packages, runs once, and exits. The local install only matters for subsequent runs, and those can be npx too. The whole bootstrap is one command.

When NOT to do this

The reconciler is right anywhere above three repos. Below three, copy-paste is genuinely cheaper, and the per-file maintenance cost is small enough that the operator can absorb drift mentally. The architecture has no real upper bound on consumer count: the CLI is stateless, every consumer is independent, and adding one is one npx away.

The real ceiling is governance, not scale. Review doesn't disappear in this model; it shifts upstream to the central standards repo, where PRs are the negotiation surface. That works fine when there's an agreed owner of the central repo. It breaks down when there isn't: the reconciler's "upstream wins" semantics turn a governance question into a technical one consumers can't appeal. A monorepo handles disagreement by making every change visible to all consumers in the same commit; the reconciler doesn't.

And the reconciler is wrong when drift is a feature, not a bug. Research codebases, demo branches, throwaway prototypes: these benefit from divergence. A pull-based "upstream wins" CLI run against a demo branch will quietly undo the demo. The right fix there is to mark the consumer out of scope, not to weaken the reconciler.

Outro: the Building Lionshead thread

The reconciler is one chapter in the Building Lionshead series. Post one was a sketch of why I built a company-specific BMAD module; the post you are reading is what the distribution mechanism for that module looks like in code. The series's spine is "small-company process at small-company cost"; this post is one chapter of it.

The next post in the series will be about the BMAD module itself: what is in it, what the boundary is between Lionshead-shaped behavior and stock BMAD behavior, and why I keep that boundary thin. After that, a post on the rule files in .claude/rules/ and the way they get loaded into agent context. Both posts depend on the reconciler existing; this post is the prerequisite.