DEV Community: Allen Brooks

I handed my backlog to an agent fleet for ten days. Three of its safeguards were lying to me.

Allen Brooks — Mon, 10 Aug 2026 10:27:41 +0000

I have a launch with a date that does not move. I also have a job between eight and five, a two year old, and a four month old. Everything I build, I build around those. So the binding constraint is not taste or architecture. It is how many stories I can carry from "ready" to "merged" in the hours that are left, which for the next couple of months is most of the hours I am not at work or holding a baby.

The obvious lever is to stop being in the loop for every story. Ten days ago I wired up a fleet that pulls from my story queue, builds each story, reviews it, and merges it. Then I pointed it at the product with the deadline and let it run.

It worked, in the sense that 503 commits landed across 168 pull requests in the ten days that followed.

I should say plainly what that means, because it is the fact most likely to be misread. I did not write any of them. Not one line of that product's code has been hand-written or hand-edited by me in months. I instruct, and I review. Claude writes. Every commit below is an agent's work, including the three failures.

It also lied to me three times, in three different ways, and not one of the lies announced itself. This is a post about the three. I have written two posts in a row about the gap between a check existing and a check running: green is not evidence, then make it fail on purpose. I did not expect to find the same disease in a system I built with that lesson still fresh.

The shape of the machine

Two roles, deliberately unequal.

The planner is the expensive one, and the only part allowed to hold the whole graph: what is in the backlog, what the written strategy says matters, what depends on what, which stories produce something another one consumes. It fills the ready queue itself rather than waiting for me to hand-pick the next card, because a machine that needs me to curate its input every cycle has relocated the work rather than removed it. It behaves like a scrum master rather than a filter: when the roadmap needs a story that does not exist, it writes the stub.

The executor is the cheap one, and it is deliberately dumb. It carries stories through create, build, review, merge. It does not re-derive dependencies. It does not resolve ambiguity. Every unclear thing is an escalation rather than a judgment call, either back to the planner or to me. The rule I wrote at the top of its instructions is the only one that really matters: cheap but humble, never cheap but guessing.

Three things about how it spends money, because they are what people ask about and they are the reason it can run unattended at all:

It picks the model per step. Spec-writing and mechanical work go to a cheaper tier, implementation to a stronger one. The code reviewer is pinned to the top tier and nothing is permitted to cheapen it, ever, because a cheap reviewer defeats the only reason any of this is allowed to merge without me.
It watches its own spend. My usage telemetry exports to Grafana, the fleet reads it back, and it compares this week's burn against a target line. When it is running hot it cuts its own concurrency first, before it touches anything else. It throttles itself so that I do not have to notice.
It works the gaps between stories, not just the stories. Every story declares what it produces and what it consumes, and the fleet checks that anything being consumed has a producer that already landed.

I am going to write the full build-out as its own post, because the mechanism deserves more room than this one can give it. What matters here is that third bullet, because that is precisely where the third lie turns out to live.

Around that sit the safeguards, and the safeguards are the whole reason I was willing to let it merge anything without reading it first. Some of them police what the fleet is allowed to start:

A scope gate, so it only builds work inside the milestone I am currently shipping: the card that serves the launch, not the most interesting one.
A cap on work in flight, so a bad pattern can only contaminate a few stories before I see it.
Dependency gating, so a story does not start until the things it depends on have landed.

Others police what it is allowed to finish:

No self-review. The thing that reviews the code is a different instance than the thing that wrote it. Non-negotiable, and the reason review is a separate spawn rather than a continuation.
A live design review on any story that touches a real user-facing surface. A reviewer drives the running preview deployment, looks at the rendered result in light and dark, checks the empty and error states, and returns a verdict with evidence.
A human-eyes lane. Anything touching auth, billing, money movement, schema migrations, or the merge gates themselves gets built and driven to green automatically and then parked, unmerged, for me to click.

And one polices the whole run: the budget throttle above.

That is the design. Three of those safeguards were lying, and a safeguard lies by reporting a protection it is not providing.

What I am not going to claim

I went and counted, instead of trusting my memory.

The last full week before the fleet existed: 403 commits. The first full week after it: 401. Flat. If you were hoping for a throughput multiplier, it is not in this data, and I am not going to manufacture one out of a chart that does not show it.

One thing makes that flat line more interesting than it looks. I had been out of work for four months. My first day back at an eight-to-five was July 29th, and the fleet landed on the 30th. So the 403 week was a week when this product was my full-time job, and the 401 week was the first week it emphatically was not.

That is the honest claim. Not that the fleet made me faster. That it held my pace steady through losing most of my waking day to something else, which for the next couple of months is the only result I actually need.

What changed underneath the flat line is what a single unit of work costs me.

Before, every story meant opening three sessions by hand: one to write the spec, one to implement it, one to review it. Dozens of times a week. The cost was never the typing. It was that each one required me to stop what I was doing, decide it was time, and start something. That is a tax on attention, and attention is the resource I am actually short of.

The fleet instantiates all three the moment they are due, in order, without asking. I read outcomes instead of driving steps. The same output for a fraction of my attention is a trade I will take every single time, and it is the honest version of what happened here. It is also not the version that fits in a headline, which is exactly why I want it stated before the interesting part.

Lie one: the instructions the fleet was running were not the instructions I had written

The fleet's behavior lives in instruction files. Those files live in a company module that gets distributed to every product repo, and the copy the agent runtime actually loads is generated separately, into a different directory.

Except it was not generated. Nothing generated it. The copies the fleet was really loading had been made by hand months earlier, and they had drifted thousands of bytes behind the source I kept carefully editing.

So a real batch ran against an executor missing two of the safeguards I listed above as though they were operating: the scope gate and the live design review. Neither fired, because neither was in the file it was reading. The planner's copy was worse, missing four of its sections and its entire milestone model.

This is a known hazard in the underlying tooling, not a clever novel bug of mine. The same class of gap has been reported upstream more than once: skills installed into a custom module's own directory without the registration that makes them loadable (BMAD-METHOD issue #2107), and custom-module agents installed without the commands that make them invocable (issue #1131). Both are closed now, so I am not pointing at an open ticket and calling it my excuse. I am pointing at prior art: this is a well-worn hole, and my module fell in it.

What kept me in the hole was not the tooling. The instruction file's own header stated, in confident prose, that the installer generated that copy "exactly as it does" for the built-in modules. That sentence sat at the top of the file for months. It was wrong. A document asserting that a mechanism is wired, sitting inside the mechanism it is wrong about, is the purest version of the failure I wrote up two weeks ago.

Lie two: the gate that should have caught lie one read silence as approval

Two stories touching user-facing surfaces merged with no design review at all. Not a review that passed when it should have failed. No review.

That is the same incident as lie one, viewed from the other side. The drift explains why the reviewer never ran. It does not explain why nothing noticed. The gate looked for a rejection, found none, and let the story through, which means a missing reviewer and a clean pass produced identical output.

Two independent bugs stacked on one control: the reviewer was absent, and the gate was built to be silent about absence. Either one alone is survivable. Together they are indistinguishable from working. A gate that only blocks when it produces a rejection is not a gate, because the most common way for a gate to fail is not to run at all.

Underneath both sits a third temptation, and it is the one I would warn you about most: letting the code reviewer "also cover the UX." It cannot. A code reviewer reads class names and token references off a diff. A broken layout, an unreadable dark mode, an empty state that renders as a blank rectangle: none of those are visible in a diff. Folding a check into a reviewer that structurally cannot perform it is not a cheaper version of the check. It is the absence of the check, wearing its name.

The rule now is one line, and it is the rule from two posts ago in a new costume. Exactly two artifacts satisfy that gate: a recorded pass with live evidence attached, or my explicit written waiver. Anything else, silence very much included, is a block.

Lie three: a merge that made another story's premise false, with nothing to notice it

The first two came down to a control that was missing, and a gate that could not tell it was missing. This one was there, ran exactly as designed, and reported a clean board anyway. It is the one I find genuinely hard, and the one I would not have predicted.

The fleet watches the seams between stories by checking dependencies. When a story merges, it asks what that merge unblocked. That is a set-membership question and it is cheap, which is why a mechanical executor is allowed to do it.

But a merge can break another story without touching a single dependency edge. A design document is full of statements about the world: this thing is not wired up yet, nothing reads this yet, this path is inert. Every one of those was true when it was written. A merge can turn any of them false while the story that rests on it is still being built, and nothing in the dependency graph moves at all.

That happened. One story merged and quietly falsified a premise that another in-flight story's design rested on. The dependency check saw a clean board and kept pulling. A human caught it by reading, which is not a mechanism I can schedule.

The fix is embarrassingly small, which is the good kind of small. After a merge, take the names of the surfaces that the merge touched, and grep the in-flight branches' design docs and changelogs for those names. Do not evaluate whether the claim still holds in spirit, because that is judgment and the cheap executor is not allowed to have any. Report that an in-flight document names something this merge just changed, and let the planner decide. One command, and it would have caught it.

What is not built

I have ended the last few of these with an honest list of what I have not finished, because a half-built system that thinks it is finished is worse than one that knows it is halfway. Here is the list for this one.

The design calls for a retrospective pass after each batch: a set of specialist lenses that look at ten merged stories together and catch the drift that no single story's review could possibly see, because the defect only exists in the space between them. Three stories that each independently invented a slightly different pattern for one job, and each passed review on its own terms. A feature that is now complete and undocumented because no single story owned the seam. That review is written down, it is accepted, and it does not exist. Only the planner and the executor are real.

Which means the compounding half of this system, the half where what batch ten teaches gets fed back so batch eleven starts from a higher floor, is currently a document. The throughput half is running. The learning half is a plan.

There is one detail I cannot leave out. That unbuilt retrospective specifies six lenses, and one of them is a documentation lens whose stated job includes mining these posts out of what a run just taught. This post is that lens's output, produced by hand, because the thing that was supposed to produce it has not been built. I am not sure whether that is funny or just accurate.

What it costs

The fleet spends my weekly model allowance, which is why the throttle exists and why it cuts parallelism before it does anything else. What it may never do is cheapen the reviewer.

The real cost is subtler. Every safeguard I listed is a claim about a system I could not watch directly, and I was making those claims to myself, in documentation, and believing them. Three of them were reporting a protection they were not providing, and I did not learn that from an alert. I learned it from reading, from a batch behaving oddly, and in one case from a person noticing something no check was positioned to notice.

The part I did not expect

The honest summary of ten days is not "agents built my product." It is that agents built my product, and the part of the system I trusted most, the safeguards, was the part with the highest rate of quiet failure.

Which follows, if you sit with it. The feature code announces its failures, because a broken feature is visible the moment someone uses it. The safeguards are the only components whose success looks exactly like doing nothing, so they are the only ones that can fail for months in complete silence. I have now found that same shape in a Vale rule, in a CI check, and in an agent fleet, and I no longer think it is a coincidence about any of those three things. It is a property of anything whose job is to sit quietly and not fire.

What's next

The unbuilt half. I am going to build the retrospective pass and run it against a real batch. Then I will write up whether specialist lenses reading ten merged stories at once find the drift the design promises, or whether that is a fourth thing I believed because I wrote it down.

Killing the long-lived tokens in my CI, one exchange at a time

Allen Brooks — Mon, 03 Aug 2026 02:26:14 +0000

A static API token stored in GitHub Secrets is a breach that has not happened yet. The timer is already running. You do not know how long the fuse is, because a long-lived token's fuse is however long you set it at mint time, which for most people is "never thought about it."

At enterprise scale a leaked CI token is an incident: rotate it, audit the blast radius, file the postmortem. At solo scale it is the same product-ending event as any other breach. My CI can reach Doppler, which holds every secret every product uses. A token that lets it do that, sitting in a secret store, readable by any workflow, valid for months, is precisely the thing I do not want to own.

So the goal is to own zero of them. Not "rotate them on a schedule I will forget." Zero. The mechanism that gets you there is OIDC, and the good news is that it is strictly better by default: less to store, less to leak, less to remember.

The mechanism, in one paragraph

Instead of storing a token, the CI job proves who it is at runtime and trades that proof for a token that expires in minutes. GitHub Actions can mint a short-lived OIDC token for a workflow run, signed by GitHub, carrying claims about which repo and which workflow asked for it. Doppler has a matching feature, Service Account Identities (a paid capability, which I will price out in the cost section), that validates such a token against a set of expected claims and hands back a short-lived Doppler session token in exchange. The job posts the GitHub token to POST https://api.doppler.com/v3/auth/oidc with an identity slug, gets back a session token good for ten minutes to an hour, uses it, and the run ends. Nothing durable is stored anywhere. (I skip Doppler's off-the-shelf secrets-fetch-action, which only injects fetched secrets as environment variables; my workflows need the raw session token itself, to hand to doppler run and to Terraform as TF_VAR_doppler_token.)

Why a static service token is the wrong shape

Three properties make a long-lived token structurally bad, and they are the same three that pushed the GitHub side off personal access tokens in an earlier migration:

Lifetime. The token is valid until someone remembers to revoke it. Rotation is a manual ceremony with no calendar attached. The compromise window equals the lifetime, and the lifetime is effectively forever.

No identity. A Doppler service token has a label, not a workload-bound identity. When it is exercised, the audit log names the token, not the caller. A leak shows up as "that token did something," with no provenance about which workflow in which repo actually made the call.

No revocation chokepoint per purpose. One shared token feeds many consumers. Revoking it kills all of them at once. Granular, per-purpose revocation means minting per-purpose tokens, which nobody does, so in practice you get one big token and an all-or-nothing kill switch.

OIDC fixes all three at once. The credential lives for minutes, it is bound to a named service-account identity with claims that pin it to a specific repo, and each logical purpose gets its own identity you can disable independently.

What actually runs on OIDC today

Here is the concrete state, not the aspiration.

A word on "per-product" first, because it runs through everything below. Lionshead is a set of small separate repos, one per product, rather than one monorepo. I wrote about that setup and why I chose it in the reconciler post. The practical effect here is that most of this machinery is replicated per repo: one identity, one Doppler config, one token scope each, so a compromise in one product cannot reach another.

Terraform plan and apply, every product, no stored token. Every terraform-plan (on PRs) and terraform-apply (on merge to main) authenticates to Doppler by OIDC. The caller declares permissions: id-token: write, passes a per-product identity slug (for this repo, LIONSHEAD_WEB_DOPPLER_TF_IDENTITY_SLUG, a GitHub variable), and the reusable workflow does the exchange. The slug is not a secret: it is a stable identifier that is worthless without a GitHub-signed token carrying the right claims, so it lives as a plain repo variable that Terraform itself manages.

Three named service accounts, not one shared token. The setup provisions ci-platform-services-tf (admin, OIDC, session TTL 3600s, one hour), ci-credential-sync (OIDC, session TTL 600s, ten minutes, the job that writes org secrets into per-product Doppler configs), and one transitional static-token account (covered below). Each has its own claims and its own independent off switch.

The Terraform-managed identity itself. The Doppler provider has a native doppler_service_account_identity resource with a config_oidc block (discovery_url, claims, ttl_seconds). The thing that grants OIDC access is itself declared in code and reviewed in a PR, not clicked into a dashboard. The one exception is the very first admin identity, which cannot create itself: that is a one-time manual step in the Doppler dashboard, which I wrote down as the single documented exception it is, so it is not a mystery to future me.

A fallback that fails loud, not silent. The Terraform callers explicitly pass doppler-token-secret-name: ''. The reusable used to default that input to DOPPLER_TOKEN, so if an identity slug ever resolved to empty (org variable deleted, repo forked, a run context that cannot see org variables) and a bare DOPPLER_TOKEN happened to exist in scope, it would quietly slide back onto the static path. Passing an empty string forces a loud OIDC failure instead of a silent downgrade to the thing I am trying to delete.

A guard that proves it stays gone. The old workplace DOPPLER_TOKEN GitHub org secret was deleted on 2026-06-09. A scheduled audit workflow (security-audit-doppler-token.yml) watches for its reintroduction, and post-deletion Terraform runs across three products came back green with zero deprecated-static-token warnings, which is the only actual evidence that every Terraform caller is really on OIDC. (If that sounds like a theme, it is the make-it-fail-on-purpose rule from the last post: the guard is only worth something because I watched the audit catch a planted token before I trusted it.)

What still holds a long-lived token

This is the part every "we adopted OIDC" post leaves out. I have not finished. Here is exactly what is still static, and why.

The per-product Terraform bridge. Per-product Terraform still needs a Doppler token for two things: its own Doppler provider block, and its Vercel-sync REST calls. That second one is worth explaining: those calls register a Doppler-to-Vercel integration, which is what pushes each product's secrets out of Doppler and into that product's Vercel project environment, so the deployed app actually has its config at runtime. Terraform sets that sync up through Vercel's API, and the call authenticates with a Doppler token. An OIDC session token (TTL 3600s, one hour) would expire partway through a long apply, so one service account, ci-per-product-tf-bridge, keeps a long-lived token. The one mitigation that matters: that token lives only in Doppler's own state and the per-product config it populates. It is never written to a GitHub org secret. It is explicitly labeled transitional; the follow-up is per-product OIDC identities that retire it.

The Vercel build wrap. The vercel-deploy jobs still wrap vercel build in doppler run using a per-product static secret (LIONSHEAD_WEB_DOPPLER_SERVICE_TOKEN_CI) to inject build-time secrets like the Sentry auth token. This one is a plain GitHub secret and it is next on the list.

Note syndication. The workflow that cross-posts these notes to dev.to, Hashnode, and Mastodon uses that same per-product static token to pull the app tokens out of Doppler. The workflow file literally carries a # Migrate to Doppler OIDC comment, because I wrote the honest TODO in the place I would actually see it.

The Terraform state-backend keys. The R2 credentials that read and write remote Terraform state (LIONSHEAD_WEB_TFSTATE_R2_*) are static repo secrets issued by platform-services. These are S3-style keys, not Doppler tokens, so they are a different migration on a different provider's timeline, but they are static and I am not going to pretend otherwise by leaving them off the list.

So: the Terraform authorization path is done. The build, deploy, and syndication paths are not. Naming the gap is the point. A half-migrated system that thinks it is finished is worse than one that knows it is halfway.

What it costs

The exchange adds one HTTP call and a few seconds to each job. The provisioning is Terraform I write once and inherit everywhere. The GitHub side is free.

The Doppler side is not, and I want to be straight about it: Service Accounts and their OIDC identities are a paid feature. They start on Doppler's Team plan, which is $21 per user per month as of this writing, so if you are on the free or Developer tier this specific path is not open to you without upgrading. In my case the marginal cost of the OIDC work was zero, because I was already on the Team plan before this migration for an unrelated reason: a different product had outgrown the free tier's cap on secret syncs. That is luck, not a saving, and I would rather you hear the real number than a "no vendor to pay" line that only holds if you happen to be pre-upgraded.

Against that cost, what I stop owning is a set of long-lived credentials to my entire secret store, each one a standing invitation I have to remember to revoke. At solo scale I would rather have a mechanism that forgets on my behalf, every single run, than a discipline I have to sustain forever.

What's next

Next in the CI/CD sub-series: path-gated test routing across a five-product portfolio. How one reusable workflow decides which tests to run based on what a PR actually touched, why the path gate is shared but the runner command is deliberately caller-owned, and what breaks when you get that split wrong.

How do you know a passing check still works? Make it fail on purpose.

Allen Brooks — Thu, 30 Jul 2026 10:49:15 +0000

A feature makes money. A check protects it. Those are not the same job, and the difference is why checks rot.

When you ship a feature, the market tells you fast whether it works. Someone signs up or they don't. A check has no such feedback loop. A check that has quietly stopped working looks exactly like a check that is passing, and nothing outside the build can tell you which one you have.

At corporate scale a QA team is paid to close that gap. Their whole job is the negative case: break the thing on purpose and confirm the alarm rings. At solo scale there is no such team, and the negative case is the first thing to fall off the list, because it is the work that sits furthest from revenue. Features feel like money. Checks feel like insurance. The trap is believing features are money. Features are money-making. Checks are money-protecting. Skip the second one and you don't notice for months, which is exactly the failure mode I wrote about last time.

The count I had been avoiding

In Green is not evidence I found four checks that had never done anything. An em-dash linter whose regex could not match. A security scanner that was never installed. A cost gate that was only ever red. A config file I had documented but never written. The post ended on a rule I was very pleased with: make it fail on purpose. Every gate should have a known-bad fixture, something you can point it at that must fail, because if you have never watched a check fail you do not have a check, you have a decoration that emits green.

Then I did the obvious thing I had not done. I counted how many of my own gates actually had one.

The answer was humbling. Across my repos I run several dozen distinct gates. The reusable workflow every repo calls runs nine third-party scanners: Trivy twice, Gitleaks, Checkov, Hadolint, Actionlint, Shellcheck, ZAP, and an Infracost cost gate. Not one of them has a fixture. My content pipeline runs four gates on every note: prose style, spelling, frontmatter schema, link integrity. Not one of them has a fixture. The em-dash rule, the one this whole series keeps circling back to, had none.

What did have fixtures? Only the checks I had written myself. The little scripts that scan for hardcoded hex colors, or unpinned GitHub Actions, or drift in my shared config files: every one of those has a negative test that feeds it bad input and asserts it goes red. I have a unit-testing habit, and it reaches exactly as far as the code I author. Everything I bought, and every rule I configured inside someone else's tool, I took on faith. The fixture was present wherever I already had the habit, and absent everywhere I was trusting a vendor or my own YAML.

The two places I had gone further

Twice, I had wired the negative case into CI itself, so it runs forever instead of once. One is a schema gate for my product metadata: CI points the validator at a deliberately invalid file on every run and fails the build unless the validator rejects it. The other is my design-token scanner, which re-runs its own "this hex literal must be caught" test inside every repo that calls it. Two, out of several dozen. Both, again, on code I wrote.

Has any of this ever caught a regression?

Here is the part I have to be honest about, because the whole point of the last post was honesty about what a practice actually catches. Has one of these fixtures ever caught a real regression? A gate that silently broke, and a fixture that screamed?

No. Not once.

Every real catch I can point to was something else. The em-dash breakage from last post I found by hand, poking at it, not because a fixture rang. The one genuine vulnerability my pipeline caught recently, a critical CVE in a transitive dependency that showed up the week I was writing the last post, was a live scanner doing its ordinary job, not a canary. And the closest thing I have to a fixture story runs the wrong way: my hex-color scanner once misfired, flagged a pull-request number as if it were a color, and blocked a real merge. I fixed it by adding the negative tests I should have had first. The fixture did not prevent the bug. The missing fixture allowed it, and I added the fixture afterward, as an apology.

So this is not a victory lap. I am writing down a rule I have mostly not been following.

So I built one

Then I built the first one properly, on the gate that most deserved it.

There is now a file in my repo whose only job is to be wrong. It contains a single em dash and a line telling you not to remove it. It never renders and it is not a post. And there is a step in CI that points Vale at that one file and inverts the result: if Vale reports the em dash, the step passes; if Vale comes back clean, the step fails the build and tells me the gate has gone inert. It is about fifteen lines. The em-dash rule had been silently dead once already, for the entire life of the repo, and nothing would have told me if it happened again. Now something will.

The first time I ran it, I broke the rule on purpose and watched the build go red. That is the only moment in this whole exercise that constitutes actual evidence the check works. Everything before it was faith.

A fixture is a second thing that can rot

I want to be clear-eyed about the obvious objection. A known-bad file that quietly stops being known-bad is just more green, one layer down. Adding a fixture and never watching it fail leaves you exactly where you started.

So the discipline is not "add fixtures." The discipline is: break the thing once, on purpose, watch the alarm ring with your own eyes, and then wire that watching into CI so it repeats without you. The fixture is not the point. The watching is the point. The fixture is just how you make the watching automatic, and cheap enough that you keep doing it after the day you cared.

The rule

So here is the rule, and it is going into my standards where the checklist that opens every story will ask about it. No gate ships without a fixture that proves it can fail. And the smaller, harder half, the one I keep having to relearn: a fixture you have never watched fail is just another green light.

If you run any checks at all, you almost certainly have one that has never rung. Pick it. Break the thing it is meant to catch. Watch. It takes ninety seconds, and it is the only proof you are ever going to get.

I audited 2,864 CI checks. Several had never run, and green never told me.

Allen Brooks — Thu, 30 Jul 2026 10:49:14 +0000

Someone on Hacker News asked a one-line question about the security checks I run on every PR: how often do those checks actually hold your PRs back?

I didn't know. That bothered me more than it should have, so I counted every PR check I have ever run. 2,864 of them, across nine repos.

The answer to his question turned out to be boring. What I found on the way to it did not.

The boring answer

286 of the 2,864 runs failed. About 10%.

That number is useless on its own, and it took me embarrassingly long to see why. A 10% failure rate sounds like a tax, one PR in ten held up at the door. It isn't, because the failures are not spread out. They arrive in tight three or four day clusters with months of silence between them, and every cluster sits on top of a migration. Late April, when I pulled these checks into a shared reusable workflow: eighteen cost-gate failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of nothing. A second product adopting that workflow in May, with a commit two days later called fix(ci): resolve remaining PR check failures. Skeleton repos I bootstrapped in July that lit up the same way and then went quiet.

The repos with the worst failure rates in the whole org are the ones that have only ever been in setup, with no steady state to average against. The worst sits at 57%, and it is the template that the others get cloned from.

None of that is in the numbers. I only know it because I remember those weeks. Read cold, the dates say the opposite: that my oldest and busiest repo carries the most failures and is therefore the most burdened. It carries the most failures because it is where I did the extraction. The context that told the true story from its inverse lived in my head and nowhere else in the data. That is the benign version of a problem I was about to hit in a much worse form, one where the missing context was not in my head, or anywhere I could reach.

The number that didn't fit

I had the answer to the question and could have stopped. What stopped me was a small number that didn't fit.

I have a Vale rule in my content pipeline called EmDashes. It is twelve lines long. It is set to level: error, which fails the build. It exists because I don't write with em dashes, I never have, and I would rather they never reach the site. My drafts run through Claude, which likes them, so the rule is there to strip the ones I miss. That is stated plainly on the colophon; the rule is the enforcement.

The post that started all of this shipped with fifteen em dashes in it.

Vale ran on that post. Vale passed. On that same file, in that same run, Vale reported fifteen passive-voice alerts and ten weak-word alerts, so it was finding the file, parsing the MDX, and applying my custom styles. It reported zero em dashes in a document containing fifteen of them.

Here is why. Vale's existence check compiles a tokens list into a regex with word boundaries wrapped around it:

(?i)(?m)\b(?:—|–)\b

\b asserts a boundary between a word character and a non-word character. An em dash is punctuation. There is no word character on either side of the match to anchor the boundary against, so \b can never be satisfied, so the pattern matches nothing. Ever. Vale has a nonword: true option that drops the boundaries, and the rule works correctly the moment you add it.

That rule had sat in my repo, at error level, catching absolutely nothing, since the day I wrote it. And there is no observable difference between that and a rule that works. Both emit silence. Both go green.

It was worse than one bug

I fixed the regex, shipped it, felt good about myself, and then went looking for proof that the fix actually gated anything. It didn't.

Here is the CI step that ran Vale:

vale --no-exit --output=line web/content/notes/ | tee vale-report.txt
if grep -E ' error ' vale-report.txt; then
  exit 1
fi

--no-exit forces Vale to exit 0 regardless of what it finds. That was deliberate: print the report first, decide separately. The deciding is that grep. And --output=line emits path:line:col:check:message. There is no severity field anywhere in that format. The grep could never match. The two halves cancel each other perfectly: Vale is forbidden from failing the step, and the thing meant to fail the step is searching for a word the format never prints. That job had passed every build it ever ran. It could not have done anything else.

Then, while testing the fix, Vale flagged an em dash inside a fenced code block, which markdown scoping is supposed to skip. It turns out Vale has no built-in knowledge of the .mdx extension: without an explicit [formats] mapping it treats the file as plain text, applies no markdown scoping at all, and lints your code blocks and inline code as if they were prose. My config carried a comment confidently asserting that "Vale's markdown scoping handles fenced code blocks and inline code natively." It does not, because I never told Vale what it was reading. Six of the alerts on my notes tree were phantoms from that.

So: a rule that could not match, inside a runner that could not fail, parsing a format it did not know how to read. Three independent bugs stacked on one rule, every one of them silent, the whole assembly green. I only found the second and third because I went looking for evidence that my fix to the first one worked.

Once I saw it, it was everywhere

Then I looked at the rest, and found the same disease in three more checks.

Semgrep. My own security baseline mandates it. My own post names it. It is not wired into any workflow, in any repo. A Semgrep that was never installed reports zero findings. So does a Semgrep that is running clean.

The cost gate. Failing for months in two repos, red on every Terraform PR, for the excellent reason that I never provisioned the GitHub App token it needs to read my private modules. It was not gating anything. It was just red, in the corner, next to the checks that were working.

A config file that does not exist. My post says known-false-positive suppressions for the IaC scanner go in a .checkov.yaml. Zero of my nine repos contain that file. I have never written one. I documented a system I had imagined.

I audited the tools with tools that lied

It gets worse, because the tools I used to audit the tools had the same disease.

My first pass used gh run list --limit 100. It told me my secret scanner had fired twice. It has fired thirty-five times. The window silently truncated at a hundred runs, and my busiest repo has over five thousand. My second pass filtered to pull-request events, which silently discarded every push run. A sampled window told me one mature repo had a spotless record; its real number is 27 failures out of 313. Every one of those queries returned a confident, well-formatted, wrong answer. Not one of them announced what it could not see. I was auditing my instruments with instruments that shared their failure mode.

The thing I actually learned

A check that isn't running looks exactly like a check that's passing.

That is the whole post. Green means "nothing was reported." It does not mean "nothing is wrong," and it does not even mean "something looked." The absence of an alert is perfectly compatible with the absence of a scanner, and from outside the build you cannot tell those two apart.

It is the same shape as the 10% number, pushed to its limit. There, the surface reading inverted the truth, and only what I happened to remember could set it right. Here there is nothing to remember. A green check withholds the one fact you actually need, whether a check even ran, and no amount of context brings it back.

Make it fail on purpose

Make it fail on purpose. That is the entire technique.

The only reason I know my em dash rule works now is that I fed Vale a line containing an em dash and watched it report the em dash, then removed the fix and watched it report nothing. Ninety seconds, and the before-and-after is the proof. The only reason I know my one grandfathered exemption is scoped to a single file, rather than having quietly switched the rule off everywhere, is that I temporarily broke a different post and watched the gate catch it.

Every gate you own should have a known-bad fixture: something you can point it at that must fail. If you have never watched a check fail, you do not have a check. You have a decoration that emits green.

This generalizes well past linters. Any alarm you have never heard ring is in one of two states, and you do not know which one.

The punchline

I wrote most of this up as an update to the original post. In it, I said the security stack had caught two real things in four months: a CVE in a transitive dependency, and an IAM policy attached to a user instead of a role. Thin, I said, but the premium is near zero and the downside ends the product.

The pull request carrying that sentence went red.

Trivy caught CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I had changed. It was disclosed against code I merged weeks ago, which is precisely the case that a PR-time scan exists to catch. Six minutes to bump it and go green.

So it is three, not two, and the third arrived while I was typing the sentence that said there were two.

I would take this trade again tomorrow. Near-zero premium against a downside that ends the product is not a hard call. But "we run X" is a claim, not a fact, until you have watched X fail.

What's next

The preview-environment pattern that ZAP runs against is already written up: Preview environments are the dev loop. Vercel plus a fresh Neon Postgres branch per PR, a real running app talking to a real database on every review, for almost nothing. This post was written before that one and sat in a drawer for a week. The next one is on making the invisible checks announce themselves, which is the other half of this problem.

Every pull request gets its own app and its own Postgres branch, for $0

Allen Brooks — Thu, 30 Jul 2026 10:43:56 +0000

Every place I've worked plumbed everything to one shared dev environment. One database, one deploy, everyone's half-finished work landing in the same place. The database slowly filled with test accounts named asdf and orders for negative dollars, until someone declared bankruptcy and refreshed it. Then it filled up again. I've triggered more than one of those refreshes myself.

Exactly one company I worked at had per-feature environments, and they were narrow: a preview for the GraphQL schema and nothing else. The app, the database, the auth flow, all of it still pointed at shared dev. The one thing they isolated was the one thing that was easy to isolate.

Lionshead does the opposite. Every pull request gets its own running app AND its own database. Not a mock. Not a shared dev instance with a "please don't touch" Slack message. A real Next.js deploy talking to a real Postgres database that was branched, schema and data, from the parent in under a second and will be destroyed the moment the PR closes.

The point isn't fancy infrastructure. The point is that local, preview, and production are a stack, not a substitution. Local is where I write the code. Production is where users live. Preview is the rung in between, where a change proves it actually runs before anyone trusts it. Kill that middle step and every review becomes "looks good to me" against a diff, or a push-and-pray to shared dev. The preview environment is where the dev loop closes.

Here's what happens when I open a PR.

Neon branch, created or refreshed. A copy-on-write branch named preview/pr-<number> is cut from the database's parent branch. Copy-on-write means it's not a dump-and-restore; Neon clones the storage layer by reference and only diverges on write, so the clone itself is effectively instant regardless of how much data is in the parent. The step is idempotent by branch name, so every new commit on the PR refreshes the same branch instead of piling up new ones.

Atlas applies the PR's schema. Lionshead manages Postgres schema declaratively with Atlas. Against the fresh branch, atlas schema apply reads the desired schema, diffs it against what's live, and applies the difference. The branch now has exactly the schema this PR proposes, running against real Postgres, not a SQLite stand-in or a mock that lies to you about what a foreign key does.

Vercel builds and deploys against that branch. The build runs with the per-PR DATABASE_URL injected, so anything that touches the database at build time sees this PR's branch. Then vercel deploy --prebuilt ships the artifacts with the same URL wired in at runtime. The reviewer gets a link to a live app, and that app is talking to this PR's database and no one else's.

A sticky comment carries the link. One PR comment, updated in place on every commit, holds the latest preview URL and a short per-commit history. No comment spam, one place to click.

On close, the branch is deleted. When the PR merges or closes, a cleanup job deletes the Neon branch and confirms it's actually gone. The junkyard never accumulates, because the junkyard is thrown away every time.

All of this lives in one reusable workflow, lionshead-vercel-preview-deploy.yml, that every product repo calls. There's a hard-won detail buried in it worth surfacing: it's a single job, not three, because GitHub Actions refuses to pass a step output between jobs when the value looks like a secret. A Postgres connection string contains a password, so a clean "branch job hands the URL to a deploy job" design gets the URL silently scrubbed to an empty string. The fix is unglamorous: keep the URL inside one job. Most of the good infrastructure decisions I make look like that.

What it costs

Nothing worth mentioning. Neon is on the free plan. Vercel just moved to the $20/month Team plan. That's the whole bill for this capability.

The reason it's free is the same reason it's fast. Neon bills for storage and compute, not for the number of branches, and an idle branch scales its compute to zero. A per-PR branch that nobody is actively hitting costs effectively nothing while it sits there, and it shares the parent's storage by reference until it's written to. So "a database per pull request" sounds extravagant and turns out to be a rounding error.

What it has actually caught

I'll be honest about this the way I was honest about the security tooling: most bad migrations never reach a PR, because I run migrations against a local Docker Postgres before I push. The preview branch isn't where I catch a typo'd column.

Where it earns its keep is the full-workflow bugs that are miserable to test locally. Sign-up and sign-in against Firebase, for instance, are genuinely hard to exercise on a laptop; on a real preview URL with a real database, I can watch an account get created and see exactly what we persisted, including the times we captured a wrong or incomplete value. And I can watch all of that happen without dumping a single piece of that test garbage into the shared dev database.

That's the quieter, bigger win. In every corporate project I've seen, the dev database degrades into a landfill of fake accounts and abandoned test data, until it looks nothing like production and stops being useful for real testing. Per-PR branches give the trash somewhere to live and die. The messy test data stays on the ephemeral branch that gets deleted on merge, and dev stays clean enough to keep nudging it toward looking like prod. Cleaner dev means more focused development, which means bugs that show up in testing are more likely to be real.

Why almost nobody does this

I've said before that I haven't seen this pattern at solo scale anywhere else, and I still haven't. I've seen it exactly once at corporate scale.

The reason isn't cost or difficulty. It's that the enabling primitive is rare. Plenty of databases advertise "branching," but they don't mean the same thing. PlanetScale's branching is built for schema changes; its branches don't clone your data. Supabase spins up an empty database and replays your migrations and seed script. Both are useful, neither gives you a full copy of real, production-shaped data per pull request. Neon's copy-on-write branch does, in about a second, for free, and I have not found another managed provider that does the same. Most people simply don't know that per-PR data isolation this cheap is even on the menu.

The gotchas (because there are always gotchas)

Two of these bit me last week, so they're fresh.

The 10-branch cap. The Neon free plan caps a project at ten branches. I opened several PRs in quick succession, left a few in draft, and every one of them dutifully cut a branch. I hit the ceiling, and preview deploys started failing across the board. The fix was to be stricter about which PRs earn a database: filter the workflow so draft and not-yet-ready PRs don't provision a branch until they're actually up for review. The capability was never the problem. My discipline about triggering it was.

Orphaned branches from a skipped cleanup. The nastier one. A concurrency-rule interaction was canceling the cleanup job mid-run when a late push raced a merge, so the branch-delete never executed and branches leaked, which of course marched me straight back into the 10-branch cap. The fix was giving the cleanup job its own concurrency group that never cancels in progress, plus making the delete step actually authenticate instead of failing silently while reporting green. A teardown, once it starts, has to finish. That one cost me an afternoon and a good deal of "why do I have eleven branches when I have three open PRs."

It's slow, though not the part you'd guess. Full transparency: the preview environment is by far the longest thing on the PR. The deploy job runs about five minutes, and the whole preview pipeline lands closer to ten once you count waiting on sibling checks. I pulled the step timings across recent runs to see where those five minutes actually go, and almost none of it is the database. Creating or refreshing the Neon branch takes 10 to 15 seconds on a cold create and effectively nothing when it's just refreshing an existing one; the copy-on-write clone is instant, and the seconds that do pass are spent standing up the branch's compute endpoint, not copying data. Applying the schema with Atlas adds another 10 to 15 seconds, or nothing when there's no diff. The remaining four-plus minutes are all Vercel: about 20 seconds to install the CLI, then three and a half to four minutes for the Next.js production build, then about 25 seconds to upload the artifacts. So the database work tops out around half a minute and is usually far less. Building and shipping a whole app per PR is the slow part, and it would be slow with or without the branch. Everything else on the PR finishes while the build is still going. I think that's an acceptable price for a real running app on every change, but I won't pretend it's instant.

None of these are reasons not to do it. They're the tax you pay once, learn from, and encode into the workflow so the next repo inherits the fix instead of rediscovering the bug. Which, if you've read the last post, you'll know is a lesson I keep having to learn.

What's next

The preview deploy needs secrets, a Vercel token, a Neon key, to do any of this. For a long time that meant long-lived tokens sitting in repository secrets, which is exactly the kind of thing the security post frets about. The next note in this sub-series is how that got fixed: OIDC through Doppler and GitHub Actions, and how short-lived, federated credentials killed the long-lived token entirely.

The security checks in every Lionshead PR

Allen Brooks — Thu, 30 Jul 2026 10:43:55 +0000

At enterprise scale, a breach is a bad quarter. You have a legal team to coordinate disclosure. A disaster-recovery plan you drill annually. A PR team to control the narrative. A security team to quarantine, investigate, and triage the incident. A bank account big enough to absorb regulatory fines, class-action settlements, and the customer churn that follows.

At solo scale, you have none of those. A real breach of user data will almost certainly end the product. Not "hurts the brand," not "sets us back a quarter." Ends it.

So you breach yourself first.

The security stack in every Lionshead PR is an automated self-breach. Every merge attempt is scanned by tools a mid-sized security team would run against production code weekly. If any of them find something, the merge blocks. The vulnerability never reaches production, and the version of me that would have caused an incident never gets the chance.

Here's what actually runs on every PR against every Lionshead product.

Trivy filesystem — vulnerability, secret, and license scanning across the whole repo. Runs on every PR, no path gate. Catches dependency CVEs, accidental hardcoded credentials, license conflicts. Configured via trivy.yaml at the repo root; fails on any CRITICAL or HIGH severity. Trivy is Aqua Security's open-source scanner and it's the single best free vulnerability scanner I've used.

Trivy IaC (Terraform) — the same tool, second mode, against Terraform files. Catches insecure infrastructure defaults (public buckets, permissive IAM roles, unencrypted resources). Path-gated: only runs when terraform/** files change. Same severity gate.

Gitleaks — secret scanning across the entire git history, not just the current diff. Catches API keys, tokens, private keys, connection strings that someone (probably me) committed years ago and later deleted. The secret is still in the history and still exploitable. Runs on every PR. Uses the default ruleset unless the repo drops a .gitleaks.toml override.

Checkov — a second IaC security scanner, run alongside Trivy IaC for defense in depth. Checkov and Trivy IaC use different rulesets and different heuristics; they catch different classes of problem. Path-gated on terraform/**. Skip lists for known-false-positive rules go in .checkov.yaml.

OWASP ZAP baseline — active DAST against the PR's Vercel preview URL. When a repo produces a preview deploy, ZAP hits the deployed site with a baseline security scan. Catches missing security headers, cookie configuration issues, common injection vectors. Fails on any WARN+ alert; per-rule suppressions live in the repo's .zap/rules.tsv.

Actionlint — GitHub Actions workflow linting. Not strictly security, but security-adjacent: catches syntax errors, invalid contexts, and permission-scope typos that would silently over-scope your workflow's access. Path-gated on .github/workflows/**.

Shellcheck — shell script static analysis. Catches quoting errors, command-injection patterns, and the classic "unquoted variable in an rm command" bug that has ended more careers than any single vulnerability. Path-gated on *.sh files.

Hadolint — Dockerfile linting. Catches insecure image bases, missing USER directives, and other Dockerfile-shape issues that expand blast radius on container escape. Path-gated on Dockerfile*.

Action pins check — a Lionshead-authored guardrail that validates every third-party GitHub Action referenced in workflow files is pinned to a full commit SHA, not a moving tag. Prevents a supply-chain attack against an action's tag from silently propagating into Lionshead's CI. Fails PRs that add or update actions without a SHA pin.

Infracost + OPA cost gate — not security in the CVE sense, but security in the "my card gets charged $10K/month by mistake" sense. Runs on every Terraform PR, computes the cost delta, and pauses the PR on a GitHub Environment approval gate when the delta exceeds $50/month. I covered this in more detail in the last post; mentioned here because it lives in the same workflow.

Two more layers run outside the PR gate.

Scheduled security audit — every product repo runs lionshead-security-audit.yml daily against the default branch. It runs npm audit (or the language-appropriate equivalent) plus a full Trivy filesystem scan. Findings open or update a single security:dependency tracking issue per repo; clean runs close any open issue. The point is to catch CVEs disclosed after code was merged, not just what a PR introduces. The majority of real-world CVEs are disclosed against code already in production; a PR-time scan alone misses them.

Renovate — dependency-update automation. The ci-standards Renovate preset applies to every product repo via a one-line renovate.json. Renovate opens PRs to bump dependencies, pin action SHAs, and update base container images. Its github-actions manager keeps the pinned reusable workflow SHAs current, which is how new products cloned from template-product stay on the latest ci-standards workflow without manual intervention. Weekly schedule keeps the noise bounded.

Three things are on the roadmap but not shipping yet.

Semgrep (SAST) — the security baseline mandates Semgrep OSS for static-analysis coverage as a free-and-open-source substitute for GitHub Advanced Security. Rollout across all products is in progress.

OWASP ZAP as a scheduled cron against dev — today, ZAP runs at PR time against the preview URL. The plan is to also run ZAP against the shared dev environment weekly to catch drift between preview builds and the environment that lives longer.

Pre-commit hooks — a Husky-driven local hook stack that runs the most annoying failure modes (a Gitleaks pre-commit scan, at minimum) before code even gets to the PR. Removes a full round-trip when you catch a mistake before pushing. On the shortlist to ship next.

The whole stack costs zero in licensing. Every tool above is open source. The cost is time to wire them up and ongoing time to tune false-positive suppressions. That cost is measured in hours, once, not months. What it buys is the difference between "my product survives" and "my product ends" in the scenario where I fat-finger a Terraform PR at 11pm.

That is the trade at solo scale. It happens to look like the trade at enterprise scale for exactly the same reason.

Update, 16 July 2026: what it cost, and what it caught

Someone on Hacker News asked how often these checks actually hold up my PRs. I didn't know, which bothered me, so I went and counted every PR check this stack has ever run: 2,864 of them across nine repos. 286 failed, call it 10%.

The failures aren't spread out, and that's the part I didn't expect. They arrive in tight clusters of three or four days, and every cluster sits on top of a setup event. The biggest was late April, when I pulled these checks out of Vesper and into the shared ci-standards workflow. Eighteen Infracost failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of silence. Another cluster in May when Crisper adopted the same workflow. Another in July when I bootstrapped some skeleton repos that are parked until Vesper ships. Between the clusters it's quiet for months. The repos with the worst failure rates are the ones that have only ever been in setup and have no steady state to average against.

So the cost isn't a tax on every PR. It's a bill that comes due when I migrate something, and it's mostly me misconfiguring my own plumbing rather than the scanners crying wolf.

The other surprise: tests, not security, are what actually block me. Failing tests account for roughly half of every PR check failure I've ever had. All the security tooling combined is well under a quarter.

What the security tools have caught, honestly: not much. Trivy found a real CVE in a transitive form-data dependency and I bumped it. Checkov found a real IAM policy attached to a user instead of a role. It also flagged AKIAEXAMPLEEXAMPLE00 in a .env.example, and read a Terraform format("postgresql://%s:%s@...") string as hardcoded credentials when the password was a random_password reference. Which is to say it caught the correct implementation and would have sailed straight past a bad one.

And then, while I was writing this very update, the pull request carrying it went red. Trivy flagged CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I'd changed. It was disclosed against code I merged weeks ago, and the PR-time scan is simply where I happened to meet it. Bumped to 0.7.5, green again, about six minutes of work.

So: ~~two real findings in four months~~ three real findings in four months, one of them CRITICAL, and the third arrived while I was typing the sentence that said there were only two.

That's still a thin haul if you measure it the way a security team measures things when it needs headcount approved. But that was never the trade. The premium is a few hours per migration and zero dollars, and what it covers is the scenario where I fat-finger a Terraform PR at 11pm and end the product. At that price I don't need it to catch much. I need it to catch the one. It just caught one while I was mid-sentence arguing it rarely does.

A correction and a clarification to the post above, while I'm here.

The post says the tuning cost is "measured in hours, once, not months." Hours is right. Once is wrong. It's hours per migration, and every new repo keeps rediscovering the same false positives because I've written a slightly different allowlist in each one instead of fixing template-product so the next repo inherits it. That's on me, not the tooling. For the full effect: publishing this update tripped Gitleaks on the AKIAEXAMPLEEXAMPLE00 two paragraphs up, and I fixed it by writing yet another per-repo allowlist. Which tells you exactly how much I've learned.

On ZAP: what runs at PR time today is the baseline passive scan. Full active DAST is on the roadmap, not shipping yet.

What's next

Next in the CI/CD sub-series: the preview environment pattern that ZAP runs against. Vercel plus a fresh Neon Postgres branch per PR, how the reviewer gets a real running app talking to a real database on every PR, and why that costs almost nothing.

I'm one person, and my CI does what a mid-size eng org's does. Here's why.

Allen Brooks — Thu, 30 Jul 2026 10:36:36 +0000

Corporate CI is often the red-headed stepchild of the dev team. Written six years ago by a consulting firm, held together with cron jobs whose owners left in two reorgs, touched only when it breaks. Nobody on the current team could explain what half the steps do.

Lionshead's CI runs a security suite, an Infracost cost gate, IaC scanning, secret scanning, schema-migration validation, and a preview-environment deploy before any PR can merge. Categorically the same coverage as many mid-sized engineering orgs.

I am one person.

The obvious question, from other indie founders, is why. The reusable workflows in ci-standards took months to write. Nobody is going to breach my pre-launch SaaS in month three. Why not push to main and let Vercel deploy?

Because this is the way cloud software is built. I don't know how to build it any other way, and I don't want to relearn. There's no three-person dev team here, no DevOps engineer, no IT department. Nothing about being alone changes what "shipping safely" means. It only changes who has to remember it, and if the answer is me, then the answer needs to be automation.

The shops I have seen succeed treat the CI/CD pipeline as first-class code. The developers who write the product also write the pipeline. The knowledge of how one flows into the other compounds over the years. The shops I have seen fail leave CI to consultants, or hire it out to a platform team that has never shipped their product, or worst of all, both. In my experience, no exception to that pattern has stuck. The shape of the CI pipeline is a leading indicator of how the whole engineering culture works.

Lionshead's pipeline is four areas: build, test, security, deploy. Two of those areas make choices unusual at solo scale.

Security. Trivy for vulnerabilities and IaC, Gitleaks for secrets, Checkov for policy, plus linters that keep the workflow files themselves honest. All open-source, no licensing cost. Any one of these is a one-line YAML addition and takes ten minutes to wire up. The unusual choice isn't setup. It's committing to run all of them, keep them maintained across every product repo, and enforce them as merge gates when the only person they block is me.

Preview environments. Every PR spins up its own Vercel deploy AND its own fresh Neon Postgres branch: a copy-on-write clone of prod's schema. The reviewer clicks the preview URL and gets a real running app talking to a real database. Schema migrations run against actual Postgres before merge, not against a mock or a shared dev database. When the PR closes, the branch is destroyed. This costs nearly nothing because Neon bills for storage and compute, not for branches. I have not seen this pattern at solo scale anywhere else.

The check that deserves specific mention beyond those two is Infracost. It runs on every Terraform PR and requires human approval when a change would add $50 or more to my monthly cloud spend. Fifty dollars is not the number that would ruin me. Fifty dollars is the number where I would want to look twice at the diff. Between an agent auto-completing a Terraform block wrong, and me clicking through a review too fast, I would rather burn a merge-queue slot than wake up to a $10,000-a-month resource I did not mean to create. Every solo founder running on their own cash should have this exact gate. Almost none do.

All of it runs end-to-end in about five minutes. That's not despite the number of checks. It's a function of designing the pipeline with parallel jobs and path-gated conditionals from day one, not retrofitting them on after the fact.

The line between too much CI and not enough CI is a straightforward one. It lands wherever developers no longer need to remember how to get code into a live environment, or troubleshoot when it doesn't. Below that line, you pay a tax on every ship in ceremony that used to happen automatically. Above that line, you pay a tax on every setup in ceremony that never had to happen. My CI investment sits right at the line for one person. It happens to look like the CI at a mid-sized org because the line is drawn by the same physics, not by the size of the team.

What's next

This post is the "why." The rest of the CI/CD sub-series is the how. Coming up: the specific security checks that ship with every Lionshead PR, how the Vercel + Neon-branch preview environment is provisioned, the OIDC secret flow through Doppler, and the path-gated test routing that keeps a five-product portfolio building against one shared workflow.

Every website should have a colophon

Allen Brooks — Thu, 30 Jul 2026 10:36:34 +0000

Every developer I know has the same itch. Pop the hood on a website, a SaaS app, an app on the App Store, and see how it was built. Which framework. Which hosting. Which type. The craft underneath the surface.

The colophon page is the formal scratch to that itch.

The word comes from publishing. A colophon is the small inscription on the last page of a book that tells you the typeface, the paper, the press, the binding. The book's how. Stripe Press still does this. Information Architects does this. A small corner of the web that takes its lineage from print has been doing it for years.

I had never heard the word until six months ago.

That isn't a humblebrag. Ten-plus years in corporate engineering, up through senior director, and the word "colophon" never came up. The companies I worked at shipped real software to millions of users, and not one of them had a public page that said "here is how we built this." It wasn't anyone's job. It wasn't a category.

I came across it earlier this year, looked it up, read a few examples, and built one that weekend.

Lionshead's colophon is at lionshead.digital/colophon. Stack, type, color source, build method, hosting. One screen, no decoration. The same structure now ships in every Lionshead product per a company rule. They all have colophons.

I added it because building things can be a craft, not just a job. Corporate engineering taught me to ship. Lionshead is teaching me to sign my work.

Keeping 10 repos in sync without a monorepo or a platform team

Allen Brooks — Thu, 30 Jul 2026 04:39:20 +0000

Most dev shops don't have standards. At least not codified and documented standards. They have a Confluence space with someone's 2021 attempt at a style guide buried under three years of half-finished runbooks and obsolete service docs. The rules aren't enforced because nobody reads them. The shops that do enforce standards pay for it: a platform team, a custom lint server, a quarterly review cadence, and someone whose job is to argue with developers about line length.

Lionshead has standards, and they're enforced for free. They live in code, in .claude/rules/ and a BMAD module, and the AI agents doing the keystrokes read them on every story. There's no human in the middle to forget the rule; the agent reads it and writes the code in the same breath. That's not the same trade as "humans who follow the rules better." The AI is doing the writing too; the free enforcement is a side effect of swapping the actor, not a process improvement layered over a human team. That changes the problem. The question stops being "how do we get developers to follow the rules" and starts being "how do we get the same rules into ten unrelated product repos without a monorepo or a platform team."

This post is the answer. A small CLI I call the reconciler: pull-based, hash-compared, idempotent. It ships a manifest from a central repo and lets each consumer adopt the parts it wants. Here is what it does, what it costs to build, and where it ends.

I have operated at the ten-repo scale before, and the reconciler is built for that target. Lionshead today is one product (Vesper); this is the system for where I am going, not where I am.

The shape of the problem

The familiar approaches to keeping many repos in sync all trade the same three axes: friction at edit time, friction at consumer time, and how much shared infrastructure the consumers have to swallow. Pick any two, lose the third.

Copy-paste. Zero infrastructure. The cost shows up at every fix: ten repos, ten patches, and the moment one repo skips a patch the divergence starts. Worse, copy-paste makes the canonical version invisible. No one can grep it without first deciding which repo's copy to trust. I shipped two months on copy-paste before the cost compounded; it compounds fast.

Git submodules. One source of truth, a real link between consumer and producer, and a developer experience that is genuinely awful. Submodules break IDE checkouts, break clone scripts, and force every consumer to handle a sub-checkout step that someone new will get wrong on day one. I tried submodules for two days; the cost showed up faster than copy-paste.

Monorepo. Solves drift completely. Also forces every product onto a shared CI, a shared release cadence, and a shared dependency graph. That cost is paid even by products that have nothing to do with each other. The big-company default; the wrong default for the operator who runs ten small unrelated things.

All three trade the same axes in different proportions. The reconciler trades a fourth axis: it accepts a small per-consumer install cost in exchange for keeping every consumer's repo shape, CI, release cadence, and dependency graph independent. That is the only trade I was willing to make at the ten-repo scale.

What the reconciler actually does

The CLI is published to GitHub Packages under the @lionsheaddigital scope. Consumers configure their npm token once per machine. After that, npx @lionsheaddigital/setup just works.

The CLI reads a single declarative manifest, lionshead.json, from the central ci-standards repo over plain HTTPS to raw.githubusercontent.com. The manifest lists every standards file the central repo owns: BMAD module sources, the Claude Code rule files, GitHub Actions reusable workflows, the GitHub project sync script, and a few others. Each entry carries a sha256 hash of the upstream file.

For every file declared in the manifest, the CLI walks one branch:

The heart of the loop is a hash comparison:

// packages/setup/lib/reconcile.mjs (excerpt)
function computeDelta(remoteStandards, localInstalled) {
  const remoteFiles = remoteStandards?.files ?? [];
  const localFiles = localInstalled?.files ?? [];

  const localMap = new Map(localFiles.map((f) => [f.path, f.sha256]));
  // ...

  const toDownload = [];
  for (const rf of remoteFiles) {
    const localHash = localMap.get(rf.path);
    if (localHash !== rf.sha256) {
      toDownload.push(rf);
    }
  }
  // ...
}

If a consumer's local copy of a declared file already matches the upstream hash, the CLI does nothing for that file. If the hash differs (the consumer is missing the file, or upstream has moved on, or someone hand-edited the local copy), the CLI downloads the upstream version and overwrites the local one. No prompt. No merge. No three-way confusion. Upstream wins; local edits to managed files are not allowed.

The full run on a clean consumer takes about three seconds. The full run on a clean, up-to-date consumer takes about one second and prints "Standards up to date." That second case is the one that earned the reconciler its name: it is safe to run a hundred times in a row, and the only side effect is that the operator knows the consumer is current.

The hard parts

The interesting design decisions in this kind of CLI are not the visible ones (the manifest shape, the file copy). They are the boring ones underneath.

Idempotency is the first hard part. The hash-compare and the conditional download together give you "do nothing when nothing has drifted." That looks easy until you find the case where BMAD's own installer silently overwrites a managed file with stale bundled content after the main download loop has already updated it. The reconciler now does a second verify-and-restore pass after BMAD runs; without it, a clean run would show "Standards up to date" while the on-disk SHA had quietly diverged from the manifest. That pass cost a day to discover and an afternoon to write. It is the kind of bug that only shows up at the seam between two tools that both think they own the same files.

Partial adoption is the second hard part. Not every consumer wants every standard. A static marketing site does not need the Test Architect workflow; a backend service does not need the design-tokens rule. The manifest supports per-consumer opt-outs via a small lionshead.json in the consumer repo that lists explicit skip paths. The escape hatch is necessary; without it the reconciler is a coupling mechanism dressed up as a distribution one.

Upgrade safety is the third hard part. The reconciler is itself one of the things it ships. Bootstrapping a brand-new consumer is a chicken-and-egg moment: the consumer does not yet have the CLI installed, but the CLI is what installs the CLI. The answer is npx: the CLI runs without a prior install, fetches itself from GitHub Packages, runs once, and exits. The local install only matters for subsequent runs, and those can be npx too. The whole bootstrap is one command.

When NOT to do this

The reconciler is right anywhere above three repos. Below three, copy-paste is genuinely cheaper, and the per-file maintenance cost is small enough that the operator can absorb drift mentally. The architecture has no real upper bound on consumer count: the CLI is stateless, every consumer is independent, and adding one is one npx away.

The real ceiling is governance, not scale. Review doesn't disappear in this model; it shifts upstream to the central standards repo, where PRs are the negotiation surface. That works fine when there's an agreed owner of the central repo. It breaks down when there isn't: the reconciler's "upstream wins" semantics turn a governance question into a technical one consumers can't appeal. A monorepo handles disagreement by making every change visible to all consumers in the same commit; the reconciler doesn't.

And the reconciler is wrong when drift is a feature, not a bug. Research codebases, demo branches, throwaway prototypes: these benefit from divergence. A pull-based "upstream wins" CLI run against a demo branch will quietly undo the demo. The right fix there is to mark the consumer out of scope, not to weaken the reconciler.

Outro: the Building Lionshead thread

The reconciler is one chapter in the Building Lionshead series. Post one was a sketch of why I built a company-specific BMAD module; the post you are reading is what the distribution mechanism for that module looks like in code. The series's spine is "small-company process at small-company cost"; this post is one chapter of it.

The next post in the series will be about the BMAD module itself: what is in it, what the boundary is between Lionshead-shaped behavior and stock BMAD behavior, and why I keep that boundary thin. After that, a post on the rule files in .claude/rules/ and the way they get loaded into agent context. Both posts depend on the reconciler existing; this post is the prerequisite.