DEV Community: Allen Brooks

Invariant checks: the comment is the check you never wrote.

Allen Brooks — Mon, 31 Aug 2026 11:30:43 +0000

Twelve days ago I built a check that reads my company standards and asserts that every enforcement mechanism they name actually exists as code. Pointed at the repository where those standards are written, it found three rules describing gates nobody had built. I repaired all three, wired the check into that repository's pipeline, and watched it go green.

This afternoon, while drafting this post, I pointed the same check at a different repository: one that consumes those standards, and where it had never run once.

A hundred and forty-one claims across fifty files. Three of them unbacked, and two were the same fabricated gates I'd repaired twelve days earlier, sitting untouched in a tree I ship to.

The check was fine. It was correct on the first day and it's correct now. What I want to write about is the class of assertion it belongs to, why I'd been writing that class without a name for it, and the specific mechanical reason a repair I genuinely made was still not true in the place it needed to be true.

The comment

Somewhere in your codebase there's probably a comment like this:

If you add a column here, remember to add it to the SET map below.

I want to be precise about what that comment is. It isn't documentation. It's a person who, at the moment of writing it, held in their head the complete specification of a check: the two lists, the direction of the relationship, and the failure. They had everything they needed. They wrote a sentence to a stranger instead, and the stranger is in a hurry.

That comment is the check you never wrote. It marks the exact coordinate of a hazard and does nothing about it, and the reason it feels responsible is that it genuinely is more responsible than saying nothing. It's a local maximum. You can sit in one for years.

Three properties, all required

I call the thing that comment should have been an invariant check. Three properties have to hold in the check at once, and missing any one of them means you've built something else.

First, though, the word "pairing," because the whole class turns on it. A pairing is two things that have to be added together and routinely aren't. In the comment above it's the column and its entry in the SET map. Add one without the other and nothing complains, because nothing is watching the relationship between them. That relationship is the pair. A behavior test asks whether one thing does what it should; an invariant check asks whether both halves are still there.

The check is mechanically derived. It computes both sides rather than restating one. A new column, enum member, or route joins its scope without anyone editing the check. The moment you hand-enumerate one of the two lists you've created a second artifact that can drift from the first, and now you've got two problems.

The check asserts a pairing. When it fails, it names the missing half. Not "this value is wrong" but "you added this and not that." The failure message is a coordinate, not a judgment.

The check belongs to no single feature. One piece of work creates the seam and a different piece of work, months later, violates it. That's why no feature's own tests cover it: at the moment the pairing is created there's nothing to catch, and at the moment it breaks, nobody's looking at the place it was created.

That third property is what makes these invisible. A unit test belongs to a feature. An invariant check belongs to the space between two features, which belongs to nobody, which is why it doesn't get written.

Four of them already existed

When I went looking, I found I'd written four of these already. None of them were called anything.

One asserts that every database table exported by the schema has at least one reader outside the test suite, which catches a table you built, wired into tests, and never actually used. One is a runtime throw at construction time: registering a channel without a client fails immediately and loudly, rather than at three in the morning on the first real message. One asserts that every rule file over a size threshold has a sibling document holding its detail. One asserts that every deprecation tagged with a retirement issue is owned by an issue that's still open.

Four instances of one idea, in two repositories, in three completely different mechanisms, every one of them written reactively after the corresponding gap had already bitten me.

The mechanism is not the definition, which took me a while to see. A construction-time throw is as much an invariant check as a scheduled scanner. Choosing between them is a question of where the pairing can be observed most cheaply, not a question of what kind of artifact you're building.

And the reason to name the class isn't tidiness. An unnamed pattern can't be asked for at planning time, can't be audited for at the level of a whole board, and can't be hunted for by anyone who has no word for it. Naming it is what turns four accidents into something a gate can require. So now the question gets asked when work is planned: does this create a pairing that nothing asserts? "Nothing pairs here" is a fine answer. Nobody asking is the failure.

The fifth one, written on purpose

Then I built one deliberately and pointed it at my own standards.

Those standards are rule files plus longer sibling documents holding the detail, and they're the first thing my agents read before doing anything. The invariant is that every enforcement mechanism a rule names resolves to code that exists. The pairing is a claim in the corpus and an implementation in the repository.

The first sweep found three rules that specified enforcement in implementation-level detail and had shipped none of it.

The first of the three described a gate that would refuse to promote work to ready-for-dev without a test design, complete with an escape-hatch flag called --allow-no-td and a matching environment override. The script it named contained neither string.

The second described a checklist item the planning agent was supposed to add when planning new work. That agent had never been told to add it. By the time I found this, a second rule described its own gate as sitting "parallel to the IaC checkbox," which is to say parallel to nothing.

The third told developers to scaffold their first failing test with npm run new-test. No repository in the portfolio had a new-test script in any manifest.

The detail is what makes these dangerous rather than merely aspirational. A vague aspiration reads as an aspiration and gets treated like one. A documented flag name reads as a flag, and the reader's next move is to type it. I'd written three specifications convincing enough that an agent following them would have believed it was working inside a system of gates, and would have been alone.

This is the failure mode where documentation asserts coverage that never existed. Prose can't catch it, because prose is where it lives.

What it cannot do, said plainly

The check reads three kinds of claim: a named npm script, a runnable path, and a command-line flag attributed to a command I own. It doesn't read prose for meaning.

So of the three defects that motivated it, it catches the first and the third. It doesn't catch the second. A checklist item described only in sentences names nothing runnable, so there's nothing to resolve and nothing to fail on. A rule can still promise a gate in English and walk past every check I have. What it can no longer do is name a runnable thing that isn't there.

I'm spending this many words on the boundary deliberately. Overstating the coverage would be precisely the defect the check exists to catch, and I'd rather be tedious about the limit than publish a post that commits its own subject.

Making it quiet enough to survive

The engineering that mattered was not detection. It was suppression. A check that cries wolf gets disabled, and a disabled check asserts exactly as much as that comment does, which is nothing.

Flags were the hard part. gh pr checks --watch and git ls-remote --heads are flags on other people's tools and I'm not the one who implements them. An early version attributed any flag to whatever script happened to be named in the same section, which produced eight confident reports about flags that were never mine. So attribution runs command-first: if the surrounding text invokes a script I own, the flag is checked against that script; if it invokes somebody else's tool, there's no claim at all. But it can't be command-only, because a bare flag written in a sentence has no command attached, and that's exactly the shape the real defect took. --allow-no-td sat four lines below the script that was supposed to own it, in prose. The section-level fallback is load-bearing.

Environment variables are off by default. Operator-setup sections mix mine with everyone else's and nothing in a variable name tells you whose it is. Gating on them produced eleven false reports, and eleven false reports is how you teach someone to ignore a check permanently. Nothing is lost, because every environment override in these designs has a paired flag and the flag is checked. The original fabricated variable got caught anyway, through its twin.

Paths are narrow: runnable files under three directories. An earlier draft included documentation directories and reported seventy-one unbacked paths, nearly all of them either stale cross-links between rule files or claims about a consumer's tree rather than the publisher's. Both are real problems. Neither is this one, and folding them in would bury the signal under a backlog nobody triages.

I re-ran the tuned version against the authoring repository while writing this. A hundred and twenty claims across forty-five files, zero unbacked.

Why the two were still there

So the publisher is clean and the consumer isn't. There are two reasons, and the second is the one worth your time.

The first is that the corpus is authored in one repository and mostly describes the others. npm run dev resolves in a product repository and nowhere near the publisher. So the check runs in two modes, and running the consumer mode at the publisher reports seven unbacked claims that are all false: dev, lint, typecheck. Real scripts, in other repositories.

I wrote that asymmetry down when I built it, along with its consequence, which I'm going to quote because I was right and it did me no good whatsoever: at the publisher, a fabricated npm run new-test is indistinguishable from a real npm run dev, and only a consumer can tell them apart.

I had the correct theory of where this check needed to run, and I ran it in the other place.

The second reason is worse and more interesting. A failing claim only ever has two honest resolutions: build the thing, or stop saying you did. Which one is right depends on whether the gate was worth having, and deleting the sentence is only the answer when the mechanism genuinely isn't coming. Delete prose to turn a check green and you haven't fixed anything, you've just moved the lie somewhere the check can't see it.

So I repaired the three claims three different ways. One gate was worth having and I built it, though not where I'd said it was: it lives in the planning agent's own procedure rather than in the script the rule had named, and the prose changed to describe where the gate actually is. One I resolved the other way. And where the scaffolding command used to be, the upstream file now carries an explicit correction saying in as many words that the command was specified and never built, that no manifest anywhere has it, and that it had been written up as though it shipped. The claim is gone and the record of it isn't, which is the only version of deletion I trust. That correction is sitting there today.

It's sitting in the file that replaced the file the claim was written in.

A while back I consolidated thirty-three rule documents down to eight. The consolidation created the new files and distributed them everywhere. It didn't delete the old ones. My distribution manifest lists fourteen sibling documents; the consumer repository has nineteen sitting in that folder. Five are pre-consolidation ancestors: unowned, undelivered, never updated again, and completely indistinguishable from the live ones to anything that reads the directory. Including my agents.

So the correction landed on the descendant. The ancestor kept the original claim, kept shipping it, and kept reading as authoritative. Upstream: fourteen files, no orphans, zero unbacked. One repository over: nineteen files, five of them ghosts, and inside one of the ghosts, verbatim, the fabricated flag and the fabricated command.

I checked both by hand before writing this. The script contains neither the flag nor its environment variable. No manifest in that tree has a new-test script.

The third unbacked claim, for completeness, is the check being wrong. A rule describes CI running npm run typecheck --if-present, and that repository has no typecheck script. But --if-present is the idiom that means absence is fine; the claim never asserted the script exists. My check can't read that qualifier, so it flagged a gate behaving exactly as designed. Two real, one false. I'd rather print three and explain than round down to the number that flatters me.

The step after noticing

The check was right. The design note about publishers and consumers was right. The repair was right, and thorough, and honest about itself.

What failed was distribution, which is the second Sunday running that I've written that sentence in a different costume. Last week it was a retrospective whose findings were real and went nowhere. The noticing has never been my weak link. The step after the noticing is where I have no gate, no counter, and apparently no attention left over, and it keeps being the part that costs me.

There's a version of this post where I fix the orphans tonight and write all of this in the past tense. I'm not writing that one. They're the evidence, and they'll be a line on my roadmap tomorrow morning instead of a quiet commit tonight.

The comment you never turned into a check is one failure. The check you wrote and never pointed anywhere is the same failure wearing better clothes.

What's next

The cost gate: infrastructure spend as a merge requirement rather than a monthly surprise, and what it's like to have a pull request blocked by a number instead of a test.

My retro proposed three fixes and built none of them. Its health check read fine the whole time.

Allen Brooks — Mon, 24 Aug 2026 02:02:57 +0000

Last Sunday I published a post about the agent fleet that builds my backlog. It ended with a section called "What I have not run," in which I said the retrospective pass was designed, was accepted, and as of that morning was code that had never run. The next sentence said the lenses and the trigger arithmetic had shipped into my standards a few hours before the post did.

Both halves were wrong by the time anyone read them.

The first retrospective merged at 15:30:35 Eastern that afternoon. That is twenty-nine minutes before the timestamp on the post claiming it had never run, and five hours and forty minutes before I actually pressed commit. The machinery had shipped two days earlier, not a few hours. And the post closed by asking whether specialist lenses reading a batch of merged stories at once would find the drift the design promises, or whether that was one more thing I believed because I wrote it down. The answer had been sitting in a merged file on my own main branch since mid-afternoon.

I am not going to edit that post. Originals stay as they were. But it is a strange way to learn that I was writing about my own system from memory instead of from it.

So: it ran. Twice, over fifteen merged stories and then seven. It found real things. And then it found something worse than anything on either board, which is that everything it produced the first time had quietly gone nowhere.

What the thing is

After a batch of stories merges, eight specialist lenses read the whole batch at once. Architecture, integration, testing, UX, product scope, process, documentation, business. Each is a separate agent with a written charter, each looks only for the class of problem that lens exists to notice, and each must either file a finding or explicitly pass. Then everything they file lands on one board and gets sorted into work to do now, work for the roadmap, or things that are permanently out of scope.

The premise is one I still believe. Individual code review is at the wrong altitude for a whole class of defect. Three stories can each invent a slightly different way to do one job, and every review passes, because from inside any one story there is nothing to compare against. You only see it by looking at ten at once.

It is generative and never gating. It runs after the work is merged, and the only things it can produce are new work and durable learnings. It cannot block anything.

That turns out to matter more than I thought.

What it caught

It works. I want to be clear about that before I spend the rest of the post on how it failed, because the findings are real and I would not have gotten them any other way.

One authorization boundary, expressed three ways. The free-versus-paid line in the product lives in a redact ? null : ... ternary in the GraphQL resolver, in a synthesisAllowed early return for the derived facts, and in independent tier reads inside two REST mirrors. One story drew three blocking review findings, all of which its own reviewer characterized as one family: a field derived from the redacted blob, or an action gated on it, that the decision table missed. A fourth was found live after the code review passed. The defect was never in a mechanism. It was in the gaps between three, and no single review sits where the gaps are.

A route is reachable only if three registries agree. A page in this app exists for a user only if it appears in the nav array, has an entry in the sidebar icon map, and ships a segment layout that mounts the app shell. Four merged stories touched the nav registries. One satisfied both nav halves and shipped no layout, so clicking through to Purchases made the entire navigation disappear on the hub that story exists to anchor. Four more routes were still sitting in that third state when the retro ran.

A test that was empty by construction. One story's acceptance suite built a recorder sink, called the function under test without wiring the recorder, and asserted the sink was empty. It was empty because nothing could ever have written to it. That assertion carried an acceptance guarantee and could not have failed for any change to production code. On a sibling story with the same shape, deleting an entire acceptance loop left 58 suites and 1,658 tests green.

The milestone shipped its ends and not its middle. The active milestone's own sentence is that a stranger can find a county, see what intelligence exists for it, pay, and receive a bundle. The first batch shipped find and see. The second batch also did not ship pay. The two commerce stories were priority zero and ready for both batches and built in neither. Nothing went wrong to cause that. Every pull was defensible in isolation, and those two cards were simply never next when a slot freed. The fabric is very good at building what is on the board and structurally incapable of noticing that the board stopped describing the plan.

That last one is the finding I would most want a human to have caught, and no human did.

What happened to them

The first retro adjudicated three of its findings to stubs, which is the strongest disposition it has. A stub is a scoped piece of work with a title, a rationale, and a sketch, ready to become a card.

Two days later the second retro opened with this:

The 2026-08-16 board proposed three stubs. None were built.

Two of those three classes recurred in the very next batch. The third is better than that. It had proposed a single reachability check that would supersede the existing icon invariant and absorb the check a queued story was already going to write, and it warned in as many words that otherwise there would be "two checks that have to agree about the same population."

One story shipped its own check. Another shipped a third. There are now three separate tests walking the same route population, which must all agree with each other, which is precisely the class the stub existed to eliminate and precisely the outcome it predicted in writing.

Nothing failed. No step errored. The proposals were made, written to a board, committed to the repository, and then the next run started and nobody read them. A ceremony whose only outputs are proposals and learnings is worth exactly what happens to its proposals, and what happened to mine was nothing.

This is where being generative and never gating stops being a design virtue. I chose that property deliberately, because a retrospective that can block a merge becomes a thing people route around. What I did not think through is that it leaves nothing at all standing between a proposal and the floor. Every other quality mechanism I have gets its force from being able to refuse something. This one had no teeth by construction, and I never asked what would give it any.

The gauge that said this was fine

Here is the part that made me sit back.

The retro instruments its own health. It was designed with the obvious failure mode in mind: that charters accumulate observations forever and never convert any of them into a real check or a real rule, so the ceremony slowly turns into a diary. So it tracks two numbers. How many standing observations are held across the eight charters, and how many have ever graduated out into automation or process.

After the second run those numbers were nine and four. That is a healthy ratio. That is the ratio you want. The retro's own report says so, in the section named for the rot it exists to detect.

Both numbers were true. The whole time they were true, every stub the ceremony had produced was being dropped on the floor. Stubs proposed versus stubs ingested was three and zero, and no instrument anywhere in the system was watching that pair.

I did not build a broken gauge. I built a correct gauge pointed at the wrong thing, which is worse, because a broken gauge eventually reads implausible and a correct one never does. Nine and four looks like a system working. It is what a system working looks like. It is also what this looks like.

The counter that was counting in another directory

Then I went looking, the way I did last month, and found the same shape twice more in the same week.

The retro is not supposed to fire after every batch. Small batches are supposed to accumulate: runs of two and two and two should add up and fire one retrospective over all six stories. That accumulation is the mechanism, written into the design document, and it is the thing that makes the compounding claim true rather than aspirational.

The accumulator kept its state in a file resolved against the current working directory. Every session of the fleet runs from a fresh git worktree. So every run opened a brand new counter at zero, found nothing in it, and reported its source as absent, its verdict as defer, its reason as below the floor.

absent reads as "this is the first run." It actually meant "the state you want is in a different directory."

Measured against real history: three runs, two merged stories each, from three real worktrees. All three read absent. Six stories, zero retrospectives. The cross-run accumulator is the exact mechanism the design specifies, and worktree-local state defeated the design it implements, silently, for as long as it existed.

This is why the state file on my machine still says one retrospective has happened. Two have.

The check that was missing, not red

The third one came out of the second retro itself, and it is the same sentence again in a different costume.

A pull request came up for merge with every check green. One required check, the one that runs the tests, was not green and was not red. It was not there. The fan-in job that reports it had never been created, so the context simply did not exist on that pull request. Every visible signal said ready. The pull request was structurally unmergeable.

Anything reading that pull request for failures would have found none, because there were none. The absence of a failing test run and the absence of a test run are the same picture. The same shape had already hidden never-run tests on two separate stories, in both cases while the developer agent reported them passing locally.

What these three have in common

A check that isn't running looks exactly like a check that's passing. I wrote that a month ago and thought I had learned it.

I had learned it about checks. I had not learned it about counters, and a counter is worse, because a check at least has a job that a human can go look at. A counter hands you a number, and a number carries no information about whether anything was measured to produce it. Zero findings and zero looks render identically. Nine and four is arithmetic over the wrong pair. absent is the honest report of an empty file and a completely dishonest report of the situation.

Every one of these three had a defensible reason to report what it reported. None of them was a bug in the sense of doing something other than what it was written to do. The rot check counted what it was told to count. The accumulator faithfully read the file at the path it was given. The merge check correctly found no failures.

The thing they share is that each of them collapsed two states that needed to stay apart. Nothing wrong and nothing looked. Zero and unmeasured. Not failing and not present. In all three cases the collapsed pair renders as the good one.

So the rule I am taking out of this, which is narrower and more useful than anything I would have written down in advance: any counter that can read zero has to be able to distinguish zero from unmeasured, and has to say which one it means. Not as a nicety. As the primary output.

What shipped

Two things, and only one of them came from a retrospective.

The reconciliation script exists. It parses the newest retro board, extracts every stub, searches for a card carrying that stub's identifier, and reports the ones with no card behind them. It runs as a precondition before the fleet starts a batch, and again before a retro is allowed to open, so a retrospective now cannot begin without first showing what its predecessor dropped. It reports rather than files, deliberately: I stay the valve on what becomes work.

The accumulator resolves its state against the git common directory, which every linked worktree of a clone reports as the same path. It also learned a legacy adoption path, so that finding an old worktree-local file and adopting it reports as a migration rather than as silence, and absent now means genuinely absent. Fifteen new tests, built against real git repositories with real linked worktrees, because the defect was entirely about what git answers from inside a worktree, and a stubbed test would only have asserted the belief that produced the bug. Against a mutant that reverts the path resolution and changes nothing else, nine of the fifteen fail.

That second one is not a retro finding. I found it separately, which is its own small comment on the state of the loop.

What has not

I had a paragraph here claiming two more of these had landed. Before publishing I went and grepped for them, because a post about proposals evaporating is a bad place to assert one landed on vibes.

The second retrospective proposed five stubs. Five days later, one is built. It is the one whose entire purpose is to report stubs that never got built.

The merge gate is still the prototype written during the run; there is no script in the repository. The mutation-evidence rule, which would require any assertion carrying an acceptance guarantee to be recorded as observed failing against a deliberately broken tree, is not in the rules file. Neither is the one-sentence rule about how a brief may describe an in-flight sibling. The milestone verb report is not in the fleet's preconditions.

So the honest count is one of five, and I only know that because I checked by hand, five days after the mechanism that is supposed to tell me had itself shipped, and before it has had a batch to run against. That is better than three and zero. It is not a solved problem, and I would rather print the number than round it.

Would I run it again

Yes, and the honest reason is uncomfortable. The retrospective's most valuable output in two runs was not any of its findings. It was the second run telling me the first run's findings had gone nowhere, which is a thing only a repeated ceremony can tell you, and which I would never have discovered by making the individual findings better.

The compounding claim from last week is still a claim. Two runs is not a trend. What batch ten teaches batch eleven is still something I have written down rather than something I have watched. But I know one more thing now than I did, which is that the loop's weakest link was never the quality of what it noticed. It was the step after noticing, the one with no gate on it and no counter watching it, and I had spent all of my design attention on the noticing.

What's next

The invariant check: a name for the class of assertion that pins a pairing rather than a behavior, and the observation that a prose warning about a hazard is what a careful person writes at the exact moment they had everything in hand to write the check instead.

I handed my backlog to an agent fleet for ten days. Three of its safeguards were lying to me.

Allen Brooks — Mon, 10 Aug 2026 10:27:41 +0000

I have a launch with a date that does not move. I also have a job between eight and five, a two year old, and a four month old. Everything I build, I build around those. So the binding constraint is not taste or architecture. It is how many stories I can carry from "ready" to "merged" in the hours that are left, which for the next couple of months is most of the hours I am not at work or holding a baby.

The obvious lever is to stop being in the loop for every story. Ten days ago I wired up a fleet that pulls from my story queue, builds each story, reviews it, and merges it. Then I pointed it at the product with the deadline and let it run.

It worked, in the sense that 503 commits landed across 168 pull requests in the ten days that followed.

I should say plainly what that means, because it is the fact most likely to be misread. I did not write any of them. Not one line of that product's code has been hand-written or hand-edited by me in months. I instruct, and I review. Claude writes. Every commit below is an agent's work, including the three failures.

It also lied to me three times, in three different ways, and not one of the lies announced itself. This is a post about the three. I have written two posts in a row about the gap between a check existing and a check running: green is not evidence, then make it fail on purpose. I did not expect to find the same disease in a system I built with that lesson still fresh.

The shape of the machine

Two roles, deliberately unequal.

The planner is the expensive one, and the only part allowed to hold the whole graph: what is in the backlog, what the written strategy says matters, what depends on what, which stories produce something another one consumes. It fills the ready queue itself rather than waiting for me to hand-pick the next card, because a machine that needs me to curate its input every cycle has relocated the work rather than removed it. It behaves like a scrum master rather than a filter: when the roadmap needs a story that does not exist, it writes the stub.

The executor is the cheap one, and it is deliberately dumb. It carries stories through create, build, review, merge. It does not re-derive dependencies. It does not resolve ambiguity. Every unclear thing is an escalation rather than a judgment call, either back to the planner or to me. The rule I wrote at the top of its instructions is the only one that really matters: cheap but humble, never cheap but guessing.

Three things about how it spends money, because they are what people ask about and they are the reason it can run unattended at all:

It picks the model per step. Spec-writing and mechanical work go to a cheaper tier, implementation to a stronger one. The code reviewer is pinned to the top tier and nothing is permitted to cheapen it, ever, because a cheap reviewer defeats the only reason any of this is allowed to merge without me.
It watches its own spend. My usage telemetry exports to Grafana, the fleet reads it back, and it compares this week's burn against a target line. When it is running hot it cuts its own concurrency first, before it touches anything else. It throttles itself so that I do not have to notice.
It works the gaps between stories, not just the stories. Every story declares what it produces and what it consumes, and the fleet checks that anything being consumed has a producer that already landed.

I am going to write the full build-out as its own post, because the mechanism deserves more room than this one can give it. What matters here is that third bullet, because that is precisely where the third lie turns out to live.

Around that sit the safeguards, and the safeguards are the whole reason I was willing to let it merge anything without reading it first. Some of them police what the fleet is allowed to start:

A scope gate, so it only builds work inside the milestone I am currently shipping: the card that serves the launch, not the most interesting one.
A cap on work in flight, so a bad pattern can only contaminate a few stories before I see it.
Dependency gating, so a story does not start until the things it depends on have landed.

Others police what it is allowed to finish:

No self-review. The thing that reviews the code is a different instance than the thing that wrote it. Non-negotiable, and the reason review is a separate spawn rather than a continuation.
A live design review on any story that touches a real user-facing surface. A reviewer drives the running preview deployment, looks at the rendered result in light and dark, checks the empty and error states, and returns a verdict with evidence.
A human-eyes lane. Anything touching auth, billing, money movement, schema migrations, or the merge gates themselves gets built and driven to green automatically and then parked, unmerged, for me to click.

And one polices the whole run: the budget throttle above.

That is the design. Three of those safeguards were lying, and a safeguard lies by reporting a protection it is not providing.

What I am not going to claim

I went and counted, instead of trusting my memory.

The last full week before the fleet existed: 403 commits. The first full week after it: 401. Flat. If you were hoping for a throughput multiplier, it is not in this data, and I am not going to manufacture one out of a chart that does not show it.

One thing makes that flat line more interesting than it looks. I had been out of work for four months. My first day back at an eight-to-five was July 29th, and the fleet landed on the 30th. So the 403 week was a week when this product was my full-time job, and the 401 week was the first week it emphatically was not.

That is the honest claim. Not that the fleet made me faster. That it held my pace steady through losing most of my waking day to something else, which for the next couple of months is the only result I actually need.

What changed underneath the flat line is what a single unit of work costs me.

Before, every story meant opening three sessions by hand: one to write the spec, one to implement it, one to review it. Dozens of times a week. The cost was never the typing. It was that each one required me to stop what I was doing, decide it was time, and start something. That is a tax on attention, and attention is the resource I am actually short of.

The fleet instantiates all three the moment they are due, in order, without asking. I read outcomes instead of driving steps. The same output for a fraction of my attention is a trade I will take every single time, and it is the honest version of what happened here. It is also not the version that fits in a headline, which is exactly why I want it stated before the interesting part.

Lie one: the instructions the fleet was running were not the instructions I had written

The fleet's behavior lives in instruction files. Those files live in a company module that gets distributed to every product repo, and the copy the agent runtime actually loads is generated separately, into a different directory.

Except it was not generated. Nothing generated it. The copies the fleet was really loading had been made by hand months earlier, and they had drifted thousands of bytes behind the source I kept carefully editing.

So a real batch ran against an executor missing two of the safeguards I listed above as though they were operating: the scope gate and the live design review. Neither fired, because neither was in the file it was reading. The planner's copy was worse, missing four of its sections and its entire milestone model.

This is a known hazard in the underlying tooling, not a clever novel bug of mine. The same class of gap has been reported upstream more than once: skills installed into a custom module's own directory without the registration that makes them loadable (BMAD-METHOD issue #2107), and custom-module agents installed without the commands that make them invocable (issue #1131). Both are closed now, so I am not pointing at an open ticket and calling it my excuse. I am pointing at prior art: this is a well-worn hole, and my module fell in it.

What kept me in the hole was not the tooling. The instruction file's own header stated, in confident prose, that the installer generated that copy "exactly as it does" for the built-in modules. That sentence sat at the top of the file for months. It was wrong. A document asserting that a mechanism is wired, sitting inside the mechanism it is wrong about, is the purest version of the failure I wrote up two weeks ago.

Lie two: the gate that should have caught lie one read silence as approval

Two stories touching user-facing surfaces merged with no design review at all. Not a review that passed when it should have failed. No review.

That is the same incident as lie one, viewed from the other side. The drift explains why the reviewer never ran. It does not explain why nothing noticed. The gate looked for a rejection, found none, and let the story through, which means a missing reviewer and a clean pass produced identical output.

Two independent bugs stacked on one control: the reviewer was absent, and the gate was built to be silent about absence. Either one alone is survivable. Together they are indistinguishable from working. A gate that only blocks when it produces a rejection is not a gate, because the most common way for a gate to fail is not to run at all.

Underneath both sits a third temptation, and it is the one I would warn you about most: letting the code reviewer "also cover the UX." It cannot. A code reviewer reads class names and token references off a diff. A broken layout, an unreadable dark mode, an empty state that renders as a blank rectangle: none of those are visible in a diff. Folding a check into a reviewer that structurally cannot perform it is not a cheaper version of the check. It is the absence of the check, wearing its name.

The rule now is one line, and it is the rule from two posts ago in a new costume. Exactly two artifacts satisfy that gate: a recorded pass with live evidence attached, or my explicit written waiver. Anything else, silence very much included, is a block.

Lie three: a merge that made another story's premise false, with nothing to notice it

The first two came down to a control that was missing, and a gate that could not tell it was missing. This one was there, ran exactly as designed, and reported a clean board anyway. It is the one I find genuinely hard, and the one I would not have predicted.

The fleet watches the seams between stories by checking dependencies. When a story merges, it asks what that merge unblocked. That is a set-membership question and it is cheap, which is why a mechanical executor is allowed to do it.

But a merge can break another story without touching a single dependency edge. A design document is full of statements about the world: this thing is not wired up yet, nothing reads this yet, this path is inert. Every one of those was true when it was written. A merge can turn any of them false while the story that rests on it is still being built, and nothing in the dependency graph moves at all.

That happened. One story merged and quietly falsified a premise that another in-flight story's design rested on. The dependency check saw a clean board and kept pulling. A human caught it by reading, which is not a mechanism I can schedule.

The fix is embarrassingly small, which is the good kind of small. After a merge, take the names of the surfaces that the merge touched, and grep the in-flight branches' design docs and changelogs for those names. Do not evaluate whether the claim still holds in spirit, because that is judgment and the cheap executor is not allowed to have any. Report that an in-flight document names something this merge just changed, and let the planner decide. One command, and it would have caught it.

What is not built

I have ended the last few of these with an honest list of what I have not finished, because a half-built system that thinks it is finished is worse than one that knows it is halfway. Here is the list for this one.

The design calls for a retrospective pass after each batch: a set of specialist lenses that look at ten merged stories together and catch the drift that no single story's review could possibly see, because the defect only exists in the space between them. Three stories that each independently invented a slightly different pattern for one job, and each passed review on its own terms. A feature that is now complete and undocumented because no single story owned the seam. That review is written down, it is accepted, and it does not exist. Only the planner and the executor are real.

Which means the compounding half of this system, the half where what batch ten teaches gets fed back so batch eleven starts from a higher floor, is currently a document. The throughput half is running. The learning half is a plan.

There is one detail I cannot leave out. That unbuilt retrospective specifies six lenses, and one of them is a documentation lens whose stated job includes mining these posts out of what a run just taught. This post is that lens's output, produced by hand, because the thing that was supposed to produce it has not been built. I am not sure whether that is funny or just accurate.

What it costs

The fleet spends my weekly model allowance, which is why the throttle exists and why it cuts parallelism before it does anything else. What it may never do is cheapen the reviewer.

The real cost is subtler. Every safeguard I listed is a claim about a system I could not watch directly, and I was making those claims to myself, in documentation, and believing them. Three of them were reporting a protection they were not providing, and I did not learn that from an alert. I learned it from reading, from a batch behaving oddly, and in one case from a person noticing something no check was positioned to notice.

The part I did not expect

The honest summary of ten days is not "agents built my product." It is that agents built my product, and the part of the system I trusted most, the safeguards, was the part with the highest rate of quiet failure.

Which follows, if you sit with it. The feature code announces its failures, because a broken feature is visible the moment someone uses it. The safeguards are the only components whose success looks exactly like doing nothing, so they are the only ones that can fail for months in complete silence. I have now found that same shape in a Vale rule, in a CI check, and in an agent fleet, and I no longer think it is a coincidence about any of those three things. It is a property of anything whose job is to sit quietly and not fire.

What's next

The unbuilt half. I am going to build the retrospective pass and run it against a real batch. Then I will write up whether specialist lenses reading ten merged stories at once find the drift the design promises, or whether that is a fourth thing I believed because I wrote it down.

Killing the long-lived tokens in my CI, one exchange at a time

Allen Brooks — Mon, 03 Aug 2026 02:26:14 +0000

A static API token stored in GitHub Secrets is a breach that has not happened yet. The timer is already running. You do not know how long the fuse is, because a long-lived token's fuse is however long you set it at mint time, which for most people is "never thought about it."

At enterprise scale a leaked CI token is an incident: rotate it, audit the blast radius, file the postmortem. At solo scale it is the same product-ending event as any other breach. My CI can reach Doppler, which holds every secret every product uses. A token that lets it do that, sitting in a secret store, readable by any workflow, valid for months, is precisely the thing I do not want to own.

So the goal is to own zero of them. Not "rotate them on a schedule I will forget." Zero. The mechanism that gets you there is OIDC, and the good news is that it is strictly better by default: less to store, less to leak, less to remember.

The mechanism, in one paragraph

Instead of storing a token, the CI job proves who it is at runtime and trades that proof for a token that expires in minutes. GitHub Actions can mint a short-lived OIDC token for a workflow run, signed by GitHub, carrying claims about which repo and which workflow asked for it. Doppler has a matching feature, Service Account Identities (a paid capability, which I will price out in the cost section), that validates such a token against a set of expected claims and hands back a short-lived Doppler session token in exchange. The job posts the GitHub token to POST https://api.doppler.com/v3/auth/oidc with an identity slug, gets back a session token good for ten minutes to an hour, uses it, and the run ends. Nothing durable is stored anywhere. (I skip Doppler's off-the-shelf secrets-fetch-action, which only injects fetched secrets as environment variables; my workflows need the raw session token itself, to hand to doppler run and to Terraform as TF_VAR_doppler_token.)

Why a static service token is the wrong shape

Three properties make a long-lived token structurally bad, and they are the same three that pushed the GitHub side off personal access tokens in an earlier migration:

Lifetime. The token is valid until someone remembers to revoke it. Rotation is a manual ceremony with no calendar attached. The compromise window equals the lifetime, and the lifetime is effectively forever.

No identity. A Doppler service token has a label, not a workload-bound identity. When it is exercised, the audit log names the token, not the caller. A leak shows up as "that token did something," with no provenance about which workflow in which repo actually made the call.

No revocation chokepoint per purpose. One shared token feeds many consumers. Revoking it kills all of them at once. Granular, per-purpose revocation means minting per-purpose tokens, which nobody does, so in practice you get one big token and an all-or-nothing kill switch.

OIDC fixes all three at once. The credential lives for minutes, it is bound to a named service-account identity with claims that pin it to a specific repo, and each logical purpose gets its own identity you can disable independently.

What actually runs on OIDC today

Here is the concrete state, not the aspiration.

A word on "per-product" first, because it runs through everything below. Lionshead is a set of small separate repos, one per product, rather than one monorepo. I wrote about that setup and why I chose it in the reconciler post. The practical effect here is that most of this machinery is replicated per repo: one identity, one Doppler config, one token scope each, so a compromise in one product cannot reach another.

Terraform plan and apply, every product, no stored token. Every terraform-plan (on PRs) and terraform-apply (on merge to main) authenticates to Doppler by OIDC. The caller declares permissions: id-token: write, passes a per-product identity slug (for this repo, LIONSHEAD_WEB_DOPPLER_TF_IDENTITY_SLUG, a GitHub variable), and the reusable workflow does the exchange. The slug is not a secret: it is a stable identifier that is worthless without a GitHub-signed token carrying the right claims, so it lives as a plain repo variable that Terraform itself manages.

Three named service accounts, not one shared token. The setup provisions ci-platform-services-tf (admin, OIDC, session TTL 3600s, one hour), ci-credential-sync (OIDC, session TTL 600s, ten minutes, the job that writes org secrets into per-product Doppler configs), and one transitional static-token account (covered below). Each has its own claims and its own independent off switch.

The Terraform-managed identity itself. The Doppler provider has a native doppler_service_account_identity resource with a config_oidc block (discovery_url, claims, ttl_seconds). The thing that grants OIDC access is itself declared in code and reviewed in a PR, not clicked into a dashboard. The one exception is the very first admin identity, which cannot create itself: that is a one-time manual step in the Doppler dashboard, which I wrote down as the single documented exception it is, so it is not a mystery to future me.

A fallback that fails loud, not silent. The Terraform callers explicitly pass doppler-token-secret-name: ''. The reusable used to default that input to DOPPLER_TOKEN, so if an identity slug ever resolved to empty (org variable deleted, repo forked, a run context that cannot see org variables) and a bare DOPPLER_TOKEN happened to exist in scope, it would quietly slide back onto the static path. Passing an empty string forces a loud OIDC failure instead of a silent downgrade to the thing I am trying to delete.

A guard that proves it stays gone. The old workplace DOPPLER_TOKEN GitHub org secret was deleted on 2026-06-09. A scheduled audit workflow (security-audit-doppler-token.yml) watches for its reintroduction, and post-deletion Terraform runs across three products came back green with zero deprecated-static-token warnings, which is the only actual evidence that every Terraform caller is really on OIDC. (If that sounds like a theme, it is the make-it-fail-on-purpose rule from the last post: the guard is only worth something because I watched the audit catch a planted token before I trusted it.)

What still holds a long-lived token

This is the part every "we adopted OIDC" post leaves out. I have not finished. Here is exactly what is still static, and why.

The per-product Terraform bridge. Per-product Terraform still needs a Doppler token for two things: its own Doppler provider block, and its Vercel-sync REST calls. That second one is worth explaining: those calls register a Doppler-to-Vercel integration, which is what pushes each product's secrets out of Doppler and into that product's Vercel project environment, so the deployed app actually has its config at runtime. Terraform sets that sync up through Vercel's API, and the call authenticates with a Doppler token. An OIDC session token (TTL 3600s, one hour) would expire partway through a long apply, so one service account, ci-per-product-tf-bridge, keeps a long-lived token. The one mitigation that matters: that token lives only in Doppler's own state and the per-product config it populates. It is never written to a GitHub org secret. It is explicitly labeled transitional; the follow-up is per-product OIDC identities that retire it.

The Vercel build wrap. The vercel-deploy jobs still wrap vercel build in doppler run using a per-product static secret (LIONSHEAD_WEB_DOPPLER_SERVICE_TOKEN_CI) to inject build-time secrets like the Sentry auth token. This one is a plain GitHub secret and it is next on the list.

Note syndication. The workflow that cross-posts these notes to dev.to, Hashnode, and Mastodon uses that same per-product static token to pull the app tokens out of Doppler. The workflow file literally carries a # Migrate to Doppler OIDC comment, because I wrote the honest TODO in the place I would actually see it.

The Terraform state-backend keys. The R2 credentials that read and write remote Terraform state (LIONSHEAD_WEB_TFSTATE_R2_*) are static repo secrets issued by platform-services. These are S3-style keys, not Doppler tokens, so they are a different migration on a different provider's timeline, but they are static and I am not going to pretend otherwise by leaving them off the list.

So: the Terraform authorization path is done. The build, deploy, and syndication paths are not. Naming the gap is the point. A half-migrated system that thinks it is finished is worse than one that knows it is halfway.

What it costs

The exchange adds one HTTP call and a few seconds to each job. The provisioning is Terraform I write once and inherit everywhere. The GitHub side is free.

The Doppler side is not, and I want to be straight about it: Service Accounts and their OIDC identities are a paid feature. They start on Doppler's Team plan, which is $21 per user per month as of this writing, so if you are on the free or Developer tier this specific path is not open to you without upgrading. In my case the marginal cost of the OIDC work was zero, because I was already on the Team plan before this migration for an unrelated reason: a different product had outgrown the free tier's cap on secret syncs. That is luck, not a saving, and I would rather you hear the real number than a "no vendor to pay" line that only holds if you happen to be pre-upgraded.

Against that cost, what I stop owning is a set of long-lived credentials to my entire secret store, each one a standing invitation I have to remember to revoke. At solo scale I would rather have a mechanism that forgets on my behalf, every single run, than a discipline I have to sustain forever.

What's next

Next in the CI/CD sub-series: path-gated test routing across a five-product portfolio. How one reusable workflow decides which tests to run based on what a PR actually touched, why the path gate is shared but the runner command is deliberately caller-owned, and what breaks when you get that split wrong.

How do you know a passing check still works? Make it fail on purpose.

Allen Brooks — Thu, 30 Jul 2026 10:49:15 +0000

A feature makes money. A check protects it. Those are not the same job, and the difference is why checks rot.

When you ship a feature, the market tells you fast whether it works. Someone signs up or they don't. A check has no such feedback loop. A check that has quietly stopped working looks exactly like a check that is passing, and nothing outside the build can tell you which one you have.

At corporate scale a QA team is paid to close that gap. Their whole job is the negative case: break the thing on purpose and confirm the alarm rings. At solo scale there is no such team, and the negative case is the first thing to fall off the list, because it is the work that sits furthest from revenue. Features feel like money. Checks feel like insurance. The trap is believing features are money. Features are money-making. Checks are money-protecting. Skip the second one and you don't notice for months, which is exactly the failure mode I wrote about last time.

The count I had been avoiding

In Green is not evidence I found four checks that had never done anything. An em-dash linter whose regex could not match. A security scanner that was never installed. A cost gate that was only ever red. A config file I had documented but never written. The post ended on a rule I was very pleased with: make it fail on purpose. Every gate should have a known-bad fixture, something you can point it at that must fail, because if you have never watched a check fail you do not have a check, you have a decoration that emits green.

Then I did the obvious thing I had not done. I counted how many of my own gates actually had one.

The answer was humbling. Across my repos I run several dozen distinct gates. The reusable workflow every repo calls runs nine third-party scanners: Trivy twice, Gitleaks, Checkov, Hadolint, Actionlint, Shellcheck, ZAP, and an Infracost cost gate. Not one of them has a fixture. My content pipeline runs four gates on every note: prose style, spelling, frontmatter schema, link integrity. Not one of them has a fixture. The em-dash rule, the one this whole series keeps circling back to, had none.

What did have fixtures? Only the checks I had written myself. The little scripts that scan for hardcoded hex colors, or unpinned GitHub Actions, or drift in my shared config files: every one of those has a negative test that feeds it bad input and asserts it goes red. I have a unit-testing habit, and it reaches exactly as far as the code I author. Everything I bought, and every rule I configured inside someone else's tool, I took on faith. The fixture was present wherever I already had the habit, and absent everywhere I was trusting a vendor or my own YAML.

The two places I had gone further

Twice, I had wired the negative case into CI itself, so it runs forever instead of once. One is a schema gate for my product metadata: CI points the validator at a deliberately invalid file on every run and fails the build unless the validator rejects it. The other is my design-token scanner, which re-runs its own "this hex literal must be caught" test inside every repo that calls it. Two, out of several dozen. Both, again, on code I wrote.

Has any of this ever caught a regression?

Here is the part I have to be honest about, because the whole point of the last post was honesty about what a practice actually catches. Has one of these fixtures ever caught a real regression? A gate that silently broke, and a fixture that screamed?

No. Not once.

Every real catch I can point to was something else. The em-dash breakage from last post I found by hand, poking at it, not because a fixture rang. The one genuine vulnerability my pipeline caught recently, a critical CVE in a transitive dependency that showed up the week I was writing the last post, was a live scanner doing its ordinary job, not a canary. And the closest thing I have to a fixture story runs the wrong way: my hex-color scanner once misfired, flagged a pull-request number as if it were a color, and blocked a real merge. I fixed it by adding the negative tests I should have had first. The fixture did not prevent the bug. The missing fixture allowed it, and I added the fixture afterward, as an apology.

So this is not a victory lap. I am writing down a rule I have mostly not been following.

So I built one

Then I built the first one properly, on the gate that most deserved it.

There is now a file in my repo whose only job is to be wrong. It contains a single em dash and a line telling you not to remove it. It never renders and it is not a post. And there is a step in CI that points Vale at that one file and inverts the result: if Vale reports the em dash, the step passes; if Vale comes back clean, the step fails the build and tells me the gate has gone inert. It is about fifteen lines. The em-dash rule had been silently dead once already, for the entire life of the repo, and nothing would have told me if it happened again. Now something will.

The first time I ran it, I broke the rule on purpose and watched the build go red. That is the only moment in this whole exercise that constitutes actual evidence the check works. Everything before it was faith.

A fixture is a second thing that can rot

I want to be clear-eyed about the obvious objection. A known-bad file that quietly stops being known-bad is just more green, one layer down. Adding a fixture and never watching it fail leaves you exactly where you started.

So the discipline is not "add fixtures." The discipline is: break the thing once, on purpose, watch the alarm ring with your own eyes, and then wire that watching into CI so it repeats without you. The fixture is not the point. The watching is the point. The fixture is just how you make the watching automatic, and cheap enough that you keep doing it after the day you cared.

The rule

So here is the rule, and it is going into my standards where the checklist that opens every story will ask about it. No gate ships without a fixture that proves it can fail. And the smaller, harder half, the one I keep having to relearn: a fixture you have never watched fail is just another green light.

If you run any checks at all, you almost certainly have one that has never rung. Pick it. Break the thing it is meant to catch. Watch. It takes ninety seconds, and it is the only proof you are ever going to get.

I audited 2,864 CI checks. Several had never run, and green never told me.

Allen Brooks — Thu, 30 Jul 2026 10:49:14 +0000

Someone on Hacker News asked a one-line question about the security checks I run on every PR: how often do those checks actually hold your PRs back?

I didn't know. That bothered me more than it should have, so I counted every PR check I have ever run. 2,864 of them, across nine repos.

The answer to his question turned out to be boring. What I found on the way to it did not.

The boring answer

286 of the 2,864 runs failed. About 10%.

That number is useless on its own, and it took me embarrassingly long to see why. A 10% failure rate sounds like a tax, one PR in ten held up at the door. It isn't, because the failures are not spread out. They arrive in tight three or four day clusters with months of silence between them, and every cluster sits on top of a migration. Late April, when I pulled these checks into a shared reusable workflow: eighteen cost-gate failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of nothing. A second product adopting that workflow in May, with a commit two days later called fix(ci): resolve remaining PR check failures. Skeleton repos I bootstrapped in July that lit up the same way and then went quiet.

The repos with the worst failure rates in the whole org are the ones that have only ever been in setup, with no steady state to average against. The worst sits at 57%, and it is the template that the others get cloned from.

None of that is in the numbers. I only know it because I remember those weeks. Read cold, the dates say the opposite: that my oldest and busiest repo carries the most failures and is therefore the most burdened. It carries the most failures because it is where I did the extraction. The context that told the true story from its inverse lived in my head and nowhere else in the data. That is the benign version of a problem I was about to hit in a much worse form, one where the missing context was not in my head, or anywhere I could reach.

The number that didn't fit

I had the answer to the question and could have stopped. What stopped me was a small number that didn't fit.

I have a Vale rule in my content pipeline called EmDashes. It is twelve lines long. It is set to level: error, which fails the build. It exists because I don't write with em dashes, I never have, and I would rather they never reach the site. My drafts run through Claude, which likes them, so the rule is there to strip the ones I miss. That is stated plainly on the colophon; the rule is the enforcement.

The post that started all of this shipped with fifteen em dashes in it.

Vale ran on that post. Vale passed. On that same file, in that same run, Vale reported fifteen passive-voice alerts and ten weak-word alerts, so it was finding the file, parsing the MDX, and applying my custom styles. It reported zero em dashes in a document containing fifteen of them.

Here is why. Vale's existence check compiles a tokens list into a regex with word boundaries wrapped around it:

(?i)(?m)\b(?:—|–)\b

\b asserts a boundary between a word character and a non-word character. An em dash is punctuation. There is no word character on either side of the match to anchor the boundary against, so \b can never be satisfied, so the pattern matches nothing. Ever. Vale has a nonword: true option that drops the boundaries, and the rule works correctly the moment you add it.

That rule had sat in my repo, at error level, catching absolutely nothing, since the day I wrote it. And there is no observable difference between that and a rule that works. Both emit silence. Both go green.

It was worse than one bug

I fixed the regex, shipped it, felt good about myself, and then went looking for proof that the fix actually gated anything. It didn't.

Here is the CI step that ran Vale:

vale --no-exit --output=line web/content/notes/ | tee vale-report.txt
if grep -E ' error ' vale-report.txt; then
  exit 1
fi

--no-exit forces Vale to exit 0 regardless of what it finds. That was deliberate: print the report first, decide separately. The deciding is that grep. And --output=line emits path:line:col:check:message. There is no severity field anywhere in that format. The grep could never match. The two halves cancel each other perfectly: Vale is forbidden from failing the step, and the thing meant to fail the step is searching for a word the format never prints. That job had passed every build it ever ran. It could not have done anything else.

Then, while testing the fix, Vale flagged an em dash inside a fenced code block, which markdown scoping is supposed to skip. It turns out Vale has no built-in knowledge of the .mdx extension: without an explicit [formats] mapping it treats the file as plain text, applies no markdown scoping at all, and lints your code blocks and inline code as if they were prose. My config carried a comment confidently asserting that "Vale's markdown scoping handles fenced code blocks and inline code natively." It does not, because I never told Vale what it was reading. Six of the alerts on my notes tree were phantoms from that.

So: a rule that could not match, inside a runner that could not fail, parsing a format it did not know how to read. Three independent bugs stacked on one rule, every one of them silent, the whole assembly green. I only found the second and third because I went looking for evidence that my fix to the first one worked.

Once I saw it, it was everywhere

Then I looked at the rest, and found the same disease in three more checks.

Semgrep. My own security baseline mandates it. My own post names it. It is not wired into any workflow, in any repo. A Semgrep that was never installed reports zero findings. So does a Semgrep that is running clean.

The cost gate. Failing for months in two repos, red on every Terraform PR, for the excellent reason that I never provisioned the GitHub App token it needs to read my private modules. It was not gating anything. It was just red, in the corner, next to the checks that were working.

A config file that does not exist. My post says known-false-positive suppressions for the IaC scanner go in a .checkov.yaml. Zero of my nine repos contain that file. I have never written one. I documented a system I had imagined.

I audited the tools with tools that lied

It gets worse, because the tools I used to audit the tools had the same disease.

My first pass used gh run list --limit 100. It told me my secret scanner had fired twice. It has fired thirty-five times. The window silently truncated at a hundred runs, and my busiest repo has over five thousand. My second pass filtered to pull-request events, which silently discarded every push run. A sampled window told me one mature repo had a spotless record; its real number is 27 failures out of 313. Every one of those queries returned a confident, well-formatted, wrong answer. Not one of them announced what it could not see. I was auditing my instruments with instruments that shared their failure mode.

The thing I actually learned

A check that isn't running looks exactly like a check that's passing.

That is the whole post. Green means "nothing was reported." It does not mean "nothing is wrong," and it does not even mean "something looked." The absence of an alert is perfectly compatible with the absence of a scanner, and from outside the build you cannot tell those two apart.

It is the same shape as the 10% number, pushed to its limit. There, the surface reading inverted the truth, and only what I happened to remember could set it right. Here there is nothing to remember. A green check withholds the one fact you actually need, whether a check even ran, and no amount of context brings it back.

Make it fail on purpose

Make it fail on purpose. That is the entire technique.

The only reason I know my em dash rule works now is that I fed Vale a line containing an em dash and watched it report the em dash, then removed the fix and watched it report nothing. Ninety seconds, and the before-and-after is the proof. The only reason I know my one grandfathered exemption is scoped to a single file, rather than having quietly switched the rule off everywhere, is that I temporarily broke a different post and watched the gate catch it.

Every gate you own should have a known-bad fixture: something you can point it at that must fail. If you have never watched a check fail, you do not have a check. You have a decoration that emits green.

This generalizes well past linters. Any alarm you have never heard ring is in one of two states, and you do not know which one.

The punchline

I wrote most of this up as an update to the original post. In it, I said the security stack had caught two real things in four months: a CVE in a transitive dependency, and an IAM policy attached to a user instead of a role. Thin, I said, but the premium is near zero and the downside ends the product.

The pull request carrying that sentence went red.

Trivy caught CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I had changed. It was disclosed against code I merged weeks ago, which is precisely the case that a PR-time scan exists to catch. Six minutes to bump it and go green.

So it is three, not two, and the third arrived while I was typing the sentence that said there were two.

I would take this trade again tomorrow. Near-zero premium against a downside that ends the product is not a hard call. But "we run X" is a claim, not a fact, until you have watched X fail.

What's next

The preview-environment pattern that ZAP runs against is already written up: Preview environments are the dev loop. Vercel plus a fresh Neon Postgres branch per PR, a real running app talking to a real database on every review, for almost nothing. This post was written before that one and sat in a drawer for a week. The next one is on making the invisible checks announce themselves, which is the other half of this problem.

Every pull request gets its own app and its own Postgres branch, for $0

Allen Brooks — Thu, 30 Jul 2026 10:43:56 +0000

Every place I've worked plumbed everything to one shared dev environment. One database, one deploy, everyone's half-finished work landing in the same place. The database slowly filled with test accounts named asdf and orders for negative dollars, until someone declared bankruptcy and refreshed it. Then it filled up again. I've triggered more than one of those refreshes myself.

Exactly one company I worked at had per-feature environments, and they were narrow: a preview for the GraphQL schema and nothing else. The app, the database, the auth flow, all of it still pointed at shared dev. The one thing they isolated was the one thing that was easy to isolate.

Lionshead does the opposite. Every pull request gets its own running app AND its own database. Not a mock. Not a shared dev instance with a "please don't touch" Slack message. A real Next.js deploy talking to a real Postgres database that was branched, schema and data, from the parent in under a second and will be destroyed the moment the PR closes.

The point isn't fancy infrastructure. The point is that local, preview, and production are a stack, not a substitution. Local is where I write the code. Production is where users live. Preview is the rung in between, where a change proves it actually runs before anyone trusts it. Kill that middle step and every review becomes "looks good to me" against a diff, or a push-and-pray to shared dev. The preview environment is where the dev loop closes.

Here's what happens when I open a PR.

Neon branch, created or refreshed. A copy-on-write branch named preview/pr-<number> is cut from the database's parent branch. Copy-on-write means it's not a dump-and-restore; Neon clones the storage layer by reference and only diverges on write, so the clone itself is effectively instant regardless of how much data is in the parent. The step is idempotent by branch name, so every new commit on the PR refreshes the same branch instead of piling up new ones.

Atlas applies the PR's schema. Lionshead manages Postgres schema declaratively with Atlas. Against the fresh branch, atlas schema apply reads the desired schema, diffs it against what's live, and applies the difference. The branch now has exactly the schema this PR proposes, running against real Postgres, not a SQLite stand-in or a mock that lies to you about what a foreign key does.

Vercel builds and deploys against that branch. The build runs with the per-PR DATABASE_URL injected, so anything that touches the database at build time sees this PR's branch. Then vercel deploy --prebuilt ships the artifacts with the same URL wired in at runtime. The reviewer gets a link to a live app, and that app is talking to this PR's database and no one else's.

A sticky comment carries the link. One PR comment, updated in place on every commit, holds the latest preview URL and a short per-commit history. No comment spam, one place to click.

On close, the branch is deleted. When the PR merges or closes, a cleanup job deletes the Neon branch and confirms it's actually gone. The junkyard never accumulates, because the junkyard is thrown away every time.

All of this lives in one reusable workflow, lionshead-vercel-preview-deploy.yml, that every product repo calls. There's a hard-won detail buried in it worth surfacing: it's a single job, not three, because GitHub Actions refuses to pass a step output between jobs when the value looks like a secret. A Postgres connection string contains a password, so a clean "branch job hands the URL to a deploy job" design gets the URL silently scrubbed to an empty string. The fix is unglamorous: keep the URL inside one job. Most of the good infrastructure decisions I make look like that.

What it costs

Nothing worth mentioning. Neon is on the free plan. Vercel just moved to the $20/month Team plan. That's the whole bill for this capability.

The reason it's free is the same reason it's fast. Neon bills for storage and compute, not for the number of branches, and an idle branch scales its compute to zero. A per-PR branch that nobody is actively hitting costs effectively nothing while it sits there, and it shares the parent's storage by reference until it's written to. So "a database per pull request" sounds extravagant and turns out to be a rounding error.

What it has actually caught

I'll be honest about this the way I was honest about the security tooling: most bad migrations never reach a PR, because I run migrations against a local Docker Postgres before I push. The preview branch isn't where I catch a typo'd column.

Where it earns its keep is the full-workflow bugs that are miserable to test locally. Sign-up and sign-in against Firebase, for instance, are genuinely hard to exercise on a laptop; on a real preview URL with a real database, I can watch an account get created and see exactly what we persisted, including the times we captured a wrong or incomplete value. And I can watch all of that happen without dumping a single piece of that test garbage into the shared dev database.

That's the quieter, bigger win. In every corporate project I've seen, the dev database degrades into a landfill of fake accounts and abandoned test data, until it looks nothing like production and stops being useful for real testing. Per-PR branches give the trash somewhere to live and die. The messy test data stays on the ephemeral branch that gets deleted on merge, and dev stays clean enough to keep nudging it toward looking like prod. Cleaner dev means more focused development, which means bugs that show up in testing are more likely to be real.

Why almost nobody does this

I've said before that I haven't seen this pattern at solo scale anywhere else, and I still haven't. I've seen it exactly once at corporate scale.

The reason isn't cost or difficulty. It's that the enabling primitive is rare. Plenty of databases advertise "branching," but they don't mean the same thing. PlanetScale's branching is built for schema changes; its branches don't clone your data. Supabase spins up an empty database and replays your migrations and seed script. Both are useful, neither gives you a full copy of real, production-shaped data per pull request. Neon's copy-on-write branch does, in about a second, for free, and I have not found another managed provider that does the same. Most people simply don't know that per-PR data isolation this cheap is even on the menu.

The gotchas (because there are always gotchas)

Two of these bit me last week, so they're fresh.

The 10-branch cap. The Neon free plan caps a project at ten branches. I opened several PRs in quick succession, left a few in draft, and every one of them dutifully cut a branch. I hit the ceiling, and preview deploys started failing across the board. The fix was to be stricter about which PRs earn a database: filter the workflow so draft and not-yet-ready PRs don't provision a branch until they're actually up for review. The capability was never the problem. My discipline about triggering it was.

Orphaned branches from a skipped cleanup. The nastier one. A concurrency-rule interaction was canceling the cleanup job mid-run when a late push raced a merge, so the branch-delete never executed and branches leaked, which of course marched me straight back into the 10-branch cap. The fix was giving the cleanup job its own concurrency group that never cancels in progress, plus making the delete step actually authenticate instead of failing silently while reporting green. A teardown, once it starts, has to finish. That one cost me an afternoon and a good deal of "why do I have eleven branches when I have three open PRs."

It's slow, though not the part you'd guess. Full transparency: the preview environment is by far the longest thing on the PR. The deploy job runs about five minutes, and the whole preview pipeline lands closer to ten once you count waiting on sibling checks. I pulled the step timings across recent runs to see where those five minutes actually go, and almost none of it is the database. Creating or refreshing the Neon branch takes 10 to 15 seconds on a cold create and effectively nothing when it's just refreshing an existing one; the copy-on-write clone is instant, and the seconds that do pass are spent standing up the branch's compute endpoint, not copying data. Applying the schema with Atlas adds another 10 to 15 seconds, or nothing when there's no diff. The remaining four-plus minutes are all Vercel: about 20 seconds to install the CLI, then three and a half to four minutes for the Next.js production build, then about 25 seconds to upload the artifacts. So the database work tops out around half a minute and is usually far less. Building and shipping a whole app per PR is the slow part, and it would be slow with or without the branch. Everything else on the PR finishes while the build is still going. I think that's an acceptable price for a real running app on every change, but I won't pretend it's instant.

None of these are reasons not to do it. They're the tax you pay once, learn from, and encode into the workflow so the next repo inherits the fix instead of rediscovering the bug. Which, if you've read the last post, you'll know is a lesson I keep having to learn.

What's next

The preview deploy needs secrets, a Vercel token, a Neon key, to do any of this. For a long time that meant long-lived tokens sitting in repository secrets, which is exactly the kind of thing the security post frets about. The next note in this sub-series is how that got fixed: OIDC through Doppler and GitHub Actions, and how short-lived, federated credentials killed the long-lived token entirely.

The security checks in every Lionshead PR

Allen Brooks — Thu, 30 Jul 2026 10:43:55 +0000

At enterprise scale, a breach is a bad quarter. You have a legal team to coordinate disclosure. A disaster-recovery plan you drill annually. A PR team to control the narrative. A security team to quarantine, investigate, and triage the incident. A bank account big enough to absorb regulatory fines, class-action settlements, and the customer churn that follows.

At solo scale, you have none of those. A real breach of user data will almost certainly end the product. Not "hurts the brand," not "sets us back a quarter." Ends it.

So you breach yourself first.

The security stack in every Lionshead PR is an automated self-breach. Every merge attempt is scanned by tools a mid-sized security team would run against production code weekly. If any of them find something, the merge blocks. The vulnerability never reaches production, and the version of me that would have caused an incident never gets the chance.

Here's what actually runs on every PR against every Lionshead product.

Trivy filesystem — vulnerability, secret, and license scanning across the whole repo. Runs on every PR, no path gate. Catches dependency CVEs, accidental hardcoded credentials, license conflicts. Configured via trivy.yaml at the repo root; fails on any CRITICAL or HIGH severity. Trivy is Aqua Security's open-source scanner and it's the single best free vulnerability scanner I've used.

Trivy IaC (Terraform) — the same tool, second mode, against Terraform files. Catches insecure infrastructure defaults (public buckets, permissive IAM roles, unencrypted resources). Path-gated: only runs when terraform/** files change. Same severity gate.

Gitleaks — secret scanning across the entire git history, not just the current diff. Catches API keys, tokens, private keys, connection strings that someone (probably me) committed years ago and later deleted. The secret is still in the history and still exploitable. Runs on every PR. Uses the default ruleset unless the repo drops a .gitleaks.toml override.

Checkov — a second IaC security scanner, run alongside Trivy IaC for defense in depth. Checkov and Trivy IaC use different rulesets and different heuristics; they catch different classes of problem. Path-gated on terraform/**. Skip lists for known-false-positive rules go in .checkov.yaml.

OWASP ZAP baseline — active DAST against the PR's Vercel preview URL. When a repo produces a preview deploy, ZAP hits the deployed site with a baseline security scan. Catches missing security headers, cookie configuration issues, common injection vectors. Fails on any WARN+ alert; per-rule suppressions live in the repo's .zap/rules.tsv.

Actionlint — GitHub Actions workflow linting. Not strictly security, but security-adjacent: catches syntax errors, invalid contexts, and permission-scope typos that would silently over-scope your workflow's access. Path-gated on .github/workflows/**.

Shellcheck — shell script static analysis. Catches quoting errors, command-injection patterns, and the classic "unquoted variable in an rm command" bug that has ended more careers than any single vulnerability. Path-gated on *.sh files.

Hadolint — Dockerfile linting. Catches insecure image bases, missing USER directives, and other Dockerfile-shape issues that expand blast radius on container escape. Path-gated on Dockerfile*.

Action pins check — a Lionshead-authored guardrail that validates every third-party GitHub Action referenced in workflow files is pinned to a full commit SHA, not a moving tag. Prevents a supply-chain attack against an action's tag from silently propagating into Lionshead's CI. Fails PRs that add or update actions without a SHA pin.

Infracost + OPA cost gate — not security in the CVE sense, but security in the "my card gets charged $10K/month by mistake" sense. Runs on every Terraform PR, computes the cost delta, and pauses the PR on a GitHub Environment approval gate when the delta exceeds $50/month. I covered this in more detail in the last post; mentioned here because it lives in the same workflow.

Two more layers run outside the PR gate.

Scheduled security audit — every product repo runs lionshead-security-audit.yml daily against the default branch. It runs npm audit (or the language-appropriate equivalent) plus a full Trivy filesystem scan. Findings open or update a single security:dependency tracking issue per repo; clean runs close any open issue. The point is to catch CVEs disclosed after code was merged, not just what a PR introduces. The majority of real-world CVEs are disclosed against code already in production; a PR-time scan alone misses them.

Renovate — dependency-update automation. The ci-standards Renovate preset applies to every product repo via a one-line renovate.json. Renovate opens PRs to bump dependencies, pin action SHAs, and update base container images. Its github-actions manager keeps the pinned reusable workflow SHAs current, which is how new products cloned from template-product stay on the latest ci-standards workflow without manual intervention. Weekly schedule keeps the noise bounded.

Three things are on the roadmap but not shipping yet.

Semgrep (SAST) — the security baseline mandates Semgrep OSS for static-analysis coverage as a free-and-open-source substitute for GitHub Advanced Security. Rollout across all products is in progress.

OWASP ZAP as a scheduled cron against dev — today, ZAP runs at PR time against the preview URL. The plan is to also run ZAP against the shared dev environment weekly to catch drift between preview builds and the environment that lives longer.

Pre-commit hooks — a Husky-driven local hook stack that runs the most annoying failure modes (a Gitleaks pre-commit scan, at minimum) before code even gets to the PR. Removes a full round-trip when you catch a mistake before pushing. On the shortlist to ship next.

The whole stack costs zero in licensing. Every tool above is open source. The cost is time to wire them up and ongoing time to tune false-positive suppressions. That cost is measured in hours, once, not months. What it buys is the difference between "my product survives" and "my product ends" in the scenario where I fat-finger a Terraform PR at 11pm.

That is the trade at solo scale. It happens to look like the trade at enterprise scale for exactly the same reason.

Update, 16 July 2026: what it cost, and what it caught

Someone on Hacker News asked how often these checks actually hold up my PRs. I didn't know, which bothered me, so I went and counted every PR check this stack has ever run: 2,864 of them across nine repos. 286 failed, call it 10%.

The failures aren't spread out, and that's the part I didn't expect. They arrive in tight clusters of three or four days, and every cluster sits on top of a setup event. The biggest was late April, when I pulled these checks out of Vesper and into the shared ci-standards workflow. Eighteen Infracost failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of silence. Another cluster in May when Crisper adopted the same workflow. Another in July when I bootstrapped some skeleton repos that are parked until Vesper ships. Between the clusters it's quiet for months. The repos with the worst failure rates are the ones that have only ever been in setup and have no steady state to average against.

So the cost isn't a tax on every PR. It's a bill that comes due when I migrate something, and it's mostly me misconfiguring my own plumbing rather than the scanners crying wolf.

The other surprise: tests, not security, are what actually block me. Failing tests account for roughly half of every PR check failure I've ever had. All the security tooling combined is well under a quarter.

What the security tools have caught, honestly: not much. Trivy found a real CVE in a transitive form-data dependency and I bumped it. Checkov found a real IAM policy attached to a user instead of a role. It also flagged AKIAEXAMPLEEXAMPLE00 in a .env.example, and read a Terraform format("postgresql://%s:%s@...") string as hardcoded credentials when the password was a random_password reference. Which is to say it caught the correct implementation and would have sailed straight past a bad one.

And then, while I was writing this very update, the pull request carrying it went red. Trivy flagged CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I'd changed. It was disclosed against code I merged weeks ago, and the PR-time scan is simply where I happened to meet it. Bumped to 0.7.5, green again, about six minutes of work.

So: ~~two real findings in four months~~ three real findings in four months, one of them CRITICAL, and the third arrived while I was typing the sentence that said there were only two.

That's still a thin haul if you measure it the way a security team measures things when it needs headcount approved. But that was never the trade. The premium is a few hours per migration and zero dollars, and what it covers is the scenario where I fat-finger a Terraform PR at 11pm and end the product. At that price I don't need it to catch much. I need it to catch the one. It just caught one while I was mid-sentence arguing it rarely does.

A correction and a clarification to the post above, while I'm here.

The post says the tuning cost is "measured in hours, once, not months." Hours is right. Once is wrong. It's hours per migration, and every new repo keeps rediscovering the same false positives because I've written a slightly different allowlist in each one instead of fixing template-product so the next repo inherits it. That's on me, not the tooling. For the full effect: publishing this update tripped Gitleaks on the AKIAEXAMPLEEXAMPLE00 two paragraphs up, and I fixed it by writing yet another per-repo allowlist. Which tells you exactly how much I've learned.

On ZAP: what runs at PR time today is the baseline passive scan. Full active DAST is on the roadmap, not shipping yet.

What's next

Next in the CI/CD sub-series: the preview environment pattern that ZAP runs against. Vercel plus a fresh Neon Postgres branch per PR, how the reviewer gets a real running app talking to a real database on every PR, and why that costs almost nothing.

I'm one person, and my CI does what a mid-size eng org's does. Here's why.

Allen Brooks — Thu, 30 Jul 2026 10:36:36 +0000

Corporate CI is often the red-headed stepchild of the dev team. Written six years ago by a consulting firm, held together with cron jobs whose owners left in two reorgs, touched only when it breaks. Nobody on the current team could explain what half the steps do.

Lionshead's CI runs a security suite, an Infracost cost gate, IaC scanning, secret scanning, schema-migration validation, and a preview-environment deploy before any PR can merge. Categorically the same coverage as many mid-sized engineering orgs.

I am one person.

The obvious question, from other indie founders, is why. The reusable workflows in ci-standards took months to write. Nobody is going to breach my pre-launch SaaS in month three. Why not push to main and let Vercel deploy?

Because this is the way cloud software is built. I don't know how to build it any other way, and I don't want to relearn. There's no three-person dev team here, no DevOps engineer, no IT department. Nothing about being alone changes what "shipping safely" means. It only changes who has to remember it, and if the answer is me, then the answer needs to be automation.

The shops I have seen succeed treat the CI/CD pipeline as first-class code. The developers who write the product also write the pipeline. The knowledge of how one flows into the other compounds over the years. The shops I have seen fail leave CI to consultants, or hire it out to a platform team that has never shipped their product, or worst of all, both. In my experience, no exception to that pattern has stuck. The shape of the CI pipeline is a leading indicator of how the whole engineering culture works.

Lionshead's pipeline is four areas: build, test, security, deploy. Two of those areas make choices unusual at solo scale.

Security. Trivy for vulnerabilities and IaC, Gitleaks for secrets, Checkov for policy, plus linters that keep the workflow files themselves honest. All open-source, no licensing cost. Any one of these is a one-line YAML addition and takes ten minutes to wire up. The unusual choice isn't setup. It's committing to run all of them, keep them maintained across every product repo, and enforce them as merge gates when the only person they block is me.

Preview environments. Every PR spins up its own Vercel deploy AND its own fresh Neon Postgres branch: a copy-on-write clone of prod's schema. The reviewer clicks the preview URL and gets a real running app talking to a real database. Schema migrations run against actual Postgres before merge, not against a mock or a shared dev database. When the PR closes, the branch is destroyed. This costs nearly nothing because Neon bills for storage and compute, not for branches. I have not seen this pattern at solo scale anywhere else.

The check that deserves specific mention beyond those two is Infracost. It runs on every Terraform PR and requires human approval when a change would add $50 or more to my monthly cloud spend. Fifty dollars is not the number that would ruin me. Fifty dollars is the number where I would want to look twice at the diff. Between an agent auto-completing a Terraform block wrong, and me clicking through a review too fast, I would rather burn a merge-queue slot than wake up to a $10,000-a-month resource I did not mean to create. Every solo founder running on their own cash should have this exact gate. Almost none do.

All of it runs end-to-end in about five minutes. That's not despite the number of checks. It's a function of designing the pipeline with parallel jobs and path-gated conditionals from day one, not retrofitting them on after the fact.

The line between too much CI and not enough CI is a straightforward one. It lands wherever developers no longer need to remember how to get code into a live environment, or troubleshoot when it doesn't. Below that line, you pay a tax on every ship in ceremony that used to happen automatically. Above that line, you pay a tax on every setup in ceremony that never had to happen. My CI investment sits right at the line for one person. It happens to look like the CI at a mid-sized org because the line is drawn by the same physics, not by the size of the team.

What's next

This post is the "why." The rest of the CI/CD sub-series is the how. Coming up: the specific security checks that ship with every Lionshead PR, how the Vercel + Neon-branch preview environment is provisioned, the OIDC secret flow through Doppler, and the path-gated test routing that keeps a five-product portfolio building against one shared workflow.

Every website should have a colophon

Allen Brooks — Thu, 30 Jul 2026 10:36:34 +0000

Every developer I know has the same itch. Pop the hood on a website, a SaaS app, an app on the App Store, and see how it was built. Which framework. Which hosting. Which type. The craft underneath the surface.

The colophon page is the formal scratch to that itch.

The word comes from publishing. A colophon is the small inscription on the last page of a book that tells you the typeface, the paper, the press, the binding. The book's how. Stripe Press still does this. Information Architects does this. A small corner of the web that takes its lineage from print has been doing it for years.

I had never heard the word until six months ago.

That isn't a humblebrag. Ten-plus years in corporate engineering, up through senior director, and the word "colophon" never came up. The companies I worked at shipped real software to millions of users, and not one of them had a public page that said "here is how we built this." It wasn't anyone's job. It wasn't a category.

I came across it earlier this year, looked it up, read a few examples, and built one that weekend.

Lionshead's colophon is at lionshead.digital/colophon. Stack, type, color source, build method, hosting. One screen, no decoration. The same structure now ships in every Lionshead product per a company rule. They all have colophons.

I added it because building things can be a craft, not just a job. Corporate engineering taught me to ship. Lionshead is teaching me to sign my work.

Keeping 10 repos in sync without a monorepo or a platform team

Allen Brooks — Thu, 30 Jul 2026 04:39:20 +0000

Most dev shops don't have standards. At least not codified and documented standards. They have a Confluence space with someone's 2021 attempt at a style guide buried under three years of half-finished runbooks and obsolete service docs. The rules aren't enforced because nobody reads them. The shops that do enforce standards pay for it: a platform team, a custom lint server, a quarterly review cadence, and someone whose job is to argue with developers about line length.

Lionshead has standards, and they're enforced for free. They live in code, in .claude/rules/ and a BMAD module, and the AI agents doing the keystrokes read them on every story. There's no human in the middle to forget the rule; the agent reads it and writes the code in the same breath. That's not the same trade as "humans who follow the rules better." The AI is doing the writing too; the free enforcement is a side effect of swapping the actor, not a process improvement layered over a human team. That changes the problem. The question stops being "how do we get developers to follow the rules" and starts being "how do we get the same rules into ten unrelated product repos without a monorepo or a platform team."

This post is the answer. A small CLI I call the reconciler: pull-based, hash-compared, idempotent. It ships a manifest from a central repo and lets each consumer adopt the parts it wants. Here is what it does, what it costs to build, and where it ends.

I have operated at the ten-repo scale before, and the reconciler is built for that target. Lionshead today is one product (Vesper); this is the system for where I am going, not where I am.

The shape of the problem

The familiar approaches to keeping many repos in sync all trade the same three axes: friction at edit time, friction at consumer time, and how much shared infrastructure the consumers have to swallow. Pick any two, lose the third.

Copy-paste. Zero infrastructure. The cost shows up at every fix: ten repos, ten patches, and the moment one repo skips a patch the divergence starts. Worse, copy-paste makes the canonical version invisible. No one can grep it without first deciding which repo's copy to trust. I shipped two months on copy-paste before the cost compounded; it compounds fast.

Git submodules. One source of truth, a real link between consumer and producer, and a developer experience that is genuinely awful. Submodules break IDE checkouts, break clone scripts, and force every consumer to handle a sub-checkout step that someone new will get wrong on day one. I tried submodules for two days; the cost showed up faster than copy-paste.

Monorepo. Solves drift completely. Also forces every product onto a shared CI, a shared release cadence, and a shared dependency graph. That cost is paid even by products that have nothing to do with each other. The big-company default; the wrong default for the operator who runs ten small unrelated things.

All three trade the same axes in different proportions. The reconciler trades a fourth axis: it accepts a small per-consumer install cost in exchange for keeping every consumer's repo shape, CI, release cadence, and dependency graph independent. That is the only trade I was willing to make at the ten-repo scale.

What the reconciler actually does

The CLI is published to GitHub Packages under the @lionsheaddigital scope. Consumers configure their npm token once per machine. After that, npx @lionsheaddigital/setup just works.

The CLI reads a single declarative manifest, lionshead.json, from the central ci-standards repo over plain HTTPS to raw.githubusercontent.com. The manifest lists every standards file the central repo owns: BMAD module sources, the Claude Code rule files, GitHub Actions reusable workflows, the GitHub project sync script, and a few others. Each entry carries a sha256 hash of the upstream file.

For every file declared in the manifest, the CLI walks one branch:

The heart of the loop is a hash comparison:

// packages/setup/lib/reconcile.mjs (excerpt)
function computeDelta(remoteStandards, localInstalled) {
  const remoteFiles = remoteStandards?.files ?? [];
  const localFiles = localInstalled?.files ?? [];

  const localMap = new Map(localFiles.map((f) => [f.path, f.sha256]));
  // ...

  const toDownload = [];
  for (const rf of remoteFiles) {
    const localHash = localMap.get(rf.path);
    if (localHash !== rf.sha256) {
      toDownload.push(rf);
    }
  }
  // ...
}

If a consumer's local copy of a declared file already matches the upstream hash, the CLI does nothing for that file. If the hash differs (the consumer is missing the file, or upstream has moved on, or someone hand-edited the local copy), the CLI downloads the upstream version and overwrites the local one. No prompt. No merge. No three-way confusion. Upstream wins; local edits to managed files are not allowed.

The full run on a clean consumer takes about three seconds. The full run on a clean, up-to-date consumer takes about one second and prints "Standards up to date." That second case is the one that earned the reconciler its name: it is safe to run a hundred times in a row, and the only side effect is that the operator knows the consumer is current.

The hard parts

The interesting design decisions in this kind of CLI are not the visible ones (the manifest shape, the file copy). They are the boring ones underneath.

Idempotency is the first hard part. The hash-compare and the conditional download together give you "do nothing when nothing has drifted." That looks easy until you find the case where BMAD's own installer silently overwrites a managed file with stale bundled content after the main download loop has already updated it. The reconciler now does a second verify-and-restore pass after BMAD runs; without it, a clean run would show "Standards up to date" while the on-disk SHA had quietly diverged from the manifest. That pass cost a day to discover and an afternoon to write. It is the kind of bug that only shows up at the seam between two tools that both think they own the same files.

Partial adoption is the second hard part. Not every consumer wants every standard. A static marketing site does not need the Test Architect workflow; a backend service does not need the design-tokens rule. The manifest supports per-consumer opt-outs via a small lionshead.json in the consumer repo that lists explicit skip paths. The escape hatch is necessary; without it the reconciler is a coupling mechanism dressed up as a distribution one.

Upgrade safety is the third hard part. The reconciler is itself one of the things it ships. Bootstrapping a brand-new consumer is a chicken-and-egg moment: the consumer does not yet have the CLI installed, but the CLI is what installs the CLI. The answer is npx: the CLI runs without a prior install, fetches itself from GitHub Packages, runs once, and exits. The local install only matters for subsequent runs, and those can be npx too. The whole bootstrap is one command.

When NOT to do this

The reconciler is right anywhere above three repos. Below three, copy-paste is genuinely cheaper, and the per-file maintenance cost is small enough that the operator can absorb drift mentally. The architecture has no real upper bound on consumer count: the CLI is stateless, every consumer is independent, and adding one is one npx away.

The real ceiling is governance, not scale. Review doesn't disappear in this model; it shifts upstream to the central standards repo, where PRs are the negotiation surface. That works fine when there's an agreed owner of the central repo. It breaks down when there isn't: the reconciler's "upstream wins" semantics turn a governance question into a technical one consumers can't appeal. A monorepo handles disagreement by making every change visible to all consumers in the same commit; the reconciler doesn't.

And the reconciler is wrong when drift is a feature, not a bug. Research codebases, demo branches, throwaway prototypes: these benefit from divergence. A pull-based "upstream wins" CLI run against a demo branch will quietly undo the demo. The right fix there is to mark the consumer out of scope, not to weaken the reconciler.

Outro: the Building Lionshead thread

The reconciler is one chapter in the Building Lionshead series. Post one was a sketch of why I built a company-specific BMAD module; the post you are reading is what the distribution mechanism for that module looks like in code. The series's spine is "small-company process at small-company cost"; this post is one chapter of it.

The next post in the series will be about the BMAD module itself: what is in it, what the boundary is between Lionshead-shaped behavior and stock BMAD behavior, and why I keep that boundary thin. After that, a post on the rule files in .claude/rules/ and the way they get loaded into agent context. Both posts depend on the reconciler existing; this post is the prerequisite.