CogniPrep is 1,074 commits old, 617 of them in the last 30 days. It has 353 test files and 7,715 unit tests that run in 26 seconds. .github/workflows/ contains exactly one file, 111 lines long, and it is a nightly pg_dump to Cloudflare R2.
Nothing runs the tests except me.
"typecheck": "tsc --noEmit",
"test": "vitest run",
"verify": "pnpm typecheck && pnpm test && pnpm build"
There is also no linter and no formatter: no ESLint, no Prettier, no config file for either, no pre-commit hook. That part is deliberate and I have no regrets about it. The missing CI is a different thing, because I did not decide it. It just never happened.
What is actually gating a deploy
Two things, and it is worth being precise about them because they catch different categories of problem.
Vercel runs next build on every push, and next build type checks. That is a real gate, enforced by a machine, and it cannot be skipped or forgotten. A type error does not ship.
Nothing on a server has ever run vitest. Every one of those 7,715 tests is a check that ran on a laptop at some point before a push, which is to say: a check that ran if I remembered. The tests that assert content invariants the typechecker cannot see (that a provider page never claims a thing we must not claim, that a link points where a registry says it should) are exactly the ones a tired human skips, because they pass so reliably that running them feels like a formality.
The failure mode is not what you would expect
The interesting part is what the gap produced in practice, because it was not a broken production.
A commit landed where a type was imported in a component and never exported from the module it came from. pnpm typecheck failed on main. Vercel's build therefore failed too, exactly as designed, so nothing deployed for days. Production kept serving the previous good build, perfectly, while two commits' worth of work sat in main believing itself to be live.
The way that was discovered is the useful bit. One of those commits had deleted a price line from a public page, so the check was: load the page, search for the string that should be gone, find it still there. One byte of evidence, no dashboard, no build log.
That trick generalises, and it is now the first thing I do before writing anything that claims the live site does something:
# which deployment is serving right now
curl -s https://cogniprep.app/ | grep -o 'data-dpl-id="[^"]*"'
# data-dpl-id="dpl_GkMzLhK3oE65bMMF4P7bepdwZiDZ"
# does a header only the newest commit could have produced exist?
curl -s -D - -o /dev/null https://cogniprep.app/ | grep -i content-security-policy
The deployment id travels in the html element on every page and in a ?dpl= parameter on every asset URL, so a diff of that one string tells you whether a deploy actually landed. For "is this change live", pick a byte the change itself altered: a header value, a word on a page, a route that should now answer. Pick a public one. Pages behind authentication answer the same way whether or not the route you are testing even exists, which makes them useless as a probe.
So why no CI, honestly
Not for any of the usual reasons. A 26 second test suite on a private repository would cost a rounding error of the free Actions minutes, the install step is cached, and the workflow would be fifteen lines. The real reason is that the one workflow in that folder exists because losing the database is terrifying, and no equivalent fear ever attached to merging a commit that fails a test I had just run by hand.
What makes that defensible for now, if anything does, is the shape of the gate: a type error cannot reach production, and the next worst thing (a content invariant test failing) produces a wrong sentence on a page rather than an outage. What makes it indefensible is the honest version of the same sentence: the only reason a failing test has never shipped is that I have not yet forgotten on the wrong day.
Three things I would tell anyone in the same position:
- Know which of your gates are machines and which are habits. Write them in two columns. The habits are your real risk surface, and most people cannot name theirs without doing this.
- A build that fails loudly is not the worst outcome; a build that fails and goes unnoticed is. Our failure was invisible precisely because production stayed healthy. Add the probe before you add the pipeline, because the probe takes thirty seconds and catches the silent case.
- Backups first, CI second. If you only ever write one workflow, the one that protects data you cannot regenerate beats the one that protects code you can.
See it
The probes in this post are all public, so you can run them against the live site: the homepage carries the deployment id in its html element, the provider catalogue and pricing page are the pages I check strings on, and the health endpoint answers without authentication and today reports "environment":{"status":"healthy"}, which is the shape it takes when nothing is missing; when something is, it adds a count and never a name. Between the deployment id and that endpoint, you can tell from a terminal whether what you just pushed is the thing answering requests.
Top comments (0)