CI Paid for Itself on the First Run: 4 Repos, Real Bugs

#cicd #testing #ai #discuss

I added CI to four small apps that had been living without it. The first run went red before it executed a single test — and not because my code was broken. Because a file I had never actually tested was.

Here's what a first CI run catches that your machine has been quietly hiding.

The pipeline was nothing special

Same shape for all four: fresh checkout, cp .env.example .env, install dependencies, run the checks, run a dependency audit. The kind of thing you'd copy off a template in ten minutes.

The value wasn't the sophistication. It was the word fresh. My laptop has a real .env, an installed node_modules, a warm cache, and years of accumulated state. CI has none of that. It starts from what's actually committed — which turns out to be a very different thing.

Bug type 1: `.env.example` broke a clean boot

Two of the four repos couldn't boot from their own example environment file.

The values had unquoted spaces — think APP_NAME=My App — and the dotenv parser rejects that outright. On my machine it never mattered, because I have a real .env from months ago and never re-copy the example. CI copies the example on every run, so the first thing it did was fail to parse it, which failed to boot the app, which took a chunk of the test suite down with it before any real test ran.

That's the whole "works on my machine" trap in one line: the example env is a file you ship to every new contributor and every deploy, and it's the one file your own environment guarantees you'll never exercise.

Bug type 2: a required variable was in the example, but blank

Related, in one of those same repos: a required encryption key was listed in .env.example but deliberately left blank — it's a secret you don't commit — so a fresh checkout had no usable value for it. The app booted fine for me — my environment had a real key — and fell over in the clean room, where there was nothing to fill it in with.

CI made the missing value explicit the only way that ever really happens: by being a machine that starts from nothing.

Bug type 3: the audit gate caught real, known vulnerabilities

The pipelines ran a dependency audit and failed on high-severity findings. This wasn't uniform, and I want to be precise — including about the limits of what I can still reproduce:

One repo's audit flagged a high in its build-tool chain — a vite advisory of the kind npm audit fix clears. This is the one I can point you straight at: it's documented in that repo's CI-setup PR.
My notes from the same week flagged advisories in a second repo's test-runner / vite chain, but the pinned lockfile I can re-audit today comes back clean. So I'll only claim what the PR trail still shows, and leave that one as "flagged then, not reproducible now."
The other repos turned up moderate/low at most.

So the honest version is smaller than "every repo was hiding a critical": at least one repo had a real high-severity advisory that the first gated npm audit surfaced — sitting there silently because nobody had ever run npm audit in a gate that could fail.

The pattern, not the scoreboard

None of these were logic bugs. Every one lived in the gap between "runs on my machine" and "runs from a clean checkout": a broken example env, an undocumented required variable, a stale dependency nobody re-audited.

My machine had exactly the state needed to hide all three.

I'll be honest about the counting, too. I'm not going to hand you a tidy "13 products now have CI" number — the exact figure depends on what you call a product, and it was a self-tally at the end of that week, not a measurement. What I can stand behind is smaller and more useful: four repos that had no CI, one afternoon of adding the most boring possible pipeline, and real defects on the first run of each.

That's the argument for CI I actually believe. Not "it catches regressions someday" — that's the slow, abstract payoff. The immediate one is that CI is the only environment that starts from nothing, and starting from nothing is where your .env.example, your undocumented variables, and your un-audited dependencies all come due at once.

The four CI-setup PRs, if you want the workflow files and the audit output: nene-field, nene-contact, nene-deal, nene-vault.

It's the same bet I made on my documentation, for the same reason: 261 docs in 6 languages with one maintainer, where the index is generated and CI goes red when it drifts. Different problem, identical logic — a machine that checks every single time doesn't have a bad week, and I do.

If you've got a repo running without CI right now, the cheapest pipeline you can write will probably pay for itself on run #1.

What did your first CI run catch that your machine had been hiding?

── Hideyuki Mori (Ayane International) 🔗 hideyuki-mori.com