My Generator Hid a Lint Error: CI Never Checks What Nobody Commits

#bugsmash #devchallenge #typescript #testing

The lint error lived in a file my CI had never seen. It couldn't have seen it — the file only exists after you run a generator with a particular flag, and nobody had ever committed that flag's output.

The setup

NENE2, my framework for typed business apps, ships frontend scaffolding generators: gen:entity builds the typed API layer for a resource, gen:feature builds a state-machine-shaped feature on top. They're template-based and deterministic on purpose — deterministic output is what makes generated code reviewable and lets AI agents use the generators safely.

And like most generators, they have flags. gen:entity <noun> --write adds mutation hooks and write handlers on top of the read-only default. Flags mean branches. Branches mean output shapes that may exist nowhere in your repo.

That's where the bug was.

The bug

The entity mutations template emitted this:

UseMutationResult<Noun, AppError, unknown>

The third type argument, unknown, is exactly the parameter's default — so @typescript-eslint/no-unnecessary-type-arguments flags it. Which means every single output of gen:entity <noun> --write failed lint, in a codebase where CI runs ESLint with --max-warnings 0.

And CI was green the whole time. The generator's determinism tests passed — they verify the templates produce stable, expected output. The committed exemplar for the default archetype passed lint, because it's real committed code and CI lints it like everything else. But the --write branch's output had never been committed anywhere, so no type-checker and no linter had ever run over a single line of it.

A determinism test proves your generator is consistent. It will happily prove it's consistently wrong.

This is a cousin of a broader failure mode I keep running into: gates that glow green while checking nothing in the branch that matters.

How it surfaced

Not by auditing. By building the next feature.

In PR #1580 I was adding a --mutation archetype to gen:feature — a four-state union (idle | submitting | error | success) for form-style features. That archetype consumes what gen:entity --write produces, so for once, verifying the new thing forced me to actually generate the old thing.

The verification was a full rehearsal on real output: run gen:entity payment and gen:feature submit-payment payment --mutation, then put the generated files through the same gauntlet as committed code — type-check, ESLint with --max-warnings 0, Prettier, and the generated transition tests against a real mock server.

Lint went red on code no human had written and no CI had judged. There it was.

The fix

One line in the template. Drop the third type argument — TVariables defaults to unknown anyway, so the emitted types are identical and behavior doesn't change. The PR body records the before/after; the generated output now comes out type-check clean, lint clean at --max-warnings 0, with its four transition tests passing (4/4, exercised against mocked 200 and 500 responses).

The fix took a minute. The bug had unlimited shelf life. That asymmetry is the whole story.

The general lesson: enumerate your generator's branches

If your codebase has a generator with flags or optional archetypes, each branch of its output is code you're shipping to your future self — and CI's default posture toward it is total blindness. Two ways to close the gap:

Keep a committed exemplar per branch. Generate one real instance of every archetype/flag combination and commit it as living code. CI then type-checks, lints, and tests it forever, for free. Drift between templates and exemplar shows up as a normal red build.
Add a generate-and-check smoke to CI. Generate every branch into a temp dir, run type-check + lint + tests over the output, throw it away. No repo noise; costs CI minutes instead.

Either works. What doesn't work is what I had: determinism tests that hold the generator's output stable while nothing ever asks whether that output is valid.

Full honesty about where I actually am: PR #1580 did option 2 manually, once — a rehearsal, not a gate. The PR body itself flags a committed exemplar for the mutation branch as a follow-up. So the branch that bit me is verified today and unguarded tomorrow, and I'm writing this partly so I can't quietly forget that.

Takeaways

Every generator flag is a code branch CI has never met. Green CI says nothing about output nobody commits.
Determinism/golden tests check stability, not validity — pair them with a committed exemplar per branch, or a generate → type-check → lint → test smoke.
The cheapest audit is a rehearsal: generate each archetype for real and run your normal check over the output. My first rehearsal of an old flag found a lint error of unknown age.

Primary sources

NENE2 PR #1580 (merge c2b5311) — the --mutation archetype, the rehearsal, and the one-line template fix

If you have a generator in your codebase: when did the output of every one of its flags last pass your linter? I'd honestly love to know if anyone gates this properly.

── Hideyuki Mori (Ayane International) 🔗 hideyuki-mori.com