My regression test passed for 3 weeks while the code it tested was deleted

Ryulhwan Kim — Thu, 10 Sep 2026 11:13:11 +0000

I had a check guarding a bug fix. It was one line:

text.includes("trySendWithRetry(")

It was matching the import statement.

The actual call had been removed during a refactor weeks earlier. The check reported green the entire time, while user messages were being dropped whenever the network blipped once.

What bothers me is not that I wrote a sloppy check. It is that there was no way to notice.

A check that cannot fail and a check that passes produce byte-identical output. There is no signal to look for. And the better your enforcement layer gets, the worse this becomes — you stop reading the green and start trusting it.

Why I was writing these checks at all

Not new bugs. The same ones, weeks apart, in different screens of the same app.

The cause was almost never a bad fix. It was that the same five lines had been copy-pasted into eight places and only one got patched. Or a later cleanup removed a guard clause whose reason nobody had written down.

Once I started using a coding agent daily it got noticeably worse, because the agent has no memory of why a line is there. It removes it while simplifying, every test still passes, and the bug you paid for three weeks ago comes back.

So each fix became an executable rule that carries the incident with it:

why: "2026-04-02: the retry loop had no idempotency key, so the gateway treated each
      attempt as a new charge. 38 customers double-charged, ~$4,100 refunded."

Not "prevents silent failures." The date and the cost.

That difference matters more than it looks. An agent — or a tired human — deciding whether to delete a line has nothing to weigh "this is important" against. It has something to weigh $4,100 against.

The part that fixes the real problem

Every rule declares mutations that should break it. One command applies each one, asserts the rule fails, and restores the file.

$ npm run demo:verify
  ✓ key removed entirely
  ✓ key declared but not passed
  ✓ key made random
  caught 3/3 deliberate breaks

And here is the same rule written the obvious way — src.includes("idempotencyKey"), which is what I wrote first:

$ npm run demo:leak
  ✗ LEAK — the code was broken and the rule still passed
  caught 0/1 deliberate breaks

Both of those pass a normal run. Only one of them is real. That is the entire point.

Three checks that leaked on me

These are not hypothetical. Each one looked correct in review and each one was worthless.

1. The file-level check

if (runs(text, rawApi) && !runs(text, wrapper)) flag(file)

Reads fine: "this file calls the raw API and doesn't use the wrapper." But a file that calls the wrapper in one place and the raw API in another satisfies both conditions and passes — which is exactly what a half-applied fix looks like, and the single most likely shape of the bug. It has to be line by line.

2. The comment match

A rule checked /PREVIOUS/i.test(source) to confirm a previous-secret fallback still existed. The file's own header comment said "accepting the previous secret for one overlap window…".

So deleting the actual fallback code changed nothing. The rule stayed green as long as the comment survived. Scan only lines that execute.

3. Evidence of a bound is not a bound

A rule looked for the identifier YEARS as proof that a year range was limited. Widening the range to a hardcoded list left YEARS in place, so the rule found its own evidence and passed.

Mentioning a bound is not having one. Check the values, not the vocabulary.

And one that only broke for other people

On a CRLF checkout, any mutation whose anchor spans a newline matched nothing. So verify reported "mutation changed nothing" instead of testing the rule, and the headline demo caught 0/0 for anyone who cloned the repo on Windows.

Everything passed on my machine. I found it only by cloning fresh into a clean directory.

That is precisely the class of bug the tool exists to catch, and it shipped anyway. Make of that what you will — I mostly take it as evidence that "it works here" is not a test.

Try it

git clone https://github.com/igotojapan123-web/regression-guard
cd regression-guard
npm run demo:verify   # careful rule: caught 3/3
npm run demo:leak     # naive rule: exposed as fake

MIT, no dependencies, Node 18+. No install step, no signup, two npm scripts.

Where this does and does not belong

If what you need is "ban this API", ESLint's no-restricted-syntax is better and you should reach for it first. Semgrep and CodeQL are vastly stronger pattern engines than this 400-line thing.

The only thing I could not find in any of them is a first-class way to prove a rule fails when it should. If that exists somewhere, please tell me — I would genuinely rather use it than maintain this.

A guard rule is also not a lint rule. A linter says "this is bad style." A guard says "this specific past fix is still present, and here is what it cost when it wasn't."

The thing I am least comfortable with: the mutations are string-based, so a rule's breaks rot when the code around them is rewritten. verify warns about that instead of failing, and I suspect that is the wrong default.

Curious whether anyone has solved the "prove the check can fail" problem in a cleaner way. Happy to be told the whole premise is wrong.