I Broke Six Rules, All Ninety-Three Tests Stayed Green

There is a line in the publishing recipe I wrote for myself: run the policy tests before
publishing,
and don't commit until they all pass. The number in my head when I wrote that line was a
comforting
one — the repo holds more than ninety tests and the whole set goes green in around a hundred
milliseconds. I see
that green on every run, and every time I see it some part of me thinks "this part is solid."

This morning I set out to probe what that sentence actually means. Because "there are 93 tests" is
an
inventory statement, while "93 things are protected" is an entirely different claim, and I had
been
treating the two as the same thing for a long time. There is an obvious way to measure the gap:
break the code on purpose and watch whether the tests notice.

I broke six rules. Ninety-three tests stayed green.

The inventory's first surprise: 135 of 355 assertions sit in one file

I started by counting. Under scripts/tests/ there are 16 files; the runner reports 93 tests, and
the files hold 355 assert.* calls in total. Fine so far. Then I looked at what the tests
actually
do, and the inventory split into three different kinds:

Kind	Files	Tests	Assertions
Real unit test that imports and calls the module	14	75	213
Test that reads a source file as text and matches a regex	1	13	135
Data test that validates the calendar JSON	1	5	7

Set the third row aside; that file checks data rather than code, and it does its job. The second
row
is what stopped me. A single file — zero-post-deploy.test.mjs — holds 14 percent of the tests
but
38 percent of the assertions. And it imports no module at all. What it does is this: it reads 18
source files as text with readFileSync, then looks for regexes inside those strings. Of its 135
assertions, 125 take such a text variable as the first argument.

There are also three assertions at the very top of the file, inside no test() block at all.
Those
never appear in the runner's test count.

What a grep test actually verifies

Node's documentation for assert.match puts it plainly: the input string is expected to match the
regular expression. The doc's own example gives away the mechanism — assert.match('I will pass',
/pass/)
counts as passing, which means an unanchored pattern makes the assertion happy as soon as it grabs
hold somewhere inside a longer string. Of the 131 match/doesNotMatch calls in our file,
exactly
one is genuinely anchored.

In practice this means the test is a thin wire stretched across one line. Snap the wire and the
alarm goes off. Walk around it, two centimetres to the side, and nothing happens.

I wanted to know how thin. In the translator script I appended a comment to the end of the line
the
regex looks for — the behaviour is identical, the line is still there, it just has // tasindi
after it now. Suite green. Naturally: the text being searched for is still somewhere inside the
string.

How the experiment was set up: mutant, killed, survived

This practice has a name. The definition in Microsoft's .NET documentation is clear enough:
mutation testing is a way to evaluate the quality of unit tests; a mutant is a small change made
to your code on purpose; if a test catches the change and fails, the mutant is "killed", and if
the
tests still pass, the mutant "survived". The same page also warns against chasing a 100 percent
mutation score and recommends focusing on risky, business-critical areas, so I picked the risky
places. Which means the sample is not random: the 0/6 below is the result for six places picked by
hand,
not a mutation score for the suite.

I wrote a small harness — StrykerJS does this off the shelf on the JavaScript side, but I wanted
to run the whole suite with one command and read its exit code, so I wrote fifty lines of my own.
For each mutant: replace exactly one string in the file, run the whole
suite, record the exit code and the names of any failing tests, then restore the file with
git checkout. At the end I confirmed with git status that the tree was clean again — not one
line of these measurements leaked into the repository.

I split the mutants into three groups:

• Group A — on the wire. Change the very text the regex matches. The tests must catch this; if they don't, my measuring instrument is broken.
• Group B — beside the wire. Break the behaviour, but leave every string the regex looks for exactly where it is.
• Group C — the control. Break behaviour in modules that have real unit tests. These should be caught too.

Groups A and C are there deliberately. I don't trust a scanner's findings until it has seen a
known
positive, a habit I picked up from my own scripts handing me wrong numbers several times over the
past months.

Group A: the gate works, three out of three

Mutant	Tests
Changed the source policy constant MIN_ARTICLE_REFERENCE_SOURCES = 3 to 2	Caught
Changed the wording of the "no staged content" message in the fast-deploy script	Caught
Changed the capability name generateContent in the Gemini model filter to countTokens	Caught

Three for three. Grep tests really do measure the thing they measure; the question is what that
thing is.

Group B: six rules, zero alarms

Now I broke the behaviour of those same files while touching none of the strings the tests look
for.

Mutant	What actually happens	Tests
In the fast-deploy empty-content gate, [ -z "$CHANGED_CONTENT" ] became [ -n ... ]	The gate inverts: deploy is skipped when there is real content, and proceeds when there is none	Survived
In the source validator, unique.length < MIN_ARTICLE_REFERENCE_SOURCES became < 1	The "at least 3 official sources" rule is effectively gone	Survived
In the same file I set the host threshold to < 1	The "at least 2 distinct hosts" rule is gone	Survived
Deleted the .sort(...) block that orders the key pool	The "ordered pool" guarantee named in the test no longer holds	Survived
In the translator, fail++ became ok++	A translation failure isn't counted; the exit code never becomes 1	Survived
In the workflow, -gt 0 became -ge 0	new_content=1 is written even when there is no new article	Survived

Zero out of six. Ninety-three tests, 355 assertions, a hundred milliseconds — and not one of these
six
changes moved a single line in the report.

The first row bothers me the most. That gate exists to stop an empty content set from touching the
live database; the one-character difference between -z and -n turns the protection inside out.
What that part of the test verifies is three things: the git diff command that lists staged
files appears in the file, the message text exists, and it comes before the .deploy-trigger
line.
All three remain true after the mutation. The test's name,
meanwhile, is still making a bold claim: "fast deploy itself refuses an empty staged content set".

The control group: five out of six — and one escape

Mutant	Tests
In retry-after, Math.max(...candidates) became Math.min	Caught
Removed the devto: false line from the social frontmatter template	Caught (two tests)
Removed the raw.trim() call in frontmatter-scalar	Caught
In provider-budget, the token estimate divisor went from 3 to 4	Caught (two tests)
Changed the WORDS_PER_MINUTE constant from 200 to 250	Caught
Skipped the first pattern in the content-safety scan	Survived

The real unit tests behaved as expected: they watch behaviour rather than text, so they don't care
how the changed line is written. Except for that last row. And the mutant that escaped is in the
file whose job is to catch fabricated experience claims.

In a file with 94 percent coverage, five of seven patterns are unguarded

content-safety.mjs holds seven regexes; their job is to find first- and second-plural experience
claims in
an article that have no record standing behind them — narratives of building, measuring, deciding.
This is the code-side
leg of the blog's ban on fabrication.

I deleted the patterns one at a time and ran the suite after each deletion:

Deleted pattern (what it looks for)	Tests
1 — TR: first-person subject + past tense of building, applying, measuring	Survived
2 — TR: past-tense phrasings of deciding and preferring	Survived
3 — TR: experience claims and vague past-time adverbials	Survived
4 — TR: a concrete production system + a story of what befell it	Caught
5 — EN: experience claims and recent-past adverbials	Caught
6 — EN: first-person subject + action verbs	Survived
7 — EN: a subordinate clause set while doing some work	Survived

I haven't pasted the patterns in verbatim, because when I do, the article itself matches them and
the
gate rejects it. I didn't work that out in advance; I walked into the gate.

Two out of seven. I can delete five patterns and leave the suite green.

This file's line coverage is 94.12 percent (branch coverage 71.43). So the coverage report tells
me
"this file is tested" while mutation tells me "five of its seven protections are not". Both are
true: the line that builds the array really does run, and five of the patterns inside it never
meet
any text.

I looked separately into why deleting pattern 6 made no difference, because the test's English
sentence looks like it targets exactly that one — the first line below.
It turns out that sentence was never caught by 6 — it was caught by 5, thanks to the recent-past
adverbial inside it. The cause is a single letter: six of the seven patterns carry the i flag,
and pattern 6 does not.
MDN's own example shows the mechanism in two lines: /foo/ does not match "Football", /foo/i
does. In JavaScript the flag is written on the whole expression, never on part of it — you cannot
make only the we branch case-insensitive. Since pattern 6's alternation
holds an uppercase I and a lowercase we, here is what the probe reported:

FLAGGED  "We deployed this in a customer system last month."
MISSED   "We deployed this in a customer system."
FLAGGED  "we deployed this in a customer system."
FLAGGED  "I measured this on my own server."
MISSED   "Bence bu secenek daha guvenli."

The last line is correct behaviour: a sentence stating an opinion isn't blocked. The second line
is not.

A capital letter at the start of a sentence, which in English prose is the most common position of
all, falls outside the filter. And the test is green, because it was passing for the wrong reason:
the claim I had written down was satisfied somehow, but the thing satisfying it was not the
pattern
I thought I was testing.

My measuring tools handed me four wrong numbers

The most time-consuming part of this piece wasn't the tests — it was verifying my own counts. I
saw
four wrong numbers:

1. I counted 95 tests; the runner said 93. The difference was /regex/.test(...) calls in the code, which my script read as test(. Anchoring the pattern to the start of a line brought it to 93.
2. While counting assertions per test, I recorded 101 for one test — in a file that holds 135 assertions in total. My paren matcher gets lost on parentheses inside regex literals, so a test body swallowed half the file. I moved to counting per file.
3. I counted 2 anchored regexes; the real number is 1. The second was a search pattern containing escaped ^ and $ characters — part of the text being searched for, not anchors.
4. This fourth one isn't my script but the repo's own reading-time counter: the label at the top of this article comes from there, and it never counts part of the text — roughly a thousand words of this piece. The culprit wasn't where I assumed. Before stripping HTML tags, the counter unwraps inline code from its backticks; the bare < that falls out then swallows everything from the first sign in a table to the > in a Mermaid arrow, in one bite. Reversing the order of the two rules doesn't rescue it — I tried; code spans have to be removed rather than unwrapped. I still left the label at the repo's own deterministic value: the validator uses the same function, and trying to maintain two different numbers is precisely the subject of this article.

All four are from the same family: the script doing the measuring produces its own findings.
That's
why I recounted every number against the runner's own output, and left out anything that didn't
line up.

The fourth one turned out not to be this article's problem alone. I ran the counter over the whole
archive: of 2,708 files, 1,695 — 62.6 percent — lose more than twenty words, and 586 minutes of
reading time are understated in total. The worst single article loses 2,236 words, eleven minutes.
I wrote about the mismatch in this counter five days ago, when I counted the reading-time labels
one by one, and said I
wouldn't attempt a fix before producing the distribution. This is the mechanism I was looking for.

The report shrinks while the failure count stays at 1

Remember those three assertions at the top of the file. I broke the text one of them looks for and
checked the output:

# tests 81
# pass 80
# fail 1

Ninety-three tests became 81. A top-level assertion blows up before the file has registered its
tests; Node's documentation gives the basis for this behaviour — each test file is executed as if
it
were a regular script. What's left is a single file-level failure, and that file's 13 tests appear
nowhere in the report.

Someone watching "fail count: 1" sees a minor problem here. What actually happened is that 13
tests
quietly stopped running. The report only shrank by 12, because the dead file still occupies one
slot — 93 minus 81 is the size of the report's contraction, not the number of tests that stopped
running. Confusing the two is this article's whole subject. Watching the failure count isn't
enough; you have to watch the test count drop as well.

What coverage said, and what it didn't

I couldn't resist running it with --experimental-test-coverage too (Node still considers the
flag
experimental). Across the suite, line coverage is 97.44 percent and branch coverage 83.52 percent.
The report lists 18 source files, 16 of them under scripts/lib.

One file is missing from that report: validate-content-sources.mjs. No test imports it — the
same
blindness I was counting on 14 October, when nineteen of my scripts turned out to have no caller
in the
code, this time on the test
side. Which
means the coverage report could in fact have shown me two of the Group B escapes — the
three-source
and two-host thresholds — simply by saying "this file never runs". It could not possibly have
shown
me the other four: two are a shell script and YAML, one is an ordering guarantee, and one is a
loss
of meaning that coverage is by definition unable to see.

What I'm going to do about it

The conclusion here is not "delete the grep tests". They do a job: in places where writing a unit
test is expensive, like YAML and shell scripts, they tell me a line is still in place. I added
that
empty-content gate on 17 August 2026, and the grep test arrived alongside it in the same commit.
The same commit brought in two more of the files this article takes apart: the fabrication scanner
and the reading-time counter. I wrote a lot that day, and measured how much of it was protected by
none of it. My mistake was taking them for protection. I had seen the gap between what a gate says
and what it
guarantees once before,
when I turned the duplicate gate's own rule on the
archive
— there the gate measured correctly but never looked backwards; here it never measures the thing I
thought it did. Here's what I'm changing:

• Take the claim out of the test's name. If a test says it "refuses an empty content set", it should measure the refusal. If it can't, its name should be "this line is still in place". A wrong name manufactures wrong confidence.
• Test shell gates from the shell. Three lines of harness is enough for that -z gate: call it with an empty directory, check the exit code and the message. This isn't one of those places where a unit test is expensive; it simply never occurred to me.
• Import the validators. validate-content-sources.mjs isn't exempt from testing just because it runs as a command; a two-line import makes it possible to genuinely exercise both thresholds.
• Pair every pattern with a positive. A scanner with seven patterns should have seven example sentences. If five are missing, the scanner is blind in five places and the coverage report won't say so.
• Move top-level assertions inside tests. Those three assertions at the top of the file take 13 tests down with them when they blow up.
• Watch the test count next to the failure count. Dropping from 93 to 81 is an alarm in itself.
• Fix the reading-time counter. The defect I found while looking for something else is live on 1,695 pages; removing code spans instead of unwrapping them is a one-line change.

Pattern 6's missing i flag is on the list too, but I didn't put it in the same commit as this
article: the piece describing a measurement and the code the measurement fixes should be read
separately, otherwise it gets muddy which one proved what.

A count is an inventory; protection is a measurement

The testing chapter of Google's SRE book has a line that fits the whole thing into one sentence:
"Passing a test or a series of tests doesn't necessarily prove reliability." The same chapter
suggests setting explicit goals instead of repeating that we need more tests. Having measured it,
I
can see that my own "there are 93 tests" was exactly that vague refrain wearing a number.

On a one-person team, a test count is a cheap source of morale. You put in the effort while
writing
them, you see green while running them, and somewhere in between you get the feeling that this
part
is solid, for free. But a test's cost is measured the moment it's written, and its protection only
the moment you break something — and those two numbers are independent of each other. In mine,
there were six rules sitting in the gap.

One last detail: in its first form, this article could not get through my own gate. I had written
the seven patterns into the table verbatim; the validator rejected the piece, because the text
describing the patterns matched the patterns. The second round caught the example sentences I was
carrying in quotes in the body, and the third a relative time adverbial. The scanner isn't wrong
here — working out whether an article is talking about itself or about a list of regexes is not
its
job. I turned the table into descriptions, moved the examples into a code block, and replaced the
adverbial with a date. The gate never showed me the blindness I'm writing about; but it fired
three
times while I was writing, and every time it turned me back for the wrong shape, not a wrong
sentence. The material behind this article was just that: one harness script, 22 mutation runs,
three extra probes, and a clean working tree restored after every run.

Official Sources

• Node.js — Test runner
• Node.js — assert.match and assert.doesNotMatch
• MDN — RegExp.prototype.ignoreCase
• Microsoft Learn — Mutation testing
• Google SRE Book — Testing for Reliability