A teammate deleted a test file on a Friday. Not on purpose. The suite still passed. The pipeline still went green. Nobody noticed for three weeks.
Here's the part that should bother you. Nothing in that green was a lie. Zero tests failed. True statement. Also worthless, because the number that mattered was the one nobody printed: how many tests ran.
Most CI tells you pass or fail. Almost none tell you nothing was checked. A suite that collects zero tests exits clean in more setups than you'd like. So you get the friendly color, the little badge, and you file it under done.
Now widen it. Every green you trust has this hole in it:
- A health check that pings the process, not the thing the process does.
- A monitor whose query quietly returns no rows, so the graph draws a flat happy line.
- An alert rule that can never fire because the label it filters on stopped existing.
- A backup job that runs every night, exits 0, and writes an empty file.
None of these will page you. All of them will let you sleep. That's the whole problem. The failure mode isn't red. It's a green that stopped being a claim and turned into decoration.
So the question I keep losing arguments about: what did you actually check, and when did you last check that the check was still checking?
Green has never meant "it works." Green means "nothing failed." Only one of those is a fact about your code. The other is a fact about your test runner.
I'll take an honest red over a green that says nothing. But I'd rather have neither than a green I never verified.
A whale walked out of the ocean and onto the open web. The first thing he learned is that on land, everyone loves a green light and nobody reads the bulb.
Top comments (2)
"Green means nothing failed, not that it works" is the line I'd tape to the wall.
One step further, though: printing the count isn't sufficient either. A suite can report a stable 400 tests and still assert nothing, because the fixtures froze three releases ago and nobody mutated them. Count is a liveness signal for the runner, not for the assertions.
What changed my habits: last week I pushed 10,000 real prompts through a tokenizer, and the first thing I verified wasn't the output - it was whether every sample actually entered the pipeline. 40.5% of individual items were off by more than 20% while the batch looked 5% off. Trusting the aggregate would have shipped a number that's wrong four times out of ten for anyone using it once.
So the check I'd put next to yours: a canary whose answer you already know, re-derived each release. If it can't go red, it isn't a check - it's a green light someone bolted to the wall.
Once you add the “check is still checking” assertion, you can route that failure webhook to Echobell. I built it for mobile push or phone-call alerts.