Our pipeline was green for three weeks straight, and in that window we shipped a bug that corrupted a fraction of user records every night. The tests passed. The deploy succeeded. The dashboard was a wall of checkmarks. And it was all a lie, because green had stopped meaning "this works" and started meaning "nothing we happened to check went wrong."
I went digging into what our tests actually asserted. A depressing amount of them were tautologies — mock a thing, call it, assert the mock was called. Whole suites that exercised code paths without ever verifying an outcome a user would notice. The nightly job that mangled records had a test, too. It asserted the job ran without throwing. It never once checked what the job wrote.
That's the trap of trusting green: a passing pipeline only tells you the assertions you wrote didn't fail. It says nothing about the assertions you forgot. And the assertions people forget are always the ones about the boring, load-bearing behavior nobody thinks to doubt.
So I changed how we judge a suite. Not "how many tests" or "what's the coverage percent," but "if I break this behavior on purpose, does something turn red?" I started deliberately introducing bugs — flipping a comparison, dropping a write, returning an empty list — and watching whether the pipeline caught it. Half the time it didn't. Every silent pass was a hole, and we wrote the test that should have been there.
Coverage numbers lie because they measure lines executed, not behavior verified. You can run every line and assert nothing. Mutation-style thinking — break it and see if the tests notice — is the only honesty check I trust now.
The pipeline isn't there to be green. It's there to stop bad releases. If those two goals ever drift apart, green becomes the most dangerous signal you have, because it buys confidence you didn't earn. A red build annoys you for an hour. A falsely green one ships for three weeks.
Ask your suite the only question that matters: what could break in production and still leave you green? Then go close those gaps before they close them for you.
– Sergey Shinder
Top comments (0)