There was a test suite at one company that had become a kind of weather. It failed often, for reasons unrelated to whatever you had changed, and everyone knew it. The unwritten rule, passed along to every new hire within their first week, was that a red build did not necessarily mean your code was broken. You checked whether the failure looked like your change, and if it did not, you re-ran the pipeline, and usually it went green. We had a comprehensive automated safety net, and we had collectively agreed to stop believing it.
That agreement was far more dangerous than having no tests at all, because it corroded the one thing tests are supposed to provide, which is a trustworthy signal. When failing is normal, failing stops meaning anything. The suite still ran, still burned compute, still turned red and green, but it had lost the ability to tell us anything we would act on. And on the day a test failed for a real, serious reason, it looked exactly like the hundred flaky failures before it, so it got the same treatment. Someone re-ran the build, it happened to pass on a retry, and a genuine bug walked straight into production wearing the disguise of routine noise.
Flaky tests are not a minor annoyance to be worked around. They are an active liability, because they train your team, methodically and daily, to ignore the alarm. Every time someone re-runs a build without investigating a failure, that habit gets a little stronger, until ignoring red is not a shortcut but the culture. You cannot lecture your way out of this. People are responding rationally to a signal that has lied to them too many times.
So I have come to treat a flaky test as a broken test, full stop, deserving the same urgency as a test that catches a real defect. Either it earns its place by being reliable, or it comes out of the suite entirely, because an unreliable test is worse than a missing one. A missing test is an honest gap. A flaky one is a false witness, and a safety net you have taught yourself to distrust is not protecting you.
A test you re-run instead of reading is not testing anything anymore. It is just theater with a pass rate.
– Serguey Shinder
Top comments (0)