There's a specific kind of rot that sets into a team, and it starts with a test suite that fails for no reason. Not a real failure — a flaky one. It goes red, someone reruns it, it goes green, and everyone moves on. It happens often enough that a habit forms: when the pipeline fails, don't investigate, just hit retry. That habit is quietly one of the most dangerous things that can happen to a delivery process.
Because the entire point of an automated pipeline is trust. It exists so that when it's green, you believe your change is safe, and when it's red, you stop and look. A flaky pipeline destroys both halves of that. People stop believing green means safe, because it's random anyway. And they stop respecting red, because red usually means "run it again," not "something is broken." You've trained your team, through daily repetition, to ignore the one signal that's supposed to catch their mistakes.
The failure I see most often is treating flakiness as a minor annoyance rather than a real defect. A test that fails intermittently gets a shrug and a rerun, month after month, because chasing it down is tedious and there's always something more urgent. But every one of those reruns is teaching everyone that the pipeline's word doesn't mean much. Eventually a genuine failure shows up, gets the reflexive retry, passes on some flaky rerun, and a real bug rides straight through to production on the habit you built.
I've learned to treat a flaky test as a serious bug, because it is one, even if the code it's testing is fine. Either fix it so it's reliable, or remove it, but don't leave it there poisoning everyone's trust. A test you can't believe is worse than no test at all, because no test is honest about giving you nothing, while a flaky test lies to you at random.
A pipeline is only valuable if people trust it, and trust is spent one false alarm at a time. Protect that trust ferociously. The moment "just rerun it" becomes the normal response to a failure, your automation has stopped protecting you and started training you to look away.
– Serguey Shinder
Top comments (0)