The flaky test we retried instead of fixing

Somebody added retries: 2 to our test runner config on a Thursday afternoon and the pipeline went green. It stayed green for fourteen months. In that time we shipped a data-corruption bug that a flaky test had been failing on, intermittently, the entire time. It wasn't flaky. It was correct about a race condition that only lost about one time in nine.

Retries are seductive because they work immediately. A test fails, you re-run, it passes, you merge, and the day continues. Nobody signs off on hiding failures, but the retry flag is framed as build hygiene rather than as a decision to ignore a signal, and once it's in the config nobody re-reads it. Ours applied globally, to every test, including the integration suite where the genuinely interesting failures live.

What broke the spell was measuring it. We started recording every individual attempt rather than just the final verdict, and shipped a weekly report of tests that passed only on retry. There were sixty-one. Eleven of them were real bugs: two race conditions in application code, a connection pool that leaked under load, and a date handling path that failed near midnight UTC. The rest were genuine test defects, mostly shared mutable fixtures and tests that assumed ordering the framework never promised.

We didn't remove retries in one go, because with sixty-one flaky tests the pipeline would have been unmergeable by Tuesday. Instead we quarantined. A test that fails intermittently gets moved to a separate suite that runs and reports but doesn't block, and it comes with an owner and a deadline. The main suite runs with zero retries, and a failure there is a stop. Quarantine has a hard cap, so the list can't grow forever without somebody noticing.

The thing I'd say to anyone reaching for a retry flag: a flaky test is a test telling you the system is nondeterministic, and it is almost never wrong about that. The nondeterminism is either in the test or in your product, and you cannot tell which one without looking. Retrying skips the looking.

Green is only worth something if red would have meant something.

– Sergey Shinder