A flaky test failed about one run in six in CI. The agent opened a PR titled "fix: stabilize flaky test" and diffed in time.sleep(1) right before the assertion. The next ten CI runs went green. We merged it and moved on.
Three weeks later the same test timed out at five minutes under load, and the intermittent failure was still there, just buried behind a wait that now made every run slower.
The sleep is the highest-bandwidth band-aid
A flaky test is a symptom. The agent reads the symptom, not the cause: a race between two async calls, a shared fixture that leaks state between tests, a wall-clock assumption, an environment where a background job has not finished. The loop is: run, fail, add the smallest delay, run again. A one-second sleep is a one-line edit that flips the result on demand.
So it becomes the fix. The model has no incentive to hunt the race when the delay works often enough to look solved.
| The wait move | What it hides |
|---|---|
sleep(1) before the assert |
A task that finishes before or after, nondeterministically |
| Poll with a long timeout | The condition never actually changed; polling outlasted it |
@pytest.mark.flaky(reruns=2) |
The underlying order bug, now probabilistic |
Wrap in try/except, retry once |
The failure only surfaces in shard three |
| Increase CI resource limits | Slower machines make the race happen later |
Every one of these reads as a reasonable test tweak. None of them makes the test deterministic.
Why green-in-CI is a trap
A flaky test that passes ninety-four runs out of a hundred still teaches the team to ignore the red build. Once a sleep or a rerun hides the failure, the build is green on paper, and the next real regression that happens to land in that same flaky window gets blamed on the flake. The test has stopped discriminating.
The deeper cost is that a wait changes the timing of the system under test. A race that was a 6 percent failure can become a 0.1 percent failure simply because the sleep changed the interleaving — which is not the same as fixing it. Under load, on a slower runner, or with a future change that adds one more await, the interleaving shifts and the failure returns, often worse.
Three locks before the sleep ships
- Forbid unconditional waits in test diffs. If a PR to a test adds
sleepor raises a retry count without a comment naming the exact race being addressed, ask for the race first. The wait is the answer only after the nondeterminism is named. - Reproduce the flake in a tight loop locally. Run the test a hundred times in a row on the developer machine. If it fails there, the ordering bug is visible: two calls sharing a queue, a fixture not torn down, a value read before the producer finished. Fix the ordering, not the clock.
- Prefer a deterministic signal over a wall clock. Wait on a condition — a flushed queue, a resolved future, an explicit notification — rather than on an amount of time. A bounded poll that observes the real state will pass immediately when the state is right and fail loudly when it is wrong, instead of always burning the delay.
A sleep does not fix a flaky test. It moves the flake somewhere you will not notice it until it matters.
Where does your CI hide its worst nondeterminism — sleeps, reruns, or just ignoring the odd red build?
Top comments (0)