The CI log shows a timeout. You send the failure to your agent. Two minutes later, it opens a PR: await new Promise(r => setTimeout(r, 500)) before the assertion. The test passes locally. You merge it.
Next week it flakes again, this time on a different machine, at a different hour. The sleep was 500ms. The race still takes 600ms on a loaded runner. Nobody deleted the sleep. Nobody remembered it was there.
The symptom, not the cause
A sleep() does one thing: it narrows the window in which the race can win. It doesn't close the window. On your laptop, the window might be 10ms wide. On a shared CI runner under load, it stretches. The agent picked a number that worked once and moved on.
This is the same shape as editing an assertion to match broken code. The test goes from red to green without the underlying problem moving. The difference is that a changed assertion usually gets caught in review. A sleep() looks like engineering.
What the fix should actually be
If the test depends on an async side effect — a database write, a dispatched event, a resolved promise — wait for that thing to be observable, not for a clock.
In Node, that means polling a predicate or using an event emitter, not setTimeout. In Python, it's await asyncio.wait_for(condition), not time.sleep. The rule is boring: wait on a signal, not a duration. If you can't observe the signal, that's a test design problem — the code doesn't expose what the test needs to know.
I've started asking agents to explain what state the assertion depends on before writing any fix. If they can't name it, they're going to reach for a sleep. When they can name it, the fix usually looks like a waitFor(() => expect(...)) or a callback hook, and the test stops flaking on every machine.
This happens in API workflows too
The same failure mode shows up when an agent is told to test an API endpoint that kicks off an async job. It sends the POST, then sleeps two seconds, then checks the GET. On a fast instance, the job is done in 200ms. On a cold start, it takes five seconds. The test passes sometimes.
The fix is the same: poll until the resource reaches the expected state, or use the webhook/callback the API already provides. A contract that says "the job completes asynchronously" should tell you what to wait for — a status field, a Location header, a retry-after. If the spec doesn't say, that's a spec gap, not a reason to guess a timeout.
We ran into this building Powerduck, where the spec is the source of truth for what an endpoint returns and when. An agent that has the spec in front of it can look up the async pattern instead of inventing a sleep. It's still the developer's job to reject the timeout fix, but at least the alternative is on the page.
The review trick
Next time you see sleep, wait, delay, or setTimeout in a test diff, ask one question: what is this waiting for? If the answer is "enough time," reject it. If the answer is a specific state or event, it's probably right.
How do you catch the sleep() fix in your codebase?
Top comments (1)
The "name the state the assertion depends on" prompt is a good one. We see the same failure one layer up, in agents that drive a real UI: the agent clicks, the page hasn't finished rendering, so it adds a wait of 2 seconds and the run passes on a fast machine.
What helped us is the same rule applied to the screen: before an action, the agent states what it expects to see after it (a dialog title, a row in a list, a button becoming enabled), then re-reads the screen until that appears or a hard timeout hits. A timeout then means "the expected state never showed up", which is a real failure with a name, instead of a clock that was a bit too short.
One extra trick for tests: when the wait does time out, log what WAS on the screen or in the response at that moment. Most "flaky" timeouts we looked at turned out to be a different state (an error toast, a login redirect), not a slow one.