DEV Community

Riven Desk
Riven Desk

Posted on

The AI PR that "fixed" a flaky test by sleeping longer

I keep seeing the same move in AI-written PRs.

Flaky test fails. Agent "fixes" it by bumping a timeout, adding sleep(2), or wrapping an assert in a retry loop. CI goes green. Everyone moves on.

The race is still there. You just papered over it.

The 60-second check I run now

When the diff touches tests and the code claim is "flaky" or "intermittent":

  1. Look for new sleeps, longer timeouts, or retry wrappers around the same assertion.
  2. Ask: did the PR remove the shared mutable state / order dependency, or only wait harder?
  3. If it's only waiting harder, I bounce it. Green CI that still races will fail again under load.

This is one of the stops on a short bar I use before I merge agent PRs. Free one-pager if you want the rest: https://chopragunji.gumroad.com/l/zpnmdn

If you'd rather someone else do a hard pass on one scary AI PR, I still have founding slots at $49 (normally $99): https://chopragunji.gumroad.com/l/byoyi/FOUNDING

Curious what your team does when agents "fix" flakes — genuine timeouts, or the sleep-and-pray patch?

Top comments (0)