Flaky tests aren't random: a field guide to finding and killing them

A flaky test is one that passes and fails on the same code with no change in between. It's the most corrosive thing in a test suite — not because any single failure is scary, but because flakiness teaches your team the worst possible habit: ignoring red.

Once "just re-run it" becomes the reflex, your suite stops being a safety net and becomes a slot machine. Here's a practical field guide to the usual culprits and how to actually kill them — built from years of chasing these in real automation suites.

Flakiness is never truly random

It feels random, but it almost always traces to one of a handful of non-determinisms. The job is to find which one.

1. Timing / async (the #1 cause in UI and integration tests)

The test checks for something before the app is ready. It passes when the machine is fast, fails when CI is loaded.

• Fix: never sleep for a fixed time. Wait for a condition — an element state, a network response, a value. Modern frameworks (Playwright, Cypress) auto-wait on assertions; lean on that instead of sleep(2).
• Smell to grep for: sleep, setTimeout, hard-coded waits.

2. Test order dependence / shared state

Test B passes only because Test A ran first and left data behind. Run them in a different order (or in parallel) and B falls over.

• Fix: each test sets up and tears down its own state. No reliance on execution order. Tools like pytest-randomly or iDFlakies surface these by shuffling order.

3. Shared external resources

A real database, a shared test account, a live third-party API. Two runs collide, or the API has a bad day.

• Fix: isolate data per test (unique keys), and stub the network at the boundary (cy.intercept, WireMock, VCR). Test your logic, not someone's uptime.

4. Time, timezones, randomness, locale

Tests that use now(), random seeds, or locale-dependent formatting fail at midnight, on a CI box in another timezone, or on the one day the random value hits an edge.

• Fix: inject the clock and the seed. Freeze time (freezegun, sinon), pin the seed, pin the locale.

5. Resource leaks and ordering in the app itself

Sometimes the test is honest and the code is flaky — an unawaited promise, a race in the app, a dictionary iteration order assumption. The test just exposed it.

• Fix: this one's a gift. The flaky test found a real concurrency bug. Fix the app, not the test.

A triage process that works

1. Quarantine, don't ignore. Move the flaky test to a separate suite that doesn't block the pipeline — but tracks it. The goal is to stop the "ignore red" habit, not to hide the problem.
2. Reproduce deterministically. Run it 50 times locally (pytest --count=50, or loop it). If it never fails locally but fails in CI, the difference is the clue — load, parallelism, environment.
3. Bisect the cause. Run it in isolation vs. with the suite (order dependence?). Disable the network (external dependency?). Freeze the clock (time?).
4. Fix the root cause, then de-quarantine. And add the reason to your commit message, so the next person learns the pattern.

Measure it, don't vibe it

Track a flaky rate per test (failures ÷ runs over the last N runs). It turns "the suite feels flaky" into a number you can drive down, and it tells you which tests to fix first. A test that's failed 8% of the last 200 runs is a priority; one that failed once six weeks ago is noise.

The mindset

Every flaky test is a small lie your suite tells. The fix isn't more retries — retries just make the lie quieter. The fix is removing the non-determinism, one category at a time, until green means green again.

What's the flakiest test you've ever had to hunt down — and what turned out to be the cause? I collect these war stories.

---

I'm Krishnasai Rollamadugu — a test automation architect who writes about test reliability and trust in AI-driven testing.