Originally published on the QA Guardian blog.
A flaky test is one of the most expensive items in a software organization, and one of the most underestimated. On the surface it's an annoyance: fails ten percent of the time for no apparent reason, someone re-runs CI, it passes, they ship.
This is a practical playbook: diagnose root causes, measure flakiness, prioritize fixes, write tests that are deterministic by design.
The Four Root Causes
Timing dependencies. The test interacts before the app is ready. The common "fix," waitForTimeout(2000), slows the suite and leaves the race intact. Wait for a specific condition instead.
Selector fragility. Auto-generated class names, nth-child(3), or copy marketing changes without a code review. The UI refactor breaks the test even though the feature still works.
State contamination. Shared sessions, leftover DB rows, in-memory state. One test poisons another — but only sometimes, depending on order or which parallel worker runs it.
Environment variance. Assumptions about latency, data, or CPU that hold locally and fail on CI (or the reverse).
What Stable Tests Have in Common
Semantic selectors. getByRole('button', { name: 'Place Order' }) targets what the user sees — it breaks when the product label changes (a real decision), not when a CSS module renames a class.
Condition-based waiting. Replace every waitForTimeout with a real condition: waitForResponse, a locator assertion, or Playwright's auto-waiting via expect(locator).toBeVisible().
Isolated browser contexts. Fresh context per test — no shared cookies, storage, or session bleed across workers. Required for safe parallel runs, not optional polish.
Bounded scope. A twenty-step flow has twenty places for environmental noise; a three-hundred-step monolith has three hundred. Scope to one journey.
Build a Flakiness Matrix
You can't fix what you don't measure. Track pass rate per test over a rolling thirty-day window. A test that fails two percent of the time looks fine on any given day and still burns hours and trust over a month.
Four columns: test name, thirty-day failure rate, journey criticality, owner. Triage anything below ninety-eight percent. Most teams find ~80% of flakes live in ~20% of tests — usually the oldest, most fragmented, least-owned ones.
Prioritize Fixes
Rank by journey criticality first: flaky checkout is urgent, flaky settings can wait a sprint. Then by false-positive risk — a test that goes green when the feature is broken is worse than one that fails loudly.
When you rewrite, prefer one end-to-end flow over re-stabilizing a pile of fragments.
Your First Week
- Export thirty days of CI results; sort by failure rate; pick the top ten offenders touching critical journeys.
- For each, label the root cause (timing, selector, state, environment) before changing code.
- Delete or fold presence-only checks; rewrite the journey as one isolated flow with semantic locators and condition waits.
- Re-measure for two weeks. Still under 98%? The diagnosis was wrong — dig again, don't add another timeout.
The Goal: Deterministic by Design
The best flake fix is a test that never had room to flake: real app, sequential journey, semantic selectors, isolated context, waits on real conditions. When that test fails, something in the product changed — that's the signal the suite exists to provide.
Top comments (0)