What Are Flaky Tests? The Complete 2026 Guide

If you've ever pushed code, watched CI go red, spent 20 minutes investigating, only to hit "re-run" and see it pass — you've been burned by a flaky test. And you're not alone.

According to the 2026 Flaky Test Benchmark Report, 26% of engineering teams now report flaky tests as a significant problem, up from just 10% in 2022. That's a 160% increase in four years. The problem isn't just growing — it's accelerating.

What Exactly Is a Flaky Test?

A flaky test (also called a non-deterministic test or intermittent failure) is a test that produces inconsistent results across multiple runs, even when the code under test hasn't changed. The same commit, the same environment, the same test — different outcomes.

Here's what makes a test flaky versus a real failure:

Characteristic	Real Failure	Flaky Test
Reproducible	Yes — fails every time	No — fails intermittently
Code changed?	Usually yes	No code change needed
Fix by re-running?	No	Often yes
Root cause	Code bug	Environment, timing, state, race condition

The 7 Most Common Causes of Flaky Tests

1. Async Timing Issues

The single most common cause. Your test runs faster than the application state updates. A button is clicked before the API response arrives, or an assertion runs before the DOM renders.

// FLAKY — no wait for API
await page.click('button[type="submit"]');
expect(await page.textContent('.result')).toBe('Success');

// STABLE — waits for response
await page.click('button[type="submit"]');
await expect(page.locator('.result')).toHaveText('Success');

2. Shared State Between Tests

Test B depends on data created by Test A. When tests run in a different order (or in parallel), Test B fails because its prerequisite data doesn't exist. This is especially common in integration tests and end-to-end test suites.

3. Resource Contention in Parallel Runs

Two parallel workers hit the same test database row, the same API rate limit, or the same file system path. One succeeds, one fails. The failure has nothing to do with your code — it's infrastructure noise.

4. Brittle Selectors

Tests that rely on CSS class names, DOM structure, or nth-child selectors break when the UI changes — even slightly. A designer adds a wrapper div, and suddenly your test can't find the element.

5. External Service Dependencies

Tests that call real third-party APIs (Stripe, SendGrid, Twilio) are at the mercy of those services' availability and latency. A 500ms slowdown in Stripe's response time can cascade into a test timeout.

6. Time and Date Dependencies

Tests that assert on Date.now(), new Date(), or time-based calculations fail at midnight, month boundaries, or when the system clock drifts. Timezone differences between local and CI environments add another layer of complexity.

7. Non-Deterministic Data

Random IDs, UUIDs, timestamps in test data, or randomized sort orders make assertions fragile. If your test expects the first item in a list but the sort is non-deterministic, it'll fail unpredictably.

The Real Cost of Flaky Tests (It's Worse Than You Think)

Most teams underestimate the cost because they only count the obvious metric: CI re-runs. But the real cost has four layers:

Direct Cost: CI Compute

Every re-run costs money. A team averaging 3 re-runs per day on a $0.08/minute CI runner spends $58/month just on re-running tests that were never broken. Scale to 10 teams and you're at $580/month.

Diagnostic Cost: Engineer Time

This is the hidden killer. When CI goes red, an engineer stops what they're doing, opens the build log, and investigates. Our research shows the average investigation takes 15-23 minutes before the engineer concludes "it's flaky" and re-runs. At $75/hour engineer cost, that's $19-29 per false alarm.

A team with 5 false alarms per day loses $400-725 per day in diagnostic time alone.

Trust Cost: The "Boy Who Cried Wolf" Effect

When 30-40% of your CI failures are flaky, engineers stop treating red builds as urgent. They develop "CI blindness" — a red build becomes background noise instead of an alarm. When a real regression sneaks through in a sea of flaky failures, it ships to production.

Velocity Cost: Merge Delays

Flaky tests that block PRs create a merge queue bottleneck. If 2 out of 6 daily deploys are blocked by flaky tests (averaging 40 minutes each), that's 80 minutes of blocked deployment time per day. Over a month: 40 hours of lost deployment capacity.

How to Identify Flaky Tests in Your Suite

The 30-Run Test

Run your suspect test 30 times against the same commit. If it fails more than once but doesn't fail every time, it's flaky. A 95% pass rate over 30 runs is the commonly accepted threshold.

CI History Analysis

Look at your CI history for tests that oscillate between pass and fail on the same commit. Tools like Cypress Cloud, Currents.dev, and Buildpulse can track this automatically.

The "Same Commit, Different Result" Pattern

If your CI platform shows the same commit passing on one run and failing on the next, with the same test failing, that's a flaky test signature.

The Flaky Test Spectrum: Not All Flakes Are Equal

Understanding the severity of flakiness helps prioritize your fix efforts:

• Low flakiness (95-99% pass rate): Annoying but manageable. The test provides some signal.
• Medium flakiness (80-95% pass rate): Seriously eroding trust. The test is more noise than signal.
• High flakiness (50-80% pass rate): The test is worse than useless — it actively blocks your team.
• Coin flip (<50% pass rate): Delete it. It provides zero signal and maximum noise.

What Teams Are Doing About It (2026 State of the Art)

The current industry approaches fall into three categories:

1. Quarantine: Move flaky tests to a separate pipeline that doesn't block deploys. Effective short-term, but quarantine lists grow silently and never shrink.
2. Retry: Add automatic retries (usually 2-3 attempts) to absorb flakiness. This masks the problem rather than solving it and increases CI cost.
3. Fix: Actually debug and fix the root cause. The "right" answer, but extremely time-consuming. Teams report spending 20-30% of QA engineering time just on flaky test maintenance.

What's missing? Real-time classification. None of these approaches answer the immediate question every engineer has when CI goes red: "Is this real, or is it flaky?"

The Future: AI-Powered Failure Classification

The emerging approach is using AI to classify each CI failure in real-time as "likely real" or "likely flaky" based on:

• Historical pass/fail oscillation patterns
• Failure signature analysis (same assertion, same line, different outcome)
• Environmental signals (parallel worker count, time of day, API latency)
• Test dependency graphs

This allows teams to stop real failures immediately while auto-handling flaky ones — without quarantine lists or blanket retries.

Key Takeaways

• Flaky tests are a structural problem, not a testing problem. They grow with codebase complexity.
• The real cost isn't CI compute — it's engineer diagnostic time and trust erosion.
• Quarantine and retries are band-aids. They manage symptoms, not causes.
• The most impactful improvement is real-time failure classification that tells your team immediately whether a red build is worth investigating.

Want to stop guessing whether your CI failures are real? Poly classifies every test failure in real-time — so your team only investigates what matters.