Flaky Test Detection in 2026: From Manual Reruns to AI-Powered Classification

The question used to be simple: "Is this test flaky?" You'd run it a few times, shrug, and move on. But as test suites grew from hundreds to thousands, and CI pipelines became the backbone of deployment, "is this flaky?" became the most expensive question in software development.

In 2026, the tools and approaches for answering that question have matured significantly. Here's the full landscape.

Level 0: Manual Reruns (What Most Teams Still Do)

The default approach for most teams: CI fails → engineer investigates for 15-20 minutes → gives up → clicks "re-run" → passes → moves on.

Cost per incident: $20-30 in wasted engineer time
Detection accuracy: Low (depends entirely on individual judgment)
Scalability: None (doesn't scale past ~50 CI runs/day)

This isn't really "detection" — it's "tolerance." And it costs teams an average of $50,000/year in wasted diagnostic time for a 20-person team.

Level 1: Retry Logic

The first automation step: configure your test runner to automatically retry failed tests 1-3 times before reporting failure.

How it works:

# Playwright config
retries: 2

# Cypress config
retries: { runMode: 2, openMode: 0 }

# Jest config
jest --retryTimes=2

Pros:

• Zero setup cost
• Reduces false-positive failure reports
• Every framework supports it natively

Cons:

• Masks the problem instead of solving it
• Increases CI cost by 2-3x for flaky tests
• Doesn't tell you which tests are flaky (just absorbs the failure)
• A test that fails 2 out of 3 times will still report failure — but with 3x the compute cost
• No learning — the same test stays flaky forever

Best for: Teams with <5% flakiness who need a quick fix while they work on root causes.

Level 2: Flakiness Tracking (Post-Hoc Detection)

Tools that monitor your test runs over time and flag tests that show oscillating pass/fail patterns.

Notable Tools:

Cypress Cloud — Built into Cypress's paid tier. Tracks test flakiness over time with a "Flaky" badge. Shows historical pass rate. Requires Cypress.

Currents.dev — Works with Cypress and Playwright. Shows flakiness trends, failure patterns, and test duration analytics. Strong reporting. $15-50/month.

BuildPulse — Framework-agnostic. Ingests JUnit XML reports from any test runner. Flags flaky tests based on configurable thresholds. $49-199/month.

Trunk.io — Part of a broader CI quality suite. Tracks flaky tests, code coverage, and linting in one dashboard. $15-40/developer/month.

How they work:

1. You send test results (usually JUnit XML) to their API after each CI run
2. They build a historical profile for each test
3. They flag tests that fail intermittently above a threshold you set
4. You (the human) decide what to do with the flagged tests

Pros:

• Actually identifies which tests are flaky (vs. just retrying)
• Provides historical data for prioritization
• Framework-agnostic options available

Cons:

• Post-hoc only — tells you about flakiness AFTER the fact, not in real-time
• Doesn't prevent the wasted investigation time on the current failure
• Requires manual action to quarantine or fix
• Quarantine lists grow and are never cleaned up
• Doesn't answer the immediate question: "Is THIS failure right now real or flaky?"

Cost range: $0-200/month depending on team size and features.

Level 3: Real-Time Failure Classification (Emerging)

The newest approach: instead of waiting for patterns to emerge across multiple runs, classify each individual failure in real-time as "likely real" or "likely flaky."

How It Works

When a test fails, the classification engine analyzes:

1. Historical pattern: Has this test failed on this same assertion before and then passed on the next run? What's its 20-run oscillation rate?

2. Failure signature: Is the failure at the same line, same assertion, same error message as previous failures that turned out to be flaky?

3. Contextual signals: How many parallel workers are running? What's the current API latency? Is there a known infrastructure issue?

4. Test characteristics: Does this test use sleep/waitForTimeout? Does it share state with other tests? Does it call external APIs?

The engine outputs a confidence score (e.g., "87% likely flaky") and a recommended action (block merge vs. allow with flag).

Tools in This Space:

Poly — Classifies CI failures in real-time as real or flaky. Integrates with GitHub Actions, GitLab CI, and Jenkins. Auto-handles flaky failures without blocking merges. Tracks diagnostic time saved.

Mergify Test Insights — Quarantines flaky tests and manages merge queue behavior. More focused on merge automation than classification.

Why This Is Different

The critical difference is timing. Post-hoc tools tell you about flakiness hours or days later. Real-time classification tells you immediately — the moment the failure happens.

This eliminates the diagnostic time cost, which is the single largest expense of flaky tests (over $49,000/year for a typical 20-person team).

Comparison Matrix

Feature	Manual	Retries	Tracking Tools	AI Classification
Detects flaky tests	❌	❌	✅ (delayed)	✅ (real-time)
Prevents wasted investigation	❌	Partial	❌	✅
Prevents merge blocks	❌	✅	❌	✅
Tells you WHICH tests are flaky	❌	❌	✅	✅
Auto-handles flaky failures	❌	✅ (via retry)	❌	✅ (via classification)
Learns and improves	❌	❌	✅	✅
Setup effort	None	Minimal	Moderate	Low
Monthly cost	$0	$0-50 (compute)	$15-200	$25-100

What to Choose for Your Team

< 5% flakiness, < 10 engineers:
Start with retry logic and track manually. The problem isn't big enough to justify tooling investment yet.

5-15% flakiness, 10-30 engineers:
Add a tracking tool (Currents.dev or BuildPulse) to identify your worst offenders. Fix the top 10. Then evaluate AI classification.

15%+ flakiness, 30+ engineers:
Skip straight to real-time classification. The diagnostic time cost alone justifies the investment. Fix tests in parallel based on the prioritized flakiness report.

The Trend: From Detection to Prevention

The next frontier isn't just detecting flaky tests — it's preventing them from entering your pipeline in the first place. Emerging approaches include:

• PR-level flakiness prediction: Analyzing the code changes in a PR to predict which tests might become flaky before the merge
• Environment drift detection: Monitoring test environment health and alerting when infrastructure changes might induce flakiness
• Automated test repair: Using AI to suggest or apply fixes for known flaky patterns (brittle selectors, missing waits, shared state)

The teams that adopt real-time classification now will have the historical data foundation to benefit from these preventive approaches as they mature.

The Bottom Line

In 2026, "just re-run it" is no longer an acceptable answer to flaky tests. The cost is too high, the tools are too good, and the competitive pressure to ship fast is too intense.

The right approach depends on your team size and flakiness level — but the direction is clear: move from reactive (manual reruns) to proactive (real-time classification) as fast as you can.

Ready to stop guessing? Poly classifies every CI failure in real-time — so your team only investigates what matters. Free for teams up to 5 engineers.