How I Tamed Flaky E2E Tests and Boosted Build Stability from 89% to 99%

The story of how a simple retry policy saved our team's sanity and our main branch

The Breaking Point
It was the third time in a month. Our main branch had gone red again. The culprit? A flaky E2E test that decided to fail for no good reason.

Our team was frustrated. Developers were wasting hours re-running pipelines, investigating false positives, and context-switching away from feature development. Our build stability had plummeted to a dismal 89%, and trust in our CI/CD process was at an all-time low.

Something had to change.

The Diagnosis: Flaky Tests Are Not All Created Equal
Before implementing any solution, I needed to understand why our E2E tests were flaky. After analyzing our failure patterns, I identified three main categories:

Race conditions (40%): Tests expecting UI updates faster than they could render

Network timing issues (35%): API responses taking slightly longer in CI than locally

Environment-specific failures (25%): Browser quirks and CI resource contention

The key insight? 80% of our flaky failures would pass on the very next run.

The Solution: A Smart Retry Policy
Instead of overhauling our entire test suite (which would take months), I implemented a pragmatic retry strategy:

javascript
// Example: Playwright retry configuration
import { defineConfig } from '@playwright/test';

export default defineConfig({
retries: process.env.CI ? 2 : 0, // Retry twice in CI only
workers: 4,
timeout: 60000,
// ... other config
});
The rule was simple:

Local development: 0 retries (catch issues early)

CI pipeline: 2 retries (give flaky tests a second chance)

Only retry the specific failed test, not the entire suite

The Results: What 10% Improvement Looks Like
Metric Before After Change
Build stability 89% 99% +10%
Developer productivity 3-4 hours/week wasted <30 min/week ~3 hours saved/developer
Main branch breakages 3/month 0/month (so far) 100% reduction
Team confidence Low High Priceless
Lessons Learned Along the Way

1. Retry Policies Are a Band-Aid, Not a Cure While retries immediately improved stability, they didn't fix the underlying issues. We made a pact to:

Investigate any test that needed both retries to pass

Add logging around flaky areas to identify root causes

Gradually rewrite the flakiest tests

1. Communicate the Change We sent a simple Slack message explaining the new policy:

🔧 New Flaky Test Policy: We've added automatic retries (2x) for E2E tests in CI. If a test fails twice, we need to investigate. This should reduce false positives while we continue to stabilize our test suite.

1. Track Retry Metrics We started monitoring how often retries actually helped:

javascript
// Track retry success rate
const flakyTestMetrics = {
totalRuns: 1247,
retriesUsed: 89,
flakyPasses: 74, // 83% of retries were successful
persistentFailures: 15 // Tests that truly needed fixing
};
This data helped us prioritize which tests to permanently fix.

What We Fixed Permanently
Once the immediate crisis was over, we systematically addressed the root causes:

Category Fix Impact
Race conditions Added explicit wait strategies (waitFor, expect.poll) Reduced retries needed by 60%
Network timing Increased API timeouts in CI environment Reduced retries needed by 25%
Environment issues Fixed browser versions and resource allocation Reduced retries needed by 15%
The Bottom Line
Retries are not a silver bullet. They're a pragmatic tool to stop bleeding while you heal the wound.

For our team, implementing a flaky-test retry policy was the difference between:

Chaos: Wasting time on false positives, broken builds, and blame games

Calm: Trusting our CI, shipping faster, and knowing that when a build actually fails, it's real

From 89% to 99% stability in a week—not because our tests got better overnight, but because we stopped letting perfection be the enemy of progress.

Your Turn: How to Implement This
Step 1: Choose Your Retry Strategy
Playwright: retries: 2 in config

Cypress: retries: { runMode: 2, openMode: 0 }

Jest/Puppeteer: Custom wrapper or jest.retryTimes(2)

Step 2: Set Clear Rules
CI: 2 retries

Local: 0 retries

Report: Log when retries are used

Step 3: Fix the Worst Offenders
Use retry data to identify and permanently fix the flakiest 20% of tests—they probably cause 80% of the frustration.

Step 4: Monitor and Iterate
Track your stability score and gradually reduce retries as you fix underlying issues.

Final Thoughts
Our main branch hasn't broken in 6 weeks now. The team is shipping faster, with more confidence, and less stress. The retry policy bought us the time we needed to fix things properly.

Sometimes the best solution isn't perfect—it's the one that works right now.

Have you dealt with flaky E2E tests? How did your team handle it? Share your story in the comments below! 👇