Flaky tests rarely spread evenly across a suite. In most pipelines we audit, a small group of specs produces the majority of unreliable failures. So you can usually fix the pipeline without a rewrite.
This is a practical guide to stabilizing automation you already own. No definitions of flakiness, no "rewrite it in Playwright" advice. Just triage, root cause work, quarantine rules, and the metrics that tell you whether it worked.
Examples cover Playwright, Cypress, Selenium, GitHub Actions, and GitLab CI.
How can I reduce flaky tests in CI/CD without rewriting the entire test suite?
Find the small set of specs causing most failures, classify why each one fails, fix by category instead of one test at a time, and quarantine the rest so the pipeline stays honest.
A rewrite is tempting because the suite feels untrustworthy. But a rewrite moves the same design mistakes into new syntax. Hard-coded sleeps and shared test data will reappear in any framework. Fix the causes first. Then decide if you still need new tooling.
What causes flaky tests in CI/CD pipelines?
Almost every flaky test traces back to a small number of root causes. Classify before you fix, because the remedy differs by category.
The Selenium documentation is blunt about the first row. Race conditions between browser state and driver commands are "one of the primary causes of flaky tests." That is where most teams should start.
Why do tests fail in CI but pass locally?
This question comes up in every audit, and the answer is usually one of 6 things.
- Speed. CI machines are often slower and more loaded than your laptop, so implicit timing assumptions break.
- Parallelism. Local runs are frequently single-threaded. CI runs four or eight workers that fight over the same test data.
- Headless rendering. Different viewport, no GPU, fonts missing, elements positioned differently.
- Clean state. Your local browser has cookies, cached auth, and a warm database. CI starts empty.
- Locale and timezone. Runners default to UTC. Date assertions written in your timezone drift.
- Network shape. Local hits a dev server on localhost. CI crosses a network boundary with real latency.
As a fast diagnostic, run your suite locally with the same worker count and a fresh browser profile. If it fails there too, the problem is your tests, not the runner.
How should QA teams prioritize flaky tests in an existing automation suite?
Do not fix flaky tests in the order you find them. Rank them. Start by calculating a flake rate per spec over the last 30 days:
flake rate = (runs that failed then passed on retry) / total runs
Then score each test on three factors: flake rate, how often the spec runs, and whether it blocks merges. A test that fails 4% of the time on every pull request costs far more engineering hours than one failing 30% of the time in a nightly job.
Flaky test triage checklist
Run this on each candidate before writing a fix:
- Does it fail in isolation, or only in a full parallel run?
- Does it fail on a specific worker, browser, or shard?
- Is there a hard-coded sleep or fixed timeout in the failure path?
- Does it create or reuse data that another test touches?
- Does it depend on a previous test, leaving the app in a certain state?
- Does it call a third-party service that is not stubbed?
- Do the failure timestamps cluster around deploys, backups, or cron jobs?
- Which root cause row from the table above matches?
- Fix now, quarantine, or delete?
Keep the last question open. Some flaky tests cover logic already tested at a lower level, so deleting them is a legitimate outcome.
How do you fix flaky automated tests in CI/CD?
Fix by category. The next 3 categories cover most of the work.
1. Replace sleeps with condition-based waits
A fixed sleep is a guess about timing. It fails when the app is slower and wastes time when it is faster.
Playwright assertions retry until they pass or time out, so this is usually enough:
// Fragile: assumes the row renders within 3 seconds
await page.waitForTimeout(3000);
await expect(page.locator('.order-row')).toHaveCount(1);
// Stable: retries the condition itself
await expect(page.locator('.order-row')).toHaveCount(1, { timeout: 10_000 });
In Cypress, wait on the request rather than the clock:
cy.intercept('POST', '/api/orders').as('createOrder');
cy.get('[data-cy=submit]').click();
cy.wait('@createOrder').its('response.statusCode').should('eq', 201);
cy.get('[data-cy=order-row]').should('have.length', 1);
For Selenium, use explicit waits and avoid one specific trap. The Selenium waits documentation warns against mixing implicit and explicit waits, because the combination produces unpredictable timeouts:
from selenium.common import NoSuchElementException, ElementNotInteractableException
from selenium.webdriver.support.wait import WebDriverWait
errors = [NoSuchElementException, ElementNotInteractableException]
wait = WebDriverWait(driver, timeout=10, poll_frequency=0.2, ignored_exceptions=errors)
wait.until(lambda d: d.find_element(By.CSS_SELECTOR, ".order-row").is_displayed())
Pick one strategy per project. Explicit waits give you control per interaction.
2. Give every test its own data
Shared fixtures are the most common cause of parallel-only failures. Generate data per test and seed it through the API instead of the UI:
test('user can cancel an order', async ({ page, request }) => {
const email = qa+${crypto.randomUUID()}@example.com;
const { id } = await (await request.post('/api/test/orders', {
data: { email, status: 'pending' },
})).json();
await page.goto(/orders/${id});
await page.getByRole('button', { name: 'Cancel order' }).click();
await expect(page.getByText('Order cancelled')).toBeVisible();
});
API setup removes a long UI path from the test and cuts the number of steps that can fail for reasons unrelated to what you are testing.
3. Control the network
Stub third-party calls in functional tests. Keep a separate, small set of contract tests that hit the real service on a schedule, so an outage at a payment provider does not block merges.
How can I stabilize Playwright, Cypress, or Selenium tests?
Framework-specific moves that pay off quickly.
Playwright. Use role and label locators over CSS chains. Enable trace: 'on-first-retry' so you get a full timeline for the failure without storing traces for every run. Keep tests isolated rather than reaching for serial mode, which the docs recommend for the same reason.
// playwright.config.ts
export default defineConfig({
retries: process.env.CI ? 2 : 0,
use: { trace: 'on-first-retry', video: 'retain-on-failure' },
});
Cypress. Turn on test isolation, use cy.session() for cached login, and replace cy.wait(ms) with aliased intercepts. Set retries only in run mode:
// cypress.config.js
retries: { runMode: 2, openMode: 0 }
Selenium. Centralize waits in one helper so timeouts are consistent, pin browser and driver versions in CI, and run in a container so local and pipeline environments match.
Across all three, disable CSS animations in your test build. It removes an entire class of timing failures in a single change.
Should engineering teams use retries for flaky tests?
Retries are a detection tool, not a cure. Used well, they keep the pipeline moving while you fix causes. Used badly, they hide real bugs.
The rule we apply: retries are allowed, but a passing-on-retry result must be recorded as flaky, not green. Playwright does this by default, sorting results into passed, flaky, and failed. Its Playwright test retries docs also expose testInfo.retry, which is useful for clearing server state before a second attempt:
test('checkout completes', async ({ page }, testInfo) => {
if (testInfo.retry) await resetCartState();
// ...
});
Two limits to set. Cap retries at two, since a test needing three attempts is broken rather than flaky. And never retry the whole pipeline job to get a green build, because that erases the signal you need.
In GitLab CI, scope job-level retries to infrastructure problems only:
e2e:
script: npx playwright test
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
artifacts:
when: always
paths: [playwright-report/]
reports:
junit: results.xml
That way runner crashes get retried and genuine test failures do not.
When should flaky tests be quarantined?
Quarantine when a test is unreliable enough to erode trust but too valuable to delete, and you cannot fix it this sprint. It keeps the main pipeline meaningful while the test stays visible.
Quarantine works only with an expiry date. Without one, the quarantine list becomes a graveyard.
Quarantine policy example
1. Eligibility: flake rate above 2% over 30 days, or 3+ false failures in a week.
2. Action: tag the test @quarantine and move it out of the blocking job.
**3. Ownership: **the owning team is assigned within 24 hours. No owner, no quarantine.
4. Time limit: 14 days. Fixed, deleted, or escalated at expiry.
**5. Cap: **quarantine holds no more than 2% of the suite. At the cap, stabilization work takes priority over new test development.
6. Visibility: quarantined tests run nightly and report to the team channel.
In GitHub Actions, run quarantined tests in a separate non-blocking job:
jobs:
e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npx playwright test --grep-invert @quarantine
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: playwright-report/
quarantined:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v4
- run: npx playwright test --grep @quarantine
The continue-on-error: true flag keeps these results reported without blocking the merge.
Key metrics teams should monitor
Stabilization work needs numbers, or you cannot tell progress from luck.
Track flake rate per spec alongside the suite-wide number. Averages hide the handful of tests doing the damage.
Do and Don't
Do:
- Classify root cause before writing a fix
- Rank by flake rate multiplied by run frequency
- Seed data through the API, unique per test
- Store traces and video on first retry
- Report retry passes as flaky, never as green
- Put an expiry date on every quarantined test
- Pin browser and driver versions in CI
- Review the flakiest five specs in your weekly QA sync
Don't:
- Rewrite the suite before diagnosing it
- Add sleeps to "make it stable"
- Raise global timeouts as a blanket fix
- Retry entire pipeline jobs to get green
- Mix implicit and explicit waits in Selenium
- Share user accounts or records across parallel tests
- Skip tests silently with no owner or date
- Judge progress on suite-wide averages
Where to start on Monday
Pull the last 30 days of CI results. Rank specs by flake rate times run frequency. Take the top five, classify each against the root cause table, and fix by category. Quarantine anything you cannot fix in two weeks, with an owner and a date.
In the audits we run, pipeline trust usually recovers from this alone, before anyone touches framework choice. If you would rather have an outside read on which tests to fix, quarantine, or delete, that is what test automation consulting services cover. In this case, experts audit the existing framework, stabilize what is worth keeping, and hand back the flake metrics and quarantine policy your team runs afterwards.
By the way, if you have a stabilization tactic that works on your pipeline, drop it in the comments, it would be nice to read your experience.
Top comments (0)