Why Your Playwright Tests Are Lying to You (And How to Actually Fix Flakiness)

Flaky tests aren't random — they're deterministic bugs you haven't found the trigger for yet. This post covers the 8 real root causes, how to reproduce flakiness on demand instead of guessing, and the fixes that actually stick. Code examples throughout.

---

You've been there. A test fails in CI. You re-run it. It passes. You shrug, blame "the network," and move on — until it fails again three days later, right before a release. Multiply that across a suite of 500 tests and you've got a team that no longer trusts its own automation, a Slack thread full of "is this a real bug or just flaky?", and engineers quietly ignoring red pipelines.

Flaky tests aren't a minor annoyance — they're a trust problem. The moment "red" stops meaning "something broke," your suite becomes noise instead of signal. Here's how to actually kill flakiness instead of papering over it with retries.

What flakiness actually is

A flaky test produces different results on different runs without any change to the code, the environment, or the test itself. Same commit, same browser, same spec file — different outcome.

Important distinction: if a test fails because someone genuinely broke a feature, that's not flaky — that's the test working. Flakiness is specifically non-determinism: the outcome depends on something the test author didn't account for — timing, execution order, shared state.

Mental model I use: your test is a bet on the state of the DOM at a given moment. If the DOM hasn't reached the state you're betting on and you don't wait for it properly, you lose the bet — sometimes, not always. That "sometimes" is the whole problem.

The root causes, ranked by how often I actually see them

1. Racing the DOM

Button disables during submit, re-enables after a success toast. Click lands in that window, and it either no-ops or throws.

// Flaky: assumes instant clickability
await page.click('#submit-btn');
await expect(page.locator('.toast-success')).toBeVisible();

// Reliable: wait for the signal that actually matters
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.locator('.toast-success')).toBeVisible({ timeout: 10000 });

Playwright's auto-waiting only waits for actionable — visible, stable, enabled. It has no idea your app has a 400ms debounce before the button is "really" clickable. That knowledge has to live in your assertions.

2. Animations and transitions

Playwright calls an element "stable" once its bounding box stops moving between two frames — but a fade-in or modal slide can fool this, especially on CI runners with inconsistent frame timing.

const modal = page.getByRole('dialog');
await modal.waitFor({ state: 'visible' });
await expect(modal).toHaveCSS('opacity', '1'); // confirms the transition actually finished

Better yet, if you own the app code: add a flag like aria-busy="false" that only flips once animations settle, and assert on that instead of reverse-engineering CSS state.

3. Shared state across tests

The most underestimated cause. Test A leaves a cookie or DB row that Test B didn't expect. Run in a different order (which parallel execution does by default) and your "stable" suite flakes again.

test.beforeEach(async ({ context }) => {
  await context.clearCookies();
  await context.clearPermissions();
});

Litmus test: can you run one spec file alone, in any order, on a fresh context, and get the same result every time? If not, there's a hidden dependency somewhere.

4. Unmocked network calls

Hitting a real payment gateway or analytics script in a UI test means you've outsourced your test's reliability to someone else's uptime.

await page.route('**/api/checkout', async route => {
  await route.fulfill({
    status: 200,
    body: JSON.stringify({ status: 'success', orderId: 'TEST-123' }),
  });
});

Mocking isn't cheating — it isolates the variable you're testing (your UI) from the one you're not (a third party's server health).

5. Resource contention in CI

Passes locally every time, flakes only in CI — usually resource starvation from many parallel workers. page.waitForTimeout(2000) is dangerous here because "2 seconds" locally might be 4 seconds during a noisy CI spike.

// Bad
await page.waitForTimeout(2000);

// Good
await expect(page.locator('#order-status')).toHaveText('Confirmed');

6. Selectors coupled to markup, not meaning

div:nth-child(4) button tests your current DOM layout, not your app. One refactor and it silently points at the wrong element.

// Fragile
await page.click('div:nth-child(4) button');

// Durable
await page.getByRole('button', { name: 'Save changes' }).click();

No accessible name available? data-testid is a fine fallback — still far more stable than positional CSS.

7. Environment drift between local and CI

Rock-solid on your laptop, flaky only in the pipeline? That's usually a real difference in browser version, viewport, timezone, or locale your test quietly depended on — not "CI being flaky." Pin your Playwright browser version, set an explicit viewport in config, and treat CI-only failures as a diffing clue, not a retry target.

8. Logs that don't help future-you

console.log('here') tells you nothing in six months. Log the inputs and state that let someone reconstruct the run without re-executing it:

console.log(`[checkout] user=${testUser.email} env=${process.env.TEST_ENV} orderId=${orderId}`);

Reproducing flakiness on demand (instead of guessing)

You can't fix what you can't reproduce.

Loop the suspect test:

npx playwright test tests/checkout.spec.ts --repeat-each=20

2 failures out of 20 = a confirmed, measurable failure rate — and a way to prove your fix actually worked.

Trace everything, not just failures:

// playwright.config.ts
use: {
  trace: 'retain-on-failure',
  video: 'retain-on-failure',
  screenshot: 'only-on-failure',
}

Traces turn "it just failed, IDK why" into "the button was disabled for 340ms while the API call was still pending."

Make retries visible instead of silent:

// playwright.config.ts
export default {
  retries: process.env.CI ? 2 : 0,
  reporter: [['html'], ['json', { outputFile: 'results.json' }]],
};

Parse results.json in CI and alert on any test that needed a retry to pass — even if the pipeline stayed green overall. A silent retry hides a bug; a logged retry creates accountability.

The fixes that actually stick

1. Replace every fixed wait with a condition-based wait. Grep your repo for waitForTimeout and treat every hit as a bug ticket.
2. Assert on outcomes, not intermediate states — don't check the spinner appeared, check the data it loaded is now visible.
3. Give every test its own isolated context and its own test data instead of hoping cleanup ran.
4. Mock anything outside your team's deploy boundary.
5. Quarantine flaky tests (own job, skipped from main run) with an owner and a deadline — don't let them live there forever.

Two-minute pre-merge check

• Does it pass alone, in random order, on a fresh context?
• Does every wait map to something real, or is it a guessed number?
• Would it survive a CSS class rename tomorrow?
• Does it create/clean up its own data?
• Does it talk to anything I don't control?

Can't confidently answer all five? You're not merging a test — you're merging next month's investigation.

The actual takeaway

Retries and longer timeouts are painkillers, not cures. The race condition or shared state is still there, waiting for CI to get a little slower or the app a little more complex. The teams with genuinely stable Playwright suites aren't the ones with the most retries — they're the ones who treat every flaky test as a solvable mystery: reproduce it, trace it, fix the cause.

---

Curious how others handle this — do you quarantine flaky tests in a separate CI job, or just fix-forward immediately? What's the flakiest category you've run into: animations, network, or shared state?