Why My Playwright Tests Passed Locally and Failed in CI: Field Notes on Auto-Waiting, Test Isolation, and Trace-Driven Debugging

Headline: Playwright flake is almost never randomness — it is a race between my assertion and the application's state. Web-first assertions, per-worker data isolation, and trace: 'on-first-retry' removed the large majority of the CI-only failures in my end-to-end suite.

Playwright is Microsoft's browser automation and end-to-end testing framework that drives Chromium, Firefox, and WebKit through one API. I run it against a Next.js App Router application, and for months the suite was green on my machine and red roughly one run in five on CI. These are my notes on what those failures actually were.

Key takeaways

• A flaky Playwright test is a race condition in the test, not in the browser. Playwright's auto-waiting covers element actionability; it does not wait for my application's business state.
• Never call page.waitForTimeout() in a test. Web-first assertions such as expect(locator).toBeVisible() retry until expect.timeout (5 seconds by default) and pass the instant the condition is true.
• Parallel workers need separate data. Playwright runs multiple worker processes by default, so two tests can mutate the same database row simultaneously.
• CI is a slower machine with different fonts, timezone, and network. Pin timezoneId and locale in playwright.config.ts and stub third-party requests with page.route().
• trace: 'on-first-retry' is the single highest-value config line. It records DOM snapshots, network, and console output for exactly the runs that failed, at no cost on green runs.

What actually makes a Playwright test flaky?

A flaky Playwright test is one whose result depends on timing rather than on application behaviour. Playwright already performs actionability checks before every action: before locator.click() it waits for the element to be attached, visible, stable (not animating), enabled, and able to receive pointer events. That covers the DOM. It does not cover the state I actually care about — whether the mutation committed, whether React finished hydrating, whether a toast that will cover my button has appeared yet.

The three races I hit most often are: clicking a button before Next.js hydration attaches its handler, asserting on a list before a Server Action revalidated it, and a cookie banner intercepting a click. Playwright strict mode catches a fourth class — a locator resolving to more than one element throws instead of silently picking the first — which is a feature, not flake.

// Flaky: the click handler may not be attached yet.
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Saved')).toBeVisible();

// Stable: assert an interactive signal, act, then assert the consequence.
const save = page.getByRole('button', { name: 'Save' });
await expect(save).toBeEnabled();
await save.click();
await expect(page.getByRole('status')).toHaveText('Saved');

When should I use web-first assertions instead of waitForTimeout?

Always. A web-first assertion is a Playwright assertion that retries its condition until it passes or the timeout expires — toBeVisible(), toHaveText(), toHaveCount(), toHaveURL(). page.waitForTimeout(2000) does the opposite: it is always exactly as slow as the number I guessed, and still too short on the day the runner is loaded.

Two rules I apply. First, assert the consequence, not the delay: instead of sleeping after a submit, assert the row count changed with await expect(rows).toHaveCount(4). Second, prefer role-based locators such as getByRole('button', { name: 'Save' }) over CSS selectors, because they are re-resolved on every retry and survive markup refactors. Playwright locators are lazy — page.locator() stores a query, not an element — so a stale element reference is impossible by construction. For a condition with no built-in matcher, await expect(async () => { ... }).toPass() retries a whole block instead of sleeping.

How do I keep parallel Playwright workers from corrupting each other's data?

Give each worker its own data namespace. Playwright isolates browser state for free — every test gets a fresh BrowserContext with its own cookies and storage — but it does not isolate my database. Two workers signing in as the same user and editing the same project will fail in ways that look random.

// fixtures.ts — one account per worker process, created once.
export const test = base.extend<{}, { account: Account }>({
  account: [async ({}, use, workerInfo) => {
    const acct = await createAccount(`w${workerInfo.parallelIndex}@example.test`);
    await use(acct);
    await deleteAccount(acct.id);
  }, { scope: 'worker' }],
});

Authentication is the other large win. Sign in once in a setup project, save the session with storageState, and let every test start already logged in. That removes a login flow from every single test — the most common source of both flake and wasted runtime in an end-to-end suite.

Why does the same test pass locally and fail in CI?

Because CI is a different machine with a different clock, a different font set, and fewer cores.

• Cores. Playwright derives its worker count from available CPUs. A 2-core runner executing a config I tuned on a 10-core laptop is far more contended, so I set workers: process.env.CI ? 2 : undefined.
• Locale and timezone. A date rendered 19/09/2026 locally and 9/19/2026 on the runner fails a text assertion. Pin both with use: { timezoneId: 'UTC', locale: 'en-US' }.
• Fonts. Missing fonts shift layout and break screenshot comparison. Running in the official mcr.microsoft.com/playwright image rather than a bare Node image removes that variable.
• Third-party scripts. Analytics and payment widgets are a network dependency I do not control. await page.route('**/analytics/**', route => route.abort()) takes them out of the equation.

How do I debug a flake I cannot reproduce locally?

Capture a trace of the failing run instead of trying to re-create it. Setting trace: 'on-first-retry' in playwright.config.ts writes a zip containing a DOM snapshot per action, the full network log, and console output. npx playwright show-trace trace.zip opens a time-travel viewer where I can hover any step and see the exact DOM the click landed on — which is how I found the toast that was covering a button for 300ms.

When I do want to reproduce locally, npx playwright test tests/checkout.spec.ts --repeat-each=50 --workers=4 runs the test fifty times under contention, which surfaces ordering races far faster than re-running CI. --last-failed then re-runs only what failed previously. On a sharded pipeline, the blob reporter plus npx playwright merge-reports gives one HTML report across all shards instead of four partial ones.

Should I retry a flaky test, quarantine it, or delete it?

Retries are a reporting tool, not a fix. retries: process.env.CI ? 2 : 0 keeps the pipeline usable and marks the test as flaky in the report so it stays visible. If retries hide the failure permanently, I have traded a failing test for a test that no longer means anything.

Option	When it fits	What it costs
Retry (retries: 2)	Suite is mostly stable and you want CI usable while fixing	Hides intermittent product bugs if nobody reads the flaky report
Quarantine (test.fixme or a @flaky grep tag)	The test fails daily and blocks merges	That user path is untested until someone returns to it
Fix the race	Root cause is a missing assertion or shared data	Usually an hour, and it typically fixes a whole family of tests
Delete	The test asserts nothing a user would notice	Lost coverage, but an honest report

FAQ

Q: Playwright auto-waits, so do I ever need explicit waits?

A: Playwright auto-waits for element actionability before each action, but not for application state such as a completed server mutation. Assert the consequence with a web-first assertion like expect(locator).toHaveCount(4) rather than adding a wait.

Q: What is the difference between page.waitForTimeout() and expect().toPass()?

A: waitForTimeout() sleeps a fixed duration regardless of state. expect().toPass() retries a block of assertions until they all pass or the timeout expires, so it finishes as soon as the condition becomes true.

Q: Should end-to-end tests run against a real database?

A: Yes, for confidence — but with per-worker isolation. Use a worker-scoped fixture keyed by testInfo.parallelIndex so two parallel workers never touch the same rows.

Q: How many end-to-end tests should a project have?

A: Few enough to run on every pull request. Cover authentication and revenue paths end to end, and push everything else down to component and unit tests, which are faster and far less prone to flake.

Q: Does trace: 'on' slow the suite down?

A: Yes — tracing every run adds runtime overhead and disk usage. trace: 'on-first-retry' gives identical debugging power but only on runs that already failed.

---

Originally published on devya.dev. Also on eng-ahmed.com. Built by Devya Solutions.