The Determinism Gap: Why Your Playwright Tests Lie to You (And a Systems-Engineering Fix)

Every automation engineer has said this sentence out loud: "It's flaky, just re-run it."

That sentence is doing more damage to your test suite than any bug in your application code ever could. It reframes a measurable engineering problem as an act of God — something to be appeased with retries and shrugged off in standup — instead of what it actually is: a race condition between your assertions and reality, one you can model, measure, and largely eliminate.

This article is about that race condition. I'm calling it the Determinism Gap — the window of time in which your test asserts something about application state while that state is still in motion. Everything that looks like "flakiness" lives inside this gap.

Flakiness is not randomness — it's unmeasured latency

When people say a test is "flaky," they usually mean it passes and fails on identical input with no code change. But nothing about a computer running deterministic code is actually random. What's really happening is that your test is racing against one or more of these:

• Network non-determinism — an API response arrives in 80ms on your laptop and 340ms in CI
• Rendering non-determinism — a React re-render, a CSS transition, a lazy-loaded chunk
• Data non-determinism — a shared test database where another test's teardown hasn't finished
• Scheduler non-determinism — CI runners under CPU contention, throttling event loops unpredictably

Your test doesn't fail because the universe rolled dice. It fails because you asserted at time T, and the thing you were asserting about didn't finish changing until T + Δ. The bug isn't in your app. It's in your model of time.

This matters because it changes the fix. If flakiness is randomness, the only rational response is retries. If flakiness is an unmodeled race condition, the rational response is to make the race visible and then close it — which is a fundamentally different (and much more durable) engineering activity.

Why Playwright's auto-waiting doesn't fully save you

Playwright is better at this than Selenium specifically because it ships with actionability checks — it won't click an element until it's visible, stable, and receives events. This eliminates a huge class of the old Thread.sleep(2000) flakiness that plagued Selenium suites for a decade.

But auto-waiting only solves the interaction side of the Determinism Gap. It says nothing about the assertion side. Consider:

await page.click('button[data-testid="save"]');
await expect(page.locator('.toast-success')).toBeVisible();

This looks safe — toBeVisible() is a web-first assertion, it polls. But it's polling for the symptom (a toast appearing), not the cause (the save actually completing on the server). If your toast renders optimistically before the network request resolves, your test can pass while the save silently fails. You haven't removed the gap. You've just moved it somewhere your assertion can't see it.

This is the trap: a passing test tells you your assertion was satisfied, not that your intent was fulfilled. Those are only the same thing if you've deliberately made them the same thing.

Closing the gap: assert on causes, not symptoms

The fix is to move your assertions as close as possible to the actual state transition you care about, and to make the invisible visible. Three techniques do most of the work.

1. Intercept and assert on the network layer directly.

const [response] = await Promise.all([
  page.waitForResponse(res => res.url().includes('/api/save') && res.status() === 200),
  page.click('button[data-testid="save"]'),
]);

const body = await response.json();
expect(body.status).toBe('persisted');

await expect(page.locator('.toast-success')).toBeVisible();

Now you're asserting on the actual server contract, and the UI check becomes a secondary confirmation rather than your only source of truth. If the toast is flaky but the save isn't, your failure output tells you that immediately — no guessing.

2. Replace polling assertions with expect.poll for custom conditions.

Web-first assertions cover DOM state well, but for anything derived — a computed total, a websocket-driven counter, an eventually-consistent read — use expect.poll, which gives you the same retry-until-timeout semantics for arbitrary logic:

await expect.poll(async () => {
  const res = await page.request.get('/api/orders/123');
  return (await res.json()).status;
}, { timeout: 10_000 }).toBe('confirmed');

This is strictly better than waitForTimeout because it fails fast on success and only consumes the full timeout when something is genuinely wrong.

3. Seed and tear down state through the API, never through the UI.

A huge, underrated source of "flaky" data non-determinism is tests that create their own fixtures by clicking through the app. That's two problems layered on top of each other: your setup is now also subject to the Determinism Gap, and your test's actual coverage gets diluted by incidental UI traversal. Use page.request (or a direct DB/API call in test.beforeEach) to seed state instantly and deterministically, and reserve UI interaction for the behavior you're actually testing.

Making flakiness measurable: a five-minute telemetry layer

You can't fix what you don't measure, and "it fails sometimes" is not a measurement. Playwright's JSON reporter gives you everything you need to turn flakiness from a vibe into a number.

// playwright.config.ts
export default defineConfig({
  reporter: [['json', { outputFile: 'results.json' }], ['html']],
  retries: process.env.CI ? 2 : 0,
});

Then, in CI, append each run's result to a rolling log and compute a per-test flake score — the ratio of (passed-after-retry) to (total runs) over your last N builds:

// flake-score.js
import fs from 'fs';

const results = JSON.parse(fs.readFileSync('results.json', 'utf-8'));

const flaky = results.suites
  .flatMap(s => s.specs)
  .filter(spec => spec.tests.some(t => t.results.length > 1 && t.status === 'expected'));

flaky.forEach(spec => {
  console.log(`⚠️  FLAKE: ${spec.title} — passed only after retry`);
});

Wire this into a simple dashboard or even a Slack message on merge, and you get something most teams never have: a ranked list of which tests are lying to you the most, ordered by evidence instead of by whoever complained loudest in standup this week. That list is where you spend your Determinism Gap–closing effort — not uniformly across the suite, but surgically, on the 5% of tests responsible for 80% of your CI noise.

The cultural fix is as important as the technical one

Retries have a real, legitimate place — they absorb genuine infrastructure jitter you don't control, like a CI runner having a bad five seconds. But retries used as a first response, rather than a last resort, do something insidious: they launder a real signal into noise. A test that "passes on retry" isn't passing — it's reporting that your Determinism Gap exists and is currently narrow enough not to bite you this time. Auto-retry-and-move-on treats that report as a false alarm instead of what it is: a leading indicator of a production race condition your users will eventually hit too, minus Playwright's generous timeouts.

The teams with the most reliable suites I've seen don't have less flakiness by accident — they have a policy: any test requiring a retry gets logged, and any test crossing a flake-score threshold gets quarantined into a separate, non-blocking job rather than silently re-run inside the main pipeline. Quarantine keeps the signal (your main suite stays trustworthy) while giving engineers room to actually fix the gap instead of being pressured to ship around it.

The takeaway

"Flaky" is a word we use when we haven't yet located the boundary between what our test asserted and what actually happened. Playwright gives you excellent primitives — web-first assertions, expect.poll, network interception, API-based seeding — for closing that boundary precisely. But the primitives only help if you treat flakiness as a measurement problem first and a tooling problem second.

The next time a test flakes, resist the retry reflex for thirty seconds and ask: what was still in motion when I asserted? That question, asked consistently, is the actual difference between a test suite people trust and one people route around.

---

If you're building out a Playwright suite and want a companion piece, I previously wrote about structuring data-driven tests with Excel and how Playwright compares to Selenium for modern automation — this post is meant to sit alongside those as the "why your suite still isn't reliable even after switching frameworks" piece.