DEV Community: Ankit Aloni

The Determinism Gap: Why Your Playwright Tests Lie to You (And a Systems-Engineering Fix)

Ankit Aloni — Thu, 13 Aug 2026 17:51:30 +0000

Every automation engineer has said this sentence out loud: "It's flaky, just re-run it."

That sentence is doing more damage to your test suite than any bug in your application code ever could. It reframes a measurable engineering problem as an act of God — something to be appeased with retries and shrugged off in standup — instead of what it actually is: a race condition between your assertions and reality, one you can model, measure, and largely eliminate.

This article is about that race condition. I'm calling it the Determinism Gap — the window of time in which your test asserts something about application state while that state is still in motion. Everything that looks like "flakiness" lives inside this gap.

Flakiness is not randomness — it's unmeasured latency

When people say a test is "flaky," they usually mean it passes and fails on identical input with no code change. But nothing about a computer running deterministic code is actually random. What's really happening is that your test is racing against one or more of these:

Network non-determinism — an API response arrives in 80ms on your laptop and 340ms in CI
Rendering non-determinism — a React re-render, a CSS transition, a lazy-loaded chunk
Data non-determinism — a shared test database where another test's teardown hasn't finished
Scheduler non-determinism — CI runners under CPU contention, throttling event loops unpredictably

Your test doesn't fail because the universe rolled dice. It fails because you asserted at time T, and the thing you were asserting about didn't finish changing until T + Δ. The bug isn't in your app. It's in your model of time.

This matters because it changes the fix. If flakiness is randomness, the only rational response is retries. If flakiness is an unmodeled race condition, the rational response is to make the race visible and then close it — which is a fundamentally different (and much more durable) engineering activity.

Why Playwright's auto-waiting doesn't fully save you

Playwright is better at this than Selenium specifically because it ships with actionability checks — it won't click an element until it's visible, stable, and receives events. This eliminates a huge class of the old Thread.sleep(2000) flakiness that plagued Selenium suites for a decade.

But auto-waiting only solves the interaction side of the Determinism Gap. It says nothing about the assertion side. Consider:

await page.click('button[data-testid="save"]');
await expect(page.locator('.toast-success')).toBeVisible();

This looks safe — toBeVisible() is a web-first assertion, it polls. But it's polling for the symptom (a toast appearing), not the cause (the save actually completing on the server). If your toast renders optimistically before the network request resolves, your test can pass while the save silently fails. You haven't removed the gap. You've just moved it somewhere your assertion can't see it.

This is the trap: a passing test tells you your assertion was satisfied, not that your intent was fulfilled. Those are only the same thing if you've deliberately made them the same thing.

Closing the gap: assert on causes, not symptoms

The fix is to move your assertions as close as possible to the actual state transition you care about, and to make the invisible visible. Three techniques do most of the work.

1. Intercept and assert on the network layer directly.

const [response] = await Promise.all([
  page.waitForResponse(res => res.url().includes('/api/save') && res.status() === 200),
  page.click('button[data-testid="save"]'),
]);

const body = await response.json();
expect(body.status).toBe('persisted');

await expect(page.locator('.toast-success')).toBeVisible();

Now you're asserting on the actual server contract, and the UI check becomes a secondary confirmation rather than your only source of truth. If the toast is flaky but the save isn't, your failure output tells you that immediately — no guessing.

2. Replace polling assertions with expect.poll for custom conditions.

Web-first assertions cover DOM state well, but for anything derived — a computed total, a websocket-driven counter, an eventually-consistent read — use expect.poll, which gives you the same retry-until-timeout semantics for arbitrary logic:

await expect.poll(async () => {
  const res = await page.request.get('/api/orders/123');
  return (await res.json()).status;
}, { timeout: 10_000 }).toBe('confirmed');

This is strictly better than waitForTimeout because it fails fast on success and only consumes the full timeout when something is genuinely wrong.

3. Seed and tear down state through the API, never through the UI.

A huge, underrated source of "flaky" data non-determinism is tests that create their own fixtures by clicking through the app. That's two problems layered on top of each other: your setup is now also subject to the Determinism Gap, and your test's actual coverage gets diluted by incidental UI traversal. Use page.request (or a direct DB/API call in test.beforeEach) to seed state instantly and deterministically, and reserve UI interaction for the behavior you're actually testing.

Making flakiness measurable: a five-minute telemetry layer

You can't fix what you don't measure, and "it fails sometimes" is not a measurement. Playwright's JSON reporter gives you everything you need to turn flakiness from a vibe into a number.

// playwright.config.ts
export default defineConfig({
  reporter: [['json', { outputFile: 'results.json' }], ['html']],
  retries: process.env.CI ? 2 : 0,
});

Then, in CI, append each run's result to a rolling log and compute a per-test flake score — the ratio of (passed-after-retry) to (total runs) over your last N builds:

// flake-score.js
import fs from 'fs';

const results = JSON.parse(fs.readFileSync('results.json', 'utf-8'));

const flaky = results.suites
  .flatMap(s => s.specs)
  .filter(spec => spec.tests.some(t => t.results.length > 1 && t.status === 'expected'));

flaky.forEach(spec => {
  console.log(`⚠️  FLAKE: ${spec.title} — passed only after retry`);
});

Wire this into a simple dashboard or even a Slack message on merge, and you get something most teams never have: a ranked list of which tests are lying to you the most, ordered by evidence instead of by whoever complained loudest in standup this week. That list is where you spend your Determinism Gap–closing effort — not uniformly across the suite, but surgically, on the 5% of tests responsible for 80% of your CI noise.

The cultural fix is as important as the technical one

Retries have a real, legitimate place — they absorb genuine infrastructure jitter you don't control, like a CI runner having a bad five seconds. But retries used as a first response, rather than a last resort, do something insidious: they launder a real signal into noise. A test that "passes on retry" isn't passing — it's reporting that your Determinism Gap exists and is currently narrow enough not to bite you this time. Auto-retry-and-move-on treats that report as a false alarm instead of what it is: a leading indicator of a production race condition your users will eventually hit too, minus Playwright's generous timeouts.

The teams with the most reliable suites I've seen don't have less flakiness by accident — they have a policy: any test requiring a retry gets logged, and any test crossing a flake-score threshold gets quarantined into a separate, non-blocking job rather than silently re-run inside the main pipeline. Quarantine keeps the signal (your main suite stays trustworthy) while giving engineers room to actually fix the gap instead of being pressured to ship around it.

The takeaway

"Flaky" is a word we use when we haven't yet located the boundary between what our test asserted and what actually happened. Playwright gives you excellent primitives — web-first assertions, expect.poll, network interception, API-based seeding — for closing that boundary precisely. But the primitives only help if you treat flakiness as a measurement problem first and a tooling problem second.

The next time a test flakes, resist the retry reflex for thirty seconds and ask: what was still in motion when I asserted? That question, asked consistently, is the actual difference between a test suite people trust and one people route around.

If you're building out a Playwright suite and want a companion piece, I previously wrote about structuring data-driven tests with Excel and how Playwright compares to Selenium for modern automation — this post is meant to sit alongside those as the "why your suite still isn't reliable even after switching frameworks" piece.

🚀Playwright vs Selenium in 2026: The Ultimate Guide for Modern Test Automation

Ankit Aloni — Sun, 05 Apr 2026 20:11:17 +0000

⚡ Why This Decision Matters More Than You Think

Most engineers don’t struggle with writing tests.

They struggle with trusting them.

❌ Test passes locally → fails in CI

❌ Pipeline fails → rerun → magically passes

❌ Thread.sleep() becomes your best friend

That’s not bad luck.

👉 That’s a tooling mismatch.

In 2026, choosing between Playwright vs Selenium is not just technical.

It’s a decision about:

⚡ Speed
🧠 Reliability
🏗️ Engineering culture

🧭 Two Generations of Testing Philosophy

🧱 Selenium: The Foundation

Born in 2004
W3C WebDriver standard
Enterprise adoption

Built for:

Server-rendered apps
Predictable DOM
Simpler JS

⚡ Playwright: The Modern Approach

Built for:

SPA apps
Async-heavy UI
Dynamic DOM

Key idea:

“Don’t just act on the DOM — understand its state.”

⚡ Core Difference (This Changes Everything)

Selenium	Playwright
Command-based	State-aware
External control	Internal awareness
Manual waits	Auto sync

👉 This is why Playwright feels faster, cleaner, and more reliable.

⚙️ Architecture (Where Tests Actually Break)

Selenium

[Test] → HTTP → WebDriver → Browser

Every action = network call
Slower
More fragile

WebElement btn = driver.findElement(By.id("submit"));
btn.click();

Playwright

[Test] → WebSocket/CDP → Browser

Persistent connection
Event-driven
Faster

await page.locator('#submit').click();

🔍 Why This Matters in Real Life

❌ Selenium Approach

Thread.sleep(2000);

✅ Playwright Approach

await page.locator('#submit').click();

Playwright automatically waits for:

✔️ Visibility
✔️ Stability
✔️ Interactivity

💡 Pro Tip:

If your tests need retries to pass, your architecture is broken.

🧑‍💻 Developer Experience (DX)

⚡ Setup

Playwright → Ready in minutes
Selenium → Setup heavy

🧪 Assertions

// Playwright
await expect(page.locator('.alert')).toHaveText('Success');

// Selenium
WebElement alert = driver.findElement(By.className("alert"));
assertEquals("Success", alert.getText());

👉 Playwright = cleaner + smarter assertions

⏳ Wait Handling

Selenium	Playwright
Explicit waits	Built-in waits
Manual logic	Smart defaults

🐞 Debugging

Playwright

Trace viewer
Video
Network logs

Selenium

Logs
Screenshots

📊 Feature Comparison

Feature	Playwright	Selenium
Auto-waiting	✅ Smart	❌ Manual
Parallel execution	✅ Built-in	⚠️ Grid
Network mocking	✅ Native	❌ Limited
Shadow DOM	✅ Easy	⚠️ Complex
Visual testing	✅ Built-in	❌ External

🔬 Performance Reality

Metric	Playwright	Selenium
Speed	⚡ Fast	🐢 Slower
Flakiness	Low (~3%)	Higher (~15%)
Maintenance	Low	Medium–High

🚀 CI/CD Simplicity

Playwright

- run: npm ci
- run: npx playwright install
- run: npx playwright test

Selenium

services:
  selenium:
    image: selenium/standalone-chrome
- run: mvn test

👉 More infra = more problems

🐞 Debugging Difference

npx playwright test --trace on

👉 Replay entire test (DOM + network + screenshots)

🔁 Real Code Comparison

Playwright

await page.goto('/login');
await page.fill('#user', 'admin');
await page.fill('#pass', '1234');
await page.click('#login');
await expect(page).toHaveURL('/dashboard');

Selenium

driver.get("/login");
driver.findElement(By.id("user")).sendKeys("admin");
driver.findElement(By.id("pass")).sendKeys("1234");
driver.findElement(By.id("login")).click();

new WebDriverWait(driver, Duration.ofSeconds(10))
    .until(ExpectedConditions.urlContains("/dashboard"));

🎯 When to Use What

✅ Choose Playwright if:

New project
Need speed + stability
CI-first setup

✅ Choose Selenium if:

Large existing suite
Java ecosystem
Migration cost is high

🧠 Decision Flow

New project?
 ├── Yes → Playwright
 └── No
      ├── Existing Selenium?
      │     ├── Yes → Stay / gradual migration
      │     └── No → Playwright

🏁 Final Verdict

🚀 Playwright → Modern, fast, reliable
🧱 Selenium → Stable, enterprise-ready

👉 No universal winner — only the right context.

🚀 What You Should Do Today

Install Playwright
Convert 2–3 tests
Compare:
- Speed
- Stability
- Debugging

💬 Final Thought

A test suite is not valuable because it exists.

It’s valuable because you trust it.

If you don’t trust your tests…

you don’t have automation.

You have noise.

How to Build a Playwright Framework with Excel Data-Driven Testing

Ankit Aloni — Fri, 20 Mar 2026 20:22:20 +0000

🚀 How to Build a Playwright Framework with Excel Data-Driven Testing

Modern test automation requires scalability, reusability, and flexibility.
In this guide, we’ll build a Playwright automation framework that supports:

✅ Data-driven testing using Excel (ExcelJS)
✅ Clean Page Object Model (POM) structure

🧠 Why Data-Driven Testing?

Instead of hardcoding test data, we store it in an Excel file and dynamically run tests with multiple inputs.

📌 Use Cases:

Login testing with multiple users
Form validation
Regression testing
API + UI test combinations

⚙️ Tech Stack

Playwright (E2E automation)
Node.js
ExcelJS (Excel handling)

📂 Project Structure

PlayWrightAutomation/
│
├── tests/                # Test cases
├── pages/                # Page Object Models
├── utils/                # Excel utilities
├── test-data/            # Excel test data
├── playwright.config.js
├── package.json

📊 Step 1: Create Excel Test Data

Example (testData.xlsx):

username	password
user1	pass1
user2	pass2

🔧 Step 2: Read Excel Data using ExcelJS

Create utils/excelUtil.js:

const ExcelJS = require("exceljs");

async function getExcelData(sheetName) {
    const workbook = new ExcelJS.Workbook();
    await workbook.xlsx.readFile("./test-data/testData.xlsx");

    const worksheet = workbook.getWorksheet(sheetName);
    const data = [];

    let headers = [];

    worksheet.eachRow((row, rowNumber) => {
        if (rowNumber === 1) {
            headers = row.values;
        } else {
            let rowData = {};
            row.eachCell((cell, colNumber) => {
                rowData[headers[colNumber]] = cell.value;
            });
            data.push(rowData);
        }
    });

    return data;
}

module.exports = { getExcelData };

🧪 Step 3: Use Data in Playwright Tests

const { test } = require('@playwright/test');
const { getExcelData } = require('../utils/excelUtil');

test.describe("Data Driven Tests", () => {

    let testData;

    test.beforeAll(async () => {
        testData = await getExcelData("Sheet1");
    });

    for (const data of testData) {
        test(`Login Test - ${data.username}`, async ({ page }) => {
            await page.goto("https://example.com");

            console.log("Testing with:", data.username, data.password);
        });
    }
});

⚡ Step 4: Run the Tests

npm install
npx playwright test

📊 Step 5: View HTML Report

npx playwright show-report

📌 Report Includes:

Passed / Failed tests
Execution time
Logs

⚙️ Advanced Configuration (Coming Soon 🚧)

To make this framework production-ready, upcoming features include:

📱 Mobile device emulation (viewport)
🎥 Screenshots, videos & trace debugging
🔐 SSL handling & environment configs
🤖 AI-assisted debugging

👉 These will upgrade the framework to a production-grade solution

🔄 CI/CD Integration (Coming Soon 🚧)

GitHub Actions setup
Automated test execution on push
Report publishing

🔥 Key Benefits

Scalable test design
Easy test data management
Reusable POM structure
Real-world automation ready

🔗 Full Source Code

👉 https://github.com/ankitaloni369/PlayWrightAutomation

🎯 Conclusion

Combining Playwright with Excel-based data-driven testing enables you to build a powerful, scalable automation framework.

⭐ Support

If you found this useful, consider giving the repo a ⭐ and sharing it!