From Flaky Scripts to Stable Coverage: A Practical Reliability Playbook

Keith Haag — Fri, 14 Aug 2026 15:31:47 +0000

Originally published on the QA Guardian blog.

A flaky test is one of the most expensive items in a software organization, and one of the most underestimated. On the surface it's an annoyance: fails ten percent of the time for no apparent reason, someone re-runs CI, it passes, they ship.

This is a practical playbook: diagnose root causes, measure flakiness, prioritize fixes, write tests that are deterministic by design.

The Four Root Causes

Timing dependencies. The test interacts before the app is ready. The common "fix," waitForTimeout(2000), slows the suite and leaves the race intact. Wait for a specific condition instead.

Selector fragility. Auto-generated class names, nth-child(3), or copy marketing changes without a code review. The UI refactor breaks the test even though the feature still works.

State contamination. Shared sessions, leftover DB rows, in-memory state. One test poisons another — but only sometimes, depending on order or which parallel worker runs it.

Environment variance. Assumptions about latency, data, or CPU that hold locally and fail on CI (or the reverse).

What Stable Tests Have in Common

Semantic selectors. getByRole('button', { name: 'Place Order' }) targets what the user sees — it breaks when the product label changes (a real decision), not when a CSS module renames a class.

Condition-based waiting. Replace every waitForTimeout with a real condition: waitForResponse, a locator assertion, or Playwright's auto-waiting via expect(locator).toBeVisible().

Isolated browser contexts. Fresh context per test — no shared cookies, storage, or session bleed across workers. Required for safe parallel runs, not optional polish.

Bounded scope. A twenty-step flow has twenty places for environmental noise; a three-hundred-step monolith has three hundred. Scope to one journey.

Build a Flakiness Matrix

You can't fix what you don't measure. Track pass rate per test over a rolling thirty-day window. A test that fails two percent of the time looks fine on any given day and still burns hours and trust over a month.

Four columns: test name, thirty-day failure rate, journey criticality, owner. Triage anything below ninety-eight percent. Most teams find ~80% of flakes live in ~20% of tests — usually the oldest, most fragmented, least-owned ones.

Prioritize Fixes

Rank by journey criticality first: flaky checkout is urgent, flaky settings can wait a sprint. Then by false-positive risk — a test that goes green when the feature is broken is worse than one that fails loudly.

When you rewrite, prefer one end-to-end flow over re-stabilizing a pile of fragments.

Your First Week

Export thirty days of CI results; sort by failure rate; pick the top ten offenders touching critical journeys.
For each, label the root cause (timing, selector, state, environment) before changing code.
Delete or fold presence-only checks; rewrite the journey as one isolated flow with semantic locators and condition waits.
Re-measure for two weeks. Still under 98%? The diagnosis was wrong — dig again, don't add another timeout.

The Goal: Deterministic by Design

The best flake fix is a test that never had room to flake: real app, sequential journey, semantic selectors, isolated context, waits on real conditions. When that test fails, something in the product changed — that's the signal the suite exists to provide.

Playwright vs. Selenium in 2026: What the Data Actually Shows

Keith Haag — Fri, 14 Aug 2026 15:31:22 +0000

Originally published on the QA Guardian blog.

Selenium has been the de facto standard for browser automation since 2004. Playwright arrived in 2020 and quickly became the tool of choice among QA engineers who actually write tests every day. We spent the last quarter running both frameworks on real customer test suites and documenting the results. Here's what we found.

The Core Architectural Difference

Selenium operates through the W3C WebDriver protocol: your test code talks to a driver binary (ChromeDriver, GeckoDriver), which talks to the browser. Each command is a round trip over HTTP, introducing latency at every step.

Playwright communicates directly with each browser using low-level browser-native protocols — the Chrome DevTools Protocol for Chromium, and purpose-built equivalents for Firefox and WebKit. There's no intermediary driver binary. Commands execute at near-native speed, with direct access to browser internals the WebDriver spec doesn't expose.

What the Numbers Show

Across 120 test specs on identical CI infrastructure:

Execution time: Playwright averaged 41% faster per-test. The gap widened on network interception and multi-tab flows.
Flake rate: Playwright's built-in auto-waiting cut timing-related failures by 68% with zero extra code. Selenium needed explicit WebDriverWait calls throughout.
Setup time: Playwright in a fresh CI environment: under 5 minutes (npx playwright install). Selenium meant managing driver versions alongside browser versions — a maintenance burden every team we spoke to flagged.

Where Selenium Still Has an Edge

Selenium's ecosystem is vast. IE, legacy Edge, or obscure mobile browser versions through Selenium Grid or BrowserStack — still the only practical choice. For teams with large existing Selenium codebases and no immediate pain points, migration cost may not justify the gains yet. Playwright doesn't support IE and has limited Safari parity vs. Chrome/Firefox.

The Developer Experience Gap Is Real

page.getByRole() and page.getByTestId() encourage writing tests against semantics, not implementation details. expect(locator).toBeVisible() retries automatically until timeout. Codegen records a flow by clicking through the browser.

Playwright's Trace Viewer deserves a special mention: a full timeline on failure — every network request, DOM snapshot, console log, a video, a step-by-step replay. Diagnosing a Selenium failure in headless CI often means print statements and re-runs. Diagnosing Playwright means clicking through a zip file.

Our Recommendation

For any team starting a new E2E suite in 2026, Playwright is the clear choice. For teams migrating from Selenium: write new tests in Playwright, migrate existing ones during their next maintenance cycle rather than a big-bang rewrite that costs you coverage mid-transition.

All of QA Guardian's test infrastructure runs on Playwright, powered by our parallel execution infrastructure. We made that call early and haven't regretted it.

DEV Community: QA Guardian