DEV Community: QA Guardian

Why Your App Breaks When 100% of Your Tests Pass

Keith Arters — Mon, 17 Aug 2026 13:27:27 +0000

Your CI run is green. All 340 tests passed. Your team ships the release. Twenty minutes later, a customer emails to say they can't check out.

You look at the test suite. The cart tests passed. The payment form tests passed. The order confirmation test passed. Every individual piece of the checkout journey has a green checkmark next to it.

And yet checkout is broken.

This is not a hypothetical. It is the natural outcome of optimizing for test count instead of user journey coverage — and it happens to teams with good intentions, experienced engineers, and high coverage numbers every day.

Why Isolated Tests Can't Tell You If Your Product Works

An isolated test suite is one where individual tests each cover a slice of behavior: a button renders, a form validates, a page loads. Each test is technically correct. Run in isolation, each passes consistently.

The problem is that real users do not interact with slices. They complete user journeys — sequences that span multiple pages, state changes, and API calls. They navigate from a product page to a cart, from a cart to checkout, from checkout to a payment form, from a payment form to a confirmation screen. The behavior that matters is the sequence, not the individual steps. And that sequence is exactly what isolated tests never exercise.

Consider what happens when an API response changes the shape of cart data between the cart render step and the checkout step. Your cart test passes — it does not read the checkout step. Your checkout test passes — it mocks the cart data in setup. The integration between them is broken, and no test in your suite is watching it.

Fragmentation does not just miss integration bugs. It actively creates blind spots for the bugs that matter most.

Five Things Flows Do That Isolated Tests Cannot

The alternative to testing slices is testing complete user journeys. We call these flows — a single automated Playwright script that runs one user journey from entry to measurable outcome, with real browser interactions, real API calls, and real session state throughout. No mocking between steps.

1. A passing flow proves the feature works

When a flow navigates from product page to order confirmation using real browser interactions, real API calls, and a real session — without any mocking of intermediate steps — its result is definitive. If it passes, checkout works right now, in the environment where it ran.

No fragmented test suite can make that statement. A collection of green slices is evidence that the slices worked. It is not evidence that the feature works. The distinction is not subtle. It is the entire point of having tests.

2. Flows give you precise, actionable failures

When a flow fails, you know exactly where in the user journey it broke. The step name, the screenshot at the point of failure, the trace showing every network call and DOM state — all of it points to a single location in a single journey.

When fragmented tests fail, they produce a different problem: too much signal. A UI change that shifts a button's position in the checkout layout can break the cart render test, the cart total test, the address form test, the shipping selector test, and the payment test — simultaneously, for the same root cause. Developers spend their morning triaging five test failures that share one fix.

A flow fails once, in the right place, for the right reason.

3. Flows are dramatically cheaper to maintain

Maintenance cost scales with the number of tests touching a given piece of the UI. A checkout flow with twelve fragmented tests requires twelve updates when the checkout layout changes. Selectors, assertions, setup scripts — all duplicated, all brittle, all demanding attention for the same root cause.

One flow covering the same surface requires one update. Selector changes live in a single page object. The behavior assertion lives at the end of the journey, where it belongs. The maintenance surface is a fraction of what fragmentation produces.

For teams with hundreds of fragmented tests, the hidden cost is not the CI minutes — it is the ongoing engineer time spent keeping the suite from rotting. Flows eliminate most of that overhead by design.

4. Flows catch real integration bugs

The most damaging production bugs are not "the button doesn't render" bugs. They are state propagation failures, session edge cases, API contract mismatches between consecutive steps, and race conditions that only appear in realistic navigation sequences.

Fragmented tests miss all of these by construction — they each test a single slice with the rest of the world mocked out. A flow catches them because it runs the real journey. The session is real. The API calls are real. The state transitions between steps are real. Integration bugs have nowhere to hide.

5. Flows speak the same language as your product

A flow named guest-checkout-flow.spec.ts is immediately legible to everyone involved in shipping software — engineers, product managers, QA leads, and engineering leadership. Its result maps directly to a product question anyone can ask: "Does guest checkout work?"

A suite of 340 fragmented tests does not answer that question. It answers 340 narrower questions that nobody outside of QA has context to interpret. Coverage conversations become QA-only discussions instead of product conversations.

Flow-based coverage is business-legible by default. When an engineering lead needs to know what's tested before a release, the answer is a list of flows — not a test runner output that requires decoding.

The Compounding Problem

Isolated test suites do not stay manageable. They grow. Every sprint adds more slices. The coverage gaps widen because new isolated checks get added next to the old ones rather than filling in the missing end-to-end paths. By the time a team recognizes the problem, they are sitting on 500 tests with no clear picture of which user journeys — which flows — are actually verified.

The failure mode is gradual but predictable: CI run times inflate, flakiness accumulates, developers start treating red as noise, and the suite that was supposed to catch regressions before production has trained the team to merge anyway.

Flows do not have this trajectory. A suite of 50 focused flows covering 50 real user journeys stays at 50 meaningful tests. It does not drift into entropy because there is no natural pressure to add fragments. Either a journey is covered end-to-end, or it is not.

What to Do With an Existing Isolated Suite

Rewriting hundreds of tests is not a realistic starting point. The practical approach is to start at the critical path.

Identify the three or four user journeys — the flows — that would generate a customer complaint within an hour of breaking. For most products, that list is short: login, checkout, the core action that creates business value, and password recovery. Write a single flow for each. Run them on every commit. Watch what they catch that the existing suite misses.

After the first wave, audit which fragmented tests cover the same journeys as your new flows. Most of those fragments can be deleted — they are now redundant and weaker. You have not lost coverage. You have improved it while shrinking the maintenance surface.

Expand from there by adding flows for the next tier of critical journeys. The fragmented tests that have no corresponding flow can be evaluated individually: do they cover something the flow misses, or are they checking rendering details that have no bearing on whether the feature works? Most of them are the latter.

The Standard Worth Holding

The purpose of a test suite is not to produce a large number of passing checks. It is to give your team confidence that the product works before it reaches users.

Fragmented tests can pass comprehensively while that confidence is completely unjustified — as the customer who cannot check out will tell you. Flows tie the test result directly to the outcome that matters. If the flow passes, the journey works. If it fails, something real is broken and you know exactly where.

That is the only standard worth building a test suite around.

Originally published on the QA Guardian blog.

What You Actually Own in Modern QA: Avoiding Vendor Lock-In

Keith Arters — Mon, 17 Aug 2026 13:24:33 +0000

Most SaaS QA tools have a clause buried somewhere in their value proposition: your tests only run here. Whether it's a proprietary scripting language, a closed-source selector engine, or a platform-only runner, the practical effect is the same. If you cancel, your tests stay. You start from scratch.

This isn't an accident. Lock-in through switching costs is easier to build than lock-in through genuine value. Knowing the difference is how engineering teams avoid a decision they'll regret in two years.

The Proprietary Language Trap

Some platforms require you to write tests in their own scripting DSL. It looks like code, but it compiles into something only their runner understands. You can't run it locally. You can't review it in a standard pull request. You can't grep through it the way you would a TypeScript file.

When you cancel, you have a library of files in a format nothing else reads. Your historical test coverage — two years of institutional knowledge about which flows are fragile, which selectors drift, which assertions catch real regressions — is gone. You're not migrating tests. You're rewriting them.

The Record-and-Replay Trap

Others sell no-code test creation via browser recording. Click through your app, export the test, done. This sounds compelling until you try to maintain it six months later.

Recorded tests encode the exact DOM state at the moment of recording: generated CSS class names, positional selectors, brittle XPaths. When your UI changes — and it will — the recording is invalid. Because the test was generated rather than written, there's no logic to update surgically. You re-record from scratch. Every. Time.

What Genuine Ownership Actually Requires

Ownership isn't about where the files sit day-to-day. It's about what you can do with them at any moment — and whether the code itself is yours to keep. Three things have to be true simultaneously.

Standard, non-proprietary code. Your tests are written in Playwright with TypeScript — not a wrapper, not a fork, not an abstraction layer. If you handed the files to a developer who had never heard of your vendor, they could read and run them without any explanation.
Instant, unconditional export. A dashboard that stores your tests for performance is fine — faster updates, faster self-healing, faster CI execution — but it should include a single-click export that packages every test as a standard Playwright project: directory structure, config file, spec files, everything. No support ticket. No waiting period. One click.
Zero-modification portability. The export runs immediately on GitHub Actions, CircleCI, a Kubernetes job, or a laptop with npx playwright test. No migration step. No vendor-specific runner. No changes to the files. The tests you built over two years of product development work exactly the same way outside the platform as they do inside it.

Why Centralized Storage Isn't the Same as Lock-In

Keeping your tests in a vendor's database isn't automatically a lock-in strategy — it can be a legitimate architecture decision, as long as the code format stays open.

When a selector breaks because the UI shipped a change, a self-healing system needs to update the test and re-verify it fast. That loop — detect, patch, validate, deploy — is significantly faster when the source of truth is centralized and the vendor controls the full pipeline. A PR-based workflow into your repository adds review cycles and merge queues to a process that needs to be automatic and immediate.

Centralized storage also means your full test history, failure traces, videos, and run analytics live in one place — accessible without any setup on your side. But the underlying code should remain standard Playwright TypeScript throughout. Storage location is an operational detail. Code format is the ownership guarantee.

The Three Questions to Ask Any QA Vendor

Is the test code written in a standard, open-source format? Proprietary DSLs and no-code recorders create formats only the vendor's platform understands. Standard Playwright TypeScript is readable and runnable anywhere.
Can I export all my tests right now, without asking? If the answer involves a support request, a data export form, or a waiting period — that's a red flag. Export should be self-serve and immediate.
What happens to my tests if I cancel today? You should be able to export a complete, runnable Playwright project and continue running it without modification. If the answer is "you'll need to re-create them" — you never owned them.

Your test suite is accumulated institutional knowledge about how your application behaves. It should never be held hostage by a vendor relationship — because a tool you can leave freely is a tool worth trusting.

Originally published on the QA Guardian blog.

Parallel Execution Without Chaos: Reducing CI Feedback Time for Engineering Teams

Keith Arters — Sat, 15 Aug 2026 16:28:23 +0000

Originally published on the QA Guardian blog.

The average engineering team running tests on GitHub Actions experiences something like this: a developer opens a pull request, CI triggers, tests run sequentially, forty-five minutes later a result appears. The developer has long since context-switched. They come back, see a failure, and try to remember what they were doing.

Sequential test execution is one of the most pervasive and underacknowledged sources of developer slowdown. And it is almost entirely unnecessary.

Why Sequential Testing Is the Default

Most CI platforms run tests sequentially out of the box. You configure a job, it runs your test command, tests execute one file at a time. This is the path of least resistance — and it scales terribly.

Going from twenty flows to forty doubles your CI time. Going from forty to a hundred makes CI unusable as a fast-feedback mechanism. At that point, developers stop treating CI results as a signal and start treating them as a gate to wait out. The test suite stops changing behavior.

How Parallel Execution Works

Parallel execution runs multiple test files simultaneously, each in its own isolated environment. Instead of forty flows running in sequence at roughly forty-five seconds each (thirty minutes total), ten workers each handle four flows simultaneously. Total time: three to four minutes.

The key requirement is isolation. Each parallel worker needs its own browser instance, its own test data seeded independently, and its own application state — no shared sessions or database records between workers. If your tests share state, parallelism will break them. This is actually a useful diagnostic: tests that cannot be parallelized almost always have a hidden state-sharing problem.

Implementation Approaches

Native Playwright sharding. Playwright supports --shard=1/5 arguments out of the box. Define a matrix strategy in GitHub Actions to run five parallel jobs, each handling a shard of the suite. This is the simplest approach and works well for most teams up to moderate scale.

Dedicated infrastructure. For larger suites or teams that need consistent, predictable performance, Kubernetes-based infrastructure spins up a fresh container per test file. This eliminates queue time — a common pain on shared CI runners — and delivers consistent sub-five-minute execution regardless of platform load.

The Math on Developer Time

Ten engineers opening three pull requests per day each is thirty CI runs per day. At forty-five minutes per run, that is twenty-two and a half hours of accumulated wait time — roughly two hours and fifteen minutes per engineer per day, every day.

Drop CI to four minutes and that becomes two hours total across the whole team — twelve minutes per engineer. The freed attention is significant. Engineers can review CI results before opening the next task. Feedback loops collapse from hours to minutes.

There is also a behavioral threshold at around four minutes: developers actually keep the CI tab open. When CI takes forty-five minutes, nobody waits. When it takes four, many do. That behavioral change is where the compounding value comes from.

What to Watch Out For

Database conflicts. Tests that write to a shared database will interfere with each other under parallelism. Use isolated database schemas per worker, or generate unique test data per run.

Port conflicts. If tests spin up local servers, multiple workers may collide on the same port. Target an already-running environment or use dynamic port allocation.

Rate limiting. Real external APIs may be hit in parallel far more aggressively than in sequential runs. Mock or stub third-party dependencies; reserve real API calls for dedicated integration runs.

The Target

For most applications with twenty to fifty critical flows, the benchmark is straightforward:

Sequential on shared CI: thirty to sixty minutes
Parallel with proper isolation: three to five minutes

Anything under five minutes means developers get feedback before they have fully switched context. That is the meaningful threshold — not just for efficiency, but for whether CI results actually influence developer behavior. A test suite nobody waits for is a safety net with holes.

If you'd like to see how Guardian runs your flows in parallel on dedicated infrastructure, book a demo and we'll walk through the execution model.

From Flaky Scripts to Stable Coverage: A Practical Reliability Playbook

Keith Arters — Fri, 14 Aug 2026 15:31:47 +0000

Originally published on the QA Guardian blog.

A flaky test is one of the most expensive items in a software organization, and one of the most underestimated. On the surface it's an annoyance: fails ten percent of the time for no apparent reason, someone re-runs CI, it passes, they ship.

This is a practical playbook: diagnose root causes, measure flakiness, prioritize fixes, write tests that are deterministic by design.

The Four Root Causes

Timing dependencies. The test interacts before the app is ready. The common "fix," waitForTimeout(2000), slows the suite and leaves the race intact. Wait for a specific condition instead.

Selector fragility. Auto-generated class names, nth-child(3), or copy marketing changes without a code review. The UI refactor breaks the test even though the feature still works.

State contamination. Shared sessions, leftover DB rows, in-memory state. One test poisons another — but only sometimes, depending on order or which parallel worker runs it.

Environment variance. Assumptions about latency, data, or CPU that hold locally and fail on CI (or the reverse).

What Stable Tests Have in Common

Semantic selectors. getByRole('button', { name: 'Place Order' }) targets what the user sees — it breaks when the product label changes (a real decision), not when a CSS module renames a class.

Condition-based waiting. Replace every waitForTimeout with a real condition: waitForResponse, a locator assertion, or Playwright's auto-waiting via expect(locator).toBeVisible().

Isolated browser contexts. Fresh context per test — no shared cookies, storage, or session bleed across workers. Required for safe parallel runs, not optional polish.

Bounded scope. A twenty-step flow has twenty places for environmental noise; a three-hundred-step monolith has three hundred. Scope to one journey.

Build a Flakiness Matrix

You can't fix what you don't measure. Track pass rate per test over a rolling thirty-day window. A test that fails two percent of the time looks fine on any given day and still burns hours and trust over a month.

Four columns: test name, thirty-day failure rate, journey criticality, owner. Triage anything below ninety-eight percent. Most teams find ~80% of flakes live in ~20% of tests — usually the oldest, most fragmented, least-owned ones.

Prioritize Fixes

Rank by journey criticality first: flaky checkout is urgent, flaky settings can wait a sprint. Then by false-positive risk — a test that goes green when the feature is broken is worse than one that fails loudly.

When you rewrite, prefer one end-to-end flow over re-stabilizing a pile of fragments.

Your First Week

Export thirty days of CI results; sort by failure rate; pick the top ten offenders touching critical journeys.
For each, label the root cause (timing, selector, state, environment) before changing code.
Delete or fold presence-only checks; rewrite the journey as one isolated flow with semantic locators and condition waits.
Re-measure for two weeks. Still under 98%? The diagnosis was wrong — dig again, don't add another timeout.

The Goal: Deterministic by Design

The best flake fix is a test that never had room to flake: real app, sequential journey, semantic selectors, isolated context, waits on real conditions. When that test fails, something in the product changed — that's the signal the suite exists to provide.

Playwright vs. Selenium in 2026: What the Data Actually Shows

Keith Arters — Fri, 14 Aug 2026 15:31:22 +0000

Originally published on the QA Guardian blog.

Selenium has been the de facto standard for browser automation since 2004. Playwright arrived in 2020 and quickly became the tool of choice among QA engineers who actually write tests every day. We spent the last quarter running both frameworks on real customer test suites and documenting the results. Here's what we found.

The Core Architectural Difference

Selenium operates through the W3C WebDriver protocol: your test code talks to a driver binary (ChromeDriver, GeckoDriver), which talks to the browser. Each command is a round trip over HTTP, introducing latency at every step.

Playwright communicates directly with each browser using low-level browser-native protocols — the Chrome DevTools Protocol for Chromium, and purpose-built equivalents for Firefox and WebKit. There's no intermediary driver binary. Commands execute at near-native speed, with direct access to browser internals the WebDriver spec doesn't expose.

What the Numbers Show

Across 120 test specs on identical CI infrastructure:

Execution time: Playwright averaged 41% faster per-test. The gap widened on network interception and multi-tab flows.
Flake rate: Playwright's built-in auto-waiting cut timing-related failures by 68% with zero extra code. Selenium needed explicit WebDriverWait calls throughout.
Setup time: Playwright in a fresh CI environment: under 5 minutes (npx playwright install). Selenium meant managing driver versions alongside browser versions — a maintenance burden every team we spoke to flagged.

Where Selenium Still Has an Edge

Selenium's ecosystem is vast. IE, legacy Edge, or obscure mobile browser versions through Selenium Grid or BrowserStack — still the only practical choice. For teams with large existing Selenium codebases and no immediate pain points, migration cost may not justify the gains yet. Playwright doesn't support IE and has limited Safari parity vs. Chrome/Firefox.

The Developer Experience Gap Is Real

page.getByRole() and page.getByTestId() encourage writing tests against semantics, not implementation details. expect(locator).toBeVisible() retries automatically until timeout. Codegen records a flow by clicking through the browser.

Playwright's Trace Viewer deserves a special mention: a full timeline on failure — every network request, DOM snapshot, console log, a video, a step-by-step replay. Diagnosing a Selenium failure in headless CI often means print statements and re-runs. Diagnosing Playwright means clicking through a zip file.

Our Recommendation

For any team starting a new E2E suite in 2026, Playwright is the clear choice. For teams migrating from Selenium: write new tests in Playwright, migrate existing ones during their next maintenance cycle rather than a big-bang rewrite that costs you coverage mid-transition.

All of QA Guardian's test infrastructure runs on Playwright, powered by our parallel execution infrastructure. We made that call early and haven't regretted it.