DEV Community: Kate Sai kishore

I built QAFind — a red Playwright build should mean a real bug

Kate Sai kishore — Wed, 11 Nov 2026 20:53:17 +0000

`If your team clicks Re-run failed jobs more than once a week, your end-to-end suite has stopped telling you the truth. Red no longer means "you broke something". It means "something was slow, probably".

I built QAFind to fix that. It's an open-source (MIT) CLI that attaches to the Playwright suite you already run. You don't rewrite specs, adopt a new runner or sign up for anything.

What it does, in one sentence each

attach wraps your existing playwright.config.ts. Your config and specs are read, never written.
verify-merge runs the specs a PR touches, reruns failures once, and fails CI only on a regression that reproduces.
flake aggregates runs and tells you which tests flake, the probable cause, how confident it is, and who owns them.
attach --determinism records and replays network traffic and freezes the browser clock, so fewer tests flake in the first place.
Signed evidence: every verdict can be signed (HMAC or Ed25519) and exported as an in-toto attestation you can verify offline.

60-second start

From the root of any Playwright project:

bash npx qafind attach . npx playwright test --config .qafind/playwright.qafind.config.ts npx qafind flake .qafind/reports --markdown

That's it. The first command writes a wrapper config into .qafind/. The second is your normal test run with one extra reporter. The third gives you a table like this:

text | Test | Runs | Flakes | Cause | Quarantine | | cart fixture returns totals | 3 | 1 | network_fixture | yes | | login selector exists | 3 | 1 | selector_drift | yes | | slow recommendations endpoint | 3 | 1 | timing | yes | | known checkout total mismatch | 3 | 3 | app_race | no |

Notice the last row: it failed every time, so it's a real failure and it is never suggested for quarantine.

The merge gate

Add one step to CI:

bash npx qafind verify-merge --run --base origin/main --head HEAD

First run	Rerun	Verdict	Blocks merge?
failed	failed	regression	yes
failed	passed	flake	no (but it's recorded)
failed	different failure	unreproduced	no

Flakes stop holding your PRs hostage. Real regressions still stop the merge. Add --markdown for a PR comment, or --intoto for a signed attestation.

Why not just use retries?

retries: 2 turns a flaky failure into a green exit code and throws away the information. Nobody counts how often a test needed a retry, and an intermittent bug your users will hit looks identical to a slow CI runner. QAFind keeps that information, counts it, and puts it in front of a human. Only the reproduced failures block.

What it isn't

I'd rather you trust it than be surprised by it:

Playwright only, for now.
Cause labels are suggestions, based on error text plus an optional request log. On a synthetic benchmark (400 tests × 5 runs) they're 87.8% accurate on text alone and 90.3% with the request log. Check before you quarantine.
WebSockets and service workers aren't recorded by the network replay yet.
Not a test generator. It makes the tests you already have more trustworthy.

Free for open source

The CLI is MIT and fully local: no account needed for attach, flake or verify-merge.

There's also an optional hosted GitHub App (QAFind CI) that runs QA on pull requests and comments the report on the PR. Public repositories are free with no daily quota.

Try it and tell me where it's wrong

bash npx qafind attach .

⭐ Repo: github.com/katesaikishore/QAFind
🌐 Site: qafind.si

If you run it on a real suite, the most useful thing you can send me is a test it labelled wrong. Open an issue or drop it in the comments.

What's the flakiest test in your suite right now, and what's its excuse?`

Your Playwright suite is lying to you (and retries are how)

Kate Sai kishore — Wed, 11 Nov 2026 20:49:12 +0000

`
Every team I've talked to about end-to-end tests has the same ritual.

The PR goes red. Someone opens the CI log, sees Timeout 30000ms exceeded, shrugs, and clicks Re-run failed jobs. It goes green. Merge.

Do that enough times and two things happen. First, people stop reading red builds at all. Second, the day a real regression shows up as a timeout, it gets the same shrug and the same re-run — and if it's intermittent, it merges.

The suite isn't failing. It's lying. A red check stops meaning "you broke something" and starts meaning "something, somewhere, was slow." I spent the last few months building tooling to make that verdict honest again, and this post is what I learned.

Retries hide the thing you need to see

Playwright's retries: 2 is the obvious fix, and it does make CI greener. But look at what it does to your signal:

A test that fails, then passes on retry, is reported as flaky in the HTML report — and passed by the exit code. CI only reads the exit code.
Nobody aggregates those flaky passes across runs. The test that has flaked on 30% of runs this month looks identical to the one that flaked once.
A real race condition your users will hit 1 in 5 times looks exactly like a flake. Retries launder it.

So you have two bad options: retries off (every flake blocks a merge, people learn to ignore red) or retries on (every intermittent bug ships, people never see red). What I actually wanted was a third rule:

Block the merge only on a failure you can reproduce. Everything else gets reported, counted and owned — but doesn't block.

Rule 1: rerun once, block only what reproduces

The core idea is boring on purpose. Run the specs a PR touches. If something fails, run those same specs again. Then:

First run	Rerun	Verdict	Blocks merge?
failed	failed	regression	yes
failed	passed	flake	no
failed	different failure	unreproduced	no

The important part is that a flake isn't forgotten — it's recorded with its error text and counted. It just doesn't get to hold your merge hostage.

I wired this into an open-source CLI called QAFind (MIT). In CI it's one command:

bash npx qafind verify-merge --run --base origin/main --head HEAD

--run picks the affected Playwright specs from the git diff, runs them with the JSON reporter, reruns the failures once, and exits 1 only if a regression reproduced. If you already produce Playwright JSON reports in your pipeline, you can hand it both reports instead (--results first.json --rerun-results rerun.json).

I tested it on a tiny demo suite where the same assertion fails in two specs. In one it fails twice; in the other it fails then passes:

text regression.spec.ts failed → failed 1 reproduced regression exit 1 flake.spec.ts failed → passed 1 flake, 0 regressions exit 0

Same error message. Different verdict. The only thing that changed the outcome was whether it reproduced — which is exactly the question a human asks before clicking re-run.

I also ran it against a third-party open-source Playwright suite I don't own (a page-object example that drives a public demo shop). Its add-to-cart spec failed on both runs, so the verdict was "reproduced regression, merge blocked." No special setup for that repo — which brings me to the second rule.

Rule 2: don't make me rewrite my specs

Every testing tool I evaluated wanted me to adopt its runner, its DSL or its cloud. I have hundreds of specs. I'm not porting them.

So the attach step wraps your config instead of replacing it:

bash npx qafind attach . npx playwright test --config .qafind/playwright.qafind.config.ts

attach generates a wrapper config that imports your playwright.config.ts, keeps your reporters, and appends one more reporter that writes a small JSON manifest per run. Your original config and specs are read, never written. In CI you change one flag.

The fiddly part, if you ever try this yourself: Playwright resolves paths relative to the config file that was loaded. Put the wrapper in a subfolder and suddenly testDir, outputDir, globalSetup, storageState and webServer.cwd all point to the wrong place and Playwright cheerfully finds zero tests. The wrapper has to rebase every one of those. That one cost me a weekend.

Rule 3: when it flakes, say why — and say how sure you are

Once each run leaves a manifest, you can aggregate across runs:

bash npx qafind flake .qafind/reports --markdown

Look at the last row. It failed on every run. That's not a flake — it's a stable failure, so it's never recommended for quarantine. A tool that auto-quarantines anything red is just a slower way to delete your tests.

I want to be upfront about how the "Cause" column works, because this is where tools overclaim. It's deterministic matching on the recorded error text: net::ERR_ → network, a strict-mode locator violation → selector drift, a bare timeout → timing, an expect mismatch → app race. If you run with --determinism, it also records a per-test request ledger, which can confirm a network cause (a transport failure was really recorded) or contradict it (all requests succeeded, so it wasn't the network).

On a seeded synthetic benchmark of 400 tests × 5 runs, that's 87.8% accurate on error text alone and 90.3% with the request ledger. Synthetic means I know the right answer by construction — it measures the classifier, not your suite. It's a recommendation for a human, not proof of causality. Please verify before you quarantine anything.

Bonus: make the flakes go away instead of labelling them

Labelling flakes is good. Not having them is better. The two biggest sources I see are the network and the clock, so attach --determinism can:

record every HTTP request/response per test, then replay them byte-for-byte on later runs — and fail loudly on a missing fixture rather than silently hitting the network;
freeze the browser clock before navigation, so "3 minutes ago" and end-of-month date logic stop flipping.

Again with no spec changes: a generated tsconfig path alias points @playwright/test at a shim that adds auto-use fixtures. Your import { test } from '@playwright/test' keeps working. (Limits: WebSockets and service workers aren't recorded yet.)

What I'd tell my past self

The exit code is the product. Your HTML report can be beautiful; CI only reads one bit. Decide precisely what flips it.
"Passed on retry" is data, not success. Keep it, count it, give it an owner.
Reproduction is the cheapest honest test of a failure. One rerun answers most "is this real?" questions before a human looks.
Never quarantine a test that fails every time. That's a bug report.
Say how sure you are. A cause label with no confidence attached will eventually be wrong in a way that costs someone a day.

Try it

The CLI is MIT and runs entirely locally — no account needed for attach, flake or verify-merge:

bash npx qafind attach . npx playwright test --config .qafind/playwright.qafind.config.ts npx qafind flake .qafind/reports --markdown

Repo: github.com/katesaikishore/QAFind. It's Playwright-only today. If you run it on your suite, I'd love to hear where the cause labels are wrong — real-suite misclassifications are the most useful bug reports I can get.

How does your team handle flaky e2e tests today — retries, quarantine, or just vibes? Tell me in the comments.`