DEV Community: Gary Parker

Playwright 1.65: A Filter That Matches Nothing Now Fails Every Shard

Gary Parker — Sat, 14 Nov 2026 11:34:19 +0000

Playwright 1.65 has one line in its notable changes that reads like a small tidy-up: a sharded run now fails with "No tests found" when no tests match across all shards. If your CI builds a --grep or a path filter per pull request and fans it out over a shard matrix, that line decides whether a pull request that touches no tests is green or red.

I ran the same suite on 1.64.1 and 1.65.1 to see exactly which cases changed. The harness and raw results are in the PoC repo.

The setup

32 spec files with one test each. The tests use no browser.
fullyParallel: true and workers: 1, so Playwright can put each test on any shard.
@playwright/test 1.64.1 and 1.65.1, on Node 24 and macOS.
0, 1, 2, 4, 8 and 16 selected tests, picked by tag.
2, 4, 8 and 16 shards. Every shard is its own process, as it is in a CI matrix.
Each cell with and without --pass-with-no-tests, with Playwright's own split and with playwright-shard-balancer in the config.

That is 1,464 runs. I recorded the exit code of every run and the number of tests it ran. A second script runs four real filters that can match nothing.

The numbers

Zero selected tests, failed shards out of all shards:

Version	`--pass-with-no-tests`	Unsharded	2 shards	4 shards	8 shards	16 shards
1.64.1	no	1/1	0/2	0/4	0/8	0/16
1.64.1	yes	0/1	0/2	0/4	0/8	0/16
1.65.1	no	1/1	2/2	4/4	8/8	16/16
1.65.1	yes	0/1	0/2	0/4	0/8	0/16

On 1.64.1, sharding hid the error. The unsharded run failed with "No tests found" and the same filter with --shard passed on every shard. On 1.65.1 the sharded run agrees with the unsharded one: every shard fails.

No other exit code changed. A job here is one cell of the matrix with all of its shards. With 1 to 16 selected tests, all 180 jobs had 0 failed shards on both versions, even when 15 of 16 shards had no test to run. An empty shard is still fine. A whole run with no tests is not.

Which filters hit it

The same check with four filters, as shards 1/2 and 2/2, without the flag:

Filter	1.64.1 unsharded	1.64.1 sharded	1.65.1 unsharded	1.65.1 sharded
`--grep` matches nothing	fails	0/2 fail	fails	2/2 fail
Path filter matches nothing	fails	0/2 fail	fails	2/2 fail
`--last-failed` after a green run	fails	0/2 fail	fails	0/2 fail
`--only-changed`, no test file changed	passes	0/2 fail	passes	0/2 fail

For --last-failed, the last run file from the green run was still on disk, as it is when CI restores it from a cache or an artifact.

Two results here are not in the release note:

--last-failed after a green run still passes when sharded on 1.65.1. It ran 0 tests and exited 0 on both shards. Unsharded, the same command exits 1 with "No tests found" on both versions. The new check did not apply to this flag. Without the last run file, --last-failed ran the whole suite, so this only happens when the file is restored.
--only-changed with no changed test file exited 0 in every cell, sharded or not, on both versions. It passed before the change, so it says nothing about the new rule.

The balancer

playwright-shard-balancer plans shards through Playwright's preprocess hook, so I checked whether a shard it leaves empty trips the new rule. It does not. Version 0.1.3 gave the same exit code as Playwright's split in every cell: all shards fail on a zero-test selection, and none fail when at least one test is selected. The balancer did plan differently: it left a different number of shards empty in 36 of the paired jobs.

What I'd change in CI

If a filter can legitimately match nothing, such as a docs-only pull request, add --pass-with-no-tests to the sharded command. It turned every cell green on both versions.
The flag cannot tell a docs-only pull request from a broken filter. Both select nothing and both pass. If an empty selection should never happen in your pipeline, leave the flag off: on 1.65 that mistake fails the build instead of passing on every shard.
Check your sharded jobs before you upgrade. A job that was green on 1.64 with no tests to run is red on 1.65.
Do not use the exit code of a sharded --last-failed job as proof that tests ran. With a restored last run file from a green run, it ran nothing and exited 0.

Originally published at https://garyparker.dev/blog/playwright-165-empty-shards

Hands-On with Playwright Test Agents: Planner, Generator, Healer — Measured

Gary Parker — Thu, 12 Nov 2026 14:13:41 +0000

Playwright ships official AI test agents — a planner that explores your app and writes a test plan, a generator that turns the plan into live-verified test code, and a healer that repairs broken tests against the running page. The vendor demos are impressive. What I couldn't find anywhere was a measured answer to the question that actually matters: would you keep the suite these agents produce?

So I measured it. Everything below comes from a public PoC repo you can rerun yourself.

The setup

Target: the-internet.herokuapp.com, the classic demo app. Playwright 1.62.0. Agents initialised with:

npx playwright init-agents --loop=claude

That generates three agent definitions plus a playwright-test MCP server config — the agents run inside Claude Code (VS Code, Codex and opencode loops are also supported).

The brief for both sides was identical: cover login success and failure, dynamic controls (checkbox removal, input enable), and dynamic loading (both examples). The comparison baseline is a 6-test suite written in one pass from knowledge of the app, without touching a browser — the way an engineer who knows an app writes tests. One honesty note: the baseline was authored by the orchestrating model, not a human, so treat the authoring-time comparison as indicative rather than a controlled study.

The results

	One-pass baseline	Agent-generated
Scenarios	6	7
Time to produce	minutes	planner 2m 44s + generator 43m 22s
Lines of test code	80	223
Assertions per test	~2.5	~9
Locators	role/label-first, 6 CSS ids	29 `getByRole`, 7 `getByText`, 3 CSS ids
Suite runtime	10.1s	8.8s
Pass rate	6/6	7/7

The healer experiment: I gave it a login test with six deliberately stale locators — wrong field ids, wrong button label, wrong flash selector, wrong heading text — simulating a suite that drifted from its app. It repaired all six in 1m 55s, correctly.

What genuinely impressed me

Live verification catches what knowledge misses. The generator discovered that /dynamic_controls keeps stale, hidden #loading elements in the DOM — a real flake trap — and wrote around it:

// The page keeps stale hidden #loading elements around, so target the visible one
const loadingIndicator = page.getByText('Wait for it...').filter({ visible: true });

My one-pass baseline avoided that trap only by accident, because it happened to assert on different elements. This is the strongest argument for the agents: they test against the app as it is, not as you remember it.

Guardrail judgment. The plan included "attempt to fill the disabled input" — a step that would hang forever against Playwright's actionability checks. The generator recognised that and substituted toBeDisabled() plus toHaveValue('') assertions. Same intent, no flake. That's not template-filling; that's a decision.

The planner discloses scope changes. Asked for two login scenarios, it produced three — and flagged the addition instead of burying it. Small thing, big trust signal.

What the demos won't tell you

The agentic tax is real. Forty-three minutes to generate seven tests, because every locator and assertion is verified in a live browser as it's written. It's unattended time — but it scales with suite size, and if you're picturing "regenerate the suite on every sprint", do that maths first.

Nine assertions per test cuts both ways. The generated tests document the UI exhaustively: button disabled during load, loading indicator visible, message text, element gone, other button back. Thorough — and every one of those intermediate assertions is a maintenance cost when the UI legitimately changes. My baseline's ~2.5 assertions per test encode a judgment about what matters; the agent asserts everything it saw. Nobody has taught it what not to check.

The healer repairs — it does not refactor. This is the finding I'd want every team to see before wiring a healer into CI. My stale CSS ids came back as correct CSS ids, not as the semantic locators a reviewer would ask for. And it matched one button's accessible name literally — { name: ' Login' }, leading space and all. It also did something smart: it cross-referenced the known-good login test elsewhere in the repo rather than guessing. But the pattern is clear: healed code preserves the quality level of the code that broke. Feed it a suite full of brittle selectors and it will faithfully restore your brittle selectors forever.

Verdict

Use the planner and generator to scaffold coverage on apps you don't know well — the live verification genuinely produces better selectors than writing from memory, and the wall-clock cost is unattended. Keep the output under review, because assertion density is a liability you'll own later.

Treat the healer as a colleague who proposes fixes in a PR, not as a silent auto-repair step. It's fast, accurate, and completely uncritical of what it restores — which makes it exactly as good as the suite you already have.

If you're thinking about what your test suite means in a world where agents both write and consume it, that's the thread running through the playbook — start with the cost of a slow test suite, because agent workflows make suite speed matter more, not less.

Everything here is reproducible: qa-gary-parker/poc-playwright-test-agents.

Originally published at https://garyparker.dev/blog/hands-on-with-playwright-test-agents