Playwright 1.65 onlyFailures: Smaller Reports That Say Zero Tests Passed

Playwright 1.65 adds an onlyFailures option to the HTML, JSON and JUnit reporters, and a new failures terminal reporter. The release note says the option keeps only tests with unexpected outcomes, flaky tests and interrupted tests. I wanted three things before turning it on: how much smaller the reports get, whether the run gets faster, and what the report no longer contains.

I measured it on a 2,000-test suite. The harness and raw results are in the PoC repo.

The setup

• 100 spec files with 20 tests each. Each test has 3 steps. The tests use no browser.
• @playwright/test 1.65.1, on Node 24 and macOS, with workers: 4 and retries: 0.
• 0, 20, 100 or 400 of the 2,000 tests fail, each on a toEqual of two objects.
• Eight reporter setups: dot, failures, and JSON, JUnit and HTML with and without onlyFailures. The file reporters ran beside dot, with CI=1 set.
• 5 repeats of every cell: 160 runs. I recorded the wall time, the terminal output and the size of every file the reporter wrote.

A second script runs 8 tests with one outcome each, to see which outcomes count as a failure.

How much smaller

Report size, full and with onlyFailures. HTML is the whole report folder, which holds one attachment per failed test:

Failed tests	JSON	JSON onlyFailures	JUnit	JUnit onlyFailures	HTML	HTML onlyFailures
0	2.92 MB	2.8 KB	222 KB	102 B	891 KB	517 KB
20	3.05 MB	158 KB	250 KB	34 KB	964 KB	599 KB
100	3.54 MB	766 KB	363 KB	155 KB	1.14 MB	773 KB
400	5.38 MB	3.05 MB	786 KB	609 KB	1.73 MB	1.41 MB

The saving depends on the format and on how red the run is:

• JSON gains the most. At 20 failed tests the report is 5% of the full one.
• HTML gains the least. The report is 517 KB with no tests in it, because the viewer app is in the file. At 20 failed tests onlyFailures saves 38%.
• The saving shrinks as failures grow. At 400 failed tests the JSON report is still 57% of the full one, because a failed test with its error takes far more space than a passed one.

Does the run get faster

Not on this suite. Median wall time over 5 runs, in seconds:

Failed tests	dot only	JSON	JSON onlyFailures	HTML	HTML onlyFailures
0	2.47	2.41	2.52	2.48	2.41
20	3.70	3.73	3.67	3.74	3.71
100	7.95	8.49	8.40	8.08	8.16
400	24.16	23.76	24.21	24.14	24.17

The medians of a reporter and its onlyFailures twin differ by at most 0.46 seconds (JUnit at 100 failed tests, not in the table), and the sign changes from row to row. Single runs varied a lot: the slowest took just over three times the median. A difference smaller than about 5% would not show up in 5 repeats.

Loading is faster, but from a small base. I loaded each report from the last repeat 10 times: the JSON with JSON.parse in Node, the HTML in headless Chromium from a local file. Parsing the JSON report took 4.5 ms full and 0.3 ms with onlyFailures at 20 failed tests. The HTML report showed its first test after 63 ms full and 48 ms with onlyFailures.

What counts as a failure

Eight tests, one outcome each, with one retry:

Test	With onlyFailures
passes	dropped
fails	kept
passes on retry (flaky)	kept
is skipped	dropped
is fixme	dropped
fails as expected with test.fail()	dropped
passes but is marked test.fail()	kept
times out	kept

All three reporters agreed. Flaky tests stay in. Skipped and fixme tests go, so the report can no longer tell you what was not run. The source also keeps interrupted tests; I did not test that.

The totals shrink too

This is the part I did not expect. The report does not only drop the test entries. Its totals count only what it kept:

Field	Full	onlyFailures
JSON stats	expected 2, skipped 2, unexpected 3, flaky 1	expected 0, skipped 0, unexpected 3, flaky 1
JUnit <testsuites>	tests 8, failures 2, errors 1, skipped 2	tests 4, failures 2, errors 1, skipped 0
HTML counters	All 6, Passed 2, Failed 3, Flaky 1, Skipped 2	All 4, Passed 0, Failed 3, Flaky 1, Skipped 0

On the 2,000-test suite with 20 failed tests, the JUnit report says tests="20" and the JSON report says expected: 0. Anything that reads a pass rate from those fields now reads 0% for a run that was 99% green. A fully green run produces a JUnit file with zero test cases.

What reads the report after the run

Any tool that reads passed tests or durations from the report loses them. I tested one, which I maintain. playwright-shard-balancer builds its timings file from the durations in a JSON report. From the full report with 20 failed tests it read timings for 1,980 tests. From the onlyFailures report it read timings for 0 and exited 0. It printed 0 samples from 1 input(s), gave no warning, and wrote an empty timings file.

Two things did hold up:

• Blob reports are not filtered. Two shard blobs merged to all 8 tests with --reporter json, and to 4 tests with onlyFailures: true in the merge config. You can keep full blobs and produce both reports from them. I tested this with the JSON reporter only.
• The option has an environment variable per reporter, such as PLAYWRIGHT_JSON_ONLY_FAILURES=1. It gave the same reports as the config option, so you can switch it per job without a config change.

The failures reporter

The new failures terminal reporter prints no progress output. It prints each failure as it happens, then a summary. On a green run it printed 21 bytes where dot printed 4 KB. With 20 failed tests it printed 26 KB where dot printed 30 KB, because the error output is nearly all of the log by then. It makes green logs quiet. It does not make red logs short.

What I'd do with it

• Use onlyFailures for the report a person opens after a red run. It has the failed and flaky tests and little else.
• Do not point a dashboard, a pass-rate check or a duration-based tool at an onlyFailures report. The totals and the passed tests are not in it.
• If you need both, write blob reports and merge twice: once in full for the tools, once with onlyFailures for people.
• Do not expect a faster run. Across 120 paired runs I saw no consistent difference in run time.

The limits: all numbers are from one macOS machine, one Node version and one suite with no browser. Traces, screenshots and videos change what a report weighs, and I did not measure them.

Originally published at https://garyparker.dev/blog/playwright-165-only-failures