Every morning there was a run in the Actions tab and nothing in my inbox. That is exactly what a healthy nightly looks like.
It had not tested my app since late June. I found out in September.
Here is what was happening, why nothing told me, and what changed in Playwright last week as a result.
A timeout is a kill, not a failure
My nightly job had timeout-minutes: 20. My suite had quietly grown past twenty minutes.
When a GitHub Actions runner hits that limit it does not fail the job. It kills the process and records the run's conclusion as cancelled. That single word is the whole problem, and it has three consequences that stack.
The report is never written. Playwright is stopped mid-run. The reporters never get to flush, so there is no HTML report and no results.json. Nothing on disk to look at afterwards.
The upload step skips itself. The recommended workflow guards the artifact upload with if: ${{ !cancelled() }}. That guard is right and it is still there. It exists so a run somebody cancelled by hand does not upload a half-finished report. The trouble is that a runner timeout produces the same conclusion as a hand cancellation, so the guard cannot tell them apart. You get no artifact on precisely the runs where you most want one.
Your alerting never fires. If your notification step keys off the test step failing, it is not reached. A cancelled job skips everything downstream. No failed step, no email, no Slack message.
So the run happens, the run is visible, and nothing about it says anything is wrong. It is indistinguishable from a night where the tests passed. Worse, it is indistinguishable from a run somebody cancelled by hand.
The numbers from my own repo
I pulled the run history when I finally went looking.
164 nightly runs since June
7 succeeded
76 failed
81 cancelled, the last 60-plus consecutively
Every cancelled run produced zero artifacts.
Then I set globalTimeout below the job timeout and ran the same suite. It finished in 29 minutes, reported failure with 12 of 147 tests failing, and produced an 83 MB report.
The suite had needed 29 minutes for a while. The 20 minute limit had been killing it nightly. Because a kill is not a failure, nothing said so.
The fix, and why the ordering matters
globalTimeout is a Playwright setting. It defaults to no timeout at all.
Set it below your job's timeout-minutes and the order of events changes completely. Playwright stops itself first. It exits non-zero, the reporters finish, the HTML report and results.json are written, and the run reads as an ordinary test failure. Your alerting fires because something actually failed.
The runner timeout then only has to cover the install and upload steps around the run. It becomes a backstop instead of the primary bound.
// playwright.config.ts
export default defineConfig({
// Fail the run after an hour, so that the reporters still produce a report.
globalTimeout: 60 * 60 * 1000,
});
Two lines. The entire difference between a loud failure and two months of silence.
How common this is
Before filing anything I wanted to know whether this was my mistake or a shape lots of people were in.
At the time I checked, GitHub code search reported roughly 163,000 playwright.config.ts files. About 1,700 of them set globalTimeout. That is around one percent.
Around 3,200 workflows ran npx playwright test on a cron schedule. That is the case where nobody is watching the run live, so a missing report costs the most.
Those numbers are approximate and they move. The ratio is the part that mattered: the safe configuration was the rare one, and the docs never connected the two settings.
What changed
I filed microsoft/playwright#42533 on 3 September, proposing a note in ci.md.
Playwright maintainer Dmitry Gozman went considerably further than the note I asked for, and merged two changes on 4 September.
The documentation, in microsoft/playwright#42563, across four files. ci.md now opens the CI section with this:
Always set a global timeout in CI. By default a test run has no upper bound, so a suite that hangs, or that slowly grows past the job limit of your CI provider, is killed by the runner mid-run and does not produce the test report.
And it drops the job-level timeout from the workflow examples, with the reasoning attached: if you do add one, keep it comfortably above globalTimeout so Playwright always stops first.
The scaffolding, in microsoft/create-playwright#181. This is the one that matters most. create-playwright is what runs when anyone types npm init playwright@latest. It now generates globalTimeout: 60 * 60 * 1000 in all four config templates, and no longer generates timeout-minutes in the GitHub Actions workflow.
I asked for a warning sign next to the hole. What landed was the hole being filled.
What this means for you
If you start a Playwright project from today, you get the safe default and you do not need to think about any of this.
If your project already exists, you still have the old shape. The generated config sits in your repo from the day you created it and nothing rewrites it. Two things to check:
Does your playwright.config.ts set globalTimeout? If not, your run has no upper bound of its own.
Does your CI job set timeout-minutes? If yes, and globalTimeout is unset or higher, the runner is your only limit and it kills without reporting.
There is also a check that takes ten seconds and tells you whether this already happened to you. Open your nightly workflow's run history and look at the conclusions. If you see a run of cancelled you never investigated, that is this bug, and those nights were not tested.
The part I keep thinking about
The bug was not that something broke. Something broke and the system that was supposed to tell me was structurally incapable of telling me, because the failure mode it produced was the one shape it treated as "nothing to report".
I have since gone looking for more of those. In my own tooling I found a check that verified a curl had the --fail flag by searching the whole workflow step for the word. Directly above the curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stays green.
The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.
The way you find these is not by reading them. It is by introducing the exact defect each check claims to catch and confirming it goes red. I ran that against ten checks in one repo and two of them could never have failed. When I ran it, my own test harness turned out to be broken twice before the checks were.
A check that cannot fail costs exactly as much to run as one that can, looks identical in the Actions tab, and buys you confidence it has not earned. That is worse than having no check, because no check at least leaves you appropriately nervous.
Look at what your green means. Sometimes it means nothing went wrong. Sometimes it means nothing could have told you.
Top comments (0)