My QA bot was cancelled 60 nights in a row and never said a word

I run a health platform in the Netherlands. About three months ago I set up a nightly job: Playwright runs my critical flows against production at 2am, and if something breaks, Claude opens a PR with a fix. One digest email on Monday. I wrote about it here before.

Yesterday I went looking at its run history for an unrelated reason.

164 nightly runs. 7 succeeded. 76 failed. And 81 were cancelled, the last 60-plus of them consecutively.

My app had not been checked since late June. I found out in September.

Nothing had told me. Not a red job, not an email, not a failed check on a PR. Every morning the Actions tab showed a run from the night before, and every morning my inbox was empty, and I read that as good news for ten weeks.

What went wrong is boring. Why it was silent is not.

The job had timeout-minutes: 20. My suite had grown to 24 spec files, run one at a time with a retry on CI, deliberately gentle on a production app. Somewhere in late June it crossed 20 minutes.

That is the boring part. Any monitoring setup outgrows a limit eventually. You expect it to complain.

Here is the part I did not know: a runner timeout is a kill, not a failure.

When GitHub Actions hits timeout-minutes, it stops the process. Playwright never finishes, so it writes no report. And GitHub records the run's conclusion as cancelled, not failure.

My workflow looked like this:

- name: Run tests
  id: tests
  run: npx playwright test || echo "tests_failed=true" >> $GITHUB_OUTPUT
  continue-on-error: true

- name: Trigger Claude fix on failure
  if: steps.tests.outputs.tests_failed == 'true'
  # ...

- name: Fail job if tests failed
  if: steps.tests.outputs.tests_failed == 'true'
  run: exit 1

Read that again with a kill in mind. The test step never gets to write tests_failed=true, because it never gets to finish at all. So the Claude dispatch is skipped. The job-failure step is skipped. No email. The run is indistinguishable from a night where everything passed, except it is also indistinguishable from a night where nothing ran.

That is worse than having no QA. With no QA you at least know you are not covered, so you check things yourself. I had stopped checking things myself, because I believed something else was.

The real bug was one level up

I could have fixed the timeout and moved on. Bigger number, done.

But sit with the shape of it for a second. Every alert in my setup hung off exactly one thing: a test run that finished and reported a result. So the entire category of "the run never reported" had no alarm attached to it, by construction. The timeout was one member of that category. There are others, and I had none of them covered either:

• GitHub silently disables scheduled workflows on repositories with no pushes for 60 days. Your cron just stops. No notification.
• A secret expires and the job dies during setup, before any test runs.
• Someone disables a workflow to debug something and forgets to turn it back on.

All four look identical from the outside: a quiet dashboard, an empty inbox, and an app nobody is checking.

If you have a nightly check of any kind, this is the question worth asking, and it is a different question from "does it catch bugs":

What does it look like when this check stops running, and who finds out?

If the honest answer is "it looks exactly like everything being fine", the check is not finished.

The two fixes

The first one is ordering, not a bigger number.

Playwright has a globalTimeout. Set it below the job's timeout-minutes, and Playwright stops itself before the runner can kill it. It exits non-zero, the reporters still write results.json, and the whole chain behind it fires normally. A suite that got too slow now reads as a loud failure with a full report instead of silence.

// playwright.config.ts
const GLOBAL_TIMEOUT_MINUTES = Number(process.env.MURAQIB_GLOBAL_TIMEOUT_MIN ?? 35);

export default defineConfig({
  globalTimeout: GLOBAL_TIMEOUT_MINUTES * 60 * 1000,
  // ...
});

# .github/workflows/nightly.yml
jobs:
  test:
    timeout-minutes: 45   # must stay above globalTimeout

Those two numbers can drift apart later, so there is a check in CI now that fails the build if globalTimeout is missing, is not below the job timeout, or does not leave enough room for the install and upload steps around the run.

The second fix is the one that actually matters. A watchdog, running daily, asking one question: has the nightly produced a pass or a fail recently?

It alerts if the recent runs were all inconclusive, if nothing has reported inside the window, or if the workflow has no runs at all. It also flags a check that has been red for seven runs straight, because at that point it has stopped being an alert and become furniture. Everyone has learned to scroll past it, which is the same outcome as silence by a different road.

The watchdog installs nothing. No dependencies, no npm step, just the Actions API and the fetch built into Node 20. Whatever watches the watchman needs fewer moving parts than the watchman, or you have only added another thing that can go quiet.

I fed it my own run history from June onwards. It returns the alarm I needed ten weeks ago.

If you have a nightly Playwright setup

You can check yours without installing anything:

npx muraqib doctor

It reads your workflow and your Playwright config and tells you whether a slow night would fail loudly or just disappear. Two seconds. It writes nothing to your repo.

I built it because I could not be the only person whose job timeout is quietly eating their monitoring. If you run it and it comes back green, good, that is a real answer and it cost you nothing.

The whole thing is MIT licensed and runs in your own GitHub Actions on your own key. There is no service and no bill, because there is nothing to bill for.

https://github.com/holistis/muraqib

The uncomfortable part

I write software for a health platform under Dutch medical regulation. I am careful about a lot of things. And I still had a guard that had been dead for ten weeks, sitting in plain sight in my Actions tab, looking exactly like it was working.

I am fairly sure I am not unusual here. If you set up a nightly check six months ago and have not looked at it since, go look at the conclusion column, not just the green ticks. Count how many say cancelled.