On 19 August our GitHub Actions hosted-minutes quota ran out.
Everything you would expect to break broke, loudly, and was fixed inside two days. The thing that broke quietly was the uptime probe, which ran on a schedule on those same hosted runners. Every scheduled run since then died in about three seconds with no runner and no logs. By the time anyone looked, that was 165 consecutive failures.
Two consequences, and the second is the one worth the post.
The status page froze. status.tamperlens.com still said "Last checked" on 19 August at 17:13 UTC. Four days later it still said it. A frozen status page does not look broken. It looks like a service that is fine and nobody has poked recently, which is a strictly worse failure than a page that is down.
The probe was also the alarm. A failing run emailed the operator. Once the runs stopped starting, nothing was checking the API at all, so if production had gone down, nothing would have told anyone. That is the more urgent half, and it is the half that a "status page frozen" ticket does not describe.
The obvious fix does not work here
We already had a self-hosted runner on the production box, and the two workflows that mattered moved onto it two days later.
The uptime probe could not follow, for a reason that is not about our setup: that runner is the box being watched. A machine cannot meaningfully report its own outage. Worse, the status page's own copy makes the claim that it is probed from outside our network and served from outside it too, so it stays up when we do not. Moving the probe onto the box would have made the page lie rather than fixed it.
That constraint is the whole design. A status page is only worth having if the thing that measures and the thing that serves are both independent of the thing being measured. Ours had one of those properties (the page was served elsewhere) and had been getting the other from a CI provider, for free, as a side effect.
Which is the general lesson, and it applies well beyond monitoring: if a property of your system is a side effect of somebody's free tier, you have not designed for it, you have been lent it.
The move: a Worker with a cron
The probe now lives in a Cloudflare Worker with a scheduled trigger, writing history to KV. Cloudflare's edge is neither the box being watched nor the CI that deploys it, so the premise survives.
Two decisions inside that are more interesting than the migration itself.
The renderer did not move. The page is rendered by a module both the Worker and the existing Node CLI import, so the served page cannot drift from the one the CLI produces out of the same history. Migrating a monitor is a great way to end up with two subtly different renderings of the same data, and the fix is boring: extract the pure part first, make it depend on no filesystem, no clock and no process, and have both callers import it.
History moved from an orphan git branch to KV, and the reason is credentials. The old design kept history.json on an orphan branch, which was auditable and cheap to clone. A Worker cannot push to git without carrying a token that can write to the repository, and a monitor holding repo-write credentials is a worse trade than losing the git audit trail. The raw data stays public and one fetch away at /history.json, served from KV instead of GitHub.
The thing you are tempted to preserve during a migration is usually the thing whose cost you have not re-priced. A write credential in a component whose only job is to read is worth more to an attacker than the audit trail is worth to you.
What got worse, said out loud
The Worker's checker component is weaker than the Node probe it replaces, and that belongs in the post rather than in a footnote.
The Node probe fetches every script the page loads, parses each one, requires something to assign the bootstrap globals, and then executes them against a DOM and requires the demo report to actually render, plus a leg that drives a fixture PDF through a live POST /api/v1/inspect. A Worker has no DOM and no eval, so it stops after the parse-and-assign layer.
That layer is not the consolation prize, and here is why.
On 4 August a syntax error in one client script left every checker page dead for two days, while the origin kept answering 200 for the HTML. Four probes were green throughout, correctly: they asserted one HTTP status on one HTML document. One of them was even labelled "Website and free checker". A page whose scripts do not parse returns that status perfectly happily.
The monitor was watching the wrapper.
So the component that came out of that outage does something narrower and much more useful. It fetches the page, reads the <script src> list out of the HTML the origin actually served rather than from a hardcoded list, fetches every one, asserts each parses, and asserts that the bootstrap globals are assigned somewhere across them.
That last word is doing real work. const T = window.Tamperlens is a read, and it does not count. Every consumer script contains a line like it, so counting reads would make the check pass on exactly the day the file that does the assigning stopped loading.
The parse-and-assign layer is the layer that failed in the real outage. Losing the DOM execution on top of it costs something. Losing the layer underneath would have cost the only outage this component has ever had to catch.
The deeper probe did not get deleted, either. It kept a manual trigger and still runs the whole thing on the self-hosted runner on request, for use before a release or when the cheap probe says the checker is down and you want to know how far the damage goes.
The test that keeps two vantage points honest
There are now two implementations of "does this script parse": node:vm in the CLI, and a parser bundled into the Worker, because Workers have no vm.
Two parsers means two opinions, and two opinions about the same file means the repo and the edge can report different verdicts about it. So the load-bearing test in the Worker suite asserts that the two agree on what parses, including the exact historical breakage that caused the two-day outage.
That is the shape I would reuse anywhere a check gets reimplemented for a second runtime: do not test the new implementation against your expectations, test it against the old implementation, on the input that made you write the check in the first place.
The part the move did not fix, for one more day
The Worker as first deployed logged a failing probe with console.error, which reaches wrangler tail and the Workers logs, and nothing else. The old workflow at least emailed somebody when it failed. So for a day the move had restored the page and not the alarm, and that deserves saying out loud: the four-day outage above was invisible precisely because nobody was watching the watcher, and a fix that left that property intact would have been the same mistake with newer infrastructure.
I had two Cloudflare routes in mind for closing it and both were wrong for the job. Workers alerts fire on invocation errors, and a failing probe is not one: the Worker ran perfectly, the site did not, so console.error never trips it. Logpush is an Enterprise feature, which is why the menu item could not be found.
What got wired instead, on 25 August, is the dead man's switch we already run for the nightly backup and for the box's own guard script. The cron pings a healthchecks.io check when everything is up, pings its /fail URL with the failing component names when something is down, and, the part neither Cloudflare option offered, alerts by silence if the Worker itself stops firing. That last one is the exact failure mode that produced this post. The ping URL is a secret, and with it unset the Worker behaves as before, so it was safe to deploy before the check existed.
Where this landed
On 23 August the code was done and the cutover was not: creating the KV namespace, deploying, watching one real probe land, moving the custom domain off the old project. Five dashboard steps and one CLI call, which needed credentials and a DNS change that code cannot do for itself.
They happened that night. The first probe landed at 23:07 UTC with all five components up, on the same quarter-hour cadence the old workflow had asked for, and the four probes that ran before midnight all came back clean. status.tamperlens.com moved onto the Worker route at about 02:30 UTC on 24 August, and the "Last checked" line has read a timestamp from the last fifteen minutes ever since. It has also recorded a handful of short timeouts on its own since then, which is the page doing the thing the frozen one could not.
I wrote most of this before any of that, and I am posting it with the ending attached rather than rewriting it as a migration story, because "we migrated our status page" is a much less useful post than "our status page was frozen for four days and here is the whole reason it could be".
The four things I would check on your own setup
- Is your probe also your alarm? If the same job both measures and notifies, its failure is silent by construction.
- Does anything watch the watcher? A dead man's switch that expects a regular ping is a few lines and would have caught this on day one. Ours has one now, and it took the whole episode to get it.
- Is your monitor asserting on the wrapper? An HTTP 200 on an HTML document says nothing about whether the page works. Assert on the thing that actually broke last time.
- Which of your invariants are on loan? Independence, retention, alerting: if the property comes from someone else's free tier, write down what happens when the tier ends. Ours ended on a Wednesday.
What our monitoring commits to, what it deliberately does not, and the limits the status page states about its own measurement are on the security page. The product it watches is a document trust API: drop a PDF into the free checker and it returns the structural findings with the evidence under each one, no account, nothing stored.
If you have a monitor that stayed green through a real outage, I would like to hear which layer it was asserting on. Mine was asserting on the wrapper for months.
Top comments (0)