A Dead-Man's Switch That Pages Once and Goes Quiet Is Worse Than None. Ours Went Silent for 43 Days.

Most monitoring watches for something bad to appear: a 500, a timeout, an expired certificate, a slow response. A heartbeat monitor does the opposite. It watches for something good to stop appearing. Your cron runs, your backup completes, your embedded device phones home, your queue worker drains — and each of those pings a URL to say "I'm still alive." The monitor's job is to notice when the pings go quiet.

That inversion is the entire value. A cron that fails throws an error you can catch. A cron that stops being scheduled — the box got reimaged, the systemd timer got disabled, the container never came back after a deploy, the account got suspended for an unrelated billing issue — throws nothing at all. There is no log line, no exception, no non-zero exit. There is only the absence of the thing that used to happen. You cannot alert on an event that does not fire. You can only alert on the silence.

So heartbeat monitoring looks trivial: store a timestamp on every ping, and if now - last_seen > expected_interval, fire an alert. It is about ten lines. And it is exactly those ten lines that will let 43 days of downtime pass without a second word — because the hard part of a dead-man's switch is not detecting the death. It is staying loud after it.

I know because it happened to our own.

Three states, and why the third one must stay silent

Start with the check itself. A naive heartbeat has two states — alive or dead — and both are wrong at the edges.

The real answer set has three:

• alive — a beat arrived within period + grace. Everything is fine.
• dead — the last beat is older than period + grace. The thing stopped. Page someone.
• unknown — the monitor exists but has never received a single beat.

That third state is where two-state heartbeat monitors self-immolate. A brand-new heartbeat you just created has no last_seen timestamp. If your rule is "alert when last_seen is too old," a null last_seen is infinitely old, so the monitor pages you the instant you create it — before you have even wired the ping into your cron. Every new heartbeat cries wolf on birth. Users learn to ignore the first alert from every new monitor, which is precisely the alert you least want them to ignore.

The fix is to treat never-beat-yet as its own answer. A heartbeat with no last_seen is unknown: recorded, charted, but not alerted. It becomes alertable only once a first beat establishes a baseline. "I have never heard from this" and "I used to hear from this and now I don't" are different facts, and only the second one is an outage.

There is a fourth degenerate case worth naming: a beat has arrived, but nobody configured a period. You have a timestamp but no threshold to judge it against. That is alive — chartable, never alerting — not a silent dead. Absence of a rule is not evidence of death.

The evaluator that decides this should be a pure, total function: last-seen in, status out, no I/O, no throws. An unparseable timestamp degrades to "never seen," not to a 500 that takes the whole sweep down. When the thing that reports on your dead crons can itself die, you have built a monitor that needs a monitor.

The part that actually bit us: the alert dedupe key

Here is the ten-line trap. You detect dead. You insert an alert. Your alerting layer, sensibly, deduplicates — you do not want one email per evaluation while the outage persists, and a heartbeat sweep might run every minute. So you dedupe on a key, and the obvious key is:

heartbeat:<asset_id>:dead

One string. It identifies "this asset is dead." Insert-with-dedupe means the second, third, and thousandth dead evaluation all collapse onto the first row, and the customer gets one page instead of a thousand. Clean. Correct. Shipped.

It is also a monitor that can only ever report an outage once in its lifetime.

The bug is the interaction with a second, easy-to-miss fact: most alerting layers never actually resolve alerts. They insert an alert row and mark it delivered; nothing ever sets resolved_at. Our dedupe ran WHERE resolved_at IS NULL, and resolved_at was null forever — 1,540 of 1,542 alert rows still open when we measured it. So the very first outage's row stays open permanently. Every subsequent outage hashes to the same heartbeat:<asset>:dead key, finds that still-open row, and is deduplicated against an alert from a different outage weeks ago. The delivery guard sees "an alert for this key already went out" and suppresses the send.

The monitor's own comment claimed it "re-fires only after a recovery." It re-fired after nothing.

What production did, on our own account's dead-man's switch:

• 2026-06-22 20:30Z — one dead alert. Delivered sent at 21:45Z. This is the only page that ever rang.
• 2026-07-04 → 2026-08-05 — 6,052 more dead rows written, every one deduplicated into silence.
• max staleness on those rows: 3,742,701 seconds — 43.3 days.
• Then, at the far end, an "is checking in again" all-clear — a recovery notice for an alarm that had never sounded.

Forty-three days of a monitored thing being down, on the exact monitor sold to catch that, and the humans got one email at the start and a cheerful all-clear at the end. That is worse than no monitor, because "we have a heartbeat on it" is a reason to stop worrying.

Why heartbeat is the worst place for this bug

This dedupe-key mistake shows up in any monitor with a persistent-outage alert — certificate expiry, domain expiry, uptime. Heartbeat is where it does the most damage, and the reason is cadence.

A domain registration lapses about once a year. A Let's Encrypt certificate rotates roughly every 60 days. Those outages are slow; even a monitor that can only fire once per outage-lifetime mostly gets it right, because outages are rare and far apart. A cron is the opposite. A flaky scheduled job misses, recovers, and misses again in a single afternoon. The precise failure a heartbeat is bought to catch — an intermittent job that keeps dying — is exactly the pattern a fire-once monitor is blindest to. The mildest-looking of the three sites is the one that breaks the most.

The fix: make the key move with the outage

The dedupe key has to identify the outage, not the asset. Within one outage, the last-seen timestamp is frozen by definition — no beats are arriving, so it does not change — which makes it a stable, natural outage identifier: one page per outage, not one per sweep. A recovery advances last_seen, so the next outage mints a fresh key and dispatches normally.

heartbeat:<asset_id>:<last_seen_instant>

Use the full instant, not a truncated date: two outages in one afternoon are two outages and should be two pages. Normalise the timestamp through epoch-ms before formatting, because it can reach your alerting layer as a database string in one code path and a native Date in another, and a key that changes with its rendering pages on every sweep — the opposite failure, equally bad.

The rule set that falls out:

• dead → warning, keyed on the last-seen instant → one page per distinct outage.
• alive after dead → one info "recovered" notice, keyed on the recovery.
• unknown / no-threshold → recorded, never alerted.

What this looks like in practice

I build Merlonix, which runs this as an inbound heartbeat / dead-man's-switch monitor. Your cron, backup, worker, or device POSTs to a private high-entropy URL and the token is the only credential: a valid one stamps the last-seen time and returns 204, an unknown one returns a generic 404 with no enumeration detail, and the ingest is rate-capped per IP so a flood never reaches the database. A sweep then evaluates staleness against the period and grace you set, resolves to alive / dead / unknown, and alerts on a missed beat — once per outage — and again when it starts checking in.

And yes: the 43-day silence above was ours, on our own switch, before we keyed the alert on the outage instead of the asset. The three-state evaluator and the moving dedupe key in this post are not a whiteboard design. They are the diff.