A dead man's switch alert is built backwards from most alerting logic. Instead of firing when something bad happens, it fires when an expected good thing stops happening. That distinction matters enormously for cron jobs and scheduled automation, where the scariest failure isn't a crash, it's a job that silently stops running, or runs and silently stops accomplishing anything, without a single error anywhere in the chain.
Why Normal Alerting Misses This
A standard alert watches for a bad event: an exception, a non-2xx response, a timeout. A dead man's switch watches for the absence of a good event within an expected window. If your nightly job normally reports a heartbeat by 6am and it's 8am with no heartbeat, that's the alert condition, regardless of whether anything technically threw an error anywhere. This catches the job that got silently disabled, the cron entry that got accidentally deleted during a server migration, and the job that's stuck in an infinite retry loop that never actually fails.
Step 1: Emit a Heartbeat, Not Just a Result
The job needs to actively report "I ran" as a distinct signal from "I succeeded" or "I processed N rows." A simple heartbeat, a timestamped ping to a monitoring endpoint at the start or end of every run, is enough. This is a small addition to any job and doesn't require restructuring the job's core logic.
Step 2: Configure an Expected-Interval Check
Prometheus supports this pattern through the concept of a metric that's expected to update on a schedule; an alerting rule can be written to fire specifically when that metric's timestamp goes stale beyond an expected window. The same pattern exists in most monitoring platforms under names like "heartbeat monitoring" or "scheduled task monitoring," so you likely don't need custom tooling to implement this if you already have a metrics stack.
- alert: NightlySyncMissedHeartbeat
expr: time() - nightly_sync_last_heartbeat > 3600 * 10
labels:
severity: warning
annotations:
summary: "Nightly sync job has not reported in over 10 hours"
Step 3: Set the Window With Real Slack
Set the missed-heartbeat window wider than the job's normal runtime variance, but tight enough to catch a genuine failure quickly. A job that usually finishes by 2am shouldn't trigger an alert at 2:05am over ordinary jitter, but should trigger well before the next business day starts if it genuinely didn't run. A window of roughly two to three times the job's typical runtime variance is a reasonable starting point, tuned per job.
Step 4: Pair It With a Volume-Floor Check
A dead man's switch alone catches the job that stopped running entirely. It doesn't catch the job that keeps running, keeps reporting a heartbeat, and quietly processes zero real records because of a broken filter or an expired credential. Pairing the heartbeat check with a rolling row-count floor closes both halves of the problem: one alert for "did it run," a separate alert for "did it actually do anything."
Step 5: Route This Alert Differently Than a Crash Alert
A missed heartbeat and a thrown exception are both worth paging someone about, but they benefit from slightly different framing in the alert message itself. A crash alert usually comes with a stack trace pointing at the fix. A missed heartbeat alert has no such lead, the first step is almost always "go find out why this job didn't run," which is a different investigation than debugging a visible exception. Tools like Sentry let you tag and route these differently so the on-call engineer knows which kind of investigation they're starting before they open the first log line.
"A dead man's switch is one of the few alerts I've never seen a team regret adding. It's cheap, it's boring, and it catches exactly the failure mode that every other alert in the stack is blind to." - Dennis Traina, founder of 137Foundry
Step 6: Test It the Same Way You'd Test a Fire Alarm
A dead man's switch that's never been deliberately triggered is a check nobody actually trusts. Periodically disabling a job on purpose in a staging environment, or pausing a heartbeat manually, confirms the alert actually fires and reaches the right person, rather than discovering during a real incident that a routing rule was misconfigured months ago.
Step 7: Decide Who Actually Owns the Response
A dead man's switch that pages someone with no clear next step tends to get snoozed after the second false start. Before rolling this out broadly, we make sure whoever's on call knows the first three things to check when a heartbeat goes missing: is the scheduler itself running, did the job's credentials expire, did an upstream dependency change its schedule or shape. Writing that runbook down, even briefly, turns a vague "something didn't run" alert into an actionable one.
A Common Mistake: Alerting on the Job's Own Reported Status
An easy trap is having the job itself report "I ran successfully" as the heartbeat signal, rather than an independent timestamp ping that happens regardless of what the job's internal logic decides. If the job hangs before reaching its own success-reporting code, or gets killed by the process manager before it finishes, a heartbeat tied to the job's own final line never fires, and neither does your alert. The more robust pattern pings at the start of the run, not just the end, or uses an external scheduler-level check that doesn't depend on the job's own code executing correctly at all.
Comparing This to Managed Alternatives
Building a dead man's switch from scratch on top of Prometheus is a reasonable default if you already run that stack, but it's not the only path. Managed services like PagerDuty offer heartbeat-style checks as a built-in feature, which can be faster to stand up for a team that doesn't want to own the alerting infrastructure itself. The underlying pattern, expected-interval monitoring rather than error-based monitoring, is the same regardless of which tool ends up hosting it.
Where This Fits Into a Broader Monitoring Strategy
A dead man's switch is one piece of a small set of checks that together close the silent-failure gap: a heartbeat for "did it run," a row-count floor for "did it do enough," and a watermark check for "did it actually advance." None of these individually is sophisticated. Together, they cover almost every version of a job that quietly stops doing its job without ever throwing an exception. Building all three in as a standard part of every scheduled job is exactly the kind of reliability work this data automation team at 137Foundry treats as non-negotiable on client pipelines, and it's covered in more depth in our guide to catching silent job failures.
What Happens When You Get the Window Wrong
Setting the missed-heartbeat window too tight produces false alarms on ordinary variance, a job that normally finishes in twenty minutes taking forty because of unrelated infrastructure load. Setting it too loose delays real detection unnecessarily, letting a genuinely stuck job sit unnoticed for hours longer than it needs to. Neither mistake is catastrophic on its own, but a team that gets burned by false alarms tends to start ignoring the alert entirely within a few weeks, which defeats the entire purpose. Erring slightly toward a looser window at first, then tightening it once you've observed several weeks of real runtime variance, is a safer default than guessing tight and hoping.
The Takeaway
A dead man's switch is a small, mechanical piece of monitoring that catches an entire class of failure invisible to exception-based alerting. If your cron jobs and scheduled automation don't have one yet, it's one of the highest-leverage additions available, cheap to build, boring to maintain, and exactly the kind of check that turns a multi-week silent outage into a same-day fix.
Top comments (0)