Your Cron Jobs Are Failing Right Now and You Won't Know for 3 Weeks

Here's a stat that should ruin your morning: in every postmortem I've run for small teams, silent cron failure is a top-3 cause of data loss. Not hackers. Not cloud outages. A backup script that died on a Tuesday and nobody noticed until they needed the backup.

The failure mode is always the same:

1. A cron job depends on something invisible — a rotated credential, a moved binary, a full disk, a changed PATH.
2. It fails. Cron sends output to /dev/null or a mailbox nobody reads.
3. Weeks pass. The dashboard stays green because the dashboard only checks the thing that depends on the cron job, not the cron job itself.

The dead-man's-switch pattern (30 minutes to deploy)

Flip the model: every important job must check in when it SUCCEEDS, and a watchdog screams when a check-in is late.

• Job finishes OK -> writes a timestamp file (or pings a URL).
• Watchdog runs every 5 minutes -> compares age of timestamp vs. expected interval.
• Late? -> alert goes to the channel people actually read, with the job name and last-known-good time.

That's the whole design. It survives credential rotation, disk-full, and the intern who "just tweaked" the crontab — because any of those break the check-in, and the break is loud within minutes, not weeks.

The 4 additions that make it production-grade

1. Exit-code capture. Log EXIT:$? on every run. A job that "runs" but exits 2 is not a success.
2. Output fingerprinting. Log line-count + last line. Empty output from a job that used to write 400 lines is a failure in disguise.
3. Runtime deltas. A job that took 40s and now takes 40ms is probably failing at the first step. Alert on +-50%.
4. Weekly evidence report. One page: every job, last success, last failure, current runtime. This is the artifact that saves you in the postmortem.

Why teams still don't do it

Because the scripts are 30 minutes but the system is the hard part: which jobs matter, what's the right interval, what threshold is "late," who gets paged, and what the report looks like. Same reason most teams' monitoring covers everything except the money path.

I got tired of re-explaining this on calls, so I packaged the exact kit we run — a dead-man's-switch watchdog, disk canary, cert check, and the weekly evidence report generator — in the Automation Starter Pack (hive80lab.gumroad.com). It's the 30-minute version of this entire article: drop-in scripts, sane defaults, and the report your next incident review will ask for.

If you run even three cron jobs in production, one of them is lying to you. Make the check-in mandatory.