Your Cron Jobs Are Failing Right Now and You Won't Know for 3 Weeks

Here's a stat that should ruin your morning: in every postmortem I've run for small teams, silent cron failure is a top-3 cause of data loss. Not hackers. Not cloud outages. A backup script that died on a Tuesday and nobody noticed until they needed the backup.

The failure mode is always the same:

1. A cron job depends on something invisible — a rotated credential, a moved binary, a full disk, a changed PATH.
2. It fails. Cron sends output to /dev/null or a mailbox nobody reads.
3. Weeks pass. The dashboard stays green because the dashboard only checks the thing that depends on the cron job, not the cron job itself.

The dead-man's-switch pattern (30 minutes to deploy)

Flip the model: every important job must check in when it SUCCEEDS, and a watchdog screams when a check-in is late.

• Job finishes OK → writes a timestamp file (or pings a URL).
• Watchdog runs every 5 minutes → compares age of timestamp vs. expected interval.
• Late? → alert goes to the channel people actually read, with the job name and last-known-good time.

That's the whole design. It survives credential rotation, disk-full, and the intern who "just tweaked" the crontab — because any of those break the check-in, and the break is loud within minutes, not weeks.

The 4 additions that make it production-grade

1. Exit-code capture. Log EXIT:$? on every run. A job that "runs" but exits 2 is not a success.
2. Output fingerprinting. Log line-count + last line. Empty output from a job that used to write 400 lines is a failure in disguise.
3. Runtime deltas. A job that took 40s and now takes 40ms is probably failing at the first step. Alert on ±50%.
4. Weekly evidence report. One page: every job, last success, last failure, current runtime. This is the artifact that saves you in the postmortem.

Why teams still don't do it

Because the scripts are 30 minutes but the system is the hard part: which jobs matter, what's the right interval, what threshold is "late," who gets paged, and what the report looks like. Same reason most teams' monitoring covers everything except the money path.

I packaged the exact version I deploy for clients — watchdog script, per-job config format, the threshold table from a year of real incidents, and the weekly evidence report template — in the Automation Starter Pack ($19). If you also want the heartbeat/SSL/disk checks and the runbook layer, the Ops Mega Bundle is everything at bundle pricing. One-time purchase, deploys in an afternoon, no subscription.

---

Question for the comments: what's the longest a silent failure ran before someone noticed? My record to beat is 11 weeks (a DB backup that had been writing to a read-only mount).