Here's the outage nobody monitors: the cron job that fails quietly every night for three months while the dashboard stays green.
Backups that never ran. Syncs that half-finished. Reports emailed to an address that was decommissioned in March. Silent cron failures are the most common unmonitored outage in small ops teams — and they don't page you, they just quietly rot your safety nets until the day you need them.
Why cron fails silently (the 4 classics)
1. Exit codes nobody reads. Cron runs your script, the script fails, cron doesn't care. Unless something reads the exit code, the failure never happened as far as the universe is concerned.
2. stdout goes to /dev/null. That > /dev/null 2>&1 you added to "stop the email spam"? You also deleted the only evidence that anything was wrong. 90% of silent cron failures live in /dev/null.
3. Environment drift. Works in your shell, fails under cron: different PATH, different HOME, no credentials loaded. The job "runs" — exit 127, command not found — and has been running into a wall for months.
4. The job that finishes too fast. A sync that takes 40 minutes taking 3 seconds isn't a performance win. It's crashing at line 1. Duration is a health signal nobody graphs.
The 15-minute cron audit
Step 1 — Inventory. crontab -l on every server that matters. List every job with: what it does, when it runs, what breaks if it doesn't.
Step 2 — The heartbeat wrapper. Every cron job gets a wrapper: capture exit code, capture duration, and on failure write one line to a file a human actually looks at. Not email. A file. Email is where alerts go to die.
Step 3 — The staleness alarm. For each job, record its last successful run. If a backup's last success is older than its schedule, that's an outage — even though nothing "went down." Stale success beats fresh failure at predicting your next bad morning.
Step 4 — The restore test. One cron-audited backup restored today beats ten unaudited backups restored never.
The pattern behind all four: your monitoring is watching your front door while the back door is ajar. Dashboards watch services. Nobody watches the jobs that make recovery possible.
The fill-in-the-blanks version of this — the cron audit checklist, the heartbeat wrapper script, and a staleness tracker — is in the Ops Starter Kit. If you want the automation to run itself instead of running you, the Automation Starter Pack is the same problem solved with scripts instead of vigilance.
Which cron job on your systems would fail for a month before anyone noticed? Name it in the comments — I'll reply with the heartbeat wrapper for it.
Top comments (0)