The lie is silence
A cron job that runs is not a cron job that worked. Most small teams discover this during the worst possible week: the backup was failing for 40 days, and every morning's log line looked fine because the script exited 0 even when the transfer was empty.
Here are the four silent failure modes, and the tiny guard that catches each one.
1. Exit code 0 on partial failure
Scripts that loop over items and continue on error end the loop successfully. Guard: after the loop, test ${FAIL_COUNT} -eq 0 || exit 1. Track failures explicitly or the exit code lies.
2. Empty success
The backup "completed" — and produced a 0-byte file. Guard: check output size and modification time, not just exit status. test $(stat -f%z out.tar.gz) -gt 1000000 || exit 2.
3. The cron daemon itself is dead
Nothing runs, nothing logs, nobody notices — because your alerting is inside the thing that died. Guard: an external heartbeat. The job pings a dead-man's-switch URL (healthchecks-style, or a cron on a different box) on success. Silence becomes the alarm.
4. Silent hang
The job never exits, so it never fails, so it never alerts. Guard: wrap with timeout 1800 ./job.sh and alert on 124. A hung job is a failed job; most setups never encode that.
One guard script to rule them
Wrap every cron entry in a tiny supervisor: run, time, check output size, ping heartbeat, write one summary line. Twenty lines of shell. The difference between "we have backups" and "we have backups we can prove ran" is roughly this guard.
If you'd rather not write it from scratch, I package exactly this — the guard script, the heartbeat pattern, and the failure-playbook templates:
- Ops Starter Kit — https://hive80lab.gumroad.com/l/ops-starter-kit
- Automation Starter Pack — https://hive80lab.gumroad.com/l/automation-starter-pack
- Agent-Ops 24/7 — https://hive80lab.gumroad.com/l/agent-ops-24-7
Either way: audit your crontab today for these four modes. One of them is probably live right now.
Boring automation for small IT teams. More at https://hive80lab.gumroad.com
Top comments (0)