There's a gap between a script that works when you run it and a script that survives running unattended for a year. The first only has to succeed once, with you watching, on a good day. The second has to keep working through slow disks, hung sockets, recovering services, and reboots — with nobody watching, because the entire point of cron is that nobody is watching.
Almost every script I've seen take down a server was fine in a manual run and fell apart the first time conditions weren't ideal and there was no human there to notice. I've hit three versions of this personally: a sync that stacked copies until the box hit load 41, a backup that hung and didn't run for nine days without a peep, and a deploy that raced a database's boot and paged me at 11pm. Three commands, three failures, one underlying truth — "works on my machine" and "survives cron" are different engineering problems.
Cron jobs die quietly in exactly three ways, and there's a fourth problem underneath all of them that keeps the first three invisible.
The invisible problem first: cron eats your output
When cron runs a job, anything it prints to stdout or stderr goes nowhere unless you've configured otherwise. No log, no record, no trace. So a job can fail every night for a month and the only signal is the absence of whatever it was supposed to produce — and you notice that when something downstream breaks, not when the job fails.
Two things turn cron from silent to legible. MAILTO at the top of the crontab mails you any output a job produces. More reliably, redirect every job's output to a log file you control:
*/5 * * * * /usr/local/bin/safe-sync.sh >> /var/log/sync.log 2>&1
The 2>&1 is the part people forget — it folds stderr into the same stream as stdout, so your errors land in the log instead of evaporating. Everything below assumes you've done this. A locked, time-bounded, retried job that still logs nothing is a job whose failures you'll discover by accident.
Overlap: lock it to one instance
This is the failure that turns a transient slow patch into a self-inflicted outage. A sync scheduled every minute normally finishes in twenty seconds; one slow-disk afternoon it starts taking ninety, and cron launches a new copy every minute on top of the last, until a half-dozen copies are fighting over the same disk and the load climbs into the dozens.
The guard is flock: a kernel-held lock on an open file descriptor that lets exactly one copy run and makes the rest skip. Unlike a PID file, there's nothing to clean up — the kernel releases the lock automatically when the process exits, crashes, or is killed, so you never inherit a stale lock from a run that died badly. But a lock has a sharp edge: if the locked job hangs, it holds the lock forever and every future run skips. The job stops running entirely, silently. Which is the next problem.
Hang: bound the runtime
A hung command is worse than a failed one, because a failed command at least exits and frees its lock. A backup wedged on a held database lock, a curl against a dead socket, an ssh into a black hole — these never return. Under cron, "never returns" means the slot is jammed, the lock is held, and the job produces zero signal because it never gets far enough to log anything.
timeout is the outside bound the command can't set on itself:
# SIGTERM at 5 minutes; if it's ignored (process stuck in I/O), SIGKILL 20s later
timeout -k 20s 5m mysqldump --single-transaction mydb > /backup/mydb.sql.partial
It exits 124 when it had to step in and 137 when the command ignored the polite SIGTERM and had to be force-killed — codes worth branching on, because "slow" and "wedged" are different problems. The -k grace matters specifically for processes stuck in uninterruptible I/O, which can't act on SIGTERM at all. Timeout is what makes a lock safe: with both in place, a hang gets killed on a deadline, the lock always releases on time, and the failure becomes a loud 124 instead of a silent gap.
Transient failure: retry with backoff
Not every failure deserves to kill the run. A fresh database container that isn't accepting connections for six seconds, a 503 during a rolling restart, a 429 explicitly asking you to back off — these are transient. Treating the first one as fatal is how a normal six-second boot delay becomes a failed deploy at eleven at night.
The guard is a bounded retry with exponential backoff: try the command, and on failure wait a delay that doubles each round, with jitter so parallel callers don't retry in lockstep, capped so a genuinely dead dependency fails fast instead of looping forever.
# Wait for the port to be ready before doing the work that depends on it
retry 6 nc -z -w 2 db.internal 5432
The discipline that makes retries safe rather than dangerous: retry transient failures only. A 404 fails identically on every attempt, so retrying it just delays the real error and buries it under retry noise.
Composing all three — and knowing which job needs which
Locking, bounding, and retrying are independent guards, and the real value is stacking them: a single run that can't overlap, can't hang, and rides out a blip — and tells you, by mail or log, when it finally does give up. That composition is mechanical enough to generate, which is exactly what the Hardened Cron Wrapper Generator does: paste the command, toggle the guards, get a ShellCheck-clean wrapper and the crontab line.
But not every job needs all of it — wrapping a one-line date echo in flock and timeout is ceremony. Add a lock to anything that mutates shared state and can outrun its interval. Add a timeout to anything that touches the network or a database. Add retries to anything that depends on something which boots or recovers on its own clock. Log everything, always. The full decision table — guard by guard, when to add it and when to skip it — is in the guide.
Read the whole thing, including the "know when it broke" alerting layer and the decision matrix: https://bashsnippets.xyz/guides/bash-scripts-that-survive-cron
The three guards as standalone snippets: flock, timeout, retry with backoff. To write a new hardened script from a blank slate rather than wrap an existing command, the Bash Boilerplate Generator and the error-handling snippet cover the skeleton. The rest of the library is at https://bashsnippets.xyz
Top comments (0)