How to Know When Your Cron Job Silently Stops Running

#cronmonitoring #scheduledtasks #uptime #devops

Had a backup script running every night at 2 AM for about six months. Worked great. Then one day I needed to restore something and discovered the backups stopped three weeks ago. The cron entry was still there. No errors in syslog. The script just... wasn't running.

Turned out a server update changed the PATH and the script couldn't find pg_dump anymore. It failed silently because I was redirecting stderr to /dev/null like a genius.

The core problem

Cron doesn't care if your job succeeds. It cares if it runs. And even "runs" is generous — if cron fires the command and it exits immediately with an error, cron considers that a success.

Most setups have zero monitoring on whether scheduled jobs actually complete. You find out they broke when the thing they were supposed to do didn't happen, which could be days or weeks later.

Dead man's switch — the simplest fix

The idea is backwards from normal monitoring. Instead of checking if something fails, you check if it doesn't report success.

At the end of your cron script, ping a URL:

#!/bin/bash
pg_dump mydb > /backups/mydb_$(date +%F).sql

if [ $? -eq 0 ]; then
    curl -fsS --retry 3 https://your-monitoring/ping/abc123 > /dev/null
fi

If the monitoring service doesn't receive that ping within the expected window (say, 30 minutes after 2 AM), it alerts you.

That's it. No agent to install, no daemon running, no config files. One curl at the end of your script.

What trips people up

Forgetting to check the exit code. If you just put the curl at the end without the if check, it pings even when the job fails. Defeats the entire purpose.

Timeout too tight. Your job usually takes 5 minutes but sometimes takes 20. If your monitoring window is 10 minutes, you'll get false alerts. Set it to 2x your worst case.

Multiple schedules. A job that runs every hour needs a different timeout than one that runs daily. I've seen people set up monitoring for a daily job and use a 1-hour timeout — so it alerts 23 times before the next run.

Swallowed output. Stop doing 2>/dev/null on cron jobs. Send stderr somewhere useful:

*/5 * * * * /path/to/script.sh >> /var/log/myjob.log 2>&1

DIY vs hosted

You can build this yourself. A small endpoint that logs pings, a checker that runs every minute and compares last-ping time against expected schedule, and a notification sender. Maybe 200 lines of code.

I did that once. It worked until the monitoring server itself went down and nobody noticed because nothing was monitoring the monitor. Classic.

Hosted services handle the reliability part — redundant infrastructure, multiple notification channels, escalation if you don't acknowledge. Worth it if you have more than a handful of jobs to track.