Monitoring cron jobs with heartbeats: how to notice the job that silently stopped

The worst cron failure is not the one that throws an error. It's the one that simply doesn't happen.

The backup that hasn't run since the server was migrated. The import that stopped when someone redeployed and the crontab wasn't part of the image. The cleanup script that hangs forever on a lock, so the next runs pile up behind it. Nothing crashes, no error mail goes out, and you find out weeks later when you actually need that backup.

I run a cron service, and "it stopped and nobody noticed" is by far the most common story I hear. The fix is a simple pattern called a heartbeat (also known as a dead man's switch).

The idea in one sentence

Your job sends a small "I'm done" signal to a URL every time it finishes, and a monitor alerts you when that signal doesn't arrive on time.

The important part is the direction. Classic monitoring asks "did something go wrong?". A heartbeat asks "did the thing that should have happened actually happen?". That catches every failure mode at once: the server is down, cron isn't running, the crontab is gone, the script crashed before it could report anything, the job hangs.

The simplest version

Say this is your nightly backup:

# crontab
30 2 * * * /usr/local/bin/backup.sh

Add one request at the end, and only when the script succeeded:

30 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 https://example-monitor/your-check-id

• && means the ping is only sent if backup.sh exited with code 0. If the backup fails, no ping arrives and you get an alert.
• -fsS keeps curl quiet but still shows real errors.
• -m 10 stops curl after 10 seconds, so a slow monitoring endpoint can never block your job.
• --retry 3 covers a short network hiccup, so you don't get a false alarm.

On the monitor side you set two things: the schedule (every day at 02:30) and a grace period (for example 30 minutes). If no ping arrives by 03:00, you get notified.

Report failures right away

With only &&, a failed run is noticed when the grace period is over. If your monitor supports it, report the exit code directly instead:

30 2 * * * /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 3 https://example-monitor/your-check-id/$?

$? is the exit code of backup.sh. Many heartbeat tools accept it in the URL: 0 counts as success, anything else as an immediate failure. One line, both cases covered.

Catch jobs that hang

A job that never finishes is the sneakiest case: no success, no error, just a process sitting there. Send a signal when the job starts, too:

#!/bin/sh
URL=https://example-monitor/your-check-id

curl -fsS -m 10 --retry 3 "$URL/start" > /dev/null

/usr/local/bin/import.sh
STATUS=$?

curl -fsS -m 10 --retry 3 "$URL/$STATUS" > /dev/null
exit $STATUS

Now the monitor knows the job began. If neither success nor failure follows within the grace period, it can tell you the run is stuck. As a bonus you get the duration of every run, which is great for spotting a job that slowly gets slower.

Send the output along

When something fails, the first question is "what did it print?". If your monitor accepts a POST body, send the last lines of the log with the ping:

/usr/local/bin/import.sh > /tmp/import.log 2>&1
STATUS=$?
tail -c 10000 /tmp/import.log | curl -fsS -m 10 --retry 3 --data-binary @- "$URL/$STATUS"

Then the alert and the error message arrive together, and you don't have to SSH into the server to find out what happened.

Choosing the grace period

The grace period decides how fast you hear about problems versus how many false alarms you get.

• Longer than the normal run time. If the job usually takes 20 minutes and sometimes 40, a 15 minute grace period will page you for nothing.
• Shorter than "too late". If the export must be ready by 7:00 for the morning report, the alert has to come before that.
• For frequent jobs, allow a missed run. A job every 5 minutes with a 10 minute grace period tolerates one slow or skipped run before alerting. That's usually what you want.

Pitfalls I see again and again

The ping breaks the job. If the monitoring service is unreachable and your script runs with set -e, a failing curl can fail the whole job. Keep the timeout short and add || true when the ping is inside the script and must never stop it.

Pinging at the start instead of the end. A single ping as the first line proves only that cron started the script. Put it at the end, or use the start/finish pair above.

Cron's minimal environment. Cron runs with a very small PATH. A job that works in your terminal can fail under cron because curl or your interpreter isn't found. Use absolute paths, e.g. /usr/bin/curl.

Time zones. The server's cron may run in UTC while you configured the monitor in local time, or the other way around. Then every alert comes an hour or two "late" and everyone learns to ignore them. Check both sides once.

Alerts that go nowhere. An alert to root@localhost or to the inbox of a colleague who left last year is the same as no alert. Send it to a channel people actually read, like a team mailbox, Slack or Teams, and test it once.

Overlapping runs. If a run takes longer than the interval, two copies can run at once. Wrap the job in flock -n /tmp/import.lock so a second copy exits immediately instead of piling up, and let the heartbeat tell you when runs get too long.

Tools

The pattern is simple enough to build yourself, but then you have to monitor your monitor. Some ready-made options:

• Healthchecks.io: open source, you can use the hosted version or run it yourself.
• Cronitor: commercial cron and job monitoring.
• Cronjob.de: that's mine. Heartbeats with /start, /fail and the exit code in the URL as shown above, POST output up to 10 KB, alerts by e-mail, and on the Unlimited plan also via Slack, Teams, Discord, Telegram or webhook. The free plan includes one heartbeat with e-mail alerts, servers are in the EU.

Whichever you choose: set up one heartbeat for the job you'd be most embarrassed to find broken. For most of us that's the backup.

---

I run Cronjob.de, a cron service from Germany, so I'm obviously not neutral about the tools section. The pattern itself works with any of them.