Cron jobs fail silently: monitor them with a dead man's switch

Cron jobs are the most silently dangerous services in your infrastructure. When a backup job fails, nothing alerts you, until the day you actually need that backup.

The usual fix is a dead man's switch, also called a heartbeat: instead of watching for failures, you watch for the absence of success. This post explains the pattern, shows a bash script that reports its result correctly, and uses Maintenant as the receiving end. Any heartbeat service (Healthchecks.io, Uptime Kuma push monitors, a self-written endpoint) works the same way.

The heartbeat principle

After each run, your job sends a "ping" to a unique URL. The monitor knows how often that ping should arrive. If the ping does not come within the expected window plus a grace period, you get alerted.

This catches the failures that a "notify on error" approach misses: the crontab entry that was deleted, the host that is down, the job that hangs forever, the script that crashes before it can send anything.

Creating a heartbeat

In Maintenant, a heartbeat is created through the API:

curl -X POST http://localhost:8080/api/v1/heartbeats \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Backup PostgreSQL",
    "interval_seconds": 86400,
    "grace_seconds": 3600
  }'

It returns a unique UUID, used in the ping URL:

A backup script that always reports

The obvious script has two bugs. With set -e, a failing pg_dump exits the script before the final ping is sent. And $? after the start ping no longer holds the exit code of the backup, but that of the curl that came after it.

The fix is to capture the exit code in a variable right after the command, and to send the ping no matter what happened:

#!/bin/bash
set -o pipefail
UUID=your-heartbeat-uuid
PING=http://localhost:8080/ping/${UUID}

curl -fsS -o /dev/null "$PING/start"

rc=0
pg_dump -h postgres -U app mydb | gzip > /backups/mydb-$(date +%Y%m%d).sql.gz || rc=$?

curl -fsS -o /dev/null "$PING/$rc"
exit $rc

Three details matter here:

• There is no set -e. A failure must not end the script before the ping.
• || rc=$? captures the status of the pipeline and keeps the script going. rc=0 first means a success reports 0.
• set -o pipefail makes the pipeline fail if pg_dump fails. Without it, the status is that of gzip, which succeeds on empty input.

Start and end signals

The /start ping is optional, but it lets the monitor:

• Measure the duration of each run
• Detect jobs that start but never finish (timeout, deadlock)
• Track duration trends over time

Exit codes

Appending the exit code to the ping URL (/0, /1, ...) tells the monitor whether the job succeeded. A non-zero code is recorded as a failure, even if the ping arrives on time.

Cron jobs in Docker

The same rule applies in a container: report the status whatever it is, so use ; rather than && between the job and the ping.

services:
  backup:
    image: postgres:16
    entrypoint: /bin/bash
    command: >
      -c "set -o pipefail;
          pg_dump -h postgres -U app mydb | gzip > /backups/daily.sql.gz;
          wget -qO- http://maintenant:8080/ping/YOUR-UUID/$$?"

$$? is how Compose escapes a literal $?. The image must contain wget (or replace it with curl).

Best practices

1. One heartbeat per critical job: backup, cleanup, certificate renewal, sync
2. A generous grace period: if your backup usually takes 5 minutes, set a 30-minute grace period
3. Use start and end signals to detect stuck jobs
4. Pass the exit code: a job that "succeeds" with an error code is not a success

---

Originally published on maintenant.dev.

Maintenant is an open-source, self-hosted monitoring tool for Docker and Kubernetes: containers, endpoints, cron heartbeats, TLS certificates and alerts, in one container that discovers your stack on its own. Install it in a minute: https://maintenant.dev

This project was funded by the French Government as part of the France 2030 plan operated by Bpifrance.