A cron job can exit successfully and still fail your business workflow

A scheduled process can exit with code zero while the business task is still incomplete.

A backup script may finish before its upload is durable. An import may return successfully after skipping every malformed row. A billing job may write invoices but fail before notifying customers. Process success and workflow success are different states.

Logs help explain a failure after someone knows to look. A completion heartbeat answers the earlier question: did the expected work finish within the expected window?

Send the heartbeat after the critical work

Put the heartbeat at the end of the business-critical path. If a script pings before the backup reaches remote storage, the monitor confirms only that the script started.

#!/usr/bin/env bash
set -euo pipefail

create_backup
upload_backup
verify_remote_object

curl --fail --silent --show-error \
  --retry 3 \
  "https://your-monitor.example/ping/<job-token>"

The ping should follow verification: the object exists, the export contains accepted rows, or the downstream API acknowledged the operation.

A heartbeat should prove the outcome you care about, not merely that a process woke up.

This loose coupling works across shell scripts, containers, queues, serverless functions, and old applications that are difficult to instrument.

Use an interval and a grace period

A daily job rarely completes at the exact same second. Queue delays, database locks, larger inputs, deployments, and daylight-saving changes can shift completion.

Model two values:

• the expected run interval;
• a grace period based on observed variance.

If a job normally completes in 12 minutes and occasionally needs 20, a 30-minute grace period may be reasonable. If the business deadline is stricter, the workflow needs more capacity rather than a quieter monitor.

Open one incident and close it on recovery

A missed heartbeat should open a single incident instead of sending the same notification every polling cycle. When the heartbeat returns, close the incident and send a recovery message.

The pair gives operators a useful duration and prevents an inbox full of duplicate symptoms. Keep the check idempotent so that multiple scheduler passes do not create duplicate incidents for the same missed window.

Keep the alert path outside the failed component

A monitor running on the same machine as the job cannot report a full host failure. For important work, run the monitor on another host and configure more than one notification path, such as email plus Telegram or a generic webhook.

The monitor itself also needs observation. An external uptime check for its public endpoint closes the most obvious blind spot.

Design the recovery procedure before the alert

Include the job name, expected schedule, last heartbeat, incident start time, and a link to a short runbook. The runbook should answer:

• Is rerunning the job safe?
• How do we detect partial work?
• Who owns the downstream system?

Start with one job whose silent failure already has a real cost. Measure normal completion time for a week, choose a defensible grace period, and run a controlled failure. If the alert arrives but the recovery steps remain ambiguous, the monitoring is not finished.

A working reference implementation

I build SourceBento and sell the complete source code for a small self-hosted Cron Monitor implementing this pattern with TypeScript, Express, SQLite, Docker Compose, missed-run incidents, recovery alerts, and email, Telegram, or webhook notifications.

The public Cron Monitor demo does not require registration.

Disclosure: I am the author and seller of SourceBento. The operational guidance above stands on its own whether or not the product fits your stack.