How to retry failed cron jobs (with exponential backoff, in Bash and Python)

Cron has exactly one opinion about failure: it doesn't have one. If your job exits with an error at 02:00, cron writes nothing, retries nothing and tells nobody. The next attempt is whenever the schedule says, which might be tomorrow.

Most failures in scheduled jobs are temporary: a database restarting, an API returning 503, a network blip. A couple of retries with a pause in between turns most of those into non-events. Here's how to add retries properly, and where to stop.

Rule 1: decide what's worth retrying

Retrying makes sense for errors that might fix themselves:

• timeouts and connection errors
• HTTP 429 (too many requests), 502, 503, 504
• "database is locked", "connection reset" and similar

Retrying does not help, and can make things worse, for:

• HTTP 400, 401, 403, 404: the request is wrong and will stay wrong
• validation errors, missing files, bad configuration
• bugs in your code

Retrying a permanent error just delays the alert and adds load. Fail fast on those.

Rule 2: back off, and add jitter

Retrying immediately usually hits the same problem. Wait longer after each attempt: 5 seconds, then 10, then 20. This is exponential backoff. Add a little randomness (jitter) so that many jobs failing at once don't all retry at the same moment and knock the service over again.

Retries in Bash

A small function you can drop into any cron script:

#!/usr/bin/env bash
set -uo pipefail

retry() {
  local max=$1; shift
  local delay=5 attempt=1
  while true; do
    "$@" && return 0
    local code=$?
    if (( attempt >= max )); then
      echo "$(date -u +%FT%TZ) giving up after $attempt attempts (exit $code): $*" >&2
      return "$code"
    fi
    local sleep_for=$(( delay + RANDOM % delay ))
    echo "$(date -u +%FT%TZ) attempt $attempt failed (exit $code), retrying in ${sleep_for}s" >&2
    sleep "$sleep_for"
    attempt=$(( attempt + 1 ))
    delay=$(( delay * 2 ))
  done
}

retry 4 /usr/local/bin/export-orders.sh

30 2 * * * /usr/local/bin/export-with-retry.sh >> /var/log/export.log 2>&1

Four attempts with this delay pattern finish within a couple of minutes. Keep the total retry time well below the job's interval, or the next scheduled run will start while this one is still retrying. If that's possible, wrap the job in flock -n so runs can't overlap.

For HTTP calls, curl can do it

If the job is a single request, curl's built-in retries are enough:

curl -fsS --max-time 30 --retry 4 --retry-delay 5 --retry-max-time 120 \
  https://example.com/tasks/export

--retry only retries transient problems: timeouts and responses like 408, 429, 500, 502, 503 and 504. Recent curl versions also honour a Retry-After header. Avoid --retry-all-errors unless you really want to retry 4xx responses too.

Retries in Python

For jobs written in Python, the same logic with a permanent/transient split:

import random
import time

import requests

class Permanent(Exception):
    pass

def call_api():
    r = requests.post("https://api.example.com/export", timeout=30)
    if r.status_code in (429, 500, 502, 503, 504):
        r.raise_for_status()                     # transient: let retry handle it
    if r.status_code >= 400:
        raise Permanent(f"{r.status_code}: {r.text[:200]}")
    return r.json()

def retry(fn, attempts=4, base=5):
    for i in range(1, attempts + 1):
        try:
            return fn()
        except Permanent:
            raise
        except (requests.RequestException,) as e:
            if i == attempts:
                raise
            wait = base * 2 ** (i - 1) + random.uniform(0, base)
            print(f"attempt {i} failed ({e}); retrying in {wait:.0f}s", flush=True)
            time.sleep(wait)

if __name__ == "__main__":
    print(retry(call_api))

If you'd rather not write this yourself, the tenacity library provides decorators for the same patterns.

Rule 3: make the job safe to run twice

Retries mean the same work may run more than once. A request can succeed on the server and still time out on your side, so you retry something that already happened. Protect against that:

• Use an idempotency key. Many payment and messaging APIs accept a key per operation and ignore duplicates. Derive it from the job and date, such as export-2026-10-28.
• Record what's done. Write a marker (a database row, a file) when the work completes, and check it at the start.
• Use upserts, not inserts. INSERT ... ON CONFLICT DO UPDATE survives a second run; a plain INSERT creates duplicates.

Rule 4: alert when retries run out

A retry loop hides problems by design, so you need to hear about the ones it can't fix. Two simple options.

Alert on final failure:

if ! retry 4 /usr/local/bin/export-orders.sh; then
  curl -fsS -X POST -H 'Content-Type: application/json' \
    -d '{"text":"export-orders failed after 4 attempts"}' "$SLACK_WEBHOOK_URL"
  exit 1
fi

Alert after N failed runs in a row, which avoids paging anyone for a single bad night:

STATE=/var/lib/export-orders.failures
if retry 4 /usr/local/bin/export-orders.sh; then
  rm -f "$STATE"
else
  n=$(( $(cat "$STATE" 2>/dev/null || echo 0) + 1 ))
  echo "$n" > "$STATE"
  (( n >= 3 )) && curl -fsS -X POST -d "{\"text\":\"export-orders failed $n runs in a row\"}" \
    -H 'Content-Type: application/json' "$SLACK_WEBHOOK_URL"
  exit 1
fi

Rule 5: keep failed work somewhere you can replay it

For jobs that process items (orders, emails, webhooks), don't let a failure lose the item. Write failed items to a "dead letter" table or file with the error, and add a small command to retry them once the cause is fixed. That turns an outage into a backlog instead of missing data.

Or let the scheduler handle it

If the job is an HTTP endpoint, a hosted scheduler can take over most of this. delay.run, for example, retries failed jobs with backoff (by default after 1 minute, 30 minutes and 3 hours) and moves them to a dead letter queue you can replay. Recurring crons aren't retried; instead it alerts by email, Slack or Discord after the number of failures in a row you choose, and again when the job recovers, with every run's response logged.

Summary

• Cron never retries. Add retries in the job or use a scheduler that does.
• Retry only transient errors, with exponential backoff and jitter.
• Keep total retry time shorter than the schedule interval.
• Make the job idempotent, because retries will repeat work.
• Alert when retries are exhausted, ideally after a few failures in a row.
• Save failed items so they can be replayed.