Retries and backoff FAQ: 11 questions about retrying failed jobs without making things worse

Retries look like the easy part of background jobs. Something failed? Try again. But badly designed retries turn a small outage into a big one, hide real bugs, and occasionally charge a customer twice.

Here are the questions that come up most, with short answers and code.

1. Why not just retry immediately?

Because whatever caused the failure is probably still happening. If an API is overloaded and returns errors, every client retrying instantly makes it more overloaded. You get a retry storm: the service can't recover because the retries keep it busy.

Waiting before retrying gives the other side time to recover.

2. What is exponential backoff?

Each retry waits longer than the last, usually doubling:

attempt 1 fails → wait 1s
attempt 2 fails → wait 2s
attempt 3 fails → wait 4s
attempt 4 fails → wait 8s

In code:

def backoff(attempt, base=1.0, cap=300.0):
    return min(cap, base * 2 ** attempt)

The cap matters. Without it, attempt 20 waits about 12 days.

3. What is jitter, and why does everyone say to add it?

If a thousand jobs all fail at the same moment (because a dependency went down), plain exponential backoff makes them all retry at the same moments too: all at +1s, all at +2s, all at +4s. You've synchronised the storm.

Jitter adds randomness so retries spread out. The commonly recommended version is "full jitter": pick a random wait between zero and the backoff value.

import random

def backoff_with_jitter(attempt, base=1.0, cap=300.0):
    return random.uniform(0, min(cap, base * 2 ** attempt))

It looks strange (sometimes you wait almost nothing) but over many clients it smooths the load far better than fixed delays.

4. How many times should I retry?

It depends on what the job is and how long the result stays useful:

• User is waiting (an API call during a request): 1–3 quick attempts, then show an error.
• Background job, result needed soon (sending a notification): maybe 5–8 attempts over several minutes to an hour.
• Background job, result needed eventually (syncing data, delivering a webhook): attempts spread over hours or a day.

A better way to think about it: decide the total time window you're willing to keep trying, then choose attempts and backoff to fit inside it.

5. Which errors should I retry?

Retry errors that might go away on their own. Don't retry errors that won't.

function isRetryable(err, res) {
  if (err) return ["ECONNRESET", "ETIMEDOUT", "ECONNREFUSED", "EAI_AGAIN"].includes(err.code);
  if (res.status === 429) return true;          // rate limited
  if (res.status >= 500) return true;           // server errors
  if (res.status === 408) return true;          // request timeout
  return false;                                  // 400, 401, 403, 404, 422...
}

A 400 "invalid request" will fail the same way every time. Retrying it ten times just delays the moment someone notices the bug.

6. What about Retry-After?

If a server responds with 429 or 503 and a Retry-After header, it's telling you when to come back. Use it instead of your own backoff, at least as a minimum:

function waitMs(res, attempt) {
  const ra = res.headers.get("retry-after");
  if (ra) {
    const secs = Number(ra);
    if (!Number.isNaN(secs)) return secs * 1000;
    const date = Date.parse(ra);
    if (!Number.isNaN(date)) return Math.max(0, date - Date.now());
  }
  return Math.random() * Math.min(300_000, 1000 * 2 ** attempt);
}

The header can be a number of seconds or an HTTP date, so handle both.

7. Is it safe to retry anything?

Only if doing it twice has the same effect as doing it once. Reading data: safe. Sending an email or charging a card: not safe unless you've made it so.

The usual approach is an idempotency key: a unique ID for the operation that the receiver uses to detect duplicates. Many payment APIs support this directly. For your own jobs, store "this job ID already completed" and check before doing the work.

The important case to think about is the ambiguous failure: your request timed out, but did the other side process it? Without idempotency, you can't safely tell.

8. Should I retry inside the job or reschedule it?

Two patterns:

• In-process retry: loop with sleep inside the job. Simple, but the worker is busy waiting, and if the process restarts, the retry state is lost.
• Reschedule: on failure, record the attempt count and schedule the job to run again later. Workers are free in between, and retries survive restarts and deploys.

For anything beyond a few seconds of waiting, reschedule. This is exactly what delayed-job queues and schedulers are for. Many queue libraries support it with a built-in attempts/backoff option, and an HTTP scheduler like delay.run can call your endpoint again at the next retry time.

9. What happens after the last attempt?

Something deliberate. Common options:

• Move the job to a "failed" list for a human to look at.
• Alert someone if it's important.
• Mark the related record (order, user, sync) with a clear failed state.

The worst option is the default in many quick implementations: log an error and forget about it.

10. How do I know if retries are hiding a problem?

Track first-attempt success rate separately from final success rate. If final success is 99.9% but first-attempt success dropped from 99% to 70%, something is wrong and retries are covering for it. You're also doing far more work than you think.

metrics.increment("job.attempt", tags={"job": name, "attempt": attempt, "ok": ok})

One metric, tagged by attempt number, makes this visible.

11. What's a sensible default if I just need something?

For a typical background job calling an external API:

• Retry on network errors, 408, 429 and 5xx. Don't retry other 4xx.
• Respect Retry-After.
• Full-jitter exponential backoff, base 1–5 seconds, cap 5–10 minutes.
• 6–8 attempts total.
• Reschedule rather than sleep.
• Idempotency key on anything with side effects.
• A clear final failed state, with an alert for important jobs.

Then adjust based on what you see in the metrics.

Related reading

• How to retry failed cron jobs (with exponential backoff, in Bash and Python)
• Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging
• Make scheduled job handlers idempotent: safe to run twice
• Case study: redesigning a failed-payment retry schedule that retried every card at 00:00 UTC
• Our nightly cleanup job deleted the wrong things. Here's how I'd build it today

What's your default retry policy, and has it ever caused an incident instead of preventing one? I'd love to hear the retry-storm stories in the comments.