How to retry failed jobs in Node.js without making it worse

Part 3 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The signup scenario is made up. It is the same one used in Part 1 and Part 2. The timings and the provider limits come from a simulation with stated assumptions, not from a real email provider.

In Part 2, the queue sent 20 welcome emails at once, and the provider rejected 15 of them. The queue wrote those 15 failures into a list called failed and moved on. Nobody ever tried them again.

The fix looks obvious: try again. But a retry can also make things worse. If all the failed jobs retry at the same moment, they all hit the provider again at once. And if the error is permanent, the retry only wastes time.

This post adds job queue retries in Node.js, with three things that keep them safe: a growing wait between attempts (backoff), a random element in that wait (jitter), and a way to stop early when trying again cannot help.

The simplest retry

I keep the same made-up assumptions as in Part 2. Each email call takes 300 ms. Twenty welcome emails arrive at the same moment, and the queue runs up to 20 at once. The provider accepts at most 5 calls in flight and rejects a sixth with a "429, too many concurrent requests" error.

Without retries, 5 emails went out, and 15 were lost. Now let each job try up to 10 times, with a wait before every retry (I explain the wait in the next section). Everything goes out:

Strategy	Time	Emails sent	Lost	Calls to the provider	Rejected calls
No retries (Part 2)	0.3 s	5	15	20	15
Retries with backoff	1.8 s	20	0	65	45

Retries are not free. The queue took 1.8 seconds instead of 0.3, made 65 calls instead of 20, and rejected 45 of them. A retry trades time and extra calls for not losing the job. For a welcome email, that is a good trade.

How long to wait: backoff

If a job fails and tries again right away, it will most likely fail again, because the cause is still there. So the job waits, and it waits longer after each failure. The usual rule is to double the wait: 100 ms, 200 ms, 400 ms, and so on, with a maximum so the wait never grows without limit. BullMQ's exponential backoff uses the same doubling.

The code is a wrapper. withRetry takes a handler like the one from Part 2 and returns a new handler. The queue itself does not change.

import type { Handler } from "./queue.ts";

/** Throw this when trying again cannot help: bad input, a mailbox that does not exist. */
export class PermanentError extends Error {}

/** What ends up in `queue.failed` when a job gives up, with the reason. */
export class JobFailedError extends Error {
  constructor(
    readonly reason: "permanent" | "attempts exhausted",
    readonly attempts: number,
    readonly lastError: unknown,
  ) {
    super(`${reason} after ${attempts} attempt(s): ${(lastError as Error).message}`);
  }
}

export type RetryOptions = {
  maxAttempts: number; // total tries, including the first one
  baseDelayMs: number; // wait before the second try
  maxDelayMs: number; // the wait never grows past this
  jitter: boolean; // pick a random wait between 0 and the limit
  random?: () => number; // returns a number in [0, 1); replaceable in tests
  sleep?: (ms: number) => Promise<void>; // replaceable in tests
};

/** Wait after attempt number `attempt` (1 = the first try) has failed. */
export function backoffDelay(attempt: number, options: RetryOptions): number {
  const { baseDelayMs, maxDelayMs, jitter, random = Math.random } = options;
  const limit = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));
  return jitter ? Math.floor(random() * limit) : limit;
}

const defaultSleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

/** Wrap a handler so it retries transient errors and gives up on permanent ones. */
export function withRetry<T>(handler: Handler<T>, options: RetryOptions): Handler<T> {
  const sleep = options.sleep ?? defaultSleep;
  return async (job) => {
    // No loop condition: it ends only by return (success) or throw (give up).
    for (let attempt = 1; ; attempt++) {
      try {
        return await handler(job);
      } catch (error) {
        if (error instanceof PermanentError) {
          throw new JobFailedError("permanent", attempt, error);
        }
        if (attempt >= options.maxAttempts) {
          throw new JobFailedError("attempts exhausted", attempt, error);
        }
        await sleep(backoffDelay(attempt, options));
      }
    }
  };
}

Here are four things to notice:

• backoffDelay doubles the wait after each failure. With a base of 100 ms and a maximum of 2,000 ms, the waits are 100, 200, 400, 800, and 1,600 ms. From then on, every wait is 2,000 ms.
• withRetry runs the handler in a loop. If it succeeds, the loop ends. If it fails, the job waits and tries again until it runs out of attempts.
• When a job gives up, it throws a JobFailedError with the reason, the number of attempts, and the last error. So queue.failed now says why a job failed, not only that it did. A reader on dev.to asked for exactly this after Part 1.
• Using it takes one line around the handler:

import { SimpleQueue, type Job } from "./queue.ts";
import { withRetry } from "./retry.ts";

async function sendWelcomeEmail(job: Job<string>): Promise<void> {
  console.log(`sending to ${job.data}`);
}

const queue = new SimpleQueue<string>(
  withRetry(sendWelcomeEmail, { maxAttempts: 5, baseDelayMs: 100, maxDelayMs: 2000, jitter: true }),
  3, // from Part 2: at most 3 jobs at the same time
);
queue.add("[e-mail]");

Not every error deserves a retry

A "429, too many requests" will probably pass if you wait. A "550, mailbox does not exist" never will. Retrying it burns attempts and keeps a slot busy for nothing.

So the handler throws a PermanentError when trying again cannot help, and withRetry gives up at once. Everything else is treated as temporary. Here is a burst of three emails, one of them to an address that does not exist, with up to 10 attempts each:

Strategy	Calls for the bad address	Time until the queue is idle
Every error is retried	10	14.1 s
PermanentError, no retry	1	0.3 s

The bad address uses ten calls, and the queue stays busy for 14.1 seconds. The other two emails finish in the first 0.3 seconds.

Deciding which errors are permanent is a job for the handler, because only the handler knows what the provider's answer means. My rule: if the cause is in the request itself (bad input, a missing mailbox), it is permanent. If the cause is the state of the other side (a rate limit, a timeout, a server error), it is temporary. BullMQ has the same idea: throwing UnrecoverableError moves a job to the failed set without more retries, even when attempts is set.

Why jitter

Backoff has a weak spot. When many jobs fail at the same moment, they all wait the same 100 ms, then all retry together, then all wait 200 ms. They keep arriving in clusters. Marc Brooker described this problem in the AWS Architecture Blog: backoff alone leaves clusters of calls, and adding jitter spreads them out.

Jitter means the wait is not exactly the backoff value. In the "Full Jitter" version, the wait is a random time between 0 and that value. In the code above, that is jitter: true.

I tested it in the same simulation, with two providers: one that accepts 5 calls at once and one that accepts only 2. Jitter is random, so each jittered row is the average of 20 runs with fixed seeds.

Provider accepts	Strategy	Time	Emails sent (of 20)	Calls
5 at once	Backoff, no jitter	1.8 s	20	65
5 at once	Backoff, full jitter	2.3 s	20	80.8
2 at once	Backoff, no jitter	11.4 s	18	126
2 at once	Backoff, full jitter	4.6 s	19.9	113.9

The result is mixed. With the provider that accepts 5, jitter did not help: it was slower (2.3 s compared with 1.8 s) and made more calls (80.8 compared with 65). With the provider that accepts only 2, it helped: 4.6 seconds instead of 11.4, fewer calls, and nearly every email sent.

My reading is that jitter pays off when many jobs compete for very little room. When there is room, a short random wait mostly adds calls. The AWS article tests a harsher case, 100 clients fighting over one database row, and reports that jitter cut the number of calls by more than half. My simulation is smaller, and most of the gain here is in time, not in calls.

I would still turn jitter on by default. The cost in the mild case is half a second, and the gain in the harsh case is nearly seven seconds. That is my judgment from this simulation, not a rule.

What the simulation does not show

• The 20 jobs live in one queue, and I use them to represent 20 independent callers. In production, the callers that retry in step are often separate servers recovering from the same outage.
• Retries do not replace a sensible limit. In Part 2, a concurrency limit of 5 finished in 1.2 seconds with no rejected calls. That is faster than the 1.8 seconds above. Retries are for failures you cannot prevent.
• The number of attempts should match how long the problem lasts. Change maxAttempts to 5 in the simulation and run it again. In the first table, backoff without jitter still sends all 20 emails. In the jitter table, the full-jitter row for the provider that accepts 5 drops to 16.3 emails sent on average because some jobs run out of attempts before a slot opens.

What this queue still does not do

• A job that waits to retry still holds its slot. The other jobs keep running, and a test covers that. But a job that waits 2 seconds uses one of your concurrency slots for 2 seconds. That is fine for short waits. For waits of minutes, you want delayed jobs, which Part 9 covers.
• close() waits for jobs that are waiting to retry. A test covers this too. A shutdown can take as long as the running jobs need to finish their retries.
• It lives in memory. A restart loses every waiting job, retries included. Part 6 moves the queue to Redis.
• A retry can run a job twice. If the provider sent the email but the answer got lost, the retry sends it again. Part 5 covers idempotency.
• You will not write this by hand in production. BullMQ has attempts and backoff options, with a jitter setting, so the ideas here will look familiar in Part 6.

Takeaways

1. Retry a job when the cause is temporary, and give up at once when it is permanent.
2. Wait longer after each failure, and add jitter when many jobs can fail at the same moment.
3. Record why a job gave up: the reason, the number of attempts, and the last error.

Try it

This is a teaching version of retries. A more complete implementation comes later in the series. Keep the queue from Part 2 as queue.ts and save the retry.ts file from above next to it. Then save this as simulate.ts. It uses a virtual clock and a seeded random generator, so it prints the same numbers every time:

import { mock } from "node:test";
import { SimpleQueue } from "./queue.ts";
import { PermanentError, withRetry, type RetryOptions } from "./retry.ts";

// Illustrative assumptions (not a real provider):
const EMAIL_MS = 300;    // each email call takes 300 ms
const BURST = 20;        // 20 welcome emails arrive at the same moment
const CONCURRENCY = 20;  // the "all at once" setting that lost 15 emails in Part 2
const SEEDS = 20;        // jitter is random, so we repeat it with 20 fixed seeds
// The provider rejects a call with "429" when more than `limit` calls are in flight.

mock.timers.enable({ apis: ["setTimeout"] });
const flush = () => new Promise<void>((resolve) => setImmediate(resolve));
let now = 0;
async function advanceUntil(done: () => boolean) {
  await flush();
  while (!done()) {
    mock.timers.tick(10);
    now += 10;
    await flush();
  }
}
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

// A small seeded random generator, so every run prints the same numbers.
function seededRandom(seed: number) {
  let a = seed;
  return () => {
    a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

type Result = { seconds: number; sent: number; lost: number; calls: number; rejected: number };

async function runBurst(providerLimit: number, retry: RetryOptions | null): Promise<Result> {
  now = 0;
  let inFlight = 0;
  let sent = 0;
  let calls = 0;
  let rejected = 0;
  const sendEmail = async () => {
    calls++;
    inFlight++;
    try {
      if (inFlight > providerLimit) {
        rejected++;
        throw new Error("429 too many concurrent requests");
      }
      await sleep(EMAIL_MS);
      sent++;
    } finally {
      inFlight--;
    }
  };
  const queue = new SimpleQueue<number>(retry ? withRetry(sendEmail, retry) : sendEmail, CONCURRENCY);
  for (let i = 0; i < BURST; i++) queue.add(i);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { seconds: now / 1000, sent, lost: queue.failed.length, calls, rejected };
}

// Average over many seeds, rounded to one decimal.
async function runJittered(providerLimit: number, options: RetryOptions): Promise<Result> {
  const results: Result[] = [];
  for (let seed = 1; seed <= SEEDS; seed++) {
    results.push(await runBurst(providerLimit, { ...options, jitter: true, random: seededRandom(seed) }));
  }
  const avg = (pick: (r: Result) => number) =>
    Math.round((results.reduce((sum, r) => sum + pick(r), 0) / SEEDS) * 10) / 10;
  return {
    seconds: avg((r) => r.seconds), sent: avg((r) => r.sent), lost: avg((r) => r.lost),
    calls: avg((r) => r.calls), rejected: avg((r) => r.rejected),
  };
}

// Three emails, one of them to an address that does not exist.
async function runBadAddress(permanent: boolean, options: RetryOptions) {
  now = 0;
  let badCalls = 0;
  const queue = new SimpleQueue<string>(
    withRetry(async (job) => {
      await sleep(EMAIL_MS);
      if (job.data === "[e-mail]") {
        badCalls++;
        const message = "550 mailbox does not exist";
        throw permanent ? new PermanentError(message) : new Error(message);
      }
    }, options),
    CONCURRENCY,
  );
  for (const to of ["[e-mail]", "[e-mail]", "[e-mail]"]) queue.add(to);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { seconds: now / 1000, callsForTheBadAddress: badCalls };
}

async function main() {
  const options: RetryOptions = { maxAttempts: 10, baseDelayMs: 100, maxDelayMs: 2000, jitter: false };

  console.log("1. Retries on the burst from Part 2 (provider accepts 5 calls at once)");
  console.table([
    { strategy: "no retries (Part 2)", ...(await runBurst(5, null)) },
    { strategy: "retries, backoff", ...(await runBurst(5, options)) },
  ]);

  console.log(`2. Jitter (average of ${SEEDS} seeds for the jittered rows)`);
  const rows = [];
  for (const providerLimit of [5, 2]) {
    rows.push({ providerLimit, strategy: "backoff, no jitter", ...(await runBurst(providerLimit, options)) });
    rows.push({ providerLimit, strategy: "backoff, full jitter", ...(await runJittered(providerLimit, options)) });
  }
  console.table(rows);

  console.log("3. A mailbox that does not exist");
  console.table([
    { strategy: "every error is retried", ...(await runBadAddress(false, options)) },
    { strategy: "PermanentError, no retry", ...(await runBadAddress(true, options)) },
  ]);
  mock.timers.reset();
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

Node prints a warning that its mock timers API is experimental. You can ignore it. You should see these three tables:

1. Retries on the burst from Part 2 (provider accepts 5 calls at once)
┌─────────┬───────────────────────┬─────────┬──────┬──────┬───────┬──────────┐
│ (index) │ strategy              │ seconds │ sent │ lost │ calls │ rejected │
├─────────┼───────────────────────┼─────────┼──────┼──────┼───────┼──────────┤
│ 0       │ 'no retries (Part 2)' │ 0.3     │ 5    │ 15   │ 20    │ 15       │
│ 1       │ 'retries, backoff'    │ 1.8     │ 20   │ 0    │ 65    │ 45       │
└─────────┴───────────────────────┴─────────┴──────┴──────┴───────┴──────────┘
2. Jitter (average of 20 seeds for the jittered rows)
┌─────────┬───────────────┬────────────────────────┬─────────┬──────┬──────┬───────┬──────────┐
│ (index) │ providerLimit │ strategy               │ seconds │ sent │ lost │ calls │ rejected │
├─────────┼───────────────┼────────────────────────┼─────────┼──────┼──────┼───────┼──────────┤
│ 0       │ 5             │ 'backoff, no jitter'   │ 1.8     │ 20   │ 0    │ 65    │ 45       │
│ 1       │ 5             │ 'backoff, full jitter' │ 2.3     │ 20   │ 0    │ 80.8  │ 60.8     │
│ 2       │ 2             │ 'backoff, no jitter'   │ 11.4    │ 18   │ 2    │ 126   │ 108      │
│ 3       │ 2             │ 'backoff, full jitter' │ 4.6     │ 19.9 │ 0.1  │ 113.9 │ 94       │
└─────────┴───────────────┴────────────────────────┴─────────┴──────┴──────┴───────┴──────────┘
3. A mailbox that does not exist
┌─────────┬────────────────────────────┬─────────┬───────────────────────┐
│ (index) │ strategy                   │ seconds │ callsForTheBadAddress │
├─────────┼────────────────────────────┼─────────┼───────────────────────┤
│ 0       │ 'every error is retried'   │ 14.1    │ 10                    │
│ 1       │ 'PermanentError, no retry' │ 0.3     │ 1                     │
└─────────┴────────────────────────────┴─────────┴───────────────────────┘

Next: Part 4 (coming soon) looks at why a workflow with ten steps can fail much more often than any single step suggests.