DEV Community: Sandro Dezerio

How to retry failed jobs in Node.js without making it worse

Sandro Dezerio — Wed, 14 Oct 2026 17:00:08 +0000

Part 3 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The signup scenario is made up. It is the same one used in Part 1 and Part 2. The timings and the provider limits come from a simulation with stated assumptions, not from a real email provider.

In Part 2, the queue sent 20 welcome emails at once, and the provider rejected 15 of them. The queue wrote those 15 failures into a list called failed and moved on. Nobody ever tried them again.

The fix looks obvious: try again. But a retry can also make things worse. If all the failed jobs retry at the same moment, they all hit the provider again at once. And if the error is permanent, the retry only wastes time.

This post adds job queue retries in Node.js, with three things that keep them safe: a growing wait between attempts (backoff), a random element in that wait (jitter), and a way to stop early when trying again cannot help.

The simplest retry

I keep the same made-up assumptions as in Part 2. Each email call takes 300 ms. Twenty welcome emails arrive at the same moment, and the queue runs up to 20 at once. The provider accepts at most 5 calls in flight and rejects a sixth with a "429, too many concurrent requests" error.

Without retries, 5 emails went out, and 15 were lost. Now let each job try up to 10 times, with a wait before every retry (I explain the wait in the next section). Everything goes out:

Strategy	Time	Emails sent	Lost	Calls to the provider	Rejected calls
No retries (Part 2)	0.3 s	5	15	20	15
Retries with backoff	1.8 s	20	0	65	45

Retries are not free. The queue took 1.8 seconds instead of 0.3, made 65 calls instead of 20, and rejected 45 of them. A retry trades time and extra calls for not losing the job. For a welcome email, that is a good trade.

How long to wait: backoff

If a job fails and tries again right away, it will most likely fail again, because the cause is still there. So the job waits, and it waits longer after each failure. The usual rule is to double the wait: 100 ms, 200 ms, 400 ms, and so on, with a maximum so the wait never grows without limit. BullMQ's exponential backoff uses the same doubling.

The code is a wrapper. withRetry takes a handler like the one from Part 2 and returns a new handler. The queue itself does not change.

import type { Handler } from "./queue.ts";

/** Throw this when trying again cannot help: bad input, a mailbox that does not exist. */
export class PermanentError extends Error {}

/** What ends up in `queue.failed` when a job gives up, with the reason. */
export class JobFailedError extends Error {
  constructor(
    readonly reason: "permanent" | "attempts exhausted",
    readonly attempts: number,
    readonly lastError: unknown,
  ) {
    super(`${reason} after ${attempts} attempt(s): ${(lastError as Error).message}`);
  }
}

export type RetryOptions = {
  maxAttempts: number; // total tries, including the first one
  baseDelayMs: number; // wait before the second try
  maxDelayMs: number; // the wait never grows past this
  jitter: boolean; // pick a random wait between 0 and the limit
  random?: () => number; // returns a number in [0, 1); replaceable in tests
  sleep?: (ms: number) => Promise<void>; // replaceable in tests
};

/** Wait after attempt number `attempt` (1 = the first try) has failed. */
export function backoffDelay(attempt: number, options: RetryOptions): number {
  const { baseDelayMs, maxDelayMs, jitter, random = Math.random } = options;
  const limit = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));
  return jitter ? Math.floor(random() * limit) : limit;
}

const defaultSleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

/** Wrap a handler so it retries transient errors and gives up on permanent ones. */
export function withRetry<T>(handler: Handler<T>, options: RetryOptions): Handler<T> {
  const sleep = options.sleep ?? defaultSleep;
  return async (job) => {
    // No loop condition: it ends only by return (success) or throw (give up).
    for (let attempt = 1; ; attempt++) {
      try {
        return await handler(job);
      } catch (error) {
        if (error instanceof PermanentError) {
          throw new JobFailedError("permanent", attempt, error);
        }
        if (attempt >= options.maxAttempts) {
          throw new JobFailedError("attempts exhausted", attempt, error);
        }
        await sleep(backoffDelay(attempt, options));
      }
    }
  };
}

Here are four things to notice:

backoffDelay doubles the wait after each failure. With a base of 100 ms and a maximum of 2,000 ms, the waits are 100, 200, 400, 800, and 1,600 ms. From then on, every wait is 2,000 ms.
withRetry runs the handler in a loop. If it succeeds, the loop ends. If it fails, the job waits and tries again until it runs out of attempts.
When a job gives up, it throws a JobFailedError with the reason, the number of attempts, and the last error. So queue.failed now says why a job failed, not only that it did. A reader on dev.to asked for exactly this after Part 1.
Using it takes one line around the handler:

import { SimpleQueue, type Job } from "./queue.ts";
import { withRetry } from "./retry.ts";

async function sendWelcomeEmail(job: Job<string>): Promise<void> {
  console.log(`sending to ${job.data}`);
}

const queue = new SimpleQueue<string>(
  withRetry(sendWelcomeEmail, { maxAttempts: 5, baseDelayMs: 100, maxDelayMs: 2000, jitter: true }),
  3, // from Part 2: at most 3 jobs at the same time
);
queue.add("ana@example.com");

Not every error deserves a retry

A "429, too many requests" will probably pass if you wait. A "550, mailbox does not exist" never will. Retrying it burns attempts and keeps a slot busy for nothing.

So the handler throws a PermanentError when trying again cannot help, and withRetry gives up at once. Everything else is treated as temporary. Here is a burst of three emails, one of them to an address that does not exist, with up to 10 attempts each:

Strategy	Calls for the bad address	Time until the queue is idle
Every error is retried	10	14.1 s
`PermanentError`, no retry	1	0.3 s

The bad address uses ten calls, and the queue stays busy for 14.1 seconds. The other two emails finish in the first 0.3 seconds.

Deciding which errors are permanent is a job for the handler, because only the handler knows what the provider's answer means. My rule: if the cause is in the request itself (bad input, a missing mailbox), it is permanent. If the cause is the state of the other side (a rate limit, a timeout, a server error), it is temporary. BullMQ has the same idea: throwing UnrecoverableError moves a job to the failed set without more retries, even when attempts is set.

Why jitter

Backoff has a weak spot. When many jobs fail at the same moment, they all wait the same 100 ms, then all retry together, then all wait 200 ms. They keep arriving in clusters. Marc Brooker described this problem in the AWS Architecture Blog: backoff alone leaves clusters of calls, and adding jitter spreads them out.

Jitter means the wait is not exactly the backoff value. In the "Full Jitter" version, the wait is a random time between 0 and that value. In the code above, that is jitter: true.

I tested it in the same simulation, with two providers: one that accepts 5 calls at once and one that accepts only 2. Jitter is random, so each jittered row is the average of 20 runs with fixed seeds.

Provider accepts	Strategy	Time	Emails sent (of 20)	Calls
5 at once	Backoff, no jitter	1.8 s	20	65
5 at once	Backoff, full jitter	2.3 s	20	80.8
2 at once	Backoff, no jitter	11.4 s	18	126
2 at once	Backoff, full jitter	4.6 s	19.9	113.9

The result is mixed. With the provider that accepts 5, jitter did not help: it was slower (2.3 s compared with 1.8 s) and made more calls (80.8 compared with 65). With the provider that accepts only 2, it helped: 4.6 seconds instead of 11.4, fewer calls, and nearly every email sent.

My reading is that jitter pays off when many jobs compete for very little room. When there is room, a short random wait mostly adds calls. The AWS article tests a harsher case, 100 clients fighting over one database row, and reports that jitter cut the number of calls by more than half. My simulation is smaller, and most of the gain here is in time, not in calls.

I would still turn jitter on by default. The cost in the mild case is half a second, and the gain in the harsh case is nearly seven seconds. That is my judgment from this simulation, not a rule.

What the simulation does not show

The 20 jobs live in one queue, and I use them to represent 20 independent callers. In production, the callers that retry in step are often separate servers recovering from the same outage.
Retries do not replace a sensible limit. In Part 2, a concurrency limit of 5 finished in 1.2 seconds with no rejected calls. That is faster than the 1.8 seconds above. Retries are for failures you cannot prevent.
The number of attempts should match how long the problem lasts. Change maxAttempts to 5 in the simulation and run it again. In the first table, backoff without jitter still sends all 20 emails. In the jitter table, the full-jitter row for the provider that accepts 5 drops to 16.3 emails sent on average because some jobs run out of attempts before a slot opens.

What this queue still does not do

A job that waits to retry still holds its slot. The other jobs keep running, and a test covers that. But a job that waits 2 seconds uses one of your concurrency slots for 2 seconds. That is fine for short waits. For waits of minutes, you want delayed jobs, which Part 9 covers.
close() waits for jobs that are waiting to retry. A test covers this too. A shutdown can take as long as the running jobs need to finish their retries.
It lives in memory. A restart loses every waiting job, retries included. Part 6 moves the queue to Redis.
A retry can run a job twice. If the provider sent the email but the answer got lost, the retry sends it again. Part 5 covers idempotency.
You will not write this by hand in production. BullMQ has attempts and backoff options, with a jitter setting, so the ideas here will look familiar in Part 6.

Takeaways

Retry a job when the cause is temporary, and give up at once when it is permanent.
Wait longer after each failure, and add jitter when many jobs can fail at the same moment.
Record why a job gave up: the reason, the number of attempts, and the last error.

Try it

This is a teaching version of retries. A more complete implementation comes later in the series. Keep the queue from Part 2 as queue.ts and save the retry.ts file from above next to it. Then save this as simulate.ts. It uses a virtual clock and a seeded random generator, so it prints the same numbers every time:

import { mock } from "node:test";
import { SimpleQueue } from "./queue.ts";
import { PermanentError, withRetry, type RetryOptions } from "./retry.ts";

// Illustrative assumptions (not a real provider):
const EMAIL_MS = 300;    // each email call takes 300 ms
const BURST = 20;        // 20 welcome emails arrive at the same moment
const CONCURRENCY = 20;  // the "all at once" setting that lost 15 emails in Part 2
const SEEDS = 20;        // jitter is random, so we repeat it with 20 fixed seeds
// The provider rejects a call with "429" when more than `limit` calls are in flight.

mock.timers.enable({ apis: ["setTimeout"] });
const flush = () => new Promise<void>((resolve) => setImmediate(resolve));
let now = 0;
async function advanceUntil(done: () => boolean) {
  await flush();
  while (!done()) {
    mock.timers.tick(10);
    now += 10;
    await flush();
  }
}
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

// A small seeded random generator, so every run prints the same numbers.
function seededRandom(seed: number) {
  let a = seed;
  return () => {
    a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

type Result = { seconds: number; sent: number; lost: number; calls: number; rejected: number };

async function runBurst(providerLimit: number, retry: RetryOptions | null): Promise<Result> {
  now = 0;
  let inFlight = 0;
  let sent = 0;
  let calls = 0;
  let rejected = 0;
  const sendEmail = async () => {
    calls++;
    inFlight++;
    try {
      if (inFlight > providerLimit) {
        rejected++;
        throw new Error("429 too many concurrent requests");
      }
      await sleep(EMAIL_MS);
      sent++;
    } finally {
      inFlight--;
    }
  };
  const queue = new SimpleQueue<number>(retry ? withRetry(sendEmail, retry) : sendEmail, CONCURRENCY);
  for (let i = 0; i < BURST; i++) queue.add(i);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { seconds: now / 1000, sent, lost: queue.failed.length, calls, rejected };
}

// Average over many seeds, rounded to one decimal.
async function runJittered(providerLimit: number, options: RetryOptions): Promise<Result> {
  const results: Result[] = [];
  for (let seed = 1; seed <= SEEDS; seed++) {
    results.push(await runBurst(providerLimit, { ...options, jitter: true, random: seededRandom(seed) }));
  }
  const avg = (pick: (r: Result) => number) =>
    Math.round((results.reduce((sum, r) => sum + pick(r), 0) / SEEDS) * 10) / 10;
  return {
    seconds: avg((r) => r.seconds), sent: avg((r) => r.sent), lost: avg((r) => r.lost),
    calls: avg((r) => r.calls), rejected: avg((r) => r.rejected),
  };
}

// Three emails, one of them to an address that does not exist.
async function runBadAddress(permanent: boolean, options: RetryOptions) {
  now = 0;
  let badCalls = 0;
  const queue = new SimpleQueue<string>(
    withRetry(async (job) => {
      await sleep(EMAIL_MS);
      if (job.data === "nobody@example.invalid") {
        badCalls++;
        const message = "550 mailbox does not exist";
        throw permanent ? new PermanentError(message) : new Error(message);
      }
    }, options),
    CONCURRENCY,
  );
  for (const to of ["ana@example.com", "nobody@example.invalid", "luis@example.com"]) queue.add(to);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { seconds: now / 1000, callsForTheBadAddress: badCalls };
}

async function main() {
  const options: RetryOptions = { maxAttempts: 10, baseDelayMs: 100, maxDelayMs: 2000, jitter: false };

  console.log("1. Retries on the burst from Part 2 (provider accepts 5 calls at once)");
  console.table([
    { strategy: "no retries (Part 2)", ...(await runBurst(5, null)) },
    { strategy: "retries, backoff", ...(await runBurst(5, options)) },
  ]);

  console.log(`2. Jitter (average of ${SEEDS} seeds for the jittered rows)`);
  const rows = [];
  for (const providerLimit of [5, 2]) {
    rows.push({ providerLimit, strategy: "backoff, no jitter", ...(await runBurst(providerLimit, options)) });
    rows.push({ providerLimit, strategy: "backoff, full jitter", ...(await runJittered(providerLimit, options)) });
  }
  console.table(rows);

  console.log("3. A mailbox that does not exist");
  console.table([
    { strategy: "every error is retried", ...(await runBadAddress(false, options)) },
    { strategy: "PermanentError, no retry", ...(await runBadAddress(true, options)) },
  ]);
  mock.timers.reset();
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

Node prints a warning that its mock timers API is experimental. You can ignore it. You should see these three tables:

1. Retries on the burst from Part 2 (provider accepts 5 calls at once)
┌─────────┬───────────────────────┬─────────┬──────┬──────┬───────┬──────────┐
│ (index) │ strategy              │ seconds │ sent │ lost │ calls │ rejected │
├─────────┼───────────────────────┼─────────┼──────┼──────┼───────┼──────────┤
│ 0       │ 'no retries (Part 2)' │ 0.3     │ 5    │ 15   │ 20    │ 15       │
│ 1       │ 'retries, backoff'    │ 1.8     │ 20   │ 0    │ 65    │ 45       │
└─────────┴───────────────────────┴─────────┴──────┴──────┴───────┴──────────┘
2. Jitter (average of 20 seeds for the jittered rows)
┌─────────┬───────────────┬────────────────────────┬─────────┬──────┬──────┬───────┬──────────┐
│ (index) │ providerLimit │ strategy               │ seconds │ sent │ lost │ calls │ rejected │
├─────────┼───────────────┼────────────────────────┼─────────┼──────┼──────┼───────┼──────────┤
│ 0       │ 5             │ 'backoff, no jitter'   │ 1.8     │ 20   │ 0    │ 65    │ 45       │
│ 1       │ 5             │ 'backoff, full jitter' │ 2.3     │ 20   │ 0    │ 80.8  │ 60.8     │
│ 2       │ 2             │ 'backoff, no jitter'   │ 11.4    │ 18   │ 2    │ 126   │ 108      │
│ 3       │ 2             │ 'backoff, full jitter' │ 4.6     │ 19.9 │ 0.1  │ 113.9 │ 94       │
└─────────┴───────────────┴────────────────────────┴─────────┴──────┴──────┴───────┴──────────┘
3. A mailbox that does not exist
┌─────────┬────────────────────────────┬─────────┬───────────────────────┐
│ (index) │ strategy                   │ seconds │ callsForTheBadAddress │
├─────────┼────────────────────────────┼─────────┼───────────────────────┤
│ 0       │ 'every error is retried'   │ 14.1    │ 10                    │
│ 1       │ 'PermanentError, no retry' │ 0.3     │ 1                     │
└─────────┴────────────────────────────┴─────────┴───────────────────────┘

Next: Part 4 (coming soon) looks at why a workflow with ten steps can fail much more often than any single step suggests.

How many jobs should a Node.js queue run at once?

Sandro Dezerio — Sat, 10 Oct 2026 02:13:01 +0000

Part 2 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The signup scenario is made up, the same one as in Part 1. The timings and the provider limit come from a simulation with stated assumptions, not from a real email provider.

The queue from Part 1 sends one email at a time. When 20 signups arrive together, the last welcome email leaves six seconds later.

The obvious fix is to send them all at once. That works until the other side says no.

This post adds job queue concurrency to the queue: a setting for how many jobs may run at the same time. It also shows how to choose that number, and how to shut the queue down without losing work that is already running.

Why one at a time is slow

I will keep the made-up assumptions from Part 1. Each email call takes 300 ms. This time 20 welcome emails arrive at the same moment, and nothing fails.

One at a time, that is 20 x 300 ms, or 6.0 seconds. For most of that time, the server is only waiting for the provider to answer. The queue is safe, but it wastes the wait.

Why all at once is worse

Now say the email provider accepts at most 5 calls in flight at the same time. A sixth call is rejected right away with a "429, too many concurrent requests" error. This limit is made up, but real services have limits like it.

If the queue starts all 20 emails together, everything finishes in 0.3 seconds. It also loses 15 of the 20 emails because the provider rejected them.

All at once looks like the fastest option, and it loses the most work. So the goal is not raw speed. It is the fastest rate the other side accepts.

Picture a small app that sends a welcome email on signup. A newsletter mentions it, and 200 people sign up within a few minutes. With one worker and 300 ms per email, the last person waits a full minute. So someone changes the limit to "as many as possible". Now the provider answers with 429 errors, and with no retries the rejected emails are simply lost. Both versions of the queue fail, in opposite ways. This scenario is made up, like the rest of this post.

The change: a job queue concurrency limit

Here is the queue again. The rest of the class works as in Part 1, but the worker loop changed, so I show the whole file once.

/**
 * Part 2: the same queue, now with a concurrency limit and a clean shutdown.
 * Still in memory, still no retries. Those are Part 3 and Part 6.
 */

export type Job<T> = { id: number; data: T };
export type Handler<T> = (job: Job<T>) => Promise<void>;

export class SimpleQueue<T> {
  private jobs: Job<T>[] = [];
  private nextId = 1;
  private active = 0; // jobs running right now
  private closing = false;
  private idleWaiters: Array<() => void> = [];
  readonly failed: Array<{ job: Job<T>; error: unknown }> = [];

  constructor(
    private handler: Handler<T>,
    private concurrency = 1, // how many jobs may run at the same time
  ) {}

  /** Put a job in line and return right away. The work happens later. */
  add(data: T): number {
    if (this.closing) throw new Error("queue is closing");
    const job = { id: this.nextId++, data };
    this.jobs.push(job);
    this.pump();
    return job.id;
  }

  /** Start jobs until the limit is reached or the line is empty. */
  private pump(): void {
    while (!this.closing && this.active < this.concurrency && this.jobs.length > 0) {
      const job = this.jobs.shift()!;
      this.active++;
      void this.run(job);
    }
  }

  private async run(job: Job<T>): Promise<void> {
    try {
      await this.handler(job);
    } catch (error) {
      this.failed.push({ job, error }); // remember it, keep going
    }
    this.active--;
    this.pump(); // a slot is free: start the next job
    this.notifyIfIdle();
  }

  private notifyIfIdle(): void {
    if (this.active === 0 && (this.closing || this.jobs.length === 0)) {
      this.idleWaiters.splice(0).forEach((resolve) => resolve());
    }
  }

  /** Resolves when every job added so far has been handled. */
  onIdle(): Promise<void> {
    if (this.active === 0 && this.jobs.length === 0) return Promise.resolve();
    return new Promise((resolve) => this.idleWaiters.push(resolve));
  }

  /**
   * Graceful shutdown: start nothing new, let running jobs finish,
   * and return the jobs that never started so the caller can decide what to do.
   */
  async close(): Promise<Job<T>[]> {
    this.closing = true;
    if (this.active > 0) {
      await new Promise<void>((resolve) => this.idleWaiters.push(resolve));
    }
    return this.jobs.splice(0);
  }
}

There are three ideas in it:

active counts the jobs running right now, and concurrency is the limit. The default is 1, so the queue behaves like Part 1 unless you ask for more.
pump starts jobs until the limit is reached or the line is empty.
When a job ends, whether it worked or failed, it frees a slot and calls pump again.

What the limit does

The simulation below runs the same burst of 20 emails with three limits. Every number depends on the made-up assumptions above.

Concurrency	Time to finish 20 emails	Emails sent	Rejected by the provider
1 (Part 1)	6.0 s	20	0
5	1.2 s	20	0
20	0.3 s	5	15

A limit of 5 is five times faster than one at a time and loses nothing. A limit of 20 is the fastest and loses 75% of the emails.

How to choose the number

Start from the limit of the thing you call, such as a provider or a database pool. Treat it as a ceiling, not as a target. If the provider accepts 5 calls at once, run 3 or 4, because other code may call the same provider too.

Then measure instead of guessing. Raise the number one step at a time and watch the error rate and the time to finish. Stop when errors appear or when the time stops improving.

This advice fits jobs that mostly wait for the network. Jobs that burn CPU are different, because Node.js runs your code on one event loop, and long CPU work blocks everything else. A higher limit does not help there.

You will meet the same setting in production libraries. A BullMQ worker has a concurrency option that defaults to 1 and helps with asynchronous work. Part 6 uses BullMQ, so this number will look familiar.

Shutting down without losing running work

Deployments restart your process. If jobs are running at that moment, you want them to finish, and you want to know which jobs never started.

close() does that. It stops starting new jobs, waits for the running ones, and returns the jobs that never started. After that, add throws.

In a test with a limit of 2 and 6 jobs, the 2 running jobs finish, and the other 4 come back to the caller.

Those 4 jobs still live only in memory. If the process exits, they are gone. Part 6 fixes that.

What this queue still does not do

It never retries. The 15 rejected emails above are the kind of failure worth retrying later. The reason matters: a rate limit is worth retrying, but a mailbox that does not exist is not. Part 3 covers retries with backoff.
It lives in memory. Part 6 moves the queue to Redis.
It cannot run a job twice yet, but retries will change that. Part 5 covers idempotency.

Takeaways

Running jobs one at a time is safe but slow, and running them all at once can overload what you depend on.
Job queue concurrency controls the speed of a queue, so start from the limit of the dependency and leave some room below it.
A queue that shuts down cleanly finishes what is running and tells you what it never started.

Try it

Save the queue from above as queue.ts. Next to it, save this as simulate.ts. It uses a virtual clock, so it prints the same numbers every time:

import { mock } from "node:test";
import { SimpleQueue } from "./queue.ts";

// Illustrative assumptions (not a real provider):
const EMAIL_MS = 300;        // each email call takes 300 ms
const PROVIDER_LIMIT = 5;    // the provider rejects calls when more than 5 are in flight
const BURST = 20;            // 20 welcome emails arrive at the same moment

mock.timers.enable({ apis: ["setTimeout"] });
const flush = () => new Promise<void>((resolve) => setImmediate(resolve));
let now = 0;
async function advanceUntil(done: () => boolean) {
  await flush();
  while (!done()) {
    mock.timers.tick(10);
    now += 10;
    await flush();
  }
}
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

async function run(concurrency: number) {
  now = 0;
  let inFlight = 0;
  let sent = 0;
  const queue = new SimpleQueue<number>(async () => {
    inFlight++;
    const rejected = inFlight > PROVIDER_LIMIT;
    try {
      if (rejected) throw new Error("429 too many concurrent requests");
      await sleep(EMAIL_MS);
      sent++;
    } finally {
      inFlight--;
    }
  }, concurrency);

  for (let i = 0; i < BURST; i++) queue.add(i);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { concurrency, seconds: now / 1000, sent, failed: queue.failed.length };
}

async function main() {
  const rows = [];
  for (const c of [1, 5, 20]) rows.push(await run(c));
  console.table(rows);
  mock.timers.reset();
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

Node prints a warning that its mock timers API is experimental. You can ignore it. You should see this table:

┌─────────┬─────────────┬─────────┬──────┬────────┐
│ (index) │ concurrency │ seconds │ sent │ failed │
├─────────┼─────────────┼─────────┼──────┼────────┤
│ 0       │ 1           │ 6       │ 20   │ 0      │
│ 1       │ 5           │ 1.2     │ 20   │ 0      │
│ 2       │ 20          │ 0.3     │ 5    │ 15     │
└─────────┴─────────────┴─────────┴──────┴────────┘

To see the ceiling move, change PROVIDER_LIMIT to 8 and the list [1, 5, 20] to [1, 5, 8, 20], then run it again. A limit of 8 finishes in 0.9 seconds and loses nothing, while 20 now loses 12 emails.

Next: Part 3, "How to retry failed jobs in Node.js without making it worse"

What is a job queue, and why does my API need one?

Sandro Dezerio — Tue, 06 Oct 2026 03:00:21 +0000

Part 1 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior fullstack engineer.

The signup scenario in this post is made up. The timings and failure rates come from a simulation with stated assumptions, not from a real email provider.

Most APIs start the same way. A user signs up, you save them in the database, and you send a welcome email before returning a response. It works on your laptop. It keeps working for a while in production.

Then the email provider gets slow for an afternoon, and your signup page gets slow with it.

This post builds the smallest useful fix: a job queue in about 40 lines of TypeScript. It is the first step in a series that starts here and ends with reliable, multi-step workflows, including ones that call an LLM.

The problem

Here is the handler, in plain terms:

Save the user (fast)
Send the welcome email (slow, and it can fail)
Return "ok" to the user

Two things are wrong with this.

The user waits for work they don't care about. They asked to sign up. They did not ask to wait for an email to leave your system.

A failure in a side task becomes a failure in the main task. If the email call times out, the handler throws, and the user sees an error. Their account exists, but they think it doesn't. They try again and hit "email already in use".

I ran into this myself building a PDF export feature: generating the file took long enough that the request routinely timed out, and the UI just sat there frozen waiting for an answer nobody needed right away. Moving the generation into a background job was the actual fix. The request returned immediately, and the UI picked up the finished file once it was ready, instead of holding the whole page hostage for a slow report.

The idea

Split the work into two parts:

The request does the part the user needs: save the user, answer "ok".
Everything else goes into a queue, a line of jobs that a worker processes in the background.

The handler adds a job and returns straight away. The worker takes jobs from the line, one at a time, and does the slow work.

A queue in about 40 lines

export type Job<T> = { id: number; data: T };
export type Handler<T> = (job: Job<T>) => Promise<void>;

export class SimpleQueue<T> {
  private jobs: Job<T>[] = [];
  private nextId = 1;
  private running = false;
  private idleWaiters: Array<() => void> = [];
  readonly failed: Array<{ job: Job<T>; error: unknown }> = [];

  constructor(private handler: Handler<T>) {}

  /** Put a job in line and return right away. The work happens later. */
  add(data: T): number {
    const job = { id: this.nextId++, data };
    this.jobs.push(job);
    void this.drain(); // start the worker if it is not already running
    return job.id;
  }

  /** The worker: take jobs one at a time until the line is empty. */
  private async drain(): Promise<void> {
    if (this.running) return;
    this.running = true;
    while (this.jobs.length > 0) {
      const job = this.jobs.shift()!;
      try {
        await this.handler(job);
      } catch (error) {
        this.failed.push({ job, error }); // remember it, keep going
      }
    }
    this.running = false;
    this.idleWaiters.splice(0).forEach((resolve) => resolve());
  }

  /** Resolves when every job added so far has been handled. */
  onIdle(): Promise<void> {
    if (!this.running && this.jobs.length === 0) return Promise.resolve();
    return new Promise((resolve) => this.idleWaiters.push(resolve));
  }
}

There are three ideas in it:

add puts the job at the end of the line and returns. It never waits for the work.
drain is the worker. It runs until the line is empty, and only one drain runs at a time.
If a job throws, the queue records it in failed and moves on. One bad job does not stop the rest.

Does it help? A simulation

I simulated a signup endpoint with these assumptions. All of them are illustrative:

Saving the user takes 20 ms.
The email provider takes 300 ms to answer.
10% of email calls fail with a timeout.
20 signups arrive one after another.

The simulation runs on a virtual clock and a seeded random generator, so it gives the same numbers every time. The code for the simulation is not included in this post.

	Email inside the request	Email in a queue
Average response time	320 ms	20 ms
Signups that returned an error	2 of 20	0 of 20

The response time drops because the user no longer waits for the email. The errors disappear from the signup path because a failed email can no longer fail a signup.

The emails still fail, though. In the queued version, 2 emails failed in the background, and nothing retries them yet. The problem did not go away. It moved to a place where it can be handled without hurting the user.

What this queue does not do

This is a teaching tool, and its limits are the plan for the rest of the series:

It runs one job at a time. After the last signup, the worker needed about 5.6 more seconds to send 20 emails. Part 2 covers concurrency.
It never retries. A failed job is only recorded. Part 3 covers retries with backoff.
It lives in memory. If the process restarts, every job still in line is lost. Part 6 moves the queue to Redis with BullMQ.
It cannot run a job twice yet, but retries will change that. Once failed jobs are retried, the same job can run more than once. Part 5 covers idempotency.

In production you will usually reach for a library such as BullMQ instead of writing your own. Building the small version first makes it clear what the library is doing for you.

Takeaways

Do only what the user needs inside the request. Queue the rest.
A queue separates a slow or failing side task from the main task.
A queue does not remove failures. It gives you a place to handle them.

Try it

Save the queue from above as queue.ts. Next to it, save this as example.ts. It is a signup that saves the user, adds the welcome email to the queue, and answers right away:

import { SimpleQueue } from "./queue.ts";

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

const emailQueue = new SimpleQueue<{ email: string }>(async (job) => {
  await sleep(300); // pretend to call the email provider
  console.log(`  [worker] welcome email sent to ${job.data.email}`);
});

async function signup(email: string) {
  await sleep(20); // pretend to save the user
  emailQueue.add({ email }); // returns immediately
  console.log(`signup ok: ${email}`);
}

async function main() {
  await signup("ana@example.com");
  await signup("luis@example.com");
  await signup("sofia@example.com");
  console.log("all signups answered, emails still on their way");
  await emailQueue.onIdle();
  console.log("queue is empty");
}

main();

Run it:

pnpm dlx tsx example.ts

You should see all three signups answered first, and the emails go out afterwards, one at a time:

signup ok: ana@example.com
signup ok: luis@example.com
signup ok: sofia@example.com
all signups answered, emails still on their way
  [worker] welcome email sent to ana@example.com
  [worker] welcome email sent to luis@example.com
  [worker] welcome email sent to sofia@example.com
queue is empty

Try making the handler throw for one email. The other two still go out, and the failed job ends up in queue.failed.

The simulation behind the table is not part of this post. A more complete implementation, with the simulation, tests and the features from the next parts, will come later in the series.

Next: Part 2, "Workers and concurrency: process jobs without overloading your server."