DEV Community: Sandro Dezerio

Jobs that survive a crash: Node.js and BullMQ

Sandro Dezerio — Sun, 01 Nov 2026 19:04:20 +0000

Part 6 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up example is the one from Part 1, Part 2, Part 3, Part 4, and Part 5. The numbers in this post come from a test that starts and kills real processes against a real Redis, not from a simulation. Tested with Node 22, BullMQ 6.5.3, ioredis 6.0.0, and Redis 7.

Every queue in this series so far has lived in the memory of the process that runs it. That works until the process stops. A deploy, a crash, or an out-of-memory kill empties the queue, and every sign-up still waiting for its welcome email is gone, with no error and no log line. This post moves the queue to Redis with BullMQ, and then kills the worker on purpose to see what survives.

What a restart does to an in-memory queue

The queue from Part 2 keeps its jobs in an array inside the Node.js process. When the process dies, the array dies with it. Nothing was ever written down, so nothing can be recovered.

Imagine a store with 20 new sign-ups waiting in the queue. The worker handles them one at a time, and it dies right after it sends the welcome email for the sixth one. The process restarts with an empty queue.

Six users get their email. The other 14 never will, and there is no failed job to look at, because queue.failed lived in the same memory.

That is the first row of the results further down. The next sections build the other two.

A queue that lives in Redis

Redis is a data store that runs as its own process. BullMQ is a Node.js library that keeps a queue inside it: the jobs, their state, the retry settings, and a lock that says which worker is handling which job. The process that adds jobs and the process that runs them no longer share memory. They share Redis, so when the worker dies, the jobs are still there.

In BullMQ 6, the Redis client is a separate package. The documentation says, "By default, BullMQ creates Redis connections with ioredis," so you install both with pnpm add bullmq ioredis. I tried without ioredis: the first new Queue(...) throws an error that asks for it.

Redis itself runs in Docker. This docker-compose.yml starts it with the two settings that BullMQ's production guide asks for:

services:
  redis:
    image: redis:7
    command: ["redis-server", "--appendonly", "yes", "--maxmemory-policy", "noeviction"]
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data

volumes:
  redis-data:

--appendonly yes makes Redis write its changes to disk. The guide says, "We recommend enabling AOF (Append Only File), which provides a robust and fast solution." --maxmemory-policy noeviction stops Redis from deleting keys when memory runs out, and a deleted key can be a waiting job. The guide calls it "the only setting that guarantees the correct behavior of the queues." The volume keeps the data when the container is replaced.

The queue and the worker

Two files do the work. signup-queue.ts creates the queue and holds the options that every job gets:

import { Queue } from "bullmq";

export type Signup = { userId: string; email: string };

export const connection = { host: "localhost", port: 6379 };

/** The same sign-ups for every run: user_1, user_2, and so on. */
export function makeSignups(count: number): Signup[] {
  return Array.from({ length: count }, (_, i) => ({ userId: `user_${i + 1}`, email: `user${i + 1}@example.com` }));
}

/** The jobs live in Redis. These options apply to every job unless a job overrides them. */
export function createSignupQueue(name: string): Queue<Signup> {
  return new Queue<Signup>(name, {
    connection,
    defaultJobOptions: {
      attempts: 3, // up to 3 tries per job, the same limit as in Part 3
      backoff: { type: "exponential", delay: 1000, jitter: 1 }, // up to 1 s, then up to 2 s, chosen at random
      removeOnComplete: { count: 1000 }, // keep the last 1,000 finished jobs, not all of them
    },
  });
}

/** A job ID that names the business action, so the same sign-up added twice is one job. */
export const jobIdFor = (signup: Signup) => `welcome-email-${signup.userId}`;

Four things in it are new:

Retries are options now. With attempts: 3, a job that throws is tried up to three times. With exponential backoff, the documentation says BullMQ retries "after 2 ^ (attempts - 1) * delay milliseconds." Here, that means 1 second before the second attempt and 2 seconds before the third.
jitter: 1 is the full jitter from Part 3. The same page says that with a value of 1, "random delays will be generated between 0 and max generated value."
removeOnComplete: { count: 1000 } limits what Redis keeps. By default, BullMQ keeps finished jobs in Redis. This keeps the last 1,000.
jobIdFor gives each sign-up a job ID that names the business action, the same idea as the key in Part 5. The documentation says that if you add a job with an existing ID, "that job will just be ignored." A sign-up that arrives twice, for example because a client timed out and sent the request again, becomes one job. The ID cannot contain a colon or consist only of digits, which is why it is welcome-email-user_1 and not welcome-email:user_1. Tests I wrote for this post confirm all three rules, and a fourth: once a job is removed, its ID can be used again. The protection lasts only as long as the job stays in Redis.

The second file is the worker. It runs as its own process:

import { Worker } from "bullmq";
import { connection, type Signup } from "./signup-queue.ts";
import { closeEmails, sendEmail } from "./emails.ts";

const run = process.env.RUN!; // names the queue and the email records of this run
const crashUser = process.env.CRASH_USER; // die while handling this user
const useKey = process.env.USE_KEY === "1"; // send an idempotency key to the provider

const worker = new Worker<Signup>(
  run,
  async (job) => {
    await sendEmail(run, job.data.userId, useKey ? `welcome-email:${job.data.userId}` : undefined);
    // The email went out. Now the process dies before it can tell the queue.
    if (job.data.userId === crashUser) process.kill(process.pid, "SIGKILL");
  },
  // The defaults for both timers are 30,000 ms. They are shorter here so the demo finishes quickly.
  { connection, lockDuration: 2000, stalledInterval: 1000 },
);
worker.on("error", (error) => console.error(error));

// The parent process sends a message when the queue is empty: close cleanly.
process.on("message", async () => {
  await worker.close();
  await closeEmails();
  process.exit(0);
});

The worker takes one job at a time, as the queue from Part 2 did by default. The two lines with crashUser are test scaffolding: they kill the process after the email goes out and before the job is reported as done.

The two timers need an explanation. While a worker runs a job, it keeps renewing a lock on it. If the worker dies, the lock expires, and a check that runs every stalledInterval finds the job and puts it back in line. The defaults are 30,000 ms for both (worker options). Here they are 2,000 and 1,000 ms, so the test takes seconds, not minutes.

In real use, keep the defaults unless you have a reason to change them: a worker that blocks the event loop for longer than the lock lasts looks dead, and its job runs again.

Kill the worker

kill-test.ts is in the Try it section below. It runs three cases with 20 sign-ups each, and in every case the worker process dies right after it sends the welcome email for user_6. In the Redis cases, the process that adds the jobs stays alive, and a new worker takes over after the crash and runs until the queue is empty. The fake email provider records its emails in Redis, so the count survives the crash.

Strategy	Users emailed	Users never emailed	Duplicate emails
In-memory queue	6	14	0
BullMQ, no key	20	0	1
BullMQ, key sent to the provider	20	0	0

Here is what each row says:

In-memory queue. Six users got an email, and the other 14 sign-ups vanished. The restarted process had nothing to run.
BullMQ, no key. Every user got an email, and the sixth got two.
BullMQ, key sent to the provider. Every user got exactly one.

Why the sixth user got two emails

The worker sent the email for user_6 and died before it told Redis. From Redis's point of view, the job was still running when its lock expired. The documentation says, "A stalled job is moved back to the waiting status and will be processed again by another worker." So the job ran again, and the second email went out.

This is the lost reply from Parts 4 and 5 in another form: the process that did the work was killed before it could report. The queue cannot tell "the email was never sent" from "the email was sent, and I was not told," so it runs the job again. The store from Part 5 would not have helped either, because it lived in the memory of the process that died.

The fix is the one from Part 5: the key has to reach whoever performs the side effect. The third row sends welcome-email:user_6 with the email. The provider has seen that key, so the second run sends nothing.

A job that stalls more than once does not loop forever: the same page says it is moved to the failed set, and the default allows one recovery per job.

Retries move into the queue

Part 3 wrapped the handler in withRetry. BullMQ does the same work from the outside: when a job throws, the queue holds it for the backoff delay and then tries it again.

Two differences matter. First, the wait no longer blocks the worker. In Part 3, a job waiting for its retry kept its concurrency slot. A test I wrote for this post shows the opposite with BullMQ: with one worker and one slot, user_1 fails its first attempt, user_2 runs while user_1 waits, and then user_1 gets its second attempt.

Second, BullMQ retries the whole job, like the whole-job retry from Part 4. A job with several steps repeats the steps that already worked. You can keep retryEachStep from Part 4 and the keys from Part 5 inside the job handler, and let BullMQ's attempts catch what is left.

What this does not fix

Redis is now a part you run. AOF and noeviction lower the risk of losing jobs, and they do not remove it. While the single Redis container is down, no job can be added or run. Replication is outside this post.

The key store from Part 5. It lives in memory, so a restart forgets it. This post sends the key to the provider instead, and the provider remembers it. A provider that takes no key needs a shared store, and Redis can be that store.

Failed jobs. After the last attempt, a job stays in the failed set, and nobody is told. A test I wrote for this post confirms it: a job that always fails is tried 3 times and then stays there. Part 7 covers where failed jobs go and how to notice them.

Seeing it happen. In the test, the next worker took over without a single log line. Part 8 adds structured logs, so you can tell which job ran twice and why.

Takeaways

A queue in memory loses every waiting job when the process stops, and nothing reports it.
A queue in Redis keeps the jobs, but a job that was running when the worker died runs again. The job has to be safe to run twice, and the key has to reach the provider.
Retries become options on the queue. They retry the whole job, so the lessons of Parts 4 and 5 still apply.

Try it

This is a teaching version of a Redis-backed queue. A more complete implementation comes later in the series. You need Node 22, pnpm, and Docker. Create a folder and run these commands inside it:

pnpm init
pnpm add bullmq ioredis
pnpm add -D tsx

Open the package.json that pnpm init created and add the line "type": "module", next to "name". The example uses top-level await, which only works in a module.

Keep queue.ts from Part 2 in the folder. Save the docker-compose.yml, signup-queue.ts, and worker.ts files from above, and then save these three. emails.ts is the fake email provider:

import { Redis } from "ioredis";

const redis = new Redis({ host: "localhost", port: 6379 });

/** A fake email provider. It writes every email to Redis, so the record survives a crash of our process. */
export async function sendEmail(run: string, userId: string, key?: string): Promise<void> {
  // A provider that honors idempotency keys (Part 5) sends nothing for a key it has already seen.
  if (key && (await redis.set(`${run}:seen:${key}`, "1", "NX")) === null) return;
  await redis.hincrby(`${run}:emails`, userId, 1);
}

/** How many emails each user received in this run. */
export async function emailsPerUser(run: string): Promise<Record<string, number>> {
  const counts = await redis.hgetall(`${run}:emails`);
  return Object.fromEntries(Object.entries(counts).map(([userId, count]) => [userId, Number(count)]));
}

export async function forgetEmails(run: string): Promise<void> {
  await redis.del(`${run}:emails`, ...(await redis.keys(`${run}:seen:*`)));
}

export const closeEmails = () => redis.quit();

memory-worker.ts is the in-memory case. It receives the sign-ups, handles them with the queue from Part 2, and dies at user_6:

import { SimpleQueue } from "./queue.ts";
import { makeSignups, type Signup } from "./signup-queue.ts";
import { closeEmails, sendEmail } from "./emails.ts";

const run = process.env.RUN!;
const crashUser = process.env.CRASH_USER;
const count = Number(process.env.SIGNUPS ?? 0); // how many sign-ups this process receives

const queue = new SimpleQueue<Signup>(async (job) => {
  await sendEmail(run, job.data.userId);
  // The email went out. Now the process dies, and so does every job waiting in memory.
  if (job.data.userId === crashUser) process.kill(process.pid, "SIGKILL");
});

for (const signup of makeSignups(count)) queue.add(signup);
await queue.onIdle();
await closeEmails();

kill-test.ts runs the three cases and prints the table:

import { spawn, type ChildProcess } from "node:child_process";
import { fileURLToPath } from "node:url";
import { createSignupQueue, jobIdFor, makeSignups } from "./signup-queue.ts";
import { closeEmails, emailsPerUser, forgetEmails } from "./emails.ts";

const SIGNUPS = 20;
const CRASH_USER = "user_6";

/** Start worker.ts or memory-worker.ts as a separate process. */
function start(file: string, env: Record<string, string>): ChildProcess {
  return spawn(process.execPath, ["--import", "tsx", fileURLToPath(new URL(file, import.meta.url))], {
    env: { ...process.env, ...env },
    stdio: ["ignore", "inherit", "inherit", "ipc"],
  });
}
const exited = (child: ChildProcess) => new Promise<void>((resolve) => child.once("exit", () => resolve()));

/** The queue is in memory: the process dies, and a new process starts with an empty queue. */
async function memoryQueue(run: string) {
  await exited(start("memory-worker.ts", { RUN: run, CRASH_USER, SIGNUPS: String(SIGNUPS) }));
  await exited(start("memory-worker.ts", { RUN: run, SIGNUPS: "0" })); // the restart has no jobs to run
}

/** The queue is in Redis: a worker dies, and a new worker takes over. */
async function redisQueue(run: string, useKey: boolean) {
  const queue = createSignupQueue(run);
  await queue.obliterate({ force: true });
  for (const signup of makeSignups(SIGNUPS)) await queue.add("welcome-email", signup, { jobId: jobIdFor(signup) });

  const env = { RUN: run, USE_KEY: useKey ? "1" : "0" };
  await exited(start("worker.ts", { ...env, CRASH_USER })); // dies while handling user_6

  const restarted = start("worker.ts", env);
  for (;;) {
    const { waiting, active, delayed } = await queue.getJobCounts("waiting", "active", "delayed");
    if (waiting + active + delayed === 0) break;
    await new Promise((resolve) => setTimeout(resolve, 100));
  }
  restarted.send("stop");
  await exited(restarted);
  await queue.obliterate({ force: true });
  await queue.close();
}

async function row(strategy: string, run: string) {
  const perUser = Object.values(await emailsPerUser(run));
  return {
    strategy,
    "users emailed": perUser.length,
    "users never emailed": SIGNUPS - perUser.length,
    "duplicate emails": perUser.reduce((sum, count) => sum + count - 1, 0),
  };
}

const runs = ["signups-memory", "signups-redis", "signups-redis-key"];
for (const run of runs) await forgetEmails(run);

await memoryQueue(runs[0]);
await redisQueue(runs[1], false);
await redisQueue(runs[2], true);

console.log(`${SIGNUPS} sign-ups. The worker dies right after it sends the email for ${CRASH_USER}.`);
console.table([
  await row("in-memory queue", runs[0]),
  await row("BullMQ, no key", runs[1]),
  await row("BullMQ, key sent to the provider", runs[2]),
]);
for (const run of runs) await forgetEmails(run);
await closeEmails();

Start Redis and run the test:

docker compose up -d
pnpm exec tsx kill-test.ts

It takes a few seconds and prints:

20 sign-ups. The worker dies right after it sends the email for user_6.
┌─────────┬────────────────────────────────────┬───────────────┬─────────────────────┬──────────────────┐
│ (index) │ strategy                           │ users emailed │ users never emailed │ duplicate emails │
├─────────┼────────────────────────────────────┼───────────────┼─────────────────────┼──────────────────┤
│ 0       │ 'in-memory queue'                  │ 6             │ 14                  │ 0                │
│ 1       │ 'BullMQ, no key'                   │ 20            │ 0                   │ 1                │
│ 2       │ 'BullMQ, key sent to the provider' │ 20            │ 0                   │ 0                │
└─────────┴────────────────────────────────────┴───────────────┴─────────────────────┴──────────────────┘

Change SIGNUPS or CRASH_USER in kill-test.ts and see which rows change. When you are done, docker compose down stops Redis.

Previous: Jobs that run twice: idempotency keys in Node.js (Part 5).

Next: Part 7 (coming soon) looks at failed jobs: where they go after the last attempt, and how to find out before a customer does.

Jobs that run twice: idempotency keys in Node.js

Sandro Dezerio — Sun, 25 Oct 2026 16:57:21 +0000

Part 5 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up scenario is made up. It is the same one used in Part 1, Part 2, Part 3, and Part 4. The failure rates come from a simulation with stated assumptions, not from real services.

At the end of Part 4, a simulation of 1,000,000 sign-ups sent the welcome email twice to 4,939 users. Nothing had failed in the way you would expect. The email went out, the reply got lost, and the retry did what a retry does.

A retry promises that the work will be done. It does not promise that the work will be done once. Parts 3 and 4 made jobs more reliable by running them again, so now a job can run twice, and the code has to make that harmless. The tool for this is an idempotency key. This post builds a small one, shows where it helps, and shows where it does not.

Where the duplicates come from

Three things can make the sign-up job repeat a side effect:

A lost reply. The provider did the work, but the answer never arrived. This is the case from Part 4.
A replay. Retrying the whole job, or running it again after a crash, repeats steps that already worked.
A repeated request. The same sign-up is added to the queue twice, for example because a client timed out and sent the request again.

Some repeats are harmless: recording an analytics event twice only skews a chart. Emailing a customer twice, or charging a card twice, is not harmless.

What an idempotency key is

An operation is idempotent when doing it twice has the same effect as doing it once. Setting a user's plan to "pro" is idempotent. Sending an email is not.

An idempotency key makes an operation behave as if it were. The key is a name for one specific action, such as "the welcome email for user 1001." The first time the system sees the key, it does the work and remembers that the key finished. Every later time, it skips the work.

Stripe's API works this way. Its documentation says the API supports idempotency "for safely retrying requests without accidentally performing the same operation twice," and that "Retries with the same key return the saved response."

A small store

The code adds one file, idempotency.ts. The queue from Part 2, the retry wrapper from Part 3, and the workflow file from Part 4 do not change:

import type { Job } from "./queue.ts";
import type { Step } from "./workflow.ts";

type Entry = { value: unknown; expiresAt: number };

/**
 * Remembers which keys have finished, so the same key never runs its work twice.
 * A run that throws is not remembered, so a retry can try again.
 */
export class IdempotencyStore {
  private finished = new Map<string, Entry>();
  private running = new Map<string, Promise<unknown>>();

  constructor(
    private ttlMs: number, // how long a finished key is remembered
    private now: () => number = Date.now, // replaceable in tests
  ) {}

  async once<T>(key: string, work: () => Promise<T>): Promise<T> {
    const saved = this.finished.get(key);
    if (saved && saved.expiresAt > this.now()) return saved.value as T; // finished before: skip the work
    this.finished.delete(key); // forget an expired key
    const inFlight = this.running.get(key);
    if (inFlight) return inFlight as Promise<T>; // the same key is running right now: share that run
    const run = work()
      .then((value) => {
        this.finished.set(key, { value, expiresAt: this.now() + this.ttlMs });
        return value;
      })
      .finally(() => this.running.delete(key));
    this.running.set(key, run);
    return run;
  }
}

/** Skip any step that already finished for this key. `keyOf` names the business action, not the attempt. */
export function idempotentSteps<T>(
  steps: Step<T>[],
  store: IdempotencyStore,
  keyOf: (job: Job<T>) => string,
): Step<T>[] {
  return steps.map((step) => ({
    name: step.name,
    run: (job) => store.once(`${step.name}:${keyOf(job)}`, () => step.run(job)),
  }));
}

Here are three things to notice:

once has three cases. If the key finished before, it returns the saved result and skips the work. If the same key is running right now, it shares that run. Otherwise, it runs the work and remembers the result.
A run that throws is not remembered. If it were, the retry from Part 3 would get the same error back and could never succeed.
keyOf names the business action: here, the user ID. It is not the job ID, because the same sign-up added twice gets two job IDs, and it is not the attempt number, because that changes on every retry.

I use an internal user ID and not the email address, because Stripe's documentation advises against putting sensitive data, such as email addresses, in keys. The same page suggests random keys. A random key and a user ID follow the same rule: the key is created once per action and sent unchanged on every attempt.

Here is the sign-up added twice, plus one more user:

import { SimpleQueue, type Job } from "./queue.ts";
import { IdempotencyStore, idempotentSteps } from "./idempotency.ts";
import { workflow, type Step } from "./workflow.ts";

type Signup = { userId: string; email: string };

const steps: Step<Signup>[] = [
  { name: "save the user", run: async (job: Job<Signup>) => console.log(`saving ${job.data.email}`) },
  { name: "send the welcome email", run: async (job: Job<Signup>) => console.log(`emailing ${job.data.email}`) },
];

const store = new IdempotencyStore(24 * 60 * 60 * 1000); // remember finished keys for 24 hours
const queue = new SimpleQueue<Signup>(workflow(idempotentSteps(steps, store, (job) => job.data.userId)));

queue.add({ userId: "user_1001", email: "ana@example.com" });
queue.add({ userId: "user_1001", email: "ana@example.com" }); // the same signup arrives twice
queue.add({ userId: "user_1002", email: "ben@example.com" });

Save it as usage.ts next to the other files and run it:

pnpm dlx tsx usage.ts

saving ana@example.com
emailing ana@example.com
saving ben@example.com
emailing ben@example.com

Ana's sign-up arrived twice, and each of her steps ran once.

The store cannot see a lost reply

The store remembers only runs that finished. In a lost reply, the email provider does the work, but the call ends with a timeout. From the store's point of view, the run has failed, so nothing is saved, and the retry sends the email again. A test I wrote for this post confirms it: with a store in place, a lost reply still sends two emails.

The key has to reach whoever performs the side effect. Stripe does this with an Idempotency-Key header: its servers remember the key, not your process. If the email provider remembers the key too, the retry is safe, because the provider sees the same key and sends nothing. Not every API accepts a key, so check the documentation of each provider you call.

What changes in the numbers

The simulation is the one from Part 4, with a stable ID for each sign-up. It runs 1,000,000 sign-ups through the queue from Part 2. Every call to a step fails 1% of the time, the email reply is lost 0.5% of the time, and each strategy has up to 3 attempts. It assumes the email provider remembers keys and sends one email per key, the way Stripe's API handles a repeated request.

Strategy	Succeeded	Failed jobs	Calls per job	Steps done again	Users with 2+ welcome emails
Retry each step, no key	99.999%	11	10.11	4,971	4,939
Retry each step, new key on every call	99.999%	11	10.11	4,971	4,939
Retry each step, same key every time	99.999%	11	10.11	0	0
Retry the whole job, same key	99.900%	995	10.6	464,233	0
Retry the whole job, same key, skip finished steps	99.975%	250	10.11	0	0

Here is what each row says:

No key. This is the last row of Part 4's results: 4,939 users with two emails.
A new key on every call. The result is the same, because the provider never sees the same key twice. A key that changes on every attempt is not a key.
The same key every time. No duplicates, and nothing else changes: the same 11 failed jobs and the same number of calls.
Retry the whole job, same key. No duplicate emails, but the other nine steps ran 464,233 times again. They send no key, so nothing stops them.
Retry the whole job, skip finished steps. The store skips every step that already finished, so nothing runs again, and the calls per job match retrying each step.

Retrying each step is still the better strategy: it gave up on 11 sign-ups and the whole-job retry gave up on 250. A whole-job retry ends at the first failure, so three failures anywhere in the job are enough to fail it. Retrying each step needs three failures in a row on the same step. The store makes the whole-job retry safe, but it does not make it better.

Designing the key

Name the business action. welcome-email:user_1001 says what and for whom. A job ID, an attempt number, or a timestamp changes when the action has not changed, and the duplicate gets through.
Include the action in the key. The store builds step name:user ID, so two steps of the same user never share a key.
Decide how long to remember. The key has to outlive the longest time a duplicate can still arrive, including the whole retry window. If the key expires first, the duplicate runs again. If you keep it forever, memory grows. Stripe says it can remove keys once they are at least 24 hours old, and the code in this post uses 24 hours as an illustration. Your number depends on your retries and your traffic.
Treat the same key with different data as a bug. Stripe compares the new request with the original one and returns an error if they differ. The store here does not check, so the second call gets the first result.

What this does not fix

A restart. The store lives in memory, like the queue. If the process dies, it forgets every key, and a second process does not share them. Part 6 moves the queue to Redis.

A provider without keys. You can ask the provider "did I already send this?" before sending. But asking and sending are two calls, and another attempt can still run between them. Prefer a provider that accepts a key, and know which of your side effects can repeat.

The simulation itself. It assumes independent failures and a provider that honors the key every time. Treat the numbers as a way to build intuition, not as a prediction.

Takeaways

A retry can run a side effect twice. An idempotency key turns "twice" into "once."
The key names the business action and stays the same on every attempt. A new key on every attempt protects nothing.
The key has to reach whoever performs the side effect. A store in your own process cannot see a lost reply.

Try it

This is a teaching version of idempotency keys. A more complete implementation comes later in the series. Keep queue.ts from Part 2, retry.ts from Part 3, and workflow.ts from Part 4. Save the idempotency.ts file from above next to them, and then save this as simulate.ts:

import { SimpleQueue, type Handler } from "./queue.ts";
import { withRetry, type RetryOptions } from "./retry.ts";
import { retryEachStep, workflow, type Step } from "./workflow.ts";
import { IdempotencyStore, idempotentSteps } from "./idempotency.ts";

// Illustrative assumptions (not a real system):
const JOBS = 1_000_000; // signups to simulate
const P_FAIL = 0.01;    // every call to a step fails 1% of the time, independently
const LOST_REPLY = 0.005; // the email goes out, but the reply is lost 0.5% of the time
const STEPS = [
  "save the user", "create the billing customer", "create the workspace", "upload the default avatar",
  "add to the CRM", "subscribe to the newsletter", "send the welcome email", "notify the sales channel",
  "record the analytics event", "schedule the day-3 email",
];
const EMAIL = "send the welcome email";
// Up to 3 attempts. The simulation counts outcomes, not time, so it skips the waits.
const retry: RetryOptions = { maxAttempts: 3, baseDelayMs: 100, maxDelayMs: 2000, jitter: true, sleep: async () => {} };

// A small seeded random generator (mulberry32), so every run prints the same numbers.
function seededRandom(seed: number) {
  let a = seed;
  return () => {
    a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// What each signup job carries: a stable ID and how many times each step really did its work.
type Signup = { userId: string; done: Record<string, number> };

// Which key the email step sends to the provider. The provider sends one email per key it has seen.
type KeyMode = "no key" | "new key on every call" | "same key every time";

async function simulate(build: (steps: Step<Signup>[], store: IdempotencyStore) => Handler<Signup>, keyMode: KeyMode) {
  const random = seededRandom(1);
  const providerKeys = new Set<string>(); // the keys the email provider has already seen
  let freshKey = 0;
  let calls = 0;
  const steps: Step<Signup>[] = STEPS.map((name) => ({
    name,
    run: async (job) => {
      calls++;
      if (random() < P_FAIL) throw new Error(`${name}: 503 service unavailable`);
      let alreadySent = false;
      if (name === EMAIL && keyMode !== "no key") {
        const key = keyMode === "same key every time" ? `welcome-email:${job.data.userId}` : `welcome-email:${job.data.userId}:${freshKey++}`;
        alreadySent = providerKeys.has(key);
        providerKeys.add(key);
      }
      if (!alreadySent) job.data.done[name] = (job.data.done[name] ?? 0) + 1; // the side effect happened
      if (name === EMAIL && random() < LOST_REPLY) throw new Error(`${name}: timeout, reply lost`);
    },
  }));
  let failed = 0;
  let repeatedSteps = 0;
  let twoEmails = 0;
  for (let start = 0; start < JOBS; start += 10_000) {
    // A new queue and store for every 10,000 signups keeps memory small. Every key belongs to one signup, so results do not change.
    const queue = new SimpleQueue<Signup>(build(steps, new IdempotencyStore(24 * 60 * 60 * 1000)));
    const batch: Signup[] = [];
    for (let i = 0; i < 10_000; i++) {
      const signup: Signup = { userId: `user_${start + i}`, done: {} };
      batch.push(signup);
      queue.add(signup);
    }
    await queue.onIdle();
    failed += queue.failed.length;
    for (const signup of batch) {
      for (const times of Object.values(signup.done)) repeatedSteps += times - 1;
      if ((signup.done[EMAIL] ?? 0) > 1) twoEmails++;
    }
    providerKeys.clear();
  }
  return {
    succeeded: `${((100 * (JOBS - failed)) / JOBS).toFixed(3)}%`,
    failed,
    callsPerJob: Math.round((calls / JOBS) * 100) / 100,
    repeatedSteps,
    twoEmails,
  };
}

async function main() {
  console.log(`${JOBS.toLocaleString("en-US")} signups, ${STEPS.length} steps, each call fails ${P_FAIL * 100}% of the time, the email reply is lost ${LOST_REPLY * 100}% of the time`);
  console.table([
    { strategy: "retry each step, no key", ...(await simulate((s) => workflow(retryEachStep(s, retry)), "no key")) },
    { strategy: "retry each step, new key on every call", ...(await simulate((s) => workflow(retryEachStep(s, retry)), "new key on every call")) },
    { strategy: "retry each step, same key every time", ...(await simulate((s) => workflow(retryEachStep(s, retry)), "same key every time")) },
    { strategy: "retry the whole job, same key", ...(await simulate((s) => withRetry(workflow(s), retry), "same key every time")) },
    {
      strategy: "retry the whole job, same key, skip finished steps",
      ...(await simulate((s, store) => withRetry(workflow(idempotentSteps(s, store, (job) => job.data.userId)), retry), "same key every time")),
    },
  ]);
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

It runs 5,000,000 jobs, so it takes about a minute. The random generator is seeded, so you should see exactly these numbers:

1,000,000 signups, 10 steps, each call fails 1% of the time, the email reply is lost 0.5% of the time
┌─────────┬──────────────────────────────────────────────────────┬───────────┬────────┬─────────────┬───────────────┬───────────┐
│ (index) │ strategy                                             │ succeeded │ failed │ callsPerJob │ repeatedSteps │ twoEmails │
├─────────┼──────────────────────────────────────────────────────┼───────────┼────────┼─────────────┼───────────────┼───────────┤
│ 0       │ 'retry each step, no key'                            │ '99.999%' │ 11     │ 10.11       │ 4971          │ 4939      │
│ 1       │ 'retry each step, new key on every call'             │ '99.999%' │ 11     │ 10.11       │ 4971          │ 4939      │
│ 2       │ 'retry each step, same key every time'               │ '99.999%' │ 11     │ 10.11       │ 0             │ 0         │
│ 3       │ 'retry the whole job, same key'                      │ '99.900%' │ 995    │ 10.6        │ 464233        │ 0         │
│ 4       │ 'retry the whole job, same key, skip finished steps' │ '99.975%' │ 250    │ 10.11       │ 0             │ 0         │
└─────────┴──────────────────────────────────────────────────────┴───────────┴────────┴─────────────┴───────────────┴───────────┘

Change LOST_REPLY or P_FAIL and see which rows change.

Previous: Ten steps at 99% is 90%: retrying multi-step jobs (Part 4).

Next: Part 6 (coming soon) moves the queue from memory to Redis with BullMQ, so a restart no longer loses waiting jobs.

Ten steps at 99% is 90%: retrying multi-step jobs

Sandro Dezerio — Sun, 18 Oct 2026 13:23:33 +0000

Part 4 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up scenario is made up. It is the same one used in Part 1, Part 2 and Part 3. The failure rates come from a simulation with stated assumptions, not from real services.

A job with ten steps, where each step works 99% of the time, finishes about 90% of the time.

Nobody looks at a step that works 99% of the time and calls it unreliable. That is why the problem hides. Each step looks fine on its own, and the job as a whole fails about once in every ten runs.

In Part 3, every job was a single call: send one welcome email. Real jobs are rarely that small. This post looks at a job with several steps, shows what goes wrong when you retry it as one piece, and fixes it with the withRetry wrapper from Part 3, unchanged.

The job grows

Picture the sign-up job some time later. Sending the welcome email is no longer enough. The job now does ten things, in order:

Save the user
Create the billing customer
Create the workspace
Upload the default avatar
Add the user to the CRM
Subscribe the user to the newsletter
Send the welcome email
Notify the sales channel
Record the analytics event
Schedule the day-3 email

Each step calls something that can fail: a database, a payment provider, an email provider, an internal API. Assume each call fails 1% of the time, independently of the others. For a single call, 99% is a respectable number.

The arithmetic

Reliability multiplies. For the job to succeed, all ten steps must succeed, so the chance is 0.99 multiplied by itself ten times:

0.99¹⁰ = 0.904

About 96 jobs in every 1,000 fail somewhere along the way. With twenty steps, it would be about 18 in every 100.

To check this, and to see what each fix does, I wrote a simulation that runs 1,000,000 sign-up jobs through the queue from Part 2. Without retries, 90.398% of the jobs succeeded. The formula gives 90.438%, so the simulation and the math agree within sampling noise.

The failure rate is only half of the problem. Of the 96,024 jobs that failed, 85,902 failed after at least one step had already done its work. The user exists but has no workspace. The billing customer exists, but nobody got the welcome email. A failed job is often a half-finished job.

The obvious fix: retry the whole job

Part 3 already has a retry wrapper. The quickest fix is to wrap the whole job in withRetry, the same way Part 3 wrapped the email handler. With up to 3 attempts, the success rate jumps from 90.398% to 99.913%. That looks like a win, but the job now has a different problem.

When step 8 fails, the retry starts again from step 1. It saves the user again, creates the billing customer again, and sends the welcome email again. In the simulation, retrying the whole job did 462,891 steps a second time, and 29,564 users received the welcome email more than once. Nobody asked for either.

Some repeats are harmless, such as recording an analytics event twice. Others are not, such as a second billing customer for the same user.

The better fix: retry each step

The fix is to retry the step that failed, not the job. If step 8 fails, try step 8 again. Steps 1 to 7 already worked, so leave them alone.

The code adds one small file, workflow.ts. A job becomes a list of named steps, and the queue from Part 2 and the retry wrapper from Part 3 do not change:

import type { Handler } from "./queue.ts";
import { withRetry, type RetryOptions } from "./retry.ts";

/** One step of a job: a name and the work it does. */
export type Step<T> = { name: string; run: Handler<T> };

/** Run the steps in order. If one throws, the job stops there and the queue records the error. */
export function workflow<T>(steps: Step<T>[]): Handler<T> {
  return async (job) => {
    for (const step of steps) {
      await step.run(job);
    }
  };
}

/** Give each step its own retries, so a failing step never repeats the steps before it. */
export function retryEachStep<T>(steps: Step<T>[], options: RetryOptions): Step<T>[] {
  return steps.map((step) => ({ name: step.name, run: withRetry(step.run, options) }));
}

workflow turns a list of steps into a normal handler, so the queue does not need to know that the job has steps. retryEachStep wraps each step with withRetry from Part 3. The two strategies differ only in where the wrapper goes:

Whole job: withRetry(workflow(steps), options)
Each step: workflow(retryEachStep(steps, options))

Here is how a job with two steps looks:

import { SimpleQueue, type Job } from "./queue.ts";
import { retryEachStep, workflow, type Step } from "./workflow.ts";

const steps: Step<string>[] = [
  { name: "save the user", run: async (job: Job<string>) => console.log(`saving ${job.data}`) },
  { name: "send the welcome email", run: async (job: Job<string>) => console.log(`emailing ${job.data}`) },
];

const queue = new SimpleQueue<string>(
  workflow(retryEachStep(steps, { maxAttempts: 3, baseDelayMs: 100, maxDelayMs: 2000, jitter: true })),
);
queue.add("ana@example.com");

The three strategies side by side

These are the results for 1,000,000 simulated sign-ups, with up to 3 attempts in both retry strategies:

Strategy	Succeeded	Failed jobs	Failed after some steps were done	Calls per job	Steps done again	Users with 2+ welcome emails
No retries	90.398%	96,024	85,902	9.56	0	0
Retry the whole job	99.913%	867	865	10.57	462,891	29,564
Retry each step	99.999%	9	8	10.1	0	0

Retrying each step beats retrying the whole job in every column. Only 9 jobs failed out of 1,000,000. No step ran twice. And it made fewer calls than retrying the whole job (10.1 per job compared with 10.57), because a retry repeats one call instead of several.

The success rates also match the formulas. Retrying the whole job needs one attempt in which all ten steps pass, so a job fails only when all three attempts fail: 1 - (1 - 0.99¹⁰)³ = 99.913%. Retrying each step needs each step to fail three times in a row, which happens once in a million calls: (1 - 0.01³)¹⁰ = 99.999%.

Inngest, a tool for running multi-step workflows, uses the same idea: "Each step.run() has its own retry counter, so one failing step does not restart earlier completed steps."

What retrying each step does not fix

A step that worked but did not say so. Sometimes the email provider sends the email, and the reply gets lost on the way back. The step sees a timeout and fails. A retry of that step sends the email again. I ran the simulation again with a lost reply on the email step 0.5% of the time:

Strategy	Succeeded	Steps done again	Users with 2+ welcome emails
Retry each step, reply lost 0.5% of the time	99.999%	4,971	4,939

Retrying each step cannot tell the difference between "it did not happen" and "it happened, but I did not hear back." That is the job of an idempotency key, which Part 5 covers.

The jobs that still fail. Of the 9 failed jobs, 8 stopped halfway through, with some steps done. Per-step retries make this rare, but not impossible, and queue.failed does not say which step failed or which steps already ran. Later parts cover where these jobs go (Part 7) and how to see what happened to each one (Part 8).

Failures that come together. The simulation assumes every call fails independently. Real failures are often correlated: if the email provider is down for five minutes, all three attempts fail. Treat these numbers as a way to build intuition, not as a prediction.

A restart. The queue still lives in memory. If the process dies after step 4, nothing remembers that steps 1 to 4 were done. Part 6 moves the queue to Redis.

Takeaways

Reliability multiplies: ten steps at 99% give a job that succeeds about 90% of the time.
Retry the step that failed, not the whole job, so the steps that worked do not run again.
Per-step retries still repeat a step whose reply was lost, so side effects need idempotency keys.

Try it

This is a teaching version of multi-step jobs. A more complete implementation comes later in the series. Keep queue.ts from Part 2 and retry.ts from Part 3, and save the workflow.ts file from above next to them. Then save this as simulate.ts:

import { SimpleQueue, type Handler } from "./queue.ts";
import { withRetry, type RetryOptions } from "./retry.ts";
import { retryEachStep, workflow, type Step } from "./workflow.ts";

// Illustrative assumptions (not a real system):
const JOBS = 1_000_000; // signups to simulate
const P_FAIL = 0.01;    // every call to a step fails 1% of the time, independently
const LOST_REPLY = 0.005; // table 2 only: the email goes out, but the reply is lost 0.5% of the time
const STEPS = [
  "save the user", "create the billing customer", "create the workspace", "upload the default avatar",
  "add to the CRM", "subscribe to the newsletter", "send the welcome email", "notify the sales channel",
  "record the analytics event", "schedule the day-3 email",
];
const EMAIL = "send the welcome email";
// Up to 3 attempts. The simulation counts outcomes, not time, so it skips the waits.
const retry: RetryOptions = { maxAttempts: 3, baseDelayMs: 100, maxDelayMs: 2000, jitter: true, sleep: async () => {} };

// A small seeded random generator (mulberry32), so every run prints the same numbers.
function seededRandom(seed: number) {
  let a = seed;
  return () => {
    a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// What each signup job carries: how many times each step really did its work.
type Signup = { done: Record<string, number> };

async function simulate(build: (steps: Step<Signup>[]) => Handler<Signup>, lostReply: number) {
  const random = seededRandom(1);
  let calls = 0;
  const steps: Step<Signup>[] = STEPS.map((name) => ({
    name,
    run: async (job) => {
      calls++;
      if (random() < P_FAIL) throw new Error(`${name}: 503 service unavailable`);
      job.data.done[name] = (job.data.done[name] ?? 0) + 1; // the side effect happened
      if (name === EMAIL && random() < lostReply) throw new Error(`${name}: timeout, reply lost`);
    },
  }));
  const queue = new SimpleQueue<Signup>(build(steps));
  let repeatedSteps = 0;
  let twoEmails = 0;
  for (let start = 0; start < JOBS; start += 10_000) {
    const batch: Signup[] = [];
    for (let i = 0; i < 10_000; i++) {
      const signup: Signup = { done: {} };
      batch.push(signup);
      queue.add(signup);
    }
    await queue.onIdle();
    for (const signup of batch) {
      for (const times of Object.values(signup.done)) repeatedSteps += times - 1;
      if ((signup.done[EMAIL] ?? 0) > 1) twoEmails++;
    }
  }
  const failed = queue.failed.length;
  return {
    succeeded: `${((100 * (JOBS - failed)) / JOBS).toFixed(3)}%`,
    failed,
    halfDone: queue.failed.filter(({ job }) => Object.keys(job.data.done).length > 0).length,
    callsPerJob: Math.round((calls / JOBS) * 100) / 100,
    repeatedSteps,
    twoEmails,
  };
}

const pct = (p: number) => `${(100 * p).toFixed(3)}%`;
const oneTry = (1 - P_FAIL) ** STEPS.length;

async function main() {
  console.log(`1. ${JOBS.toLocaleString("en-US")} signups, ${STEPS.length} steps, each call fails ${P_FAIL * 100}% of the time`);
  console.table([
    { strategy: "no retries", formula: pct(oneTry), ...(await simulate((s) => workflow(s), 0)) },
    { strategy: "retry the whole job", formula: pct(1 - (1 - oneTry) ** 3), ...(await simulate((s) => withRetry(workflow(s), retry), 0)) },
    { strategy: "retry each step", formula: pct((1 - P_FAIL ** 3) ** STEPS.length), ...(await simulate((s) => workflow(retryEachStep(s, retry)), 0)) },
  ]);
  console.log(`2. Same, but the email provider loses the reply ${LOST_REPLY * 100}% of the time`);
  console.table([
    { strategy: "retry each step", ...(await simulate((s) => workflow(retryEachStep(s, retry)), LOST_REPLY)) },
  ]);
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

It runs 4,000,000 jobs, so it takes 15 to 20 seconds. The random generator is seeded, so you should see exactly these numbers:

1. 1,000,000 signups, 10 steps, each call fails 1% of the time
┌─────────┬───────────────────────┬───────────┬───────────┬────────┬──────────┬─────────────┬───────────────┬───────────┐
│ (index) │ strategy              │ formula   │ succeeded │ failed │ halfDone │ callsPerJob │ repeatedSteps │ twoEmails │
├─────────┼───────────────────────┼───────────┼───────────┼────────┼──────────┼─────────────┼───────────────┼───────────┤
│ 0       │ 'no retries'          │ '90.438%' │ '90.398%' │ 96024  │ 85902    │ 9.56        │ 0             │ 0         │
│ 1       │ 'retry the whole job' │ '99.913%' │ '99.913%' │ 867    │ 865      │ 10.57       │ 462891        │ 29564     │
│ 2       │ 'retry each step'     │ '99.999%' │ '99.999%' │ 9      │ 8        │ 10.1        │ 0             │ 0         │
└─────────┴───────────────────────┴───────────┴───────────┴────────┴──────────┴─────────────┴───────────────┴───────────┘
2. Same, but the email provider loses the reply 0.5% of the time
┌─────────┬───────────────────┬───────────┬────────┬──────────┬─────────────┬───────────────┬───────────┐
│ (index) │ strategy          │ succeeded │ failed │ halfDone │ callsPerJob │ repeatedSteps │ twoEmails │
├─────────┼───────────────────┼───────────┼────────┼──────────┼─────────────┼───────────────┼───────────┤
│ 0       │ 'retry each step' │ '99.999%' │ 11     │ 10       │ 10.11       │ 4971          │ 4939      │
└─────────┴───────────────────┴───────────┴────────┴──────────┴─────────────┴───────────────┴───────────┘

Change P_FAIL or add steps to STEPS and see how quickly the "no retries" row drops.

Previous: How to retry failed jobs in Node.js without making it worse (Part 3).

Next: Jobs that run twice: idempotency keys in Node.js (Part 5) looks at jobs that run twice and how idempotency keys stop the duplicates.

How to retry failed jobs in Node.js without making it worse

Sandro Dezerio — Wed, 14 Oct 2026 17:00:08 +0000

Part 3 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up scenario is made up. It is the same one used in Part 1 and Part 2. The timings and the provider limits come from a simulation with stated assumptions, not from a real email provider.

In Part 2, the queue sent 20 welcome emails at once, and the provider rejected 15 of them. The queue wrote those 15 failures into a list called failed and moved on. Nobody ever tried them again.

The fix looks obvious: try again. But a retry can also make things worse. If all the failed jobs retry at the same moment, they all hit the provider again at once. And if the error is permanent, the retry only wastes time.

This post adds job queue retries in Node.js, with three things that keep them safe: a growing wait between attempts (backoff), a random element in that wait (jitter), and a way to stop early when trying again cannot help.

The simplest retry

I keep the same made-up assumptions as in Part 2. Each email call takes 300 ms. Twenty welcome emails arrive at the same moment, and the queue runs up to 20 at once. The provider accepts at most 5 calls in flight and rejects a sixth with a "429, too many concurrent requests" error.

Without retries, 5 emails went out, and 15 were lost. Now let each job try up to 10 times, with a wait before every retry (I explain the wait in the next section). Everything goes out:

Strategy	Time	Emails sent	Lost	Calls to the provider	Rejected calls
No retries (Part 2)	0.3 s	5	15	20	15
Retries with backoff	1.8 s	20	0	65	45

Retries are not free. The queue took 1.8 seconds instead of 0.3, made 65 calls instead of 20, and rejected 45 of them. A retry trades time and extra calls for not losing the job. For a welcome email, that is a good trade.

How long to wait: backoff

If a job fails and tries again right away, it will most likely fail again, because the cause is still there. So the job waits, and it waits longer after each failure. The usual rule is to double the wait: 100 ms, 200 ms, 400 ms, and so on, with a maximum so the wait never grows without limit. BullMQ's exponential backoff uses the same doubling.

The code is a wrapper. withRetry takes a handler like the one from Part 2 and returns a new handler. The queue itself does not change.

import type { Handler } from "./queue.ts";

/** Throw this when trying again cannot help: bad input, a mailbox that does not exist. */
export class PermanentError extends Error {}

/** What ends up in `queue.failed` when a job gives up, with the reason. */
export class JobFailedError extends Error {
  constructor(
    readonly reason: "permanent" | "attempts exhausted",
    readonly attempts: number,
    readonly lastError: unknown,
  ) {
    super(`${reason} after ${attempts} attempt(s): ${(lastError as Error).message}`);
  }
}

export type RetryOptions = {
  maxAttempts: number; // total tries, including the first one
  baseDelayMs: number; // wait before the second try
  maxDelayMs: number; // the wait never grows past this
  jitter: boolean; // pick a random wait between 0 and the limit
  random?: () => number; // returns a number in [0, 1); replaceable in tests
  sleep?: (ms: number) => Promise<void>; // replaceable in tests
};

/** Wait after attempt number `attempt` (1 = the first try) has failed. */
export function backoffDelay(attempt: number, options: RetryOptions): number {
  const { baseDelayMs, maxDelayMs, jitter, random = Math.random } = options;
  const limit = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));
  return jitter ? Math.floor(random() * limit) : limit;
}

const defaultSleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

/** Wrap a handler so it retries transient errors and gives up on permanent ones. */
export function withRetry<T>(handler: Handler<T>, options: RetryOptions): Handler<T> {
  const sleep = options.sleep ?? defaultSleep;
  return async (job) => {
    // No loop condition: it ends only by return (success) or throw (give up).
    for (let attempt = 1; ; attempt++) {
      try {
        return await handler(job);
      } catch (error) {
        if (error instanceof PermanentError) {
          throw new JobFailedError("permanent", attempt, error);
        }
        if (attempt >= options.maxAttempts) {
          throw new JobFailedError("attempts exhausted", attempt, error);
        }
        await sleep(backoffDelay(attempt, options));
      }
    }
  };
}

Here are four things to notice:

backoffDelay doubles the wait after each failure. With a base of 100 ms and a maximum of 2,000 ms, the waits are 100, 200, 400, 800, and 1,600 ms. From then on, every wait is 2,000 ms.
withRetry runs the handler in a loop. If it succeeds, the loop ends. If it fails, the job waits and tries again until it runs out of attempts.
When a job gives up, it throws a JobFailedError with the reason, the number of attempts, and the last error. So queue.failed now says why a job failed, not only that it did. A reader on dev.to asked for exactly this after Part 1.
Using it takes one line around the handler:

import { SimpleQueue, type Job } from "./queue.ts";
import { withRetry } from "./retry.ts";

async function sendWelcomeEmail(job: Job<string>): Promise<void> {
  console.log(`sending to ${job.data}`);
}

const queue = new SimpleQueue<string>(
  withRetry(sendWelcomeEmail, { maxAttempts: 5, baseDelayMs: 100, maxDelayMs: 2000, jitter: true }),
  3, // from Part 2: at most 3 jobs at the same time
);
queue.add("ana@example.com");

Not every error deserves a retry

A "429, too many requests" will probably pass if you wait. A "550, mailbox does not exist" never will. Retrying it burns attempts and keeps a slot busy for nothing.

So the handler throws a PermanentError when trying again cannot help, and withRetry gives up at once. Everything else is treated as temporary. Here is a burst of three emails, one of them to an address that does not exist, with up to 10 attempts each:

Strategy	Calls for the bad address	Time until the queue is idle
Every error is retried	10	14.1 s
`PermanentError`, no retry	1	0.3 s

The bad address uses ten calls, and the queue stays busy for 14.1 seconds. The other two emails finish in the first 0.3 seconds.

Deciding which errors are permanent is a job for the handler, because only the handler knows what the provider's answer means. My rule: if the cause is in the request itself (bad input, a missing mailbox), it is permanent. If the cause is the state of the other side (a rate limit, a timeout, a server error), it is temporary. BullMQ has the same idea: throwing UnrecoverableError moves a job to the failed set without more retries, even when attempts is set.

Why jitter

Backoff has a weak spot. When many jobs fail at the same moment, they all wait the same 100 ms, then all retry together, then all wait 200 ms. They keep arriving in clusters. Marc Brooker described this problem in the AWS Architecture Blog: backoff alone leaves clusters of calls, and adding jitter spreads them out.

Jitter means the wait is not exactly the backoff value. In the "Full Jitter" version, the wait is a random time between 0 and that value. In the code above, that is jitter: true.

I tested it in the same simulation, with two providers: one that accepts 5 calls at once and one that accepts only 2. Jitter is random, so each jittered row is the average of 20 runs with fixed seeds.

Provider accepts	Strategy	Time	Emails sent (of 20)	Calls
5 at once	Backoff, no jitter	1.8 s	20	65
5 at once	Backoff, full jitter	2.3 s	20	80.8
2 at once	Backoff, no jitter	11.4 s	18	126
2 at once	Backoff, full jitter	4.6 s	19.9	113.9

The result is mixed. With the provider that accepts 5, jitter did not help: it was slower (2.3 s compared with 1.8 s) and made more calls (80.8 compared with 65). With the provider that accepts only 2, it helped: 4.6 seconds instead of 11.4, fewer calls, and nearly every email sent.

My reading is that jitter pays off when many jobs compete for very little room. When there is room, a short random wait mostly adds calls. The AWS article tests a harsher case, 100 clients fighting over one database row, and reports that jitter cut the number of calls by more than half. My simulation is smaller, and most of the gain here is in time, not in calls.

I would still turn jitter on by default. The cost in the mild case is half a second, and the gain in the harsh case is nearly seven seconds. That is my judgment from this simulation, not a rule.

What the simulation does not show

The 20 jobs live in one queue, and I use them to represent 20 independent callers. In production, the callers that retry in step are often separate servers recovering from the same outage.
Retries do not replace a sensible limit. In Part 2, a concurrency limit of 5 finished in 1.2 seconds with no rejected calls. That is faster than the 1.8 seconds above. Retries are for failures you cannot prevent.
The number of attempts should match how long the problem lasts. Change maxAttempts to 5 in the simulation and run it again. In the first table, backoff without jitter still sends all 20 emails. In the jitter table, the full-jitter row for the provider that accepts 5 drops to 16.3 emails sent on average because some jobs run out of attempts before a slot opens.

What this queue still does not do

A job that waits to retry still holds its slot. The other jobs keep running, and a test covers that. But a job that waits 2 seconds uses one of your concurrency slots for 2 seconds. That is fine for short waits. For waits of minutes, you want delayed jobs, which Part 9 covers.
close() waits for jobs that are waiting to retry. A test covers this too. A shutdown can take as long as the running jobs need to finish their retries.
It lives in memory. A restart loses every waiting job, retries included. Part 6 moves the queue to Redis.
A retry can run a job twice. If the provider sent the email but the answer got lost, the retry sends it again. Part 5 covers idempotency.
You will not write this by hand in production. BullMQ has attempts and backoff options, with a jitter setting, so the ideas here will look familiar in Part 6.

Takeaways

Retry a job when the cause is temporary, and give up at once when it is permanent.
Wait longer after each failure, and add jitter when many jobs can fail at the same moment.
Record why a job gave up: the reason, the number of attempts, and the last error.

Try it

This is a teaching version of retries. A more complete implementation comes later in the series. Keep the queue from Part 2 as queue.ts and save the retry.ts file from above next to it. Then save this as simulate.ts. It uses a virtual clock and a seeded random generator, so it prints the same numbers every time:

import { mock } from "node:test";
import { SimpleQueue } from "./queue.ts";
import { PermanentError, withRetry, type RetryOptions } from "./retry.ts";

// Illustrative assumptions (not a real provider):
const EMAIL_MS = 300;    // each email call takes 300 ms
const BURST = 20;        // 20 welcome emails arrive at the same moment
const CONCURRENCY = 20;  // the "all at once" setting that lost 15 emails in Part 2
const SEEDS = 20;        // jitter is random, so we repeat it with 20 fixed seeds
// The provider rejects a call with "429" when more than `limit` calls are in flight.

mock.timers.enable({ apis: ["setTimeout"] });
const flush = () => new Promise<void>((resolve) => setImmediate(resolve));
let now = 0;
async function advanceUntil(done: () => boolean) {
  await flush();
  while (!done()) {
    mock.timers.tick(10);
    now += 10;
    await flush();
  }
}
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

// A small seeded random generator, so every run prints the same numbers.
function seededRandom(seed: number) {
  let a = seed;
  return () => {
    a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

type Result = { seconds: number; sent: number; lost: number; calls: number; rejected: number };

async function runBurst(providerLimit: number, retry: RetryOptions | null): Promise<Result> {
  now = 0;
  let inFlight = 0;
  let sent = 0;
  let calls = 0;
  let rejected = 0;
  const sendEmail = async () => {
    calls++;
    inFlight++;
    try {
      if (inFlight > providerLimit) {
        rejected++;
        throw new Error("429 too many concurrent requests");
      }
      await sleep(EMAIL_MS);
      sent++;
    } finally {
      inFlight--;
    }
  };
  const queue = new SimpleQueue<number>(retry ? withRetry(sendEmail, retry) : sendEmail, CONCURRENCY);
  for (let i = 0; i < BURST; i++) queue.add(i);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { seconds: now / 1000, sent, lost: queue.failed.length, calls, rejected };
}

// Average over many seeds, rounded to one decimal.
async function runJittered(providerLimit: number, options: RetryOptions): Promise<Result> {
  const results: Result[] = [];
  for (let seed = 1; seed <= SEEDS; seed++) {
    results.push(await runBurst(providerLimit, { ...options, jitter: true, random: seededRandom(seed) }));
  }
  const avg = (pick: (r: Result) => number) =>
    Math.round((results.reduce((sum, r) => sum + pick(r), 0) / SEEDS) * 10) / 10;
  return {
    seconds: avg((r) => r.seconds), sent: avg((r) => r.sent), lost: avg((r) => r.lost),
    calls: avg((r) => r.calls), rejected: avg((r) => r.rejected),
  };
}

// Three emails, one of them to an address that does not exist.
async function runBadAddress(permanent: boolean, options: RetryOptions) {
  now = 0;
  let badCalls = 0;
  const queue = new SimpleQueue<string>(
    withRetry(async (job) => {
      await sleep(EMAIL_MS);
      if (job.data === "nobody@example.invalid") {
        badCalls++;
        const message = "550 mailbox does not exist";
        throw permanent ? new PermanentError(message) : new Error(message);
      }
    }, options),
    CONCURRENCY,
  );
  for (const to of ["ana@example.com", "nobody@example.invalid", "luis@example.com"]) queue.add(to);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { seconds: now / 1000, callsForTheBadAddress: badCalls };
}

async function main() {
  const options: RetryOptions = { maxAttempts: 10, baseDelayMs: 100, maxDelayMs: 2000, jitter: false };

  console.log("1. Retries on the burst from Part 2 (provider accepts 5 calls at once)");
  console.table([
    { strategy: "no retries (Part 2)", ...(await runBurst(5, null)) },
    { strategy: "retries, backoff", ...(await runBurst(5, options)) },
  ]);

  console.log(`2. Jitter (average of ${SEEDS} seeds for the jittered rows)`);
  const rows = [];
  for (const providerLimit of [5, 2]) {
    rows.push({ providerLimit, strategy: "backoff, no jitter", ...(await runBurst(providerLimit, options)) });
    rows.push({ providerLimit, strategy: "backoff, full jitter", ...(await runJittered(providerLimit, options)) });
  }
  console.table(rows);

  console.log("3. A mailbox that does not exist");
  console.table([
    { strategy: "every error is retried", ...(await runBadAddress(false, options)) },
    { strategy: "PermanentError, no retry", ...(await runBadAddress(true, options)) },
  ]);
  mock.timers.reset();
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

Node prints a warning that its mock timers API is experimental. You can ignore it. You should see these three tables:

1. Retries on the burst from Part 2 (provider accepts 5 calls at once)
┌─────────┬───────────────────────┬─────────┬──────┬──────┬───────┬──────────┐
│ (index) │ strategy              │ seconds │ sent │ lost │ calls │ rejected │
├─────────┼───────────────────────┼─────────┼──────┼──────┼───────┼──────────┤
│ 0       │ 'no retries (Part 2)' │ 0.3     │ 5    │ 15   │ 20    │ 15       │
│ 1       │ 'retries, backoff'    │ 1.8     │ 20   │ 0    │ 65    │ 45       │
└─────────┴───────────────────────┴─────────┴──────┴──────┴───────┴──────────┘
2. Jitter (average of 20 seeds for the jittered rows)
┌─────────┬───────────────┬────────────────────────┬─────────┬──────┬──────┬───────┬──────────┐
│ (index) │ providerLimit │ strategy               │ seconds │ sent │ lost │ calls │ rejected │
├─────────┼───────────────┼────────────────────────┼─────────┼──────┼──────┼───────┼──────────┤
│ 0       │ 5             │ 'backoff, no jitter'   │ 1.8     │ 20   │ 0    │ 65    │ 45       │
│ 1       │ 5             │ 'backoff, full jitter' │ 2.3     │ 20   │ 0    │ 80.8  │ 60.8     │
│ 2       │ 2             │ 'backoff, no jitter'   │ 11.4    │ 18   │ 2    │ 126   │ 108      │
│ 3       │ 2             │ 'backoff, full jitter' │ 4.6     │ 19.9 │ 0.1  │ 113.9 │ 94       │
└─────────┴───────────────┴────────────────────────┴─────────┴──────┴──────┴───────┴──────────┘
3. A mailbox that does not exist
┌─────────┬────────────────────────────┬─────────┬───────────────────────┐
│ (index) │ strategy                   │ seconds │ callsForTheBadAddress │
├─────────┼────────────────────────────┼─────────┼───────────────────────┤
│ 0       │ 'every error is retried'   │ 14.1    │ 10                    │
│ 1       │ 'PermanentError, no retry' │ 0.3     │ 1                     │
└─────────┴────────────────────────────┴─────────┴───────────────────────┘

Next: Ten steps at 99% is 90%: retrying multi-step jobs (Part 4) looks at why a workflow with ten steps can fail much more often than any single step suggests.

How many jobs should a Node.js queue run at once?

Sandro Dezerio — Sat, 10 Oct 2026 02:13:01 +0000

Part 2 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up scenario is made up, the same one as in Part 1. The timings and the provider limit come from a simulation with stated assumptions, not from a real email provider.

The queue from Part 1 sends one email at a time. When 20 sign-ups arrive together, the last welcome email leaves six seconds later.

The obvious fix is to send them all at once. That works until the other side says no.

This post adds job queue concurrency to the queue: a setting for how many jobs may run at the same time. It also shows how to choose that number, and how to shut the queue down without losing work that is already running.

Why one at a time is slow

I will keep the made-up assumptions from Part 1. Each email call takes 300 ms. This time 20 welcome emails arrive at the same moment, and nothing fails.

One at a time, that is 20 x 300 ms, or 6.0 seconds. For most of that time, the server is only waiting for the provider to answer. The queue is safe, but it wastes the wait.

Why all at once is worse

Now say the email provider accepts at most 5 calls in flight at the same time. A sixth call is rejected right away with a "429, too many concurrent requests" error. This limit is made up, but real services have limits like it.

If the queue starts all 20 emails together, everything finishes in 0.3 seconds. It also loses 15 of the 20 emails because the provider rejected them.

All at once looks like the fastest option, and it loses the most work. So the goal is not raw speed. It is the fastest rate the other side accepts.

Picture a small app that sends a welcome email on sign-up. A newsletter mentions it, and 200 people sign up within a few minutes. With one worker and 300 ms per email, the last person waits a full minute. So someone changes the limit to "as many as possible". Now the provider answers with 429 errors, and with no retries the rejected emails are simply lost. Both versions of the queue fail, in opposite ways. This scenario is made up, like the rest of this post.

The change: a job queue concurrency limit

Here is the queue again. The rest of the class works as in Part 1, but the worker loop changed, so I show the whole file once.

/**
 * Part 2: the same queue, now with a concurrency limit and a clean shutdown.
 * Still in memory, still no retries. Those are Part 3 and Part 6.
 */

export type Job<T> = { id: number; data: T };
export type Handler<T> = (job: Job<T>) => Promise<void>;

export class SimpleQueue<T> {
  private jobs: Job<T>[] = [];
  private nextId = 1;
  private active = 0; // jobs running right now
  private closing = false;
  private idleWaiters: Array<() => void> = [];
  readonly failed: Array<{ job: Job<T>; error: unknown }> = [];

  constructor(
    private handler: Handler<T>,
    private concurrency = 1, // how many jobs may run at the same time
  ) {}

  /** Put a job in line and return right away. The work happens later. */
  add(data: T): number {
    if (this.closing) throw new Error("queue is closing");
    const job = { id: this.nextId++, data };
    this.jobs.push(job);
    this.pump();
    return job.id;
  }

  /** Start jobs until the limit is reached or the line is empty. */
  private pump(): void {
    while (!this.closing && this.active < this.concurrency && this.jobs.length > 0) {
      const job = this.jobs.shift()!;
      this.active++;
      void this.run(job);
    }
  }

  private async run(job: Job<T>): Promise<void> {
    try {
      await this.handler(job);
    } catch (error) {
      this.failed.push({ job, error }); // remember it, keep going
    }
    this.active--;
    this.pump(); // a slot is free: start the next job
    this.notifyIfIdle();
  }

  private notifyIfIdle(): void {
    if (this.active === 0 && (this.closing || this.jobs.length === 0)) {
      this.idleWaiters.splice(0).forEach((resolve) => resolve());
    }
  }

  /** Resolves when every job added so far has been handled. */
  onIdle(): Promise<void> {
    if (this.active === 0 && this.jobs.length === 0) return Promise.resolve();
    return new Promise((resolve) => this.idleWaiters.push(resolve));
  }

  /**
   * Graceful shutdown: start nothing new, let running jobs finish,
   * and return the jobs that never started so the caller can decide what to do.
   */
  async close(): Promise<Job<T>[]> {
    this.closing = true;
    if (this.active > 0) {
      await new Promise<void>((resolve) => this.idleWaiters.push(resolve));
    }
    return this.jobs.splice(0);
  }
}

There are three ideas in it:

active counts the jobs running right now, and concurrency is the limit. The default is 1, so the queue behaves like Part 1 unless you ask for more.
pump starts jobs until the limit is reached or the line is empty.
When a job ends, whether it worked or failed, it frees a slot and calls pump again.

What the limit does

The simulation below runs the same burst of 20 emails with three limits. Every number depends on the made-up assumptions above.

Concurrency	Time to finish 20 emails	Emails sent	Rejected by the provider
1 (Part 1)	6.0 s	20	0
5	1.2 s	20	0
20	0.3 s	5	15

A limit of 5 is five times faster than one at a time and loses nothing. A limit of 20 is the fastest and loses 75% of the emails.

How to choose the number

Start from the limit of the thing you call, such as a provider or a database pool. Treat it as a ceiling, not as a target. If the provider accepts 5 calls at once, run 3 or 4, because other code may call the same provider too.

Then measure instead of guessing. Raise the number one step at a time and watch the error rate and the time to finish. Stop when errors appear or when the time stops improving.

This advice fits jobs that mostly wait for the network. Jobs that burn CPU are different, because Node.js runs your code on one event loop, and long CPU work blocks everything else. A higher limit does not help there.

You will meet the same setting in production libraries. A BullMQ worker has a concurrency option that defaults to 1 and helps with asynchronous work. Part 6 uses BullMQ, so this number will look familiar.

Shutting down without losing running work

Deployments restart your process. If jobs are running at that moment, you want them to finish, and you want to know which jobs never started.

close() does that. It stops starting new jobs, waits for the running ones, and returns the jobs that never started. After that, add throws.

In a test with a limit of 2 and 6 jobs, the 2 running jobs finish, and the other 4 come back to the caller.

Those 4 jobs still live only in memory. If the process exits, they are gone. Part 6 fixes that.

What this queue still does not do

It never retries. The 15 rejected emails above are the kind of failure worth retrying later. The reason matters: a rate limit is worth retrying, but a mailbox that does not exist is not. Part 3 covers retries with backoff.
It lives in memory. Part 6 moves the queue to Redis.
It cannot run a job twice yet, but retries will change that. Part 5 covers idempotency.

Takeaways

Running jobs one at a time is safe but slow, and running them all at once can overload what you depend on.
Job queue concurrency controls the speed of a queue, so start from the limit of the dependency and leave some room below it.
A queue that shuts down cleanly finishes what is running and tells you what it never started.

Try it

Save the queue from above as queue.ts. Next to it, save this as simulate.ts. It uses a virtual clock, so it prints the same numbers every time:

import { mock } from "node:test";
import { SimpleQueue } from "./queue.ts";

// Illustrative assumptions (not a real provider):
const EMAIL_MS = 300;        // each email call takes 300 ms
const PROVIDER_LIMIT = 5;    // the provider rejects calls when more than 5 are in flight
const BURST = 20;            // 20 welcome emails arrive at the same moment

mock.timers.enable({ apis: ["setTimeout"] });
const flush = () => new Promise<void>((resolve) => setImmediate(resolve));
let now = 0;
async function advanceUntil(done: () => boolean) {
  await flush();
  while (!done()) {
    mock.timers.tick(10);
    now += 10;
    await flush();
  }
}
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

async function run(concurrency: number) {
  now = 0;
  let inFlight = 0;
  let sent = 0;
  const queue = new SimpleQueue<number>(async () => {
    inFlight++;
    const rejected = inFlight > PROVIDER_LIMIT;
    try {
      if (rejected) throw new Error("429 too many concurrent requests");
      await sleep(EMAIL_MS);
      sent++;
    } finally {
      inFlight--;
    }
  }, concurrency);

  for (let i = 0; i < BURST; i++) queue.add(i);
  let idle = false;
  void queue.onIdle().then(() => (idle = true));
  await advanceUntil(() => idle);
  return { concurrency, seconds: now / 1000, sent, failed: queue.failed.length };
}

async function main() {
  const rows = [];
  for (const c of [1, 5, 20]) rows.push(await run(c));
  console.table(rows);
  mock.timers.reset();
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

Node prints a warning that its mock timers API is experimental. You can ignore it. You should see this table:

┌─────────┬─────────────┬─────────┬──────┬────────┐
│ (index) │ concurrency │ seconds │ sent │ failed │
├─────────┼─────────────┼─────────┼──────┼────────┤
│ 0       │ 1           │ 6       │ 20   │ 0      │
│ 1       │ 5           │ 1.2     │ 20   │ 0      │
│ 2       │ 20          │ 0.3     │ 5    │ 15     │
└─────────┴─────────────┴─────────┴──────┴────────┘

To see the ceiling move, change PROVIDER_LIMIT to 8 and the list [1, 5, 20] to [1, 5, 8, 20], then run it again. A limit of 8 finishes in 0.9 seconds and loses nothing, while 20 now loses 12 emails.

Next: Part 3, "How to retry failed jobs in Node.js without making it worse"

What is a job queue, and why does my API need one?

Sandro Dezerio — Tue, 06 Oct 2026 03:00:21 +0000

Part 1 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior fullstack engineer.

The sign-up scenario in this post is made up. The timings and failure rates come from a simulation with stated assumptions, not from a real email provider.

Most APIs start the same way. A user signs up, you save them in the database, and you send a welcome email before returning a response. It works on your laptop. It keeps working for a while in production.

Then the email provider gets slow for an afternoon, and your sign-up page gets slow with it.

This post builds the smallest useful fix: a job queue in about 40 lines of TypeScript. It is the first step in a series that starts here and ends with reliable, multi-step workflows, including ones that call an LLM.

The problem

Here is the handler, in plain terms:

Save the user (fast)
Send the welcome email (slow, and it can fail)
Return "ok" to the user

Two things are wrong with this.

The user waits for work they don't care about. They asked to sign up. They did not ask to wait for an email to leave your system.

A failure in a side task becomes a failure in the main task. If the email call times out, the handler throws, and the user sees an error. Their account exists, but they think it doesn't. They try again and hit "email already in use".

I ran into this myself building a PDF export feature: generating the file took long enough that the request routinely timed out, and the UI just sat there frozen waiting for an answer nobody needed right away. Moving the generation into a background job was the actual fix. The request returned immediately, and the UI picked up the finished file once it was ready, instead of holding the whole page hostage for a slow report.

The idea

Split the work into two parts:

The request does the part the user needs: save the user, answer "ok".
Everything else goes into a queue, a line of jobs that a worker processes in the background.

The handler adds a job and returns straight away. The worker takes jobs from the line, one at a time, and does the slow work.

A queue in about 40 lines

export type Job<T> = { id: number; data: T };
export type Handler<T> = (job: Job<T>) => Promise<void>;

export class SimpleQueue<T> {
  private jobs: Job<T>[] = [];
  private nextId = 1;
  private running = false;
  private idleWaiters: Array<() => void> = [];
  readonly failed: Array<{ job: Job<T>; error: unknown }> = [];

  constructor(private handler: Handler<T>) {}

  /** Put a job in line and return right away. The work happens later. */
  add(data: T): number {
    const job = { id: this.nextId++, data };
    this.jobs.push(job);
    void this.drain(); // start the worker if it is not already running
    return job.id;
  }

  /** The worker: take jobs one at a time until the line is empty. */
  private async drain(): Promise<void> {
    if (this.running) return;
    this.running = true;
    while (this.jobs.length > 0) {
      const job = this.jobs.shift()!;
      try {
        await this.handler(job);
      } catch (error) {
        this.failed.push({ job, error }); // remember it, keep going
      }
    }
    this.running = false;
    this.idleWaiters.splice(0).forEach((resolve) => resolve());
  }

  /** Resolves when every job added so far has been handled. */
  onIdle(): Promise<void> {
    if (!this.running && this.jobs.length === 0) return Promise.resolve();
    return new Promise((resolve) => this.idleWaiters.push(resolve));
  }
}

There are three ideas in it:

add puts the job at the end of the line and returns. It never waits for the work.
drain is the worker. It runs until the line is empty, and only one drain runs at a time.
If a job throws, the queue records it in failed and moves on. One bad job does not stop the rest.

Does it help? A simulation

I simulated a sign-up endpoint with these assumptions. All of them are illustrative:

Saving the user takes 20 ms.
The email provider takes 300 ms to answer.
10% of email calls fail with a timeout.
20 sign-ups arrive one after another.

The simulation runs on a virtual clock and a seeded random generator, so it gives the same numbers every time. The code for the simulation is not included in this post.

	Email inside the request	Email in a queue
Average response time	320 ms	20 ms
Sign-ups that returned an error	2 of 20	0 of 20

The response time drops because the user no longer waits for the email. The errors disappear from the sign-up path because a failed email can no longer fail a sign-up.

The emails still fail, though. In the queued version, 2 emails failed in the background, and nothing retries them yet. The problem did not go away. It moved to a place where it can be handled without hurting the user.

What this queue does not do

This is a teaching tool, and its limits are the plan for the rest of the series:

It runs one job at a time. After the last sign-up, the worker needed about 5.6 more seconds to send 20 emails. Part 2 covers concurrency.
It never retries. A failed job is only recorded. Part 3 covers retries with backoff.
It lives in memory. If the process restarts, every job still in line is lost. Part 6 moves the queue to Redis with BullMQ.
It cannot run a job twice yet, but retries will change that. Once failed jobs are retried, the same job can run more than once. Part 5 covers idempotency.

In production you will usually reach for a library such as BullMQ instead of writing your own. Building the small version first makes it clear what the library is doing for you.

Takeaways

Do only what the user needs inside the request. Queue the rest.
A queue separates a slow or failing side task from the main task.
A queue does not remove failures. It gives you a place to handle them.

Try it

Save the queue from above as queue.ts. Next to it, save this as example.ts. It is a sign-up that saves the user, adds the welcome email to the queue, and answers right away:

import { SimpleQueue } from "./queue.ts";

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

const emailQueue = new SimpleQueue<{ email: string }>(async (job) => {
  await sleep(300); // pretend to call the email provider
  console.log(`  [worker] welcome email sent to ${job.data.email}`);
});

async function signup(email: string) {
  await sleep(20); // pretend to save the user
  emailQueue.add({ email }); // returns immediately
  console.log(`signup ok: ${email}`);
}

async function main() {
  await signup("ana@example.com");
  await signup("luis@example.com");
  await signup("sofia@example.com");
  console.log("all signups answered, emails still on their way");
  await emailQueue.onIdle();
  console.log("queue is empty");
}

main();

Run it:

pnpm dlx tsx example.ts

You should see all three sign-ups answered first, and the emails go out afterwards, one at a time:

signup ok: ana@example.com
signup ok: luis@example.com
signup ok: sofia@example.com
all signups answered, emails still on their way
  [worker] welcome email sent to ana@example.com
  [worker] welcome email sent to luis@example.com
  [worker] welcome email sent to sofia@example.com
queue is empty

Try making the handler throw for one email. The other two still go out, and the failed job ends up in queue.failed.

The simulation behind the table is not part of this post. A more complete implementation, with the simulation, tests and the features from the next parts, will come later in the series.

Next: Part 2, "How many jobs should a Node.js queue run at once?"