Jobs that run twice: idempotency keys in Node.js

Part 5 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up scenario is made up. It is the same one used in Part 1, Part 2, Part 3, and Part 4. The failure rates come from a simulation with stated assumptions, not from real services.

At the end of Part 4, a simulation of 1,000,000 sign-ups sent the welcome email twice to 4,939 users. Nothing had failed in the way you would expect. The email went out, the reply got lost, and the retry did what a retry does.

A retry promises that the work will be done. It does not promise that the work will be done once. Parts 3 and 4 made jobs more reliable by running them again, so now a job can run twice, and the code has to make that harmless. The tool for this is an idempotency key. This post builds a small one, shows where it helps, and shows where it does not.

Where the duplicates come from

Three things can make the sign-up job repeat a side effect:

• A lost reply. The provider did the work, but the answer never arrived. This is the case from Part 4.
• A replay. Retrying the whole job, or running it again after a crash, repeats steps that already worked.
• A repeated request. The same sign-up is added to the queue twice, for example because a client timed out and sent the request again.

Some repeats are harmless: recording an analytics event twice only skews a chart. Emailing a customer twice, or charging a card twice, is not harmless.

What an idempotency key is

An operation is idempotent when doing it twice has the same effect as doing it once. Setting a user's plan to "pro" is idempotent. Sending an email is not.

An idempotency key makes an operation behave as if it were. The key is a name for one specific action, such as "the welcome email for user 1001." The first time the system sees the key, it does the work and remembers that the key finished. Every later time, it skips the work.

Stripe's API works this way. Its documentation says the API supports idempotency "for safely retrying requests without accidentally performing the same operation twice," and that "Retries with the same key return the saved response."

A small store

The code adds one file, idempotency.ts. The queue from Part 2, the retry wrapper from Part 3, and the workflow file from Part 4 do not change:

import type { Job } from "./queue.ts";
import type { Step } from "./workflow.ts";

type Entry = { value: unknown; expiresAt: number };

/**
 * Remembers which keys have finished, so the same key never runs its work twice.
 * A run that throws is not remembered, so a retry can try again.
 */
export class IdempotencyStore {
  private finished = new Map<string, Entry>();
  private running = new Map<string, Promise<unknown>>();

  constructor(
    private ttlMs: number, // how long a finished key is remembered
    private now: () => number = Date.now, // replaceable in tests
  ) {}

  async once<T>(key: string, work: () => Promise<T>): Promise<T> {
    const saved = this.finished.get(key);
    if (saved && saved.expiresAt > this.now()) return saved.value as T; // finished before: skip the work
    this.finished.delete(key); // forget an expired key
    const inFlight = this.running.get(key);
    if (inFlight) return inFlight as Promise<T>; // the same key is running right now: share that run
    const run = work()
      .then((value) => {
        this.finished.set(key, { value, expiresAt: this.now() + this.ttlMs });
        return value;
      })
      .finally(() => this.running.delete(key));
    this.running.set(key, run);
    return run;
  }
}

/** Skip any step that already finished for this key. `keyOf` names the business action, not the attempt. */
export function idempotentSteps<T>(
  steps: Step<T>[],
  store: IdempotencyStore,
  keyOf: (job: Job<T>) => string,
): Step<T>[] {
  return steps.map((step) => ({
    name: step.name,
    run: (job) => store.once(`${step.name}:${keyOf(job)}`, () => step.run(job)),
  }));
}

Here are three things to notice:

1. once has three cases. If the key finished before, it returns the saved result and skips the work. If the same key is running right now, it shares that run. Otherwise, it runs the work and remembers the result.
2. A run that throws is not remembered. If it were, the retry from Part 3 would get the same error back and could never succeed.
3. keyOf names the business action: here, the user ID. It is not the job ID, because the same sign-up added twice gets two job IDs, and it is not the attempt number, because that changes on every retry.

I use an internal user ID and not the email address, because Stripe's documentation advises against putting sensitive data, such as email addresses, in keys. The same page suggests random keys. A random key and a user ID follow the same rule: the key is created once per action and sent unchanged on every attempt.

Here is the sign-up added twice, plus one more user:

import { SimpleQueue, type Job } from "./queue.ts";
import { IdempotencyStore, idempotentSteps } from "./idempotency.ts";
import { workflow, type Step } from "./workflow.ts";

type Signup = { userId: string; email: string };

const steps: Step<Signup>[] = [
  { name: "save the user", run: async (job: Job<Signup>) => console.log(`saving ${job.data.email}`) },
  { name: "send the welcome email", run: async (job: Job<Signup>) => console.log(`emailing ${job.data.email}`) },
];

const store = new IdempotencyStore(24 * 60 * 60 * 1000); // remember finished keys for 24 hours
const queue = new SimpleQueue<Signup>(workflow(idempotentSteps(steps, store, (job) => job.data.userId)));

queue.add({ userId: "user_1001", email: "[e-mail]" });
queue.add({ userId: "user_1001", email: "[e-mail]" }); // the same signup arrives twice
queue.add({ userId: "user_1002", email: "[e-mail]" });

Save it as usage.ts next to the other files and run it:

pnpm dlx tsx usage.ts

saving [e-mail]
emailing [e-mail]
saving [e-mail]
emailing [e-mail]

Ana's sign-up arrived twice, and each of her steps ran once.

The store cannot see a lost reply

The store remembers only runs that finished. In a lost reply, the email provider does the work, but the call ends with a timeout. From the store's point of view, the run has failed, so nothing is saved, and the retry sends the email again. A test I wrote for this post confirms it: with a store in place, a lost reply still sends two emails.

The key has to reach whoever performs the side effect. Stripe does this with an Idempotency-Key header: its servers remember the key, not your process. If the email provider remembers the key too, the retry is safe, because the provider sees the same key and sends nothing. Not every API accepts a key, so check the documentation of each provider you call.

What changes in the numbers

The simulation is the one from Part 4, with a stable ID for each sign-up. It runs 1,000,000 sign-ups through the queue from Part 2. Every call to a step fails 1% of the time, the email reply is lost 0.5% of the time, and each strategy has up to 3 attempts. It assumes the email provider remembers keys and sends one email per key, the way Stripe's API handles a repeated request.

Strategy	Succeeded	Failed jobs	Calls per job	Steps done again	Users with 2+ welcome emails
Retry each step, no key	99.999%	11	10.11	4,971	4,939
Retry each step, new key on every call	99.999%	11	10.11	4,971	4,939
Retry each step, same key every time	99.999%	11	10.11	0	0
Retry the whole job, same key	99.900%	995	10.6	464,233	0
Retry the whole job, same key, skip finished steps	99.975%	250	10.11	0	0

Here is what each row says:

• No key. This is the last row of Part 4's results: 4,939 users with two emails.
• A new key on every call. The result is the same, because the provider never sees the same key twice. A key that changes on every attempt is not a key.
• The same key every time. No duplicates, and nothing else changes: the same 11 failed jobs and the same number of calls.
• Retry the whole job, same key. No duplicate emails, but the other nine steps ran 464,233 times again. They send no key, so nothing stops them.
• Retry the whole job, skip finished steps. The store skips every step that already finished, so nothing runs again, and the calls per job match retrying each step.

Retrying each step is still the better strategy: it gave up on 11 sign-ups and the whole-job retry gave up on 250. A whole-job retry ends at the first failure, so three failures anywhere in the job are enough to fail it. Retrying each step needs three failures in a row on the same step. The store makes the whole-job retry safe, but it does not make it better.

Designing the key

• Name the business action. welcome-email:user_1001 says what and for whom. A job ID, an attempt number, or a timestamp changes when the action has not changed, and the duplicate gets through.
• Include the action in the key. The store builds step name:user ID, so two steps of the same user never share a key.
• Decide how long to remember. The key has to outlive the longest time a duplicate can still arrive, including the whole retry window. If the key expires first, the duplicate runs again. If you keep it forever, memory grows. Stripe says it can remove keys once they are at least 24 hours old, and the code in this post uses 24 hours as an illustration. Your number depends on your retries and your traffic.
• Treat the same key with different data as a bug. Stripe compares the new request with the original one and returns an error if they differ. The store here does not check, so the second call gets the first result.

What this does not fix

A restart. The store lives in memory, like the queue. If the process dies, it forgets every key, and a second process does not share them. Part 6 moves the queue to Redis.

A provider without keys. You can ask the provider "did I already send this?" before sending. But asking and sending are two calls, and another attempt can still run between them. Prefer a provider that accepts a key, and know which of your side effects can repeat.

The simulation itself. It assumes independent failures and a provider that honors the key every time. Treat the numbers as a way to build intuition, not as a prediction.

Takeaways

1. A retry can run a side effect twice. An idempotency key turns "twice" into "once."
2. The key names the business action and stays the same on every attempt. A new key on every attempt protects nothing.
3. The key has to reach whoever performs the side effect. A store in your own process cannot see a lost reply.

Try it

This is a teaching version of idempotency keys. A more complete implementation comes later in the series. Keep queue.ts from Part 2, retry.ts from Part 3, and workflow.ts from Part 4. Save the idempotency.ts file from above next to them, and then save this as simulate.ts:

import { SimpleQueue, type Handler } from "./queue.ts";
import { withRetry, type RetryOptions } from "./retry.ts";
import { retryEachStep, workflow, type Step } from "./workflow.ts";
import { IdempotencyStore, idempotentSteps } from "./idempotency.ts";

// Illustrative assumptions (not a real system):
const JOBS = 1_000_000; // signups to simulate
const P_FAIL = 0.01;    // every call to a step fails 1% of the time, independently
const LOST_REPLY = 0.005; // the email goes out, but the reply is lost 0.5% of the time
const STEPS = [
  "save the user", "create the billing customer", "create the workspace", "upload the default avatar",
  "add to the CRM", "subscribe to the newsletter", "send the welcome email", "notify the sales channel",
  "record the analytics event", "schedule the day-3 email",
];
const EMAIL = "send the welcome email";
// Up to 3 attempts. The simulation counts outcomes, not time, so it skips the waits.
const retry: RetryOptions = { maxAttempts: 3, baseDelayMs: 100, maxDelayMs: 2000, jitter: true, sleep: async () => {} };

// A small seeded random generator (mulberry32), so every run prints the same numbers.
function seededRandom(seed: number) {
  let a = seed;
  return () => {
    a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// What each signup job carries: a stable ID and how many times each step really did its work.
type Signup = { userId: string; done: Record<string, number> };

// Which key the email step sends to the provider. The provider sends one email per key it has seen.
type KeyMode = "no key" | "new key on every call" | "same key every time";

async function simulate(build: (steps: Step<Signup>[], store: IdempotencyStore) => Handler<Signup>, keyMode: KeyMode) {
  const random = seededRandom(1);
  const providerKeys = new Set<string>(); // the keys the email provider has already seen
  let freshKey = 0;
  let calls = 0;
  const steps: Step<Signup>[] = STEPS.map((name) => ({
    name,
    run: async (job) => {
      calls++;
      if (random() < P_FAIL) throw new Error(`${name}: 503 service unavailable`);
      let alreadySent = false;
      if (name === EMAIL && keyMode !== "no key") {
        const key = keyMode === "same key every time" ? `welcome-email:${job.data.userId}` : `welcome-email:${job.data.userId}:${freshKey++}`;
        alreadySent = providerKeys.has(key);
        providerKeys.add(key);
      }
      if (!alreadySent) job.data.done[name] = (job.data.done[name] ?? 0) + 1; // the side effect happened
      if (name === EMAIL && random() < LOST_REPLY) throw new Error(`${name}: timeout, reply lost`);
    },
  }));
  let failed = 0;
  let repeatedSteps = 0;
  let twoEmails = 0;
  for (let start = 0; start < JOBS; start += 10_000) {
    // A new queue and store for every 10,000 signups keeps memory small. Every key belongs to one signup, so results do not change.
    const queue = new SimpleQueue<Signup>(build(steps, new IdempotencyStore(24 * 60 * 60 * 1000)));
    const batch: Signup[] = [];
    for (let i = 0; i < 10_000; i++) {
      const signup: Signup = { userId: `user_${start + i}`, done: {} };
      batch.push(signup);
      queue.add(signup);
    }
    await queue.onIdle();
    failed += queue.failed.length;
    for (const signup of batch) {
      for (const times of Object.values(signup.done)) repeatedSteps += times - 1;
      if ((signup.done[EMAIL] ?? 0) > 1) twoEmails++;
    }
    providerKeys.clear();
  }
  return {
    succeeded: `${((100 * (JOBS - failed)) / JOBS).toFixed(3)}%`,
    failed,
    callsPerJob: Math.round((calls / JOBS) * 100) / 100,
    repeatedSteps,
    twoEmails,
  };
}

async function main() {
  console.log(`${JOBS.toLocaleString("en-US")} signups, ${STEPS.length} steps, each call fails ${P_FAIL * 100}% of the time, the email reply is lost ${LOST_REPLY * 100}% of the time`);
  console.table([
    { strategy: "retry each step, no key", ...(await simulate((s) => workflow(retryEachStep(s, retry)), "no key")) },
    { strategy: "retry each step, new key on every call", ...(await simulate((s) => workflow(retryEachStep(s, retry)), "new key on every call")) },
    { strategy: "retry each step, same key every time", ...(await simulate((s) => workflow(retryEachStep(s, retry)), "same key every time")) },
    { strategy: "retry the whole job, same key", ...(await simulate((s) => withRetry(workflow(s), retry), "same key every time")) },
    {
      strategy: "retry the whole job, same key, skip finished steps",
      ...(await simulate((s, store) => withRetry(workflow(idempotentSteps(s, store, (job) => job.data.userId)), retry), "same key every time")),
    },
  ]);
}
main();

Run it with Node.js 22, the version I tested:

pnpm dlx tsx simulate.ts

It runs 5,000,000 jobs, so it takes about a minute. The random generator is seeded, so you should see exactly these numbers:

1,000,000 signups, 10 steps, each call fails 1% of the time, the email reply is lost 0.5% of the time
┌─────────┬──────────────────────────────────────────────────────┬───────────┬────────┬─────────────┬───────────────┬───────────┐
│ (index) │ strategy                                             │ succeeded │ failed │ callsPerJob │ repeatedSteps │ twoEmails │
├─────────┼──────────────────────────────────────────────────────┼───────────┼────────┼─────────────┼───────────────┼───────────┤
│ 0       │ 'retry each step, no key'                            │ '99.999%' │ 11     │ 10.11       │ 4971          │ 4939      │
│ 1       │ 'retry each step, new key on every call'             │ '99.999%' │ 11     │ 10.11       │ 4971          │ 4939      │
│ 2       │ 'retry each step, same key every time'               │ '99.999%' │ 11     │ 10.11       │ 0             │ 0         │
│ 3       │ 'retry the whole job, same key'                      │ '99.900%' │ 995    │ 10.6        │ 464233        │ 0         │
│ 4       │ 'retry the whole job, same key, skip finished steps' │ '99.975%' │ 250    │ 10.11       │ 0             │ 0         │
└─────────┴──────────────────────────────────────────────────────┴───────────┴────────┴─────────────┴───────────────┴───────────┘

Change LOST_REPLY or P_FAIL and see which rows change.

Previous: Ten steps at 99% is 90%: retrying multi-step jobs (Part 4).

Next: Part 6 (coming soon) moves the queue from memory to Redis with BullMQ, so a restart no longer loses waiting jobs.