Muhammad Ahsan Farooq

Posted on Jan 4 Edited on Oct 16

Idempotency: Making Retries Safe in Distributed Systems

#distributedsystems #systemdesign #backend #api

Retries are unavoidable in distributed systems, so any endpoint with side effects must survive being called twice. This post shows how to build that with idempotency keys, Redis and a database constraint.

The double-charge problem

Distributed systems fail. Networks drop packets, services time out, containers restart and clients retry. None of this is exceptional; it is the normal operating environment.

The danger starts when a retry runs a side effect twice. Picture a client sending "Charge the customer $50." The server charges the card, then crashes before it can respond. The client gets no confirmation, assumes failure and retries. Without protection, the customer is charged twice.

Client                    Server                   Payment provider
  |                         |                            |
  |-- POST /charge $50 ---->|                            |
  |                         |-- charge $50 ------------->|
  |                         |<-- success ----------------|
  |                         X  crashes before replying   |
  |   ... timeout ...       |                            |
  |-- POST /charge $50 ---->|   (retry)                  |
  |                         |-- charge $50 ------------->|  <- second charge

This is not a corner case. Two facts every backend engineer should internalize:

A timeout is not a failure. It only means no response arrived, not that the work didn't happen.
A missing response is not an unprocessed request. The server may have finished the job perfectly.

Because of that ambiguity, retries are both necessary and dangerous. Idempotency is what makes them safe.

What idempotency means

An operation is idempotent if performing it many times has the same effect as performing it once.

Safe to retry:

SET balance = 100
DELETE /users/123
PUT /profiles/123

These leave the same final state however often they run. A repeated DELETE may return 404 instead of 204, but the state is unchanged, and the state is what counts.

Dangerous to retry:

balance += 100       // run twice, you get +200
POST /orders         // duplicate orders
sendEmail()          // spams the user

Most real-world POST endpoints are not naturally idempotent, so we have to design them to be. There are two routes:

Make the operation naturally idempotent when you can: set absolute values instead of incrementing, use upserts, or let the client choose the resource ID and use PUT.
Add an idempotency key when you can't, which is the case for charging a card, creating an order or sending a message.

The Idempotency-Key pattern

The standard fix, used by payment APIs such as Stripe and Adyen, is the idempotency key. The client attaches a unique key to each logical operation, and the server uses it to recognise repeats.

The client's job

Generate a unique identifier (a UUID v4 works) once per user intent.
Send it in a header such as Idempotency-Key.
Reuse the same key on every retry of that operation, and a new key for a new operation.

The server's job

Atomically claim the key before doing any work.
If the key was already completed, return the stored response without re-running the side effect.
If it is new, run the operation, store the result under the key, and return it.

Here is the full lifecycle, including the lost response:

Client                      Server                        Redis / DB
  |-- POST + key K -------->|                                |
  |                         |-- claim K (SET NX) ----------->|
  |                         |<-- claimed --------------------|
  |                         |   run side effect              |
  |                         |-- save result under K -------->|
  |<-- 201 + result --------|                                |
  |      (response lost)                                     |
  |-- retry, same key K --->|                                |
  |                         |-- claim K -------------------->|
  |                         |<-- already done + result ------|
  |<-- 201 + saved result --|   side effect NOT run again    |

The server can meet four situations, and each needs a defined answer:

Key is new: claim it, run the operation, store and return the result.
Key is complete, same payload: replay the stored response. Adding an Idempotent-Replayed: true header helps debugging.
Key is still in progress: return 409 Conflict. The client should wait briefly and retry.
Key exists but the payload differs: return 422. The client has a bug, because one key must map to one operation.

With this in place, a retry replays the stored result instead of re-running the side effect, so the outcome is effectively once. The next sections add the layers that keep that true when Redis loses data or a request outlives its lock.

Server implementation (Node.js, Express, Redis)

The middleware below handles all four situations from the lifecycle above. It claims the key atomically with SET NX, so there is no gap between "check" and "set" for a concurrent request to slip through.

import { createHash, randomUUID } from 'node:crypto';
import type { NextFunction, Request, Response } from 'express';
import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const RESULT_TTL_SECONDS = 60 * 60 * 24; // keep finished results for 24 hours
const LOCK_TTL_SECONDS = 60;             // must exceed your slowest request

type Entry =
  | { state: 'processing'; fingerprint: string; token: string }
  | { state: 'completed'; fingerprint: string; statusCode: number; body: unknown };

// Same key + different request means the client has a bug.
const fingerprintOf = (req: Request) =>
  createHash('sha256')
    .update(`${req.method}:${req.originalUrl}:${JSON.stringify(req.body)}`)
    .digest('hex');

// "Try again" answers are not results. Never store them: release the key instead.
const isFinal = (status: number) => status < 500 && ![202, 409, 429].includes(status);

// Only touch the key if we still own it. If our lock expired and a retry
// claimed the key, its token differs and we leave its claim alone.
const OWNED = `local v = redis.call('GET', KEYS[1])
if not v or cjson.decode(v).token ~= ARGV[1] then return 0 end`;
const RELEASE = `${OWNED}
return redis.call('DEL', KEYS[1])`;
const COMPLETE = `${OWNED}
return redis.call('SET', KEYS[1], ARGV[2], 'EX', ARGV[3])`;

export async function idempotency(req: Request, res: Response, next: NextFunction) {
  const key = req.header('Idempotency-Key');
  if (!key || key.length > 255) {
    return res.status(400).json({ error: 'A valid Idempotency-Key header is required' });
  }

  // Scope keys per account (req.user comes from your auth middleware)
  // so two customers can never collide on the same key.
  const cacheKey = `idem:${req.user.id}:${key}`;
  const fingerprint = fingerprintOf(req);
  const token = randomUUID(); // proves which request owns the claim

  try {
    // 1. Atomically claim the key. Only one concurrent request can win.
    const claimed = await redis.set(
      cacheKey,
      JSON.stringify({ state: 'processing', fingerprint, token } satisfies Entry),
      { NX: true, EX: LOCK_TTL_SECONDS },
    );

    // 2. Someone already claimed it: replay, reject or ask the client to wait.
    if (!claimed) {
      const raw = await redis.get(cacheKey);
      if (!raw) return res.status(409).json({ error: 'Please retry' }); // expired in between
      const entry = JSON.parse(raw) as Entry;

      if (entry.fingerprint !== fingerprint) {
        return res
          .status(422)
          .json({ error: 'Idempotency-Key was reused with a different request' });
      }
      if (entry.state === 'processing') {
        return res.status(409).json({ error: 'Request already in progress' });
      }
      res.setHeader('Idempotent-Replayed', 'true');
      return res.status(entry.statusCode).json(entry.body);
    }

    // 3. We own the key. Capture the response so it can be stored and replayed.
    const sendJson = res.json.bind(res);
    res.json = (body: unknown) => {
      const save = isFinal(res.statusCode)
        ? redis.eval(COMPLETE, {
            keys: [cacheKey],
            arguments: [
              token,
              JSON.stringify({
                state: 'completed',
                fingerprint,
                statusCode: res.statusCode,
                body,
              } satisfies Entry),
              String(RESULT_TTL_SECONDS),
            ],
          })
        : // a transient answer is not final: let a retry run again
          redis.eval(RELEASE, { keys: [cacheKey], arguments: [token] });

      // Persist before replying, so any result the client sees can be replayed.
      save
        .catch((err) => console.error('idempotency store failed', err))
        .finally(() => sendJson(body));
      return res;
    };

    next();
  } catch (err) {
    next(err); // Redis unavailable: fail closed rather than risk a duplicate charge
  }
}

Compared with a simple "check the cache, then set it" middleware, this version fixes five weaknesses:

No race window. SET NX makes claiming the key one atomic step, so two simultaneous requests cannot both run.
Payload fingerprint. Reusing a key for a different request is rejected instead of silently returning the wrong response.
Transient answers release the key. A 5xx, 409, 429 or 202 is not a result, so it is never stored and the client's retry gets a real second attempt.
Only the owner releases or completes the key. Each claim stores a random token, and a small Lua script deletes or overwrites the key only if the token still matches. Without it, a request that outlived LOCK_TTL_SECONDS could wipe out a newer retry's claim.
Store, then reply. The result is saved before the response leaves, which closes the gap where a client sees a result that was never recorded.

Two practical notes. The middleware hooks res.json, so every response from an idempotent route, including your error handler's, must go through res.json. A response sent any other way leaves the key held until the lock expires. And on Express 4, async handlers need a wrapper (Express 5 handles this natively), or a thrown error never reaches your error handler and the request hangs.

The charge endpoint, backed by the database

Redis is a fast first line of defence, but it is a cache. Keys expire, and a Redis restart can lose them. For money, the durable guarantee should live in your primary database, where a unique constraint makes a duplicate record impossible:

CREATE TABLE payments (
  id              uuid        PRIMARY KEY,
  account_id      uuid        NOT NULL,
  idempotency_key text        NOT NULL,
  amount_cents    bigint      NOT NULL CHECK (amount_cents > 0),
  status          text        NOT NULL CHECK (status IN ('charging', 'succeeded', 'failed')),
  created_at      timestamptz NOT NULL DEFAULT now(),
  updated_at      timestamptz NOT NULL DEFAULT now(),
  UNIQUE (account_id, idempotency_key)
);

The endpoint inserts first and charges second. The insert is the durable claim on the key. If it conflicts, an earlier attempt already handled this key, and the endpoint either returns that outcome or, if the attempt died mid-charge, resumes it:

import express from 'express';
import { randomUUID } from 'node:crypto';
import { idempotency } from './idempotency';

const app = express(); // Express 5, or wrap async handlers on Express 4
app.use(express.json());
// Auth middleware runs first so that req.user is set.

const LEASE_SECONDS = 30; // longer than your payment provider's timeout

// Charge the provider and record the outcome. Used by first attempts and resumed ones.
async function chargeAndSettle(paymentId: string, amountCents: number, key: string) {
  try {
    // Forward the same key so the provider deduplicates as well.
    await paymentProvider.charge({ amountCents, idempotencyKey: key });
  } catch (err) {
    if (isDefinitiveDecline(err)) {
      // e.g. card declined: retrying cannot help
      await pool.query(
        `UPDATE payments SET status = 'failed', updated_at = now() WHERE id = $1`,
        [paymentId],
      );
      return 'failed';
    }
    throw err; // outcome unknown: leave the row in 'charging' so a retry can resume it
  }
  await pool.query(
    `UPDATE payments SET status = 'succeeded', updated_at = now() WHERE id = $1`,
    [paymentId],
  );
  return 'succeeded';
}

app.post('/api/charge', idempotency, async (req, res) => {
  const { amountCents } = req.body; // integer cents, never floating-point dollars
  if (!Number.isInteger(amountCents) || amountCents <= 0) {
    return res.status(400).json({ error: 'amountCents must be a positive integer' });
  }
  const key = req.header('Idempotency-Key')!;
  const accountId = req.user.id;

  const send = (id: string, status: string, created: boolean) =>
    status === 'failed'
      ? res.status(402).json({ transactionId: id, status })
      : res.status(created ? 201 : 200).json({ transactionId: id, status });

  // 1. Insert the payment, or discover that this key already created one.
  const inserted = await pool.query(
    `INSERT INTO payments (id, account_id, idempotency_key, amount_cents, status)
     VALUES ($1, $2, $3, $4, 'charging')
     ON CONFLICT (account_id, idempotency_key) DO NOTHING
     RETURNING id`,
    [randomUUID(), accountId, key, amountCents],
  );

  if (inserted.rowCount === 1) {
    const paymentId = inserted.rows[0].id;
    const status = await chargeAndSettle(paymentId, amountCents, key);
    return send(paymentId, status, true);
  }

  // 2. The key was used before. Redis may have forgotten it, but the database remembers.
  const { rows } = await pool.query(
    `SELECT id, status, amount_cents FROM payments
      WHERE account_id = $1 AND idempotency_key = $2`,
    [accountId, key],
  );
  const existing = rows[0];

  // pg returns bigint as a string, so convert before comparing.
  if (Number(existing.amount_cents) !== amountCents) {
    return res.status(422).json({ error: 'Idempotency-Key was reused with a different amount' });
  }
  if (existing.status !== 'charging') {
    return send(existing.id, existing.status, false); // already settled: replay the outcome
  }

  // 3. Still 'charging'. Take over only if the previous attempt's lease has expired.
  //    This single UPDATE is the lock: of several simultaneous retries, exactly one matches.
  const claim = await pool.query(
    `UPDATE payments
        SET updated_at = now()
      WHERE id = $1
        AND status = 'charging'
        AND updated_at < now() - make_interval(secs => $2)
      RETURNING id`,
    [existing.id, LEASE_SECONDS],
  );

  if (claim.rowCount === 0) {
    // Another attempt is working on it right now.
    res.setHeader('Retry-After', '5');
    return res.status(409).json({ error: 'Payment is still being processed, retry shortly' });
  }

  const status = await chargeAndSettle(existing.id, amountCents, key);
  return send(existing.id, status, false);
});

app.listen(3000, () => console.log('Server running on port 3000'));

Here pool is your pg connection pool, paymentProvider is your provider's SDK, and isDefinitiveDecline is a small function that recognises your provider's "this can never succeed" errors.

A few details are worth calling out:

A crash after the insert is recoverable. The row stays in charging. Once its lease expires, the next retry takes it over and resumes the charge with the same provider key. Returning the stale row as if it were finished would report success for a payment that never happened.
The UPDATE is the lock. Two retries arriving together both try to take over the row, but Postgres applies the updates one at a time. The first moves updated_at forward and wins. The second no longer matches the lease condition and gets 409. Nothing holds a transaction open during the slow network call to the provider.
The lease bounds the damage of a second crash. If the winner also dies, the row becomes claimable again after LEASE_SECONDS. Even if two attempts did overlap, the shared provider key means the card is charged once.
The stored amount wins. A reused key with a different amount is rejected with 422 instead of silently charging a different sum.
Unknown outcomes are retried; known ones are final. A timeout leaves the row in charging so a retry can resume it. A definitive decline is recorded as failed and replayed.
Reconcile the leftovers. If a client never retries, a row can sit in charging indefinitely. A small background job that checks the provider and settles old rows covers that.

With the middleware, the unique constraint, the lease and the forwarded provider key working together, this endpoint is safe to retry.

Client implementation (TypeScript, Axios)

The client plays a crucial role. The key must be generated once per user intent, not once per request attempt. If each retry gets a fresh key, the server sees unrelated requests and the protection is gone.

import axios from 'axios';

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

// Retry only when the outcome is unknown or the server asked us to wait.
function isRetryable(err: unknown): boolean {
  if (!axios.isAxiosError(err)) return false;
  if (!err.response) return true; // network failure or timeout
  const { status } = err.response;
  return status === 409 || status === 429 || status >= 500;
}

export async function charge(amountCents: number, idempotencyKey: string, maxAttempts = 4) {
  for (let attempt = 1; ; attempt++) {
    try {
      const { data } = await axios.post(
        'https://api.example.com/api/charge',
        { amountCents },
        { headers: { 'Idempotency-Key': idempotencyKey }, timeout: 10_000 },
      );
      return data;
    } catch (err) {
      if (!isRetryable(err) || attempt >= maxAttempts) throw err;

      // Exponential backoff with jitter, so retries don't arrive in waves.
      const delay = Math.min(1000 * 2 ** attempt, 8000);
      await sleep(delay * (0.5 + Math.random() / 2));
    }
  }
}

// Create the key when the user clicks "Pay", not inside the retry loop.
// crypto.randomUUID() is global in browsers and Node 19+.
const idempotencyKey = crypto.randomUUID();
await charge(5000, idempotencyKey);

A few rules make this robust:

One key per intent. Keep it in component state or sessionStorage, so a page refresh or a double click reuses it instead of starting a second payment. If the user changes the order, that is a new intent, so create a new key.
Retry only what is safe to retry. Network errors, timeouts, 5xx, 409 and 429 qualify. A 400, 402 or 422 will fail the same way again, so surface it.
Back off with jitter. Fixed-interval retries from many clients can hammer a recovering server all at once.
Disable the button while a request is pending, but never treat that as the protection. The key is the protection.

Edge cases that bite in production

The in-flight race. Request A checks Redis and finds nothing, then request B does the same, and both process the payment. This happens with double clicks and with a client that retries while the first attempt is still running. The fix is the atomic SET NX claim shown earlier. The second request loses the claim and gets 409.
Same key, different payload. Fingerprint the method, URL and body when the key is first used, and reject mismatches with 422. Otherwise a bug in the client silently returns the response for a different operation. The database layer repeats the check on the amount.
Crash mid-request. The processing marker expires after the lock TTL, so a dead request never blocks its key forever. A row left in charging is taken over by the next retry once its lease expires, and the provider key protects the side effect itself.
Slow requests. The lock TTL must be longer than your slowest request, or a second request can claim the key while the first is still running. The ownership token stops the late request from overwriting the newer claim, and the database lease and provider key stop a duplicate charge. For genuinely long jobs, return 202 Accepted with a status URL instead.
Redis loses the key. After a restart or failover, a retry arrives with no Redis entry. The unique constraint on (account_id, idempotency_key) catches it, and the endpoint replays or resumes the existing payment.
Failed requests. A 4xx is deterministic, so replaying it is correct. Transient answers (5xx, 409, 429, 202) are never stored; they release the key so the retry can run again.
Key scope. Namespace keys by account or API client. A global namespace lets one customer's key collide with another's.
Retention. Keep results long enough to cover realistic retry windows. 24 hours is a common default. Clients must never reuse a key after that.
Side effects outside your database. Emails, webhooks and queue messages are not covered by a row constraint. Write them to an outbox table in the same transaction, and have consumers deduplicate on an event ID.

When to use it

You don't need this on every endpoint, but if retries are possible and the operation has side effects, idempotency is not optional. It is mandatory for:

Payments: never charge twice.
Order creation: no duplicate orders or shipments.
Webhooks: providers retry aggressively, so receivers should deduplicate on the event ID.
Mobile clients: unstable networks make retries routine.
Notifications: one email or push per event, not three.

A real-world reference: Stripe

Stripe's API is the best-known example. It assumes network failures are normal and that clients will retry. In API v1, once an endpoint starts executing, Stripe saves the status code and body of the first request for a given key and returns them on every retry, including 500 errors. It compares the parameters of a retry with the original and rejects a mismatch, accepts keys up to 255 characters, and may prune keys once they are at least 24 hours old.

Two of Stripe's choices differ from the middleware above, and both are defensible. Stripe replays server errors, while this middleware releases the key on 5xx. Stripe also doesn't save a result when validation fails or a concurrent request conflicts, because no endpoint ran. What matters is that you pick one behaviour and document it for your API's clients.

Production checklist

Before shipping an idempotent endpoint, confirm that:

The client generates one key per intent and reuses it on every retry.
The server claims the key atomically before doing any work.
A random ownership token ensures a request can only release or complete a claim it still holds.
Keys are scoped per account and limited in length.
A different payload under the same key is rejected, in Redis and in the database.
A unique constraint in the primary database backs the guarantee for money-critical writes.
Interrupted work can be resumed safely: a lease on in-progress rows and the same key forwarded to the downstream provider.
Transient responses (5xx, 409, 429, 202) are never stored, and the behaviour is documented for API consumers.
Every response from an idempotent route goes through res.json, including error responses.
Locks and stored results have TTLs, and the lock TTL exceeds your slowest request.

Conclusion

Idempotency is what turns an unreliable network into a dependable system. It moves correctness out of the network layer, which you can't trust, and into durable storage, which you can. Clients can then retry as aggressively as they need to without fear of duplicate side effects.

Next time you design a POST endpoint, ask yourself:

"What happens if this is called twice?"

If the answer is "disaster", it's time to add an idempotency key.

Updated: thanks to Raza Hussain and Saksham Jain for catching two bugs in the first version. A retry now resumes a payment left in progress by a crash instead of returning it as done, and the Redis claim now uses an ownership token with a compare-and-delete script.

DEV Community