Idempotency keys done right: exactly-once effects over at-least-once networks

Why "just retry" creates duplicate payments, and a PostgreSQL-backed design for idempotency keys that survives double clicks, timeouts, crashes and concurrent retries.

Key takeaways

• Networks give you at-least-once delivery at best. Exactly-once delivery isn't achievable in general; exactly-once effects are, if the receiver de-duplicates.
• An idempotency key identifies a user intent, not an HTTP attempt. Generate it once, when the user decides, and reuse it for every retry.
• Store the key in the same transaction as the side effect. If the key and the effect can commit separately, you've only moved the race.
• Fingerprint the request. Same key + same body = replay the stored response; same key + different body = reject with an error.
• Handle in-flight duplicates explicitly (return 409 or wait), and make keys recoverable after a crash.
• External calls need phases or an outbox. You can't put a payment gateway inside your database transaction, so design recovery points around it.

I'm Mohammadreza Alamnejad, a programmer and network security professional, UI/UX designer and developer, and the founder of Arka Studio in Tehran. In the café and restaurant operations platform my studio built, NABZA, several behaviours that users experience as "it just works" are idempotency underneath: a completed order becomes a sale once and only once even with a second click, duplicate payments or payments above the outstanding balance are rejected, a stock count can't be posted twice, and an offline order is submitted exactly once after reconnecting.

I mentioned idempotency briefly in a pre-launch checklist I published earlier. This post goes all the way down: the theory, the failure modes, a schema, the request lifecycle, concurrency, crash recovery and external side effects. The code is a reference implementation in TypeScript and PostgreSQL, not NABZA's source.

1. The problem: you can't tell "lost request" from "lost response"

Consider a client calling POST /payments and getting a timeout.

 Client                      Network                     Server
   │  POST /payments ───────────►│                          │
   │                             │ ───────────────────────► │  charge + commit ✔
   │                             │ ◄─────────────────────── │  201 Created
   │                    ✖ connection drops                  │
   │  (timeout)                  │                          │
   │  retry? ─────────────────── ?                          │

From the client's side, three different worlds look identical:

1. The request never reached the server.
2. The server processed it, and the response was lost.
3. The server is still processing it.

This is a practical cousin of the classic Two Generals' Problem: over an unreliable channel, no finite number of messages lets both sides be certain they agree. The consequence for APIs is simple. If the client never retries, you lose operations in world 1. If it always retries, you duplicate operations in world 2. Neither is acceptable for money or stock.

The way out is to make retries safe: at-least-once delivery + de-duplication at the receiver = exactly-once effect.

2. What HTTP already gives you

RFC 9110 defines GET, HEAD, OPTIONS, TRACE, PUT and DELETE as idempotent methods: sending the same request several times has the same intended effect on the server as sending it once. POST and PATCH are not idempotent by definition.

Two notes:

• "Idempotent by method" is a contract you have to implement. A PUT handler that appends to a log on every call breaks it.
• Most business operations ("create a payment", "complete an order", "post a stock count") are naturally POST. You need an explicit mechanism.

That mechanism is the idempotency key: a client-generated unique value sent with the request. The pattern was popularised by public payment APIs (Stripe documents it, and Brandur Leach wrote a well-known public post about implementing it on PostgreSQL), and there's an IETF HTTPAPI working group draft standardising an Idempotency-Key request header field. I'll use that header name below.

3. Rule one: one key per intent

The single most common bug I see: the key is generated inside the retry loop.

// ✖ Wrong: a new key per attempt = no de-duplication at all
async function pay(amount: number) {
  return retry(() => fetch('/payments', {
    method: 'POST',
    headers: { 'Idempotency-Key': crypto.randomUUID() },
    body: JSON.stringify({ amount }),
  }));
}

The key must be created when the user decides, and then reused by every attempt, including attempts after a page reload or an app restart.

// ✔ Right: key belongs to the intent, persisted with it
type PendingIntent = { key: string; body: unknown; createdAt: number };

export function createPaymentIntent(body: unknown): PendingIntent {
  const intent = { key: crypto.randomUUID(), body, createdAt: Date.now() };
  savePending(intent);            // IndexedDB / localStorage / device queue
  return intent;
}

export async function submit(intent: PendingIntent) {
  return retry(
    () => fetch('/payments', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'Idempotency-Key': intent.key },
      body: JSON.stringify(intent.body),
    }),
    { retries: 6, baseMs: 300, maxMs: 10_000 },
  );
}

On the UI side, a disabled button is a nice-to-have, not the guarantee. The guarantee is that two clicks that represent one decision share one key.

Retries with backoff and jitter

Retrying immediately and in lockstep from many clients turns a brief outage into a self-inflicted overload. Use exponential backoff with jitter:

export async function retry<T>(
  fn: () => Promise<Response>,
  { retries, baseMs, maxMs }: { retries: number; baseMs: number; maxMs: number },
): Promise<Response> {
  for (let attempt = 0; ; attempt++) {
    try {
      const res = await fn();
      // Retry only on outcomes that might succeed later.
      if (res.status !== 409 && res.status !== 429 && res.status < 500) return res;
      if (attempt >= retries) return res;
    } catch (e) {
      if (attempt >= retries) throw e;   // network error / timeout
    }
    const cap = Math.min(maxMs, baseMs * 2 ** attempt);
    await sleep(Math.random() * cap);    // "full jitter"
  }
}

Don't retry 400/422: those are deterministic. Do retry network errors, 5xx, 429 (honouring Retry-After) and 409 in-progress (see below).

4. The server side: a key is a small state machine

A key isn't just "seen / not seen". It moves through states:

stateDiagram-v2
    [*] --> started: first request (INSERT)
    started --> completed: effect committed + response stored
    started --> started: crash / timeout (recoverable)
    completed --> [*]: expires after retention window
    note right of started
      concurrent duplicate → 409 (in progress)
    end note
    note right of completed
      duplicate with same fingerprint → replay response
      duplicate with different fingerprint → 422
    end note

Schema

CREATE TABLE idempotency_keys (
  tenant_id        uuid        NOT NULL,
  actor_id         uuid        NOT NULL,       -- keys are scoped to who sent them
  key              text        NOT NULL CHECK (length(key) BETWEEN 8 AND 255),
  method           text        NOT NULL,
  path             text        NOT NULL,
  request_hash     bytea       NOT NULL,       -- sha256 of canonical body
  status           text        NOT NULL DEFAULT 'started'
                     CHECK (status IN ('started', 'completed')),
  recovery_point   text        NOT NULL DEFAULT 'started',
  locked_until     timestamptz,
  response_status  int,
  response_body    jsonb,
  created_at       timestamptz NOT NULL DEFAULT now(),
  updated_at       timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, actor_id, key)
);

CREATE INDEX idempotency_keys_gc ON idempotency_keys (created_at);

Design decisions worth calling out:

• Scope. The primary key includes the tenant and the actor. A key from user A must never replay user B's response. In a multi-tenant system, that's a data leak, not just a bug.
• Method and path. The same key sent to a different endpoint is a client bug; reject it rather than replaying an unrelated response.
• request_hash. A fingerprint of the canonicalised body. Canonicalise (stable key order, normalised numbers) before hashing so semantically identical retries match.
• recovery_point. Where a crashed request should resume. More in section 7.
• locked_until. A lease, so a request that died mid-flight doesn't block the key forever.

5. The happy path, in one transaction

For operations whose side effects are all inside your database (create a sale, post a stock count, record a payment you've already confirmed), the whole thing fits in one transaction. That's the case to aim for.

import { createHash } from 'node:crypto';

const canonical = (v: unknown): string =>
  JSON.stringify(v, (_k, val) =>
    val && typeof val === 'object' && !Array.isArray(val)
      ? Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b)))
      : val);

export async function idempotent<T>(
  ctx: Ctx, req: { key: string; method: string; path: string; body: unknown },
  effect: (tx: Tx) => Promise<{ status: number; body: T }>,
) {
  const hash = createHash('sha256').update(canonical(req.body)).digest();

  return db.tx({ isolationLevel: 'read committed' }, async (tx) => {
    // 1. Try to claim the key. ON CONFLICT DO NOTHING makes this race-safe.
    const claimed = await tx.oneOrNone(
      `INSERT INTO idempotency_keys (tenant_id, actor_id, key, method, path, request_hash, locked_until)
       VALUES ($1, $2, $3, $4, $5, $6, now() + interval '30 seconds')
       ON CONFLICT DO NOTHING
       RETURNING key`,
      [ctx.orgId, ctx.userId, req.key, req.method, req.path, hash],
    );

    if (!claimed) {
      // 2. Someone has this key. Lock the row and decide.
      const row = await tx.one(
        `SELECT * FROM idempotency_keys
          WHERE tenant_id = $1 AND actor_id = $2 AND key = $3
          FOR UPDATE`,
        [ctx.orgId, ctx.userId, req.key],
      );
      if (row.method !== req.method || row.path !== req.path || !row.request_hash.equals(hash)) {
        return { status: 422, body: { error: 'idempotency_key_reused_with_different_request' } };
      }
      if (row.status === 'completed') {
        return { status: row.response_status, body: row.response_body, replayed: true };
      }
      return { status: 409, body: { error: 'request_in_progress' } };
    }

    // 3. We own the key: do the work in the SAME transaction.
    const result = await effect(tx);

    // 4. Store the response; commit makes key + effect visible together.
    await tx.none(
      `UPDATE idempotency_keys
          SET status = 'completed', response_status = $4, response_body = $5,
              locked_until = NULL, updated_at = now()
        WHERE tenant_id = $1 AND actor_id = $2 AND key = $3`,
      [ctx.orgId, ctx.userId, req.key, result.status, result.body],
    );
    return result;
  });
}

What happens in each failure scenario:

Scenario	Outcome
Response lost after commit; client retries	Row is completed, fingerprint matches → stored response replayed
Server crashes before commit	Transaction rolls back: no key, no effect. Retry starts fresh
Two identical requests arrive concurrently	One inserts; the other's INSERT … ON CONFLICT blocks until the first transaction commits, then sees a completed row → replay. (In the phased design of section 7, where a started key is committed before the effect finishes, the duplicate gets 409 request_in_progress instead.)
Same key, different amount	422; the client has a bug and should not be "helped"
Effect throws a business error (e.g. insufficient stock)	Whole transaction rolls back. Decide whether deterministic 4xx outcomes should be stored so retries get the same answer (I usually store them; see below)

Should errors be stored? For deterministic failures (validation, "payment exceeds balance"), storing the response gives retries a consistent answer and prevents a later retry from succeeding after state changed, which would surprise the user. For transient failures (5xx, lock timeouts), don't store; let the retry try again. To store a 4xx, catch the business error inside the transaction, write the response, and commit.

6. Idempotency is not only keys: guard your state transitions

Keys de-duplicate requests. Your domain should also refuse to apply the same transition twice, because duplicates can come from places that don't carry your key: a second device, a background job, an admin tool.

-- Completing an order converts it to a sale exactly once.
UPDATE orders
   SET status = 'completed', completed_at = now()
 WHERE id = $1 AND tenant_id = $2 AND status = 'served'
RETURNING id;
-- 0 rows → already completed (or not ready): don't create the sale.

-- One sale per order, enforced by the database.
CREATE UNIQUE INDEX sales_one_per_order ON sales (tenant_id, order_id);

-- Payments can't exceed the outstanding balance (check inside the tx, with a lock).
SELECT total, paid FROM invoices WHERE id = $1 FOR UPDATE;

Think of these as defence in depth: the key handles retries of one request; conditional updates and unique constraints handle every other path to a duplicate. Natural keys (one sale per order, one count posting per count) are often stronger than any header.

7. External side effects: phases and recovery points

Things get harder when the effect includes a call you can't roll back: a bank gateway, an SMS provider, an email, a webhook to a partner. You can't hold a database transaction open across a network call to someone else's system, and you can't undo an SMS.

The approach: split the request into atomic phases, each its own transaction, and record a recovery point after each phase.

 Phase A (tx)                 External call              Phase B (tx)
 ┌─────────────────────┐      ┌───────────────┐          ┌──────────────────────┐
 │ claim key           │      │ gateway.charge│          │ record charge result │
 │ create payment row  │ ───► │ (idempotent   │ ───────► │ mark invoice paid    │
 │  status=pending     │      │  key = ours)  │          │ store response       │
 │ recovery=created    │      └───────────────┘          │ recovery=finished    │
 └─────────────────────┘                                 └──────────────────────┘

Rules that make this safe:

1. Pass an idempotency key downstream. Many payment and messaging APIs accept their own idempotency key or a client reference. Derive it deterministically from yours (for example payment:${paymentId}), so a resumed request reuses the same downstream key.
2. If the provider has no idempotency support, store enough to ask it ("what happened to reference X?") before trying again. Bank gateways typically have a verify/inquiry call; use it on recovery instead of charging again.
3. On a retry of a started key whose lease expired, resume from recovery_point instead of starting over.

async function resumePayment(ctx: Ctx, row: KeyRow) {
  switch (row.recovery_point) {
    case 'started':
      await phaseCreatePayment(ctx, row);              // tx: payment row + recovery='created'
    // fall through
    case 'created': {
      const p = await loadPayment(ctx, row);
      const r = await gateway.charge({ amount: p.amount, reference: `payment:${p.id}` });
      await phaseRecordResult(ctx, row, p, r);         // tx: result + recovery='finished'
    }
    // fall through
    case 'finished':
      return loadStoredResponse(ctx, row);
  }
}

Outbox for everything downstream

For effects that can happen after the response (notifications, webhooks, analytics, search indexing), use the transactional outbox pattern: write an event row in the same transaction as the business change, and let a separate worker deliver it.

CREATE TABLE outbox (
  id          bigserial   PRIMARY KEY,
  tenant_id   uuid        NOT NULL,
  event_id    uuid        NOT NULL UNIQUE,   -- consumers de-duplicate on this
  type        text        NOT NULL,
  payload     jsonb       NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now(),
  sent_at     timestamptz
);

// Worker: at-least-once delivery, with SKIP LOCKED so several workers can run.
const batch = await tx.any(
  `SELECT * FROM outbox WHERE sent_at IS NULL
    ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED`,
);
for (const e of batch) {
  await deliver(e);                                  // may be retried → include event_id
  await tx.none(`UPDATE outbox SET sent_at = now() WHERE id = $1`, [e.id]);
}

The outbox moves the problem rather than eliminating it: the worker may deliver twice (crash after deliver, before UPDATE). That's fine, because each event carries event_id and the receiver de-duplicates. Which brings us to the other side.

Inbox: idempotent consumers

If you receive webhooks or queue messages, keep an inbox table and process each event_id once:

CREATE TABLE inbox (
  source     text NOT NULL,
  event_id   text NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (source, event_id)
);

await db.tx(async (tx) => {
  const fresh = await tx.oneOrNone(
    `INSERT INTO inbox (source, event_id) VALUES ($1, $2) ON CONFLICT DO NOTHING RETURNING 1`,
    ['gateway', event.id],
  );
  if (!fresh) return;              // already processed
  await applyEvent(tx, event);     // same transaction as the inbox insert
});

Producer outbox + consumer inbox is how you get exactly-once effects across a pipeline built on at-least-once delivery.

8. Offline queues are idempotency in slow motion

An offline-capable client is a client that retries hours later. The same rules apply, plus one more:

• The key is created when the user taps "submit", stored with the queued item, and survives app restarts.
• Items are submitted in order, and each is removed from the queue only after a definitive response (2xx or a stored 4xx).
• The server re-validates business rules at submission time. A cached price on the device is not the truth. In NABZA's case, if a price changed while the order sat in the offline queue, the order is held for review rather than silently repriced. The key de-duplicates; the re-validation keeps the result correct.

9. Operational details that bite later

Retention. Keys can't live forever, and they can't expire too early. The retention window must be longer than the longest realistic retry horizon, including offline queues. Stripe, for example, documents that keys may be pruned once they are at least 24 hours old; offline POS clients may need a longer window. Clean up with a periodic job on the created_at index.

Key format. Accept opaque strings with length limits; recommend UUIDs (v4, or v7 for time-ordered locality). Never derive keys from guessable data like orderId + amount on the client side if a collision could merge two legitimate intents.

Response size. Store what you need to replay. If responses are large, store a reference (the created resource ID) and re-render it, accepting that the replay reflects current state.

Observability. Count replays, 409s and 422s per endpoint. A spike in 422 means a client generates keys incorrectly; a spike in replays may mean a network or timeout problem upstream.

Security. Treat keys as scoped identifiers, not secrets, but scope them to tenant and actor so they can't be used to fetch someone else's stored response. Don't store secrets or full card data in response_body.

10. Trade-offs and alternatives

Approach	Strength	Weakness
Idempotency key table (this post)	Works for any POST; replays exact responses	Extra write per request; retention to manage
Natural unique constraints	Simple, enforced by the DB, covers all paths	Only where a natural key exists
Conditional state transitions	Protects the domain regardless of entry point	Doesn't replay the original response
Client-chosen resource IDs + PUT	Idempotent by method semantics	Exposes ID generation to clients; not every operation is a "put"
Message de-duplication in a broker	Offloads work	Usually bounded windows; doesn't cover your DB writes

In practice you combine them: keys at the API edge, natural constraints and guarded transitions in the domain, outbox/inbox between services.

11. A test suite for idempotency

it('replays the original response for a duplicate', async () => {
  const key = crypto.randomUUID();
  const a = await post('/payments', { invoiceId, amount: 100_000 }, key);
  const b = await post('/payments', { invoiceId, amount: 100_000 }, key);
  expect(b.status).toBe(a.status);
  expect(b.body).toEqual(a.body);
  expect(await countPayments(invoiceId)).toBe(1);
});

it('rejects the same key with a different body', async () => {
  const key = crypto.randomUUID();
  await post('/payments', { invoiceId, amount: 100_000 }, key);
  const b = await post('/payments', { invoiceId, amount: 200_000 }, key);
  expect(b.status).toBe(422);
});

it('handles concurrent duplicates', async () => {
  const key = crypto.randomUUID();
  const results = await Promise.all(
    Array.from({ length: 10 }, () => post('/payments', { invoiceId, amount: 100_000 }, key)),
  );
  expect(await countPayments(invoiceId)).toBe(1);
  expect(results.every((r) => [201, 409].includes(r.status))).toBe(true);
});

it('does not replay another user\'s response', async () => {
  const key = crypto.randomUUID();
  await post('/payments', { invoiceId, amount: 100_000 }, key, userA);
  const b = await post('/payments', { invoiceId, amount: 100_000 }, key, userB);
  // The handler sets `Idempotent-Replayed: true` on replays; user B must never get one.
  expect(b.headers['idempotent-replayed']).toBeUndefined();
});

Add a chaos test: kill the server process between the effect and the response (or inject a fault after commit), then retry and assert a single effect.

Closing

Exactly-once isn't a property of the network; it's a property of your system's bookkeeping. Generate one key per intent, claim it in the same transaction as the effect, fingerprint the request, guard your transitions, and treat every external call as something that may need to be resumed or asked about. The reward is a product where a nervous double tap, a dropped connection or a crashed server never turns into a second charge.

Further reading (public): RFC 9110 (HTTP Semantics, section on idempotent methods); the IETF HTTPAPI draft "The Idempotency-Key HTTP Header Field"; Stripe's public API documentation on idempotent requests; Brandur Leach's public article "Implementing Stripe-like Idempotency Keys in Postgres"; and descriptions of the transactional outbox pattern, for example on microservices.io.

---

About the author

Mohammadreza Alamnejad is a programmer and network security professional, a UI/UX designer and developer, and the founder of Arka Studio, a design and development studio in Tehran. He built NABZA, a café and restaurant operations SaaS, along with Avenya, Natureio and Radepa. He studied Computer and IT at Sharif University of Technology and Iran University of Science and Technology. More on his founder page at Arka Studio.