Idempotency Keys: Designing APIs That Survive Retries

Every API that sits behind an unreliable network eventually faces the same problem: a client sends a request, the connection drops before the response arrives, and the client has no idea whether the operation happened. Did the payment go through? Did the order get created twice? The client's only safe move is to retry — which means your server needs a story for what happens when the same "create this thing" request arrives more than once.

That story is idempotency keys, and getting the details right is more subtle than it first looks.

The core idea

The client generates a unique token — typically a UUID — once per logical operation, and attaches it to every retry of that operation:

POST /orders
Idempotency-Key: 7c3fd9a2-df01-4b3e-9a55-1e5f9b6b6d55

{"sku": "WIDGET-1", "qty": 2}

The server's job is to guarantee that no matter how many times a request with that key arrives, the side effect (charging a card, creating an order, sending an email) happens at most once, and every retry gets back the same response the original request would have produced.

Note what this is not: it is not deduplicating by request body. Two requests with identical bodies but no key are legitimately two different orders for two widgets. The key is what marks them as "the same attempt," not the payload.

The naive approach, and why it breaks

A common first pass is a table like:

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  response_body JSONB,
  status_code INT
);

On each request: check if the key exists, and if so return the cached response; otherwise do the work and insert the result. This looks right and is wrong in a specific way: it has a race condition. Two retries can arrive concurrently (a client that timed out and fired a second attempt while the first was still in flight), both miss the cache check, and both execute the underlying operation. You've now charged the card twice.

Making the check-and-do atomic

The fix is to claim the key before doing the work, using the database's own concurrency control rather than an application-level check:

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  status TEXT NOT NULL DEFAULT 'in_progress',
  response_body JSONB,
  status_code INT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

INSERT INTO idempotency_keys (key, status)
VALUES ($1, 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;

If the INSERT returns a row, you won this key — proceed with the operation, then update the row with the real response and flip status to 'completed'. If it returns nothing, someone else already claimed this key. Now you have three sub-cases to handle explicitly:

1. status = 'completed' — return the stored response verbatim. This is the retry-after-success path, and it's the one people design for.
2. status = 'in_progress' — another request with the same key is still executing right now, most likely a genuinely concurrent retry (client-side timeout that fired a duplicate before the first attempt returned). The correct response here is usually 409 Conflict with a "retry shortly" hint, not silently blocking, because blocking ties up a connection for as long as the original request takes and can cascade under load.
3. status = 'failed' — the original attempt errored out before completing. Whether this is safe to retry depends on whether the failure happened before or after the side effect committed, which is exactly why the next section matters.

Ordering the side effect and the key update

The dangerous window is between "the side effect happened" and "the key record says it happened." If your payment provider charges the card and then your process crashes before writing status = 'completed', the key is stuck at in_progress (or failed, if you have a crash handler) forever, and a legitimate retry will either be rejected or — worse, if you designed the failed-state to allow retry — will charge the card again.

Two practical ways out:

• Same transaction, when possible. If the side effect is itself a database write (create an order row), do it in the same transaction as the key update. Either both commit or neither does, and there's no window at all.
• External side effect, idempotent downstream. If the side effect is a call to a third party (a payment processor), pass that call an idempotency key too — most payment APIs (Stripe, Braintree, Razorpay) support this natively. Then your recovery path for a crashed in_progress row is: re-issue the downstream call with the same downstream key. If it already happened, the processor returns the original result instead of double-charging. Your own key table becomes a cache in front of an idempotent downstream operation, not the sole source of truth.

Key scope and expiry

Keys should be scoped per-endpoint or per-operation-type, not global — a key valid for POST /orders colliding with an unrelated key namespace for POST /refunds is a bug waiting to happen. Prefix keys by route, or use a composite primary key of (route, key).

Expiry matters too. Clients reuse UUID generation logic, and a key space that never expires grows forever and risks accidental key reuse across unrelated operations months apart. Twenty-four hours is a common TTL: long enough to cover any realistic retry window (including a client that retries after being offline overnight), short enough to bound table growth. Expire with a background job or a partial index plus periodic delete, not by checking created_at on every read — that still requires deciding what "expired" means for a key someone is retrying right at the boundary, so most implementations simply reject requests bearing an expired key and let the client mint a new one, which is equivalent to treating it as a fresh operation.

Handling key reuse with a different payload

What if a client sends the same idempotency key twice, but with a different request body the second time? This usually indicates a client bug — reusing a key across two logically different operations — and the safe response is to reject it (422 with an explanit error) rather than either silently applying the new payload or silently returning the old response. To detect it cheaply, store a hash of the normalized request body alongside the key and compare on the second arrival; a mismatch is the signal.

Testing it

The property you actually want to verify is: N concurrent requests with the same key produce exactly one side effect and N identical successful responses (or, for the couple of concurrent duplicates, one success and N-1 409s that a client is expected to retry). Write this as an actual concurrency test — fire the same key from multiple threads or async tasks at a test server backed by a real database, not a mock, and assert on the row count in the underlying table. The race condition in the naive approach above is invisible in a single-threaded test and only shows up under real concurrency, which is exactly the condition it exists to handle.

Comments

[Mads Hansen]

This is a strong treatment of the dangerous “side effect happened, receipt did not” window.

Two production details I would add:

1. Bind the key to more than the body hash: authenticated principal/tenant, operation or route version, normalized parameters, and any authority-relevant headers. A cached response must still pass the current caller's authorization check; otherwise key guessing or accidental cross-tenant reuse can become a data leak.

2. Treat stale in_progress rows as leased work, not automatically failed work. Store an owner token, lease expiry, attempt number, and downstream operation ID. A recovery worker can acquire the expired lease with compare-and-swap, reconcile the downstream provider using the same key, and only then finalize. The old worker must be fenced from writing after ownership changes.

Expiry also deserves a distinction between deleting the full cached response and forgetting that the key ever existed. A very late retry after hard deletion can recreate the side effect. Keeping a longer-lived compact tombstone or operation receipt than the response payload can preserve duplicate protection while bounding sensitive-data retention.

The crash test matrix is the real proof: kill the process before claim, after claim, before/after local commit, before/after the downstream request, after downstream success but before receipt, during response serialization, and while two recoverers race. Assert one logical effect and a reconstructable terminal or explicitly indeterminate outcome.