DEV Community

Cover image for Idempotency Keys: Where to Put Them, How Long to Keep Them
Gabriel Anhaia
Gabriel Anhaia

Posted on

Idempotency Keys: Where to Put Them, How Long to Keep Them

#database #postgres #architecture #tutorial

You charged the same customer twice because your retry didn't carry the idempotency key. The first POST to /charges timed out at 28 seconds. Your HTTP client retried with a fresh request ID, the gateway accepted it, and now there are two ch_ rows in your ledger for the same cart. The customer sees $79.98. Support sees a refund ticket. You see a Slack ping at 11:47 PM.

The fix is one of the oldest tricks in distributed systems: an idempotency key. The part that goes wrong, the part that gets the postmortem, is almost never the concept. It is where you put the key, how long you keep it, and what happens when two requests with the same key race each other through your stack.

The failure you're really preventing

There are three retry sources in any production path. Network blips between client and edge. Internal retries from your gateway, queue, or Kafka consumer. Application-level retries in code (urllib3.Retry, resilience4j, the retry decorator someone added in 2023). Each one can fire the same operation twice. None of them coordinate.

An idempotency key is a single value the client picks once per logical operation, then carries across every retry of that operation. The server uses it to decide: have I seen this before? If yes, return the original result. If no, do the work and remember the answer.

Two failure modes you must design against:

  • Same key, different payload. The client retried with the same key but the body changed (price recalculated, currency flipped). You must reject this, not silently use the cached response.
  • Concurrent first attempts. Two requests with the same key arrive within milliseconds. Only one should execute. The other waits or gets a 409.

Hold those two in mind. They decide every implementation choice below.

Three placements, three failure modes

There are three places the key can live. Real systems use all three, layered.

HTTP header

The IETF Idempotency-Key header draft is the placement you should reach for first. Stripe, Adyen, and most payment APIs use it. The header lives outside the business payload, so middleware can read it without parsing JSON.

POST /v1/charges HTTP/1.1
Idempotency-Key: 6f1c8d6a-3a09-4b6e-9c8f-2d1f5e7b8a90
Content-Type: application/json

{"amount": 7998, "currency": "usd", "customer": "cus_123"}
Enter fullscreen mode Exit fullscreen mode

The server hashes the request body, stores (key, body_hash, response), and on a repeat request compares hashes before replaying.

Failure mode. Clients reuse keys across logical operations. A mobile SDK that generates the key once at app start and reuses it for every checkout will collide on the second purchase. Document the rule: one key per operation, not per session. Reject mismatched-body retries with HTTP 422 and a clear error code, not 200.

Business-payload field

Sometimes the key is part of the domain. A bank transfer carries an external_reference. A webhook delivery carries a delivery_id. An order import carries the customer's PO number. These are idempotency keys whether you call them that or not.

{
  "transfer": {
    "amount_cents": 250000,
    "destination_iban": "DE89370400440532013000",
    "client_reference": "po-2026-04-27-3914"
  }
}
Enter fullscreen mode Exit fullscreen mode

Putting the key in the payload survives proxy stripping, lets you index it in your domain database, and shows up in customer support tools without a separate lookup.

Failure mode. Mixing identity and intent. If client_reference is also a human-meaningful field (the PO number printed on the invoice), customers will reuse it. You need a separate request_id for retry semantics and keep the domain reference for display. The day someone copy-pastes a PO into a second order, your idempotency layer rejects a legitimate request.

Database unique constraint

The last line of defence sits in the database. Even if the header check passes and the payload check passes, two concurrent inserts will both think they are the first. A unique constraint on the key column turns the second insert into a deterministic error you can catch.

CREATE TABLE charges (
  id            BIGSERIAL PRIMARY KEY,
  idem_key      TEXT NOT NULL,
  request_hash  BYTEA NOT NULL,
  response      JSONB,
  status        TEXT NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at    TIMESTAMPTZ NOT NULL
);

CREATE UNIQUE INDEX charges_idem_key_uniq
  ON charges (idem_key);
Enter fullscreen mode Exit fullscreen mode

The shape of a safe write path:

BEGIN;

INSERT INTO charges (idem_key, request_hash, status, expires_at)
VALUES ($1, $2, 'in_progress', now() + interval '24 hours')
ON CONFLICT (idem_key) DO NOTHING
RETURNING id;
Enter fullscreen mode Exit fullscreen mode

If the insert returns a row, you own the operation. Do the work, then UPDATE the row with status succeeded and the response. If RETURNING is empty, another worker owns it; read the existing row and either replay the response (if succeeded) or return 409 (if in_progress).

Failure mode. Long-running operations leave in_progress rows that block legitimate retries forever. You need a sweep job that resets stale rows whose created_at is older than your operation's worst-case duration plus a safety margin.

TTL by use case

The retention window is not one number. It depends on how long a sane retry could take and how badly a stale replay would hurt.

Use case TTL Why
Payment authorization 24h Matches Stripe's window; covers overnight retry queues
User signup 1h Form submissions retry within minutes; longer hides real conflicts
Replayable webhook 7d Receivers may pause, debug, redeploy, then catch up
Async job submission 30d Workflow engines retry across deploys
Read-heavy GET cache 0 (don't) GETs are already idempotent; you want HTTP caching, not this

Two numbers worth defending. Payment 24 hours: anything shorter and a customer's overnight retry from a stuck mobile app will double-charge. Anything longer and you're storing PCI-adjacent data past its useful life. Webhook 7 days: receivers go down on Friday, get fixed Monday, replay the queue. A 24-hour TTL would let the same delivery_id execute twice.

Postgres and Redis sketches

Postgres table with scoped TTL 

CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  scope         TEXT NOT NULL,
  request_hash  BYTEA NOT NULL,
  response      JSONB,
  status        TEXT NOT NULL CHECK (
    status IN ('in_progress', 'succeeded', 'failed')
  ),
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at    TIMESTAMPTZ NOT NULL
);

CREATE INDEX idem_expires_idx
  ON idempotency_keys (expires_at)
  WHERE status <> 'in_progress';
Enter fullscreen mode Exit fullscreen mode

The scope column lets one table serve payment, signup, and webhook paths with different TTLs. The partial index keeps the sweep cheap; only completed rows are candidates for deletion.

The sweep itself runs every 15 minutes:

DELETE FROM idempotency_keys
WHERE expires_at < now()
  AND status <> 'in_progress'
  AND ctid IN (
    SELECT ctid FROM idempotency_keys
    WHERE expires_at < now()
      AND status <> 'in_progress'
    LIMIT 5000
  );
Enter fullscreen mode Exit fullscreen mode

The LIMIT and self-select keep the delete batched. Without it, a sweep on a busy table can lock rows your hot path needs.

A separate stale-in_progress reaper runs hourly with a longer fuse (operation max duration plus 10 minutes) and either resets the row to allow retry or moves it to failed so the client gets a deterministic error.

Redis filter in front

When you need sub-millisecond first-write detection in front of Postgres, Redis is the right second layer. It sits in front of Postgres as a filter, never in place of it.

import hashlib
import json
import redis

r = redis.Redis()

def claim(key: str, scope: str, body: dict, ttl: int) -> str:
    body_hash = hashlib.sha256(
        json.dumps(body, sort_keys=True).encode()
    ).hexdigest()
    cache_key = f"idem:{scope}:{key}"
    # SET NX returns True only on first write.
    won = r.set(
        cache_key,
        json.dumps({"hash": body_hash, "status": "in_progress"}),
        nx=True,
        ex=ttl,
    )
    if won:
        return "owner"
    existing = json.loads(r.get(cache_key))
    if existing["hash"] != body_hash:
        return "conflict"
    return "replay"
Enter fullscreen mode Exit fullscreen mode

SET NX EX is the whole trick. One round trip, atomic, with TTL baked in. The caller branches on the return: owner runs the operation and writes the response back into the same key with the same TTL; replay reads the cached response; conflict returns 422.

Failure mode. Redis is not durable enough to be the only layer for money. If your Redis primary fails over before the response is written, the second request becomes a new owner and double-executes. Keep the Postgres unique constraint as backstop. Redis filters cheap collisions; Postgres catches the expensive ones.

The full write path runs three layers in order. Header plus Redis SET NX catches sub-millisecond reuse before the request reaches your database. Postgres INSERT ... ON CONFLICT DO NOTHING is the authoritative owner that survives a Redis failover. A sweep job enforces TTL by scope, with a stale-in_progress reaper on a longer fuse to free rows the application crashed on. Most postmortems trace back to one missing layer.

The customer charged twice at 11:47 PM did not need a smarter retry policy. They needed the idempotency key to cross the proxy boundary so both attempts carried it, and they needed the Postgres constraint to be there when Redis was still propagating. Wire all three layers with a TTL the operation deserves, and the second-charge ticket stops landing in your queue.

If this was useful

The System Design Pocket Guide: Fundamentals covers the building blocks that sit underneath patterns like this one: durable writes, retry semantics, cache layering, and the failure modes you only meet at 2 AM. If you're sketching the same idempotency layer for the third time in two years, it's a useful shelf reference.

System Design Pocket Guide: Fundamentals

Top comments (0)