"Just retry it" is the most dangerous advice in distributed systems
You click "Pay $500." The request reaches the server… but the response times out on the way back. Did it work? You have no idea. So you click again.
Now you've been charged twice.
This is the problem idempotency solves - and handling it well is one of the clearest signals of a senior engineer. Let's go deep, with code.
What "idempotent" actually means
An operation is idempotent if performing it once or ten times has the same effect.
-
GET,PUT,DELETE→ naturally idempotent -
POSTthat charges a card, creates an order, or sends an email → not idempotent
The moment a network can fail (always) and a client can retry (always), non-idempotent writes become dangerous.
Idempotency keys: the Stripe pattern
The client generates a unique key per logical operation and sends it with the request:
http
POST /charges
Idempotency-Key: a1b2c3d4-....
Server logic, in pseudocode:
record = store.get(key)
if record exists and completed:
return record.response # a retry — don't reprocess
result = process(request)
store.save(key, result)
return result
Simple in theory. The depth is entirely in the edge cases.
Edge case 1 — The concurrent-retry race
Two retries land in the same millisecond. Both run store.get(key) → both see nothing → both call process() → double charge.
A naive "check-then-write" recreates the exact bug you're trying to fix. You need an atomic claim — a unique constraint:
CREATE TABLE idempotency_keys (
key TEXT PRIMARY KEY,
status TEXT NOT NULL, -- 'in_progress' | 'completed'
response JSONB,
created_at TIMESTAMPTZ DEFAULT now()
);
-- Atomically claim the key. Only ONE concurrent request wins.
INSERT INTO idempotency_keys (key, status)
VALUES ($1, 'in_progress')
ON CONFLICT (key) DO NOTHING;
If the insert affects 0 rows, someone else owns the key → you wait/poll for their result or return 409 Conflict. The database's uniqueness guarantee does the hard concurrency work for you.
Edge case 2 — Store the RESPONSE, not just "done"
On a retry you must return the same response the client missed the first time — same charge ID, same status code, same body. Otherwise the client can't reconcile what happened.
So when the first request completes, persist its full response:
UPDATE idempotency_keys
SET status = 'completed', response = $2
WHERE key = $1;
Now any later retry replays that exact response.
Edge case 3 — Partial failures & atomicity
The nastiest case: you charged the card but crashed before saving the result. The key never reaches completed, so a retry charges again.
Two defenses:
Wrap the idempotency record + the side effect in one transaction — only works when the side effect lives in the same database.
For external side effects (charging via a payment processor), record in_progress before the call, and on a retry of an in_progress key, reconcile with the processor (using your key) instead of blindly re-charging.
This is why payment APIs lean heavily on the provider also being idempotent end-to-end.
Edge case 4 — Key scope & expiry
Scope keys per endpoint + account, so the same key string can't collide across users.
Expire them (Stripe uses ~24h) — long enough to cover retries, short enough not to store forever.
Reusing a key with a different payload should error, not silently return the old result.
The mental model: stop chasing "exactly once"
Don't try to guarantee a request runs exactly once. Make it safe to run any number of times.
"Exactly-once delivery" is largely a myth. At-least-once delivery + idempotency = exactly-once effects.
That single reframe is what makes payment, order, and messaging systems reliable — and it's a favourite senior interview probe.
Wrapping up
Idempotency looks like a one-liner ("just store the key") and turns out to be four subtle problems: the race, the response replay, partial-failure atomicity, and key scope/expiry. Nail those and your retries stop being scary.
Disclosure: I work on PrepGrind, a free playground for building & stress-testing systems like this — but the patterns above stand on their own.
How do you handle the concurrent-retry race in your systems — a unique constraint, a distributed lock, or something else? Curious what's worked for others.
Top comments (3)
The concurrent-retry race bit is the one. 'Both ran get() → both saw nothing → both called process()' reads like a horror story 😅
Haha - it's the classic "passes every test, then double-charges someone at 3am". The fix is boring; the haunting part is it's nearly impossible to reproduce on demand. How do you even test for timing races in an AI test framework?
Our approach: run it 100 times with jittered timing and watch what breaks. The ones that break at iteration 97 are the scary ones — you almost stopped watching by then 😅