A client sends a transfer request. The server debits the sender, credits the receiver, commits, and starts writing the response. The connection drops half a millisecond before the client reads it. The client, seeing no response, does the only reasonable thing a retry policy knows how to do: sends the same request again.
The server has no idea any of that happened. It just sees a second, perfectly well-formed request to move money. So it does it again.
Client Server
| |
|----- POST /transfers -------->|
| | debit sender
| | credit receiver
| | commit
| X (connection drops) |
| |
| (timeout, no response seen) |
| |
|----- POST /transfers -------->| <-- identical request
| | debit sender (again)
| | credit receiver (again)
| | commit
|<---------- 200 OK -------------|
That is the whole problem. Not a rare one either; it's the default behavior of almost every HTTP client library the moment you add a timeout and a retry. The client did nothing wrong. It genuinely cannot tell the difference between "the server never got my request" and "the server got it, handled it, and I just never heard back."
Why "just check the balance first" doesn't work
The instinct is to guard the transfer with a check: does the sender have enough balance, is the receiver valid, and so on. None of that helps here. The retried request is not invalid. It's a completely legitimate transfer request that happens to be a duplicate of one that already succeeded. Every validation check passes, twice.
You also can't fix this by making the operation "fast enough" that retries become rare. Networks time out for reasons that have nothing to do with your server's speed: a client-side timeout fires, a proxy in the middle drops the connection, a mobile client loses signal for two seconds. The request already fully executed on your server. The client just never found out.
The fix is a key, not a check
An idempotency key is a value the client generates once per logical operation, before it ever calls the endpoint, and sends on every attempt of that same operation, retries included.
const idempotencyKey = crypto.randomUUID();
await fetch('/api/transfers', {
method: 'POST',
headers: { 'Idempotency-Key': idempotencyKey },
body: JSON.stringify({ fromAccount, toAccount, amount }),
});
If the request fails or times out, the client retries with the same key. Not a new one. The key represents "this transfer," not "this HTTP request."
Client Server
| |
| key = "6f2a-...-91" |
|----- POST /transfers -------->|
| Idempotency-Key: 6f2a | key seen? no
| | debit sender
| | credit receiver
| | store result under 6f2a
| X (connection drops) |
| |
|----- POST /transfers -------->| <-- same key, same body
| Idempotency-Key: 6f2a | key seen? yes
| | skip transfer logic entirely
|<---------- 200 OK -------------| return the stored result
On the server, before doing anything with money, check whether that key has been seen before:
CREATE TABLE idempotency_keys (
key text PRIMARY KEY,
request_hash text NOT NULL,
response_status int,
response_body jsonb,
created_at timestamptz DEFAULT now()
);
async function handleTransfer(key: string, payload: TransferRequest) {
const existing = await db.query(
'SELECT * FROM idempotency_keys WHERE key = $1',
[key]
);
if (existing.rows.length > 0) {
return existing.rows[0].response_body;
}
const result = await executeTransfer(payload);
await db.query(
`INSERT INTO idempotency_keys (key, request_hash, response_status, response_body)
VALUES ($1, $2, $3, $4)`,
[key, hashPayload(payload), 200, result]
);
return result;
}
Second attempt with the same key finds the stored row and returns the original result. It never touches the transfer logic again. The money moves exactly once, no matter how many times the network makes the client ask.
Same key, different payload, is a bug, not a replay
There's a failure mode this simple version misses. What if the same key shows up with a different amount, because of a bug on the client, or someone reusing a key by accident? A naive lookup would happily return the stored response from a completely different transfer and call it correct.
That's what request_hash in the table is for. On the second request, before trusting the cached response, compare the hash of the incoming payload against what was stored the first time:
const incomingHash = hashPayload(payload);
if (existing.rows.length > 0) {
if (existing.rows[0].request_hash !== incomingHash) {
throw new ConflictError('Idempotency key reused with a different request');
}
return existing.rows[0].response_body;
}
A mismatch here should be a hard error, not a silent overwrite in either direction. It means either the client has a bug generating keys, or something worse. Either way, replaying a different transfer under an old key is exactly the kind of thing idempotency exists to prevent, so the check has to be as strict as the thing it's protecting.
Three states, not two
The lookup above only really distinguishes "never seen" from "seen and done." There's a third state hiding in between: seen, but still running. It shows up the moment two attempts overlap in time instead of arriving one after another.
INSERT reserves the key
|
v
+---------------------------+
| PENDING | response_body IS NULL
| (transfer in progress) |
+---------------------------+
|
executeTransfer() finishes
|
v
+---------------------------+
| COMPLETED | response_body populated
| (safe to replay freely) |
+---------------------------+
A different payload hash arriving
against an existing key, in either
state above, is neither of these.
It is a CONFLICT, and should be
rejected outright.
A lookup that only checks "does a row exist" can't tell a PENDING row from a COMPLETED one. Reading a PENDING row's response_body gets you NULL, not "please wait." The read path has to check which state it landed in and behave differently for each.
The race no one thinks about until it bites
Everything above assumes the two attempts arrive one after another. They don't have to. A client with an aggressive retry policy can fire the retry before the first attempt has finished, especially if the timeout it's reacting to was on the client side, not because the server was actually slow. Now two requests with the same key are running concurrently, and both can pass the "have I seen this key" check before either has written a row.
Time Request A Request B
---- -------------------------- --------------------------
t0 check key "6f2a" -> none
t1 check key "6f2a" -> none
t2 begin transfer
t3 begin transfer <-- !!
t4 debit sender
t5 debit sender <-- both proceed
t6 credit receiver
t7 credit receiver
t8 INSERT key "6f2a" -> ok
t9 INSERT key "6f2a" -> UNIQUE VIOLATION
Both proceed to execute the transfer. The unique constraint on key will stop the second INSERT from succeeding, but by then the second request may have already moved money, because the transfer logic and the idempotency check aren't happening atomically. The constraint catches the write collision after the damage, not before it.
The fix is to reserve the key before doing the work, not after:
async function handleTransfer(key: string, payload: TransferRequest) {
const incomingHash = hashPayload(payload);
try {
await db.query(
`INSERT INTO idempotency_keys (key, request_hash, response_status, response_body)
VALUES ($1, $2, NULL, NULL)`,
[key, incomingHash]
);
} catch (err) {
if (isUniqueViolation(err)) {
return await waitForCompletedResult(key, incomingHash);
}
throw err;
}
const result = await executeTransfer(payload);
await db.query(
`UPDATE idempotency_keys
SET response_status = 200, response_body = $2
WHERE key = $1`,
[key, result]
);
return result;
}
Re-run the same race with the reservation moved first:
Time Request A Request B
---- -------------------------- --------------------------
t0 INSERT key "6f2a" -> ok
t1 INSERT key "6f2a" -> UNIQUE VIOLATION
t2 begin transfer
t3 wait for row to complete
t4 debit sender
t5 credit receiver
t6 UPDATE key "6f2a" -> done
t7 read completed row -> return result
Only one request ever touches the transfer logic. The other one blocks on the database's own uniqueness guarantee, which is enforced by Postgres itself, not by application code hoping nothing runs in between two steps. Whether "wait" means polling the row every hundred milliseconds or something fancier depends on how much concurrency you're actually expecting. For a personal project, polling is a perfectly honest starting point.
Idempotency is not a transaction
Worth being precise about this, because the two get blurred together.
+-----------------------------------------------------+
| One HTTP request |
| |
| +-----------------------------------------------+ |
| | Database transaction (ACID) | |
| | | |
| | debit sender --> credit receiver --> ok | |
| | (both happen, or neither does) | |
| +-----------------------------------------------+ |
| |
+-----------------------------------------------------+
^
|
Idempotency key wraps the OUTSIDE of this box,
making it safe to ask for the whole box again.
A database transaction makes a single operation atomic: the debit and the credit inside executeTransfer either both happen or neither does. Idempotency is a different guarantee, layered on top: it makes repeating the same request from the outside safe, regardless of how many times it's repeated. You need both. The transaction protects the money during one execution. The idempotency key protects the money across retries of that execution.
Keys don't live forever
One thing I glossed over: the idempotency_keys table grows with every request, forever, if nothing ever removes old rows. Most systems only need a key to stay valid for as long as a client might plausibly still be retrying an hour, maybe twenty-four, not indefinitely.
DELETE FROM idempotency_keys
WHERE created_at < now() - interval '24 hours';
Run on a schedule, this keeps the table from becoming a second, ever-growing ledger sitting next to the real one. It also means a key can safely be reused after that window closes, which matters if client code ever generates keys from something less random than a UUID, a timestamp plus a user ID, say.
What I check now
Before I add a new "money moving" endpoint anywhere, I check:
- Does the client generate the key before the first attempt, and reuse it on every retry, never a fresh key per retry?
- Is the key reserved with a unique constraint before the operation runs, not just checked and then trusted?
- Does a key collision with a different payload hash fail loudly, instead of silently returning the wrong stored result?
- Does the read path distinguish pending from completed, instead of treating "row exists" as one single state?
- Am I storing the actual response, so a retry gets back the same answer the original call would have given, not just a generic "already done"?
- Do old keys expire, so the table isn't a permanent second ledger?
The transfer logic was never the hard part. Making it safe to ask twice was.
Top comments (0)