A request that times out may still have succeeded. That single fact is why wallet and refund systems — I've built both — have to be designed around retries from day one.
Here is the sequence. A client calls POST /refunds. The server debits the merchant, credits the customer, commits — and then the response is lost: a load balancer timeout, a pod restart, a mobile network drop. The client sees an error and does the reasonable thing: it retries. The server, seeing what looks like a new request, refunds again.
Retries are not the bug. Retries are essential; networks fail all the time. The bug is an API that can't tell a retry from a new request.
The pattern
An idempotency key is a unique value the client generates once per operation — not per HTTP attempt — and sends with every attempt:
POST /refunds
Idempotency-Key: 6f1c2a9e-7d3b-4f0a-9a51-2c8e0b7d4e11
Content-Type: application/json
{"orderId": "A-1042", "amount": 1499}
The server promises: for a given key, the operation happens at most once, and every attempt gets the same response. Stripe popularised this design for its public API, and it applies to any endpoint that moves money, sends a message or creates something a user would notice twice.
Reserve first, work second, replay on retry. Explore the interactive diagram
The table
CREATE TABLE idempotency_keys (
key TEXT NOT NULL,
client_id TEXT NOT NULL,
request_hash TEXT NOT NULL, -- SHA-256 of method + path + body
status TEXT NOT NULL, -- IN_PROGRESS | COMPLETED
response_code INT,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (client_id, key)
);
Scope keys per client: two different clients may well generate the same string, and that must not leak one client's response to the other.
The flow, with the edge cases
@PostMapping("/refunds")
public ResponseEntity<RefundResponse> refund(@RequestHeader("Idempotency-Key") String key,
@RequestBody RefundRequest req,
Principal client) {
String hash = sha256(req);
// 1. Reserve the key BEFORE doing any work. The primary key makes this atomic.
int inserted = jdbc.update("""
INSERT INTO idempotency_keys (key, client_id, request_hash, status)
VALUES (?, ?, ?, 'IN_PROGRESS')
ON CONFLICT (client_id, key) DO NOTHING
""", key, client.getName(), hash);
if (inserted == 0) {
StoredKey existing = keys.find(client.getName(), key);
if (!existing.requestHash().equals(hash)) {
return ResponseEntity.unprocessableEntity().build(); // same key, different body
}
if (existing.status() == Status.IN_PROGRESS) {
return ResponseEntity.status(409).build(); // concurrent retry: try again later
}
return existing.toResponse(); // replay the stored result
}
// 2. Do the work and record the result in ONE transaction.
return tx.execute(status -> {
RefundResponse result = refunds.issue(req);
keys.complete(client.getName(), key, 201, result);
return ResponseEntity.status(201).body(result);
});
}
Each branch exists because of a real failure mode:
Reserve before you work. If you check for the key, do the refund, and only then insert the key, two retries arriving 5 ms apart both pass the check. The INSERT ... ON CONFLICT DO NOTHING against the primary key is the lock: exactly one attempt wins the reservation.
Same key, different body. A client bug that reuses a key for a different refund must not get the first refund's response — that would silently hide the second operation. Hash the request and reject mismatches with a 422.
In progress. A retry that arrives while the first attempt is still running gets a 409. The client backs off and retries; by then the first attempt has finished and the retry gets the stored response.
Work and result commit together. If the refund commits but storing the response doesn't, the next retry finds an IN_PROGRESS row forever. Keep the business write and the key update in the same database transaction.
When the work isn't in your database
The single-transaction trick only works when the side effect lives in the same database as the key table. When step 2 calls an external payment provider, you can't roll that back. Two things help:
-
Pass the key downstream. Most payment providers accept their own idempotency key. Derive it from yours (for example
refund:{clientId}:{key}), so a retry after a crash asks the provider for the same operation instead of a new one. -
Recover stuck rows. A process can die between the provider call and the commit. A sweeper that finds rows stuck
IN_PROGRESSfor longer than your longest timeout can query the provider by the derived key and complete or release the row.
How long to keep keys
Keep keys at least as long as any client will retry, plus a margin. Stripe keeps them for 24 hours; for internal services where queues can redeliver messages days later, I'd lean longer. Delete old rows with a scheduled job — not inside the request path.
Where else this applies
The same idea shows up whenever "at least once" delivery meets "exactly once" side effects:
- Queue consumers. Kafka and SQS redeliver. Use the message ID (or a business ID) as the key and record it in the same transaction as the effect.
- Webhooks. Providers retry webhooks; store the event ID before acting on it.
- Scheduled jobs. A job that runs twice after a deploy should find its own key and stop.
Takeaways
- Retries are necessary; duplicate side effects are optional.
- The key belongs to the operation, generated by the client once and reused on every attempt.
- Reserve the key atomically before the work, store the response with the work, and replay it on retry.
- Reject a reused key with a different body, and pass keys to every downstream system that moves money.
Originally published at ayushwing.com.
Top comments (3)
the stuck IN_PROGRESS row is the edge case that actually bites in production, and the 'recover stuck rows' line is the part most teams skip until the 2am page. the single-transaction trick works until the side effect moves to a provider, and deriving the downstream key from yours (refund:{clientId}:{key}) is what keeps the crash-after-commit case from double-charging. one question on the hash check: when you return 422 for a same-key-different-body retry, do you also surface which fields mismatched, or is a bare 422 enough? we found clients mostly resent the wrong payload silently until the error named the diff.
Spot on about the stuck rows - that recovery sweeper is definitely written in incident post-mortem ink!
Regarding the 422: in our setup, returning just a bare 422 led to the exact debugging loop you described. We ended up returning a structured error. We don't store or diff the entire raw JSON body in the idempotency table to keep storage lean, but we compute field-level hashes. If it's still running, a clear error code indicating the key reuse bug usually forces the client to fix their generation logic rather than blindly retrying.
that matches where we landed too: hashing the semantically meaningful subset instead of storing the whole body keeps the table lean and still catches the diff that actually matters, since clients rarely mutate an entire payload, they change one field. the error-code point is the underrated half: a client that reads every 422 as try-again-later is exactly the one that turns a key-reuse bug into a duplicate charge, so naming the reuse in the code does work a doc never will. nice writeup.