Your network will time out mid-request and your client will retry. An idempotency key turns the retry into a replay of the answer you already computed.
The customer clicked Pay once. Their card was charged twice. The receipt, of course, never arrived. Support refunded the duplicate three days later, after the customer had already told eleven people on the internet that your company steals money. The engineering postmortem found nothing wrong with the payment code. The bug was in the thirty lines that retry a timed-out request, and those thirty lines were written by someone who has never watched a timeout lie to them.
That customer is hypothetical. The postmortem is not. I have watched this exact bug ship.
A timeout is the network's way of saying "I do not know." The request may have died before the server saw it. It may have died after the server charged the card. It may have died while the server was charging the card. The client cannot tell the difference, because the one party that knows the answer is the party that stopped talking. This is why exactly-once delivery is a lie: it would require the sender to know what the receiver did, and the receiver's report can always be the thing that gets lost.
Right now you have two ways to live with this, and both are bad.
Option one: never retry. Every timeout becomes a failed payment, a failed order, a failed signup. Your on-call learns the shape of every transient network blip in your cloud region, because each one pages a human.
Option two: retry blindly. Every timeout becomes a second attempt, and some of those attempts land on work the server already did. Your on-call learns the shape of every transient network blip from your customers instead, which is worse.
Here is the gap every retry guide skips. They tell you to add retries with exponential backoff and jitter, and they tell you to "use idempotency keys," as if the key were a header you sprinkle on the problem. Almost none of them show you what the key is actually doing. It is not a header. It is a contract with four moving parts, and every double-charge story I have ever heard is a story about one of the parts being skipped.
Mint One Key Per Intent, Not Per Attempt
The first rule is the one most implementations get wrong, so let me put it first: the key identifies the operation, not the request.
When the customer clicks Pay, the client mints one UUID. That UUID travels with every attempt: the first try, the retry after the timeout, the retry after the retry. If your retry loop generates a fresh key per attempt, you have built an elaborate no-op. The server sees three different keys, concludes it has never seen any of them, and charges the card three times. This is the most common idempotency bug in the wild, and it lives in client code, not server code.
A UUID v4 per checkout click is the standard choice. Stripe allows up to 255 characters and scopes the key to the API key that sent it. Your scheme can be simpler, but the property is non-negotiable: same intent, same key, across retries, across processes, across the load balancer. If two retries can carry different keys, you have nothing.
The Ledger Does the Work, Not the Header
Here is the mechanism, precisely. The server keeps a ledger: for each key, the fingerprint of the request, the status it returned, and the body it returned. When a request arrives with a key the server has seen, the server replays the stored answer and skips the work. That is the entire trick. The header is just how the client names the row.
The part people skip is how the ledger decides "I have seen this key." Looking the key up and then inserting it is two steps, and two steps race. Two requests with the same key can arrive at two app instances in the same millisecond, both check, both find nothing, both charge. The fix is to make the decision atomic: put a unique constraint on the key column and let the database arbitrate. The insert either wins or conflicts. The loser does not do the work; it reads the row the winner is writing and replays it. Every serious writeup on this topic lands on the same sentence: the unique index is the real guarantee. The cache, the Redis, the clever in-memory map are optimizations. The constraint is the mechanism.
The fingerprint matters too. A client can reuse a key with a different request body, by bug or by malice, and then the replay answers a question nobody asked. Stripe compares the incoming parameters against the original request and errors when they differ. The IETF draft for the Idempotency-Key header says the server should answer 422 Unprocessable Content. Whatever you choose, choose it on purpose: silently replaying is the one answer you must not give.
And when the duplicate arrives while the first request is still in flight, the ledger has a row but no answer yet. Stripe returns 409 and tells the client the original is still running. Your version can wait and poll instead, but it must not start the work a second time.
Stripe's contract is the reference implementation of all of this: results kept for at least 24 hours, the original status and body replayed byte for byte, even when the original was a 5xx, and an Idempotent-Replayed header on the replayed response so you can see the mechanism working. The IETF draft reached revision 07 in October 2025 and has since expired, which makes it a description of established practice rather than a standard, but the shape is the same everywhere because the problem is the same everywhere.
Watch the Race Happen
Picture two app instances behind a load balancer, one Postgres, and a client with a three-second timeout. The first request lands on instance A. The card charge takes four seconds, because your payment provider is having a day. At three seconds the client gives up and retries. The retry lands on instance B.
Client --(key K, attempt 1)--> LB --> Instance A --INSERT K--> Postgres (wins)
|
Client --(key K, attempt 2)--> LB --> Instance B --INSERT K--> Postgres (conflicts)
|
Instance B reads K's row, waits,
replays A's stored response
Attempt 1's insert wins the unique constraint. Attempt 2's insert conflicts, so instance B does not touch the payment provider. It reads the ledger row, finds the completed charge, and returns the same 200 with the same charge ID. The customer clicked once, retried once, and was charged once. The timeout lied, and it did not matter.
Build the Ledger in an Afternoon
Here is a complete version in Flask and sqlite. It is deliberately small: one table, one endpoint, and the unique constraint doing the arbitration. The charge itself is simulated; the ledger is real.
import hashlib
import json
import sqlite3
import uuid
from flask import Flask, request, jsonify
app = Flask(__name__)
db = sqlite3.connect("ledger.db", check_same_thread=False)
db.execute("""
CREATE TABLE IF NOT EXISTS idempotency (
key TEXT PRIMARY KEY,
fingerprint TEXT NOT NULL,
status INTEGER,
body TEXT,
state TEXT NOT NULL
)
""")
def fingerprint(body: bytes) -> str:
return hashlib.sha256(body).hexdigest()
def charge_card(amount_cents: int) -> dict:
# Stand-in for your payment provider call.
return {"charge_id": f"ch_{uuid.uuid4().hex[:12]}",
"amount_cents": amount_cents, "captured": True}
@app.post("/charge")
def charge():
key = request.headers.get("Idempotency-Key")
if not key:
return jsonify(error="Idempotency-Key header required"), 400
fp = fingerprint(request.data)
try:
db.execute(
"INSERT INTO idempotency (key, fingerprint, state) "
"VALUES (?, ?, 'inflight')",
(key, fp),
)
db.commit()
first_attempt = True
except sqlite3.IntegrityError:
first_attempt = False
if not first_attempt:
row = db.execute(
"SELECT fingerprint, status, body, state FROM idempotency "
"WHERE key = ?",
(key,),
).fetchone()
if row[0] != fp:
return jsonify(error="key reused with a different request"), 422
if row[3] == "inflight":
return jsonify(error="original request still in flight"), 409
response = jsonify({**json.loads(row[2]), "replayed": True})
response.status_code = row[1]
return response
result = charge_card(json.loads(request.data)["amount_cents"])
db.execute(
"UPDATE idempotency SET status = 200, body = ?, state = 'done' "
"WHERE key = ?",
(json.dumps(result), key),
)
db.commit()
return jsonify(result), 200
if __name__ == "__main__":
app.run()
A few things are worth noting about this example. First, the INSERT either succeeds or raises IntegrityError, and there is no SELECT before it. That ordering is the whole design; the database settles the race, not your code. Second, the duplicate path checks the fingerprint before replaying, so a key reused with a different body gets a 422 instead of someone else's receipt. Third, the replay adds "replayed": true to the body, which is how you will debug this at 2 a.m. when you need to know which responses were real and which were the ledger talking.
Run it, then act out the timeout in your terminal:
pip install flask
python app.py &
KEY=$(uuidgen)
curl -s -X POST localhost:5000/charge \
-H "Idempotency-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"amount_cents": 4200}' | head -c 200; echo
curl -s -X POST localhost:5000/charge \
-H "Idempotency-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"amount_cents": 4200}'
The first call returns a fresh charge_id. The second returns the same charge_id with "replayed": true. Two requests, one charge. If you want the Redis version of the same idea, the atomic claim is one command: SET key locked NX EX 60, then SET key result EX 86400 when the work completes. Same contract, no schema.
One honest caveat about this example: if the worker crashes between the insert and the update, the key sits in inflight forever and every retry gets a 409. In production, give the lease a timeout. A reaper that deletes inflight rows older than five minutes is an afternoon's work, and it is the difference between a demo and a system. Stripe solves this at a scale you do not have. Solve it at the scale you do.
The unique constraint is not an optimization. It is the mechanism.
Where This Breaks
- You mint a new key per attempt instead of per intent. The retry loop generates the key, the server sees three strangers, and the card is charged three times. Check your client code first; this is where the bug usually lives.
- Same key, different body, no fingerprint check. You replay yesterday's answer to today's question. Stripe rejects these outright. Your homegrown version probably smiles and replays, which is worse than erroring.
- Check-then-act. You
SELECTfor the key, find nothing,INSERT, and do the work, and a second worker slips between yourSELECTand yourINSERT. Without the unique constraint arbitrating, you have a ledger that lies under load. - The key protects the API but not the side effects behind it. Your endpoint replays perfectly while the receipt email fires twice, because the email sender never consulted the ledger. The key covers exactly what you put behind it, and nothing else.
- Stored failures replay forever. Stripe replays the original response even when it was a 5xx. That is a defensible policy, but it means a transient provider outage becomes a permanent answer for that key. Decide your policy on purpose: replay 4xx, expire keys whose first attempt died with a 5xx, let the retry try again. Either way, decide it before the outage.
- 24 hours is not forever, and forever is not free. A retry on day three with the same key is a brand-new request. Keys you never expire are a table that grows until somebody gets paged about disk. Stripe keeps results for at least 24 hours and prunes after; do the same.
Build It If, Skip It If
Build it if the endpoint moves money, sends messages people read, creates records your users can see twice, or sits anywhere near a retry: client timeouts, queue redelivery, webhook retries, double-clicked buttons, a deploy that restarts a worker mid-request. If a retry can reach the endpoint, the endpoint needs the ledger.
Skip it if the endpoint is a read, if the client provably never retries, or if duplicates are harmless and cheaper to clean up than to prevent. An internal admin endpoint that lists rows does not need a ledger. Your checkout does.
The minimal viable version fits in an afternoon: one table with a unique key column, a fingerprint of the request body, a 24-hour TTL, and the header required on your two most dangerous POST endpoints. Start with the one that moves money. The rest can wait.
Try It This Week
Pick your most dangerous POST endpoint. Add the header requirement, add the table, and fire ten concurrent requests with the same key in staging. Watch nine of them get the same answer without touching the provider. Then look at the ledger and count the rows that did the work: one. That is the entire pitch. The network will keep lying to you, and from now on it will not matter.
What is the worst duplicate your system ever produced: a double charge, a double email, a double deploy?
Top comments (0)