Why double charges happen, the exact contract that prevents them, and working code you can adapt.
A customer taps “Pay.” The request reaches your server, the payment is recorded, but the response is lost on a flaky mobile connection. The app times out and retries. Your server sees a brand new request and charges the customer again.
This is one of the most common API design interview questions, and one of the most common production bugs. The fix is a small idea called an idempotency key. In this post you will learn the contract, then build a working version with PostgreSQL and Python.
Idempotent, in One Sentence
An operation is idempotent if doing it many times has the same effect as doing it once.
In HTTP, GET, PUT and DELETE are meant to be idempotent. POST is not, because posting twice usually creates two things. Payments, orders and transfers are almost always POSTs, so they need extra protection.
The Contract
The client generates a unique key (a UUID works well) for each logical operation and sends it in a header:
POST /payments
Idempotency-Key: 4d7a6c1e-9f0b-4f6e-8a55-1c2d3e4f5a6b
Content-Type: application/json
{"customer_id": 17, "amount_cents": 2000, "currency": "usd"}
The server then follows these rules:
1.First time this key is seen: Do the work, save the result with the key, and return it.
2.Same key, same request body: Do not repeat the work; return the saved result.
3.Same key, different request body: Reject with 422.
4.Same key while the first request is still running: Wait for it, or return 409 Conflict.
The client reuses the same key on every retry of the same action, and creates a new key for each new action.
The Table
We store each key together with a fingerprint of the request and the saved response:
CREATE TABLE idempotency_keys (
key TEXT PRIMARY KEY,
request_hash TEXT NOT NULL,
response_code INT,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE payments (
id BIGSERIAL PRIMARY KEY,
customer_id BIGINT NOT NULL,
amount_cents INT NOT NULL CHECK (amount_cents > 0),
currency CHAR(3) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
The PRIMARY KEY on key does the hard part for us: the database itself guarantees that a key can only be inserted once, even if two requests arrive at the same instant.
The Handler
Here is the core logic using psycopg2. The key trick is to claim the key and do the work inside one transaction.
import hashlib
import json
def create_payment(conn, idem_key, payload):
request_hash = hashlib.sha256(
json.dumps(payload, sort_keys=True).encode()
).hexdigest()
with conn: # one transaction: commit on success, rollback on error
with conn.cursor() as cur:
# 1. Try to claim the key.
cur.execute(
"""INSERT INTO idempotency_keys (key, request_hash)
VALUES (%s, %s)
ON CONFLICT (key) DO NOTHING
RETURNING key""",
(idem_key, request_hash),
)
if cur.fetchone() is None:
# 2. Key already used: replay or reject.
cur.execute(
"SELECT request_hash, response_code, response_body "
"FROM idempotency_keys WHERE key = %s",
(idem_key,),
)
saved_hash, code, body = cur.fetchone()
if saved_hash != request_hash:
return 422, {"error": "Idempotency-Key reused with a different request"}
return code, body
# 3. First time: do the real work.
cur.execute(
"INSERT INTO payments (customer_id, amount_cents, currency) "
"VALUES (%s, %s, %s) RETURNING id",
(payload["customer_id"], payload["amount_cents"], payload["currency"]),
)
payment_id = cur.fetchone()[0]
response = {"payment_id": payment_id, "status": "succeeded"}
# 4. Save the response with the key.
cur.execute(
"UPDATE idempotency_keys SET response_code = 201, response_body = %s "
"WHERE key = %s",
(json.dumps(response), idem_key),
)
return 201, response
What Happens When You Call It Twice
create_payment(conn, "4d7a...", {...}) -> (201, {"payment_id": 1, "status": "succeeded"})
create_payment(conn, "4d7a...", {...}) -> (201, {"payment_id": 1, "status": "succeeded"}) # replayed, no new row
create_payment(conn, "4d7a...", {amount changed}) -> (422, {"error": "...reused..."})
The payments table has exactly one row. The retry got the original answer back.
The Subtle Case: Two Requests at the Same Moment
Suppose the client’s retry arrives while the first request is still running. Because both try to insert the same primary key, PostgreSQL makes the second INSERT ... ON CONFLICT wait until the first transaction finishes. When the first commits, the second hits the conflict, does nothing, reads the saved response and replays it. If the first transaction fails and rolls back, the second one proceeds normally. No double charge, and no extra locking code.
A Real Gotcha: Calling an External Payment Provider
The example above works because the “work” is a database write. If your work calls an external provider, you should not hold a database transaction open during a slow network call. The usual approach is:
- Insert the key with a status of in_progress and commit.
- Call the provider, passing your own idempotency key along so it is also protected.
- Update the key with the final response and set the status to completed.
- If a retry arrives while the status is in_progress, return 409 Conflict (and tell the client to retry shortly).
- Run a recovery job for keys stuck in_progress for too long.
Don’t Forget the Client
The server is only half of the contract. A good client:
• Generates the key once per user action and reuses it for every retry of that action.
• Retries only when it makes sense: network errors, timeouts, 5xx and 429.
• Uses exponential backoff with jitter, so thousands of clients do not retry at the same instant.
Housekeeping
• Expire old keys. Keeping them forever grows the table. Many APIs keep them for a limited window (for example, 24 hours) and delete older ones with a scheduled job. Document the window for your clients.
• Decide what to store. Saving the outcome of every completed request keeps replays consistent. Let genuinely transient failures (a 503) be retried instead of cached.
• Index and size. The primary key is the only index you need for the lookup.
How to Say This in an Interview (45 Seconds)
“Retries after a timeout can double-charge a customer, because POST is not idempotent. I would require an Idempotency-Key header, a unique ID the client generates per action. On the server I store the key with a hash of the request and the saved response, with a unique constraint so concurrent attempts cannot both succeed. A repeat with the same key and body returns the saved response, the same key with a different body gets a 422, and a request still in progress gets a 409. I would expire keys after a window like 24 hours, and if I call an external provider, I would pass the key through to it.”
Common Mistakes
• Using GET or a database row count as a “double-click guard” instead of a real key.
• Generating a new key on each retry (which defeats the purpose).
• Not comparing the request body, so a reused key silently does the wrong thing.
• Holding a database transaction open while calling an external API.
• Forgetting to expire keys.
Keep Practising API Design
Idempotency is just one of the “hard parts” of API design interviews, alongside pagination, versioning, errors and rate limits. If you want to practise designing and defending APIs hands-on, the API Design Lab on ScaleDojo has 144 levels, and you can start for free.
Top comments (4)
The bit I liked most is letting the primary key do the concurrency work — ON CONFLICT making the second insert wait instead of you writing locking code. That's the kind of thing that looks obvious after someone shows it. The in_progress + 409 pattern for external providers is the part I'd actually mess up in practice. Do you expire keys after 24h in your own stuff, or keep them longer for replay safety?
Worth reading!
This is something I am looking for.
After a very long time I'll read this kind of informative blog post great work 💯