DEV Community

Cover image for Idempotency Keys Explained: Build a Payment Endpoint That Survives Retries
ScaleDojo
ScaleDojo

Posted on

Idempotency Keys Explained: Build a Payment Endpoint That Survives Retries

Why double charges happen, the exact contract that prevents them, and working code you can adapt.

A customer taps “Pay.” The request reaches your server, the payment is recorded, but the response is lost on a flaky mobile connection. The app times out and retries. Your server sees a brand new request and charges the customer again.

This is one of the most common API design interview questions, and one of the most common production bugs. The fix is a small idea called an idempotency key. In this post you will learn the contract, then build a working version with PostgreSQL and Python.

Idempotent, in One Sentence

An operation is idempotent if doing it many times has the same effect as doing it once.

In HTTP, GET, PUT and DELETE are meant to be idempotent. POST is not, because posting twice usually creates two things. Payments, orders and transfers are almost always POSTs, so they need extra protection.

The Contract

The client generates a unique key (a UUID works well) for each logical operation and sends it in a header:

POST /payments
Idempotency-Key: 4d7a6c1e-9f0b-4f6e-8a55-1c2d3e4f5a6b
Content-Type: application/json

{"customer_id": 17, "amount_cents": 2000, "currency": "usd"}
Enter fullscreen mode Exit fullscreen mode

The server then follows these rules:

1.First time this key is seen: Do the work, save the result with the key, and return it.

2.Same key, same request body: Do not repeat the work; return the saved result.

3.Same key, different request body: Reject with 422.

4.Same key while the first request is still running: Wait for it, or return 409 Conflict.

The client reuses the same key on every retry of the same action, and creates a new key for each new action.

The Table

We store each key together with a fingerprint of the request and the saved response:

CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  request_hash  TEXT NOT NULL,
  response_code INT,
  response_body JSONB,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE payments (
  id           BIGSERIAL PRIMARY KEY,
  customer_id  BIGINT NOT NULL,
  amount_cents INT NOT NULL CHECK (amount_cents > 0),
  currency     CHAR(3) NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

Enter fullscreen mode Exit fullscreen mode

The PRIMARY KEY on key does the hard part for us: the database itself guarantees that a key can only be inserted once, even if two requests arrive at the same instant.

The Handler
Here is the core logic using psycopg2. The key trick is to claim the key and do the work inside one transaction.

import hashlib
import json

def create_payment(conn, idem_key, payload):
    request_hash = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode()
    ).hexdigest()

    with conn:                       # one transaction: commit on success, rollback on error
        with conn.cursor() as cur:
            # 1. Try to claim the key.
            cur.execute(
                """INSERT INTO idempotency_keys (key, request_hash)
                   VALUES (%s, %s)
                   ON CONFLICT (key) DO NOTHING
                   RETURNING key""",
                (idem_key, request_hash),
            )

            if cur.fetchone() is None:
                # 2. Key already used: replay or reject.
                cur.execute(
                    "SELECT request_hash, response_code, response_body "
                    "FROM idempotency_keys WHERE key = %s",
                    (idem_key,),
                )
                saved_hash, code, body = cur.fetchone()
                if saved_hash != request_hash:
                    return 422, {"error": "Idempotency-Key reused with a different request"}
                return code, body

            # 3. First time: do the real work.
            cur.execute(
                "INSERT INTO payments (customer_id, amount_cents, currency) "
                "VALUES (%s, %s, %s) RETURNING id",
                (payload["customer_id"], payload["amount_cents"], payload["currency"]),
            )
            payment_id = cur.fetchone()[0]
            response = {"payment_id": payment_id, "status": "succeeded"}

            # 4. Save the response with the key.
            cur.execute(
                "UPDATE idempotency_keys SET response_code = 201, response_body = %s "
                "WHERE key = %s",
                (json.dumps(response), idem_key),
            )
            return 201, response
Enter fullscreen mode Exit fullscreen mode

What Happens When You Call It Twice

create_payment(conn, "4d7a...", {...})  ->  (201, {"payment_id": 1, "status": "succeeded"})
create_payment(conn, "4d7a...", {...})  ->  (201, {"payment_id": 1, "status": "succeeded"})   # replayed, no new row
create_payment(conn, "4d7a...", {amount changed})  ->  (422, {"error": "...reused..."})
Enter fullscreen mode Exit fullscreen mode

The payments table has exactly one row. The retry got the original answer back.

The Subtle Case: Two Requests at the Same Moment

Suppose the client’s retry arrives while the first request is still running. Because both try to insert the same primary key, PostgreSQL makes the second INSERT ... ON CONFLICT wait until the first transaction finishes. When the first commits, the second hits the conflict, does nothing, reads the saved response and replays it. If the first transaction fails and rolls back, the second one proceeds normally. No double charge, and no extra locking code.

A Real Gotcha: Calling an External Payment Provider

The example above works because the “work” is a database write. If your work calls an external provider, you should not hold a database transaction open during a slow network call. The usual approach is:

  1. Insert the key with a status of in_progress and commit.
  2. Call the provider, passing your own idempotency key along so it is also protected.
  3. Update the key with the final response and set the status to completed.
  4. If a retry arrives while the status is in_progress, return 409 Conflict (and tell the client to retry shortly).
  5. Run a recovery job for keys stuck in_progress for too long.

Don’t Forget the Client

The server is only half of the contract. A good client:
• Generates the key once per user action and reuses it for every retry of that action.
• Retries only when it makes sense: network errors, timeouts, 5xx and 429.
• Uses exponential backoff with jitter, so thousands of clients do not retry at the same instant.

Housekeeping

• Expire old keys. Keeping them forever grows the table. Many APIs keep them for a limited window (for example, 24 hours) and delete older ones with a scheduled job. Document the window for your clients.
• Decide what to store. Saving the outcome of every completed request keeps replays consistent. Let genuinely transient failures (a 503) be retried instead of cached.
• Index and size. The primary key is the only index you need for the lookup.

How to Say This in an Interview (45 Seconds)

“Retries after a timeout can double-charge a customer, because POST is not idempotent. I would require an Idempotency-Key header, a unique ID the client generates per action. On the server I store the key with a hash of the request and the saved response, with a unique constraint so concurrent attempts cannot both succeed. A repeat with the same key and body returns the saved response, the same key with a different body gets a 422, and a request still in progress gets a 409. I would expire keys after a window like 24 hours, and if I call an external provider, I would pass the key through to it.”

Common Mistakes

• Using GET or a database row count as a “double-click guard” instead of a real key.
• Generating a new key on each retry (which defeats the purpose).
• Not comparing the request body, so a reused key silently does the wrong thing.
• Holding a database transaction open while calling an external API.
• Forgetting to expire keys.

Keep Practising API Design

Idempotency is just one of the “hard parts” of API design interviews, alongside pagination, versioning, errors and rate limits. If you want to practise designing and defending APIs hands-on, the API Design Lab on ScaleDojo has 144 levels, and you can start for free.

Top comments (4)

Collapse
 
paullovelace profile image
Paul Lovelace •

The bit I liked most is letting the primary key do the concurrency work — ON CONFLICT making the second insert wait instead of you writing locking code. That's the kind of thing that looks obvious after someone shows it. The in_progress + 409 pattern for external providers is the part I'd actually mess up in practice. Do you expire keys after 24h in your own stuff, or keep them longer for replay safety?

Collapse
 
sophiabennettdev profile image
Sophia Bennett •

Worth reading!

Collapse
 
simanta_sarma_7832f640cc6 profile image
Simanta Sarma •

This is something I am looking for.

Collapse
 
deadline_fighters_fd82ba profile image
DEADLINE FIGHTER'S •

After a very long time I'll read this kind of informative blog post great work 💯