Idempotency Keys - The Double-Charge Trap

The Problem

You click Pay. The server charges your card, but the response is lost on the way back (network drop, timeout, tab freeze). The UI thinks it failed, so you click Pay again and get charged twice.

A timeout means "I don't know what happened", not "payment failed".

The Idea

Idempotent = doing the same thing twice has the same effect as doing it once.

Action	Repeat it twice	Idempotent?
Set notifications to OFF	Still OFF	Yes
Add USD 50 to the balance	USD 100 added	No
Create a payment	Two charges	No, unless we add a key

An idempotency key is a unique ID for one payment attempt. The UI sends it with the request. If the same key arrives again, the backend says "I've already handled this one" and returns the saved result instead of charging again.

• UI's job: create the key once, keep it, and resend the same key on every retry.
• Backend's job: remember each key and its result, so repeats are not executed again.

---

Full Flow Diagram

Example: user pays USD 50 for order O100.

sequenceDiagram
    actor User
    participant UI as Browser UI
    participant SS as sessionStorage
    participant API as Backend
    participant DB as Database
    participant PSP as Payment Provider

    User->>UI: Click "Pay"
    UI->>SS: Key for O100?
    SS-->>UI: None
    UI->>UI: key = crypto.randomUUID() → K1
    UI->>SS: Save payment-key:O100 = K1
    UI->>UI: Disable Pay button, show "Processing..."

    UI->>API: POST /orders/O100/pay<br/>Idempotency-Key: K1
    API->>DB: Insert (user, K1, O100, processing)
    DB-->>API: Inserted (K1 is new)
    API->>PSP: Charge USD 50 (provider key derived from K1)
    PSP-->>API: Success, payment P900
    API->>DB: Update K1 → succeeded, result P900
    API--xUI: 200 OK (response LOST)

    UI->>UI: Timeout → show "Checking payment status..."
    UI->>SS: Read key for O100
    SS-->>UI: K1 (same key)
    UI->>API: POST /orders/O100/pay<br/>Idempotency-Key: K1 (retry)
    API->>DB: Find K1
    DB-->>API: K1 = succeeded, P900
    Note over API,PSP: Provider NOT called again
    API-->>UI: 200 OK, payment P900 (saved result)
    UI->>SS: Remove payment-key:O100
    UI-->>User: Show receipt ✅ (charged once)

---

Step-by-Step

Step 1: User clicks "Pay" — UI gets a key

The UI first checks whether this order already has a key (from an earlier attempt or before a refresh).

• Key found → reuse it. This is a retry of the same payment.
• No key → generate a new one with crypto.randomUUID().

payment-key:O100  →  K1   (e.g. "6f13d95a-e6c1-4bf2-9459-99b502098dc1")

Use crypto.randomUUID() — it is random and practically never collides. Do not use Date.now(), Math.random(), or the amount; two different payments could produce the same value.

Step 2: Save the key BEFORE sending the request

sessionStorage.setItem("payment-key:O100", key);

Save first, then send. If the page crashes or reloads mid-request, the key is still there and the next attempt reuses it.

Where to store on the UI:

Place	Survives refresh?	Use it?
Variable / React state	No	Only for the current screen
sessionStorage	Yes (same tab)	Recommended — cleared when the tab closes
localStorage	Yes (even after browser restart)	Avoid — leftover keys on shared devices, shared across tabs
Cookie	Yes	Avoid — sent with every request for no reason

Also disable the Pay button while the request is in flight. This stops accidental double clicks, but it is only a UX helper; the key is the real protection.

Step 3: Send the payment request with the key

POST /orders/O100/pay
Idempotency-Key: K1
Cookie: session=...   (user is logged in)

The UI sends only the order ID and key. The backend loads the amount from the order itself; never trust a price sent by the browser.

Step 4: Backend checks the key

Why does the backend store the key? Because the UI can't know whether the first request was charged — only the backend can remember.

The backend saves one row per key:

User	Key	Order	Status	Result
U1	K1	O100	processing → succeeded	P900

When a request arrives:

Backend finds...	Backend does
K1 not found	New payment: save processing, charge, save result, respond
K1 = processing	Another request is already charging → reply "in progress, check again" (e.g. 409)
K1 = succeeded / declined	Return the saved result. Do not charge again
K1 exists but for a different order	Reject (e.g. 422) — key misuse

The insert must be atomic (a unique constraint on user + key). If two identical requests arrive at the same moment, only one insert succeeds, so only one charge happens.

Step 5: Backend charges and saves the result

The backend calls the payment provider and passes its own provider idempotency key derived from K1, saved before the call. If the backend crashes after the provider charged but before saving the result, a retry reuses that provider key, so the provider returns the original charge instead of creating a new one.

Then it updates the row to succeeded with result P900 and responds.

Step 6: UI handles the response

What the UI gets	What it means	UI action	Key?
200 success	Paid	Show receipt	Remove key
Card declined (clear failure)	Not charged	Show error, let user try again	Remove key; next try gets a new one
409 in progress	Still charging	Show "Processing...", check again shortly	Keep
Timeout / network error / 5xx	Unknown!	Show "Checking payment status...", retry	Keep

The key rule: only remove the key when you know the final outcome.

Step 7: Retry after timeout

The UI retries with the same K1 (a few times, with increasing delays like 1s, 2s, 4s).

Two possible cases, both safe:

• First request never reached the backend → K1 is new → backend charges once now.
• First request was already charged → K1 = succeeded → backend returns P900, no second charge.

That's the whole point: the UI doesn't need to know which case happened.

Step 8: User refreshes the page

flowchart TD
    A[Page loads / user clicks Pay] --> B{Key for O100 in sessionStorage?}
    B -- Yes --> C[Reuse the same key]
    B -- No --> D[Generate new key with crypto.randomUUID]
    D --> E[Save key in sessionStorage]
    C --> F[POST /orders/O100/pay with key]
    E --> F
    F --> G{Response}
    G -- Success --> H[Show receipt, remove key]
    G -- Declined --> I[Show error, remove key]
    G -- In progress / timeout / 5xx --> J[Show 'Checking...', keep key, retry later]
    J --> F

After refresh, the UI finds K1 in sessionStorage and sends the request with K1 again. The backend returns the saved result or "in progress" — never a second charge.

---

UI Code Example

Illustrative only; status codes depend on your API.

function getOrCreateKey(orderId) {
  const name = `payment-key:${orderId}`;
  let key = sessionStorage.getItem(name);
  if (!key) {
    key = crypto.randomUUID();
    sessionStorage.setItem(name, key); // save BEFORE sending
  }
  return key;
}

async function pay(orderId) {
  const key = getOrCreateKey(orderId);

  try {
    const res = await fetch(`/orders/${orderId}/pay`, {
      method: "POST",
      headers: { "Idempotency-Key": key },
      credentials: "include",
      signal: AbortSignal.timeout(15000),
    });

    if (res.ok) {
      sessionStorage.removeItem(`payment-key:${orderId}`);
      return { status: "paid", payment: await res.json() };
    }
    if (res.status === 402) {
      // Clear decline: not charged, a new attempt may use a new key
      sessionStorage.removeItem(`payment-key:${orderId}`);
      return { status: "declined" };
    }
    // 409 in progress, 5xx, etc.: outcome unknown, keep the key
    return { status: "unknown" };
  } catch {
    // Timeout or network error: outcome unknown, keep the key
    return { status: "unknown" };
  }
}

On "unknown", show "Checking payment status..." and call pay(orderId) again after a delay — it automatically reuses the same key.

sessionStorage can throw (e.g. storage full or blocked). In that case, keep the key in memory for this page and avoid letting the user start a fresh payment while one is unresolved.

---

When Is a New Key Needed?

Situation	Same key or new?
Timeout, network error, server error	Same
Page refresh during payment	Same (read from sessionStorage)
User double-clicks Pay	Same
Payment succeeded	No more payment needed
Card clearly declined, user tries again	New
User changes amount / card / order	New (old attempt must be finished first)
A different order	New

Simple rule: same intention → same key. New intention → new key.

---

Security Checklist

• Store only the key in sessionStorage. Never store card number, CVV, or tokens.
• The key is not a password. The backend must still check that the logged-in user owns order O100.
• The backend scopes keys per user, so one user's key can't read another user's payment.
• Never trust amount or status from the browser; the backend decides both.
• Clear payment keys on logout.

What If sessionStorage Is Lost?

New tab, cleared storage, or another device → the UI generates a new key for the same order. The idempotency key can't catch this, so the backend needs one more rule:

An order that is already paid, or has a payment in progress, cannot start another charge.

Keys stop duplicate requests. Order rules stop duplicate payments for the same order. You need both.

---

System Design: Key Creation, Change Detection, and Cancellation

So far, one key = one payment. Real checkouts are messier: the user changes the cart, applies a coupon, or switches cards while a payment is unclear. This section designs a setup that answers three questions:

1. When should the UI create a key?
2. How do UI and backend detect that the payment details changed?
3. How is an old attempt safely cancelled so a new key can be used?

Core idea: a key belongs to a checkout snapshot

A key represents "pay for exactly this checkout". The checkout is identified by a small snapshot:

Snapshot field	Changes when...
orderId	A different order
orderVersion	Backend bumps it on any change to items, quantity, coupon, address, or total
paymentMethodId	User picks another saved card or method (an ID like pm_123, never the card number)

Same snapshot      → same intention → reuse the key
Different snapshot → new intention  → old attempt must be finished/cancelled, then a new key

orderVersion is the important part: the UI doesn't have to compare cart items itself. The backend increases the version whenever anything affecting the price changes.

Architecture

flowchart LR
    subgraph Browser
        UI[Checkout UI]
        KM[Key Manager]
        SS[(sessionStorage)]
        UI --> KM
        KM <--> SS
    end

    KM -->|Pay / Cancel + Idempotency-Key| GW[API Gateway<br/>auth, rate limit]
    GW --> PS[Payment Service<br/>idempotency check]
    GW --> OS[Order Service<br/>cart, orderVersion, lock]
    PS <--> IDB[(Idempotency Store<br/>key, fingerprint, status, result)]
    PS <--> OS
    PS -->|charge with provider key| PSP[Payment Provider]
    PSP -->|webhook: final result| WH[Webhook Handler]
    WH --> PS
    RJ[Reconciliation Job] -->|resolve stuck attempts| PS
    CJ[Cleanup Job] -->|expire old finished keys| IDB

Component	Responsibility
Key Manager (UI)	Creates, reuses, stores, and clears keys based on the snapshot
API Gateway	Authenticates the user, rate-limits Pay clicks
Order Service	Owns the cart and orderVersion; blocks cart edits while a payment is active
Payment Service	Checks the key, compares fingerprints, charges, saves results, handles cancel
Idempotency Store	Database table with a unique constraint; the source of truth. A cache like Redis can speed up lookups but must not be the only store
Webhook Handler	Receives the provider's final result if the original response was lost
Reconciliation Job	Periodically asks the provider about attempts stuck in processing
Cleanup Job	Deletes finished keys after the retry window (e.g. 24h). Never deletes processing ones

Data model

Idempotency store (one row per key):

Column	Example	Purpose
user_id + key	U1 + K1	Unique together, so duplicates can't both insert
order_id	O100	Which order this attempt pays
fingerprint	hash of O100, v3, 5000, USD, pm_123	Detects the same key reused with different details
status	processing, succeeded, declined, failed, cancelled	Current state
response	{ paymentId: "P900" }	Saved result to replay
provider_key	derived from K1, saved before charging	Prevents duplicate charges at the provider
expires_at	now + 24h	Cleanup time (only once finished)

Order (relevant columns):

Column	Example	Purpose
version	3	Increases on every cart/price change
status	unpaid / paid	Paid orders can't be charged again
active_attempt	K1 or empty	Only one payment may be in progress per order

The backend calculates the fingerprint from its own data (loaded order + amount), not from values the browser claims.

Attempt lifecycle

stateDiagram-v2
    [*] --> processing: Pay request accepted (new key)
    [*] --> cancelled: Cancel arrives before Pay
    processing --> succeeded: Provider approved
    processing --> declined: Provider declined
    processing --> processing: Provider timeout, waiting for webhook / reconciliation
    processing --> failed: Reconciliation found no charge
    succeeded --> [*]
    declined --> [*]
    failed --> [*]
    cancelled --> [*]

• Cancel is only possible before charging starts. Once processing, the UI must wait for the final result.
• A succeeded payment cannot be "cancelled". Undoing it is a refund, a separate operation with its own key.

When should the UI create a key?

Moment	Create a key?	Why
Checkout page loads	No	User hasn't decided to pay; details may still change
User edits cart / coupon / address	No	Nothing to pay yet
User clicks Pay, no saved attempt	Yes	Intention is now fixed. Bind the key to the current snapshot
User clicks Pay again, same snapshot	No, reuse	Same intention (retry, double click, refresh)
Snapshot changed while an attempt is unresolved	Not yet	Cancel/resolve the old attempt first
Old attempt declined, failed, or cancelled	Yes, on next Pay	Previous intention is finished, nothing was charged
Old attempt succeeded	No	Order is paid. Show the receipt

Rule: create the key at the moment of commitment (the Pay click), never earlier.

Backend rules on POST /orders/O100/pay

The request now carries the snapshot:

POST /orders/O100/pay
Idempotency-Key: K1
Content-Type: application/json

{ "orderVersion": 3, "paymentMethodId": "pm_123" }

The backend checks in this order:

1. Authenticate and confirm the user owns O100.
2. Look up the key. If it already exists:
   • Fingerprint differs → 422 KEY_REUSED_WITH_DIFFERENT_DETAILS. Nothing is charged.
   • succeeded / declined / failed → replay the saved result.
   • processing → 409 IN_PROGRESS.
   • cancelled → 409 ATTEMPT_CANCELLED. This blocks a late, delayed request from charging after the user cancelled.
3. New key. In one database transaction:
   • Request orderVersion ≠ current version → 409 ORDER_CHANGED (UI must refresh the checkout).
   • Order is paid or already has an active_attempt → 409 PAYMENT_ALREADY_ACTIVE.
   • Otherwise insert the key as processing and set order.active_attempt = K1.
4. Charge using the saved provider key, then save the result:
   • Success → key succeeded, order paid.
   • Decline → key declined, clear active_attempt.
   • Timeout → stays processing; webhook or reconciliation finishes it later.

Key lookup comes before the version check, so a retry of an already-paid attempt still gets its saved receipt.

Cart edits while a payment is active

The Order Service rejects cart changes while active_attempt is set:

PATCH /orders/O100/cart  →  409 PAYMENT_IN_PROGRESS

The UI also disables editing during payment. The backend check is what actually guarantees it (another tab or device could try to edit).

Cancel endpoint

POST /orders/O100/payment-attempts/cancel
Content-Type: application/json

{ "key": "K1" }

The key goes in the body, not the URL, so it doesn't end up in access logs or browser history.

Backend finds K1...	Backend does	Response
Not found (Pay never arrived)	Inserts K1 as cancelled (a "tombstone")	cancelled
processing	Cannot cancel, a charge may be happening	processing — wait
declined / failed / cancelled	Nothing to do	Same status
succeeded	Cannot cancel	succeeded — show receipt or offer refund

Why insert a tombstone for an unknown key? Because the original Pay request might be stuck in the network and arrive later. The tombstone uses the same unique (user_id, key) slot, so whichever arrives first wins, and the late Pay is rejected instead of charging.

Scenario: user changes the cart after a timeout

sequenceDiagram
    actor User
    participant UI as Checkout UI
    participant SS as sessionStorage
    participant API as Backend

    User->>UI: Click Pay (cart v3, card pm_123)
    UI->>SS: Save {key: K1, snapshot: "v3:pm_123"}
    UI->>API: POST /pay, key K1, v3
    API--xUI: Timeout (outcome unknown)
    UI-->>User: "Checking payment status..."

    User->>UI: "I want to change my cart"
    UI->>API: Cancel K1
    alt K1 never reached backend
        API->>API: Insert K1 as cancelled (tombstone)
        API-->>UI: cancelled
        UI->>SS: Remove K1
        UI-->>User: Cart unlocked
        User->>UI: Edit cart → backend bumps to v4
        User->>UI: Click Pay
        UI->>SS: Save {key: K2, snapshot: "v4:pm_123"}
        UI->>API: POST /pay, key K2, v4
        API-->>UI: Success ✅
        Note over API: If delayed K1 arrives now → cancelled, no charge
    else K1 is processing
        API-->>UI: processing
        UI-->>User: "Payment in progress, please wait"
    else K1 already succeeded
        API-->>UI: succeeded, P900
        UI->>SS: Remove K1
        UI-->>User: Show receipt (refund if they still want changes)
    end

Scenario: backend crashes mid-payment (K1 stuck in processing)

The key reached the backend and was saved as processing, but the status was never updated. Now the user retries with K1. What happens?

Why K1 gets stuck:

1. UI sends Pay with K1
2. Backend saves K1 = processing (+ provider key PK1)   ✅ saved
3. Backend calls the provider
4. One of these happens:
   a) Backend crashes before or during the provider call  → maybe NOT charged
   b) Provider charged, backend crashed before saving      → charged
   c) Provider is slow or timed out                       → unknown
5. K1 is still "processing" in the DB

The backend itself doesn't know whether money moved.

What the retry does:

UI      -> Backend: Pay O100 with K1 (retry)
Backend -> DB:      Find K1
DB      -> Backend: K1 = processing
Backend -> UI:      409 "Payment in progress"   (provider NOT called)
UI:     shows "Checking payment status...", asks again in a few seconds

Rule: a retry request never starts a charge for a key that is already processing. It could be case (b) or (c), where money already moved or is moving.

Who fixes the stuck status?

1. Provider webhook (usually first). When the provider finishes the charge, it notifies the backend's webhook handler, which updates K1 to succeeded or declined.
2. Reconciliation job (safety net). Every minute or so, it finds attempts stuck in processing for longer than a threshold (e.g. 2 minutes) and asks the provider about them using PK1, which was saved with K1 before calling the provider.

Provider says (for PK1)	Reconciliation job does
Charge succeeded	K1 → succeeded, save P900, order → paid
Charge declined	K1 → declined, unlock the order
No charge found (crashed before calling)	Retry the charge with the same PK1, or mark K1 failed (not charged) and unlock the order

Retrying with the same PK1 is safe: if the charge already exists, the provider returns it instead of creating a second one.

If the provider can't be reached, or offers no idempotency or status lookup, K1 stays processing and the order stays locked. It is flagged for manual review, never charged blindly.

sequenceDiagram
    participant UI as Checkout UI
    participant API as Backend
    participant DB as Database
    participant PSP as Payment Provider
    participant RJ as Webhook / Reconciliation

    UI->>API: Pay with K1
    API->>DB: K1 = processing, save PK1
    API->>PSP: Charge with PK1
    PSP->>PSP: Charge succeeds
    Note over API: 💥 Backend crashes. K1 still processing
    API--xUI: No response

    UI->>API: Retry with K1
    API->>DB: Find K1 → processing
    API-->>UI: 409 In progress (no charge)
    UI->>UI: "Checking payment status..."

    RJ->>PSP: Status of PK1?
    PSP-->>RJ: Succeeded, P900
    RJ->>DB: K1 = succeeded, P900, order paid

    UI->>API: Retry with K1
    API->>DB: Find K1 → succeeded
    API-->>UI: 200 Paid, P900 (saved result)
    UI->>UI: Show receipt, remove K1

Timeline:

12:00:00  UI: Pay with K1
12:00:00  Backend: K1 = processing, PK1 saved
12:00:01  Backend calls provider with PK1 → provider charges
12:00:01  Backend CRASHES (K1 still processing)
12:00:15  UI timeout → "Checking payment status..."
12:00:20  UI retries K1 → 409 in progress (no charge)
12:00:30  UI retries K1 → 409 in progress (no charge)
12:00:40  Webhook arrives: PK1 succeeded → K1 = succeeded, P900
12:00:45  UI retries K1 → 200 "Paid, P900" (saved result)
          UI shows receipt, removes K1 from sessionStorage
          Customer charged ONCE ✅

If the webhook never arrives, the reconciliation job performs the same update around 12:02.

What the user sees:

• While processing: "Payment in progress, please don't close this page." Pay button and cart stay locked.
• After a long wait (e.g. 1-2 minutes): "We're confirming your payment. We'll email you when it's done." The UI can stop polling; the backend still finishes the work.
• On refresh: the UI reads K1 from sessionStorage and asks again, getting the latest status.
• If the attempt ends as failed (not charged), the backend replays it like a decline (e.g. 402 in the UI example), so the UI removes K1 and the next Pay creates a new key.

Why it's safe:

Protection	Prevents
K1 saved as processing before charging	A retry starting a second charge
PK1 saved before calling the provider	The provider creating two charges
Webhook + reconciliation job	K1 staying stuck forever
Order lock (active_attempt)	A new key starting another payment for the same order

Key Manager code (UI)

Illustrative; extends the earlier example with a snapshot and cancel.

const nameFor = (orderId) => `payment-attempt:${orderId}`;
const snapshotOf = (c) => `${c.orderVersion}:${c.paymentMethodId}`;

// Called only when the user clicks Pay
function getOrCreateAttempt(checkout) {
  const name = nameFor(checkout.orderId);
  const saved = JSON.parse(sessionStorage.getItem(name) ?? "null");
  const snapshot = snapshotOf(checkout);

  if (saved && saved.snapshot === snapshot) {
    return saved; // same intention: reuse key
  }
  if (saved) {
    // Details changed but the old attempt is unresolved
    throw new Error("Resolve the previous payment attempt first");
  }

  const attempt = { key: crypto.randomUUID(), snapshot };
  sessionStorage.setItem(name, JSON.stringify(attempt)); // save BEFORE sending
  return attempt;
}

// Called when the user wants to change details during an unclear payment
async function cancelAttempt(orderId) {
  const saved = JSON.parse(sessionStorage.getItem(nameFor(orderId)) ?? "null");
  if (!saved) return { status: "none" };

  try {
    const res = await fetch(`/orders/${orderId}/payment-attempts/cancel`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      credentials: "include",
      body: JSON.stringify({ key: saved.key }),
    });
    if (!res.ok) return { status: "unknown" }; // keep key, try later

    const { status } = await res.json();
    if (["cancelled", "declined", "failed", "succeeded"].includes(status)) {
      sessionStorage.removeItem(nameFor(orderId)); // final: key no longer needed
    }
    return { status }; // "processing" → keep key and wait
  } catch {
    return { status: "unknown" }; // network error: keep key
  }
}

UI reactions:

• cancelled / declined → unlock the cart; the next Pay creates a new key.
• processing / unknown → keep the cart locked, show "Payment in progress".
• succeeded → show the receipt; offer a refund if the user still wants changes.
• Pay returns 409 ORDER_CHANGED → the order changed elsewhere (another tab). Reload the checkout, show the new total, and let the user confirm. The old key was never accepted, so remove it; the confirmation Pay creates a new key.

Design summary

Concern	Handled by
Create key at the right time	UI Key Manager, on Pay click only
Reuse on retry / refresh	UI snapshot match + sessionStorage
Detect changed details with the same key	Backend fingerprint check (422)
Detect stale checkout	orderVersion check (409 ORDER_CHANGED)
Prevent edits during payment	Order lock (active_attempt)
Cancel before charging	Cancel endpoint + tombstone
Late request after cancel	Tombstone blocks it
Lost response / crash	Saved result, provider key, webhook, reconciliation
Key stuck in processing	Retries get 409, reconciliation resolves via provider key
Storage growth	Cleanup job removes finished keys after the retry window

---

Summary

Click Pay
  → UI: reuse key, or create one with crypto.randomUUID()
  → UI: save key in sessionStorage
  → UI: POST with Idempotency-Key
  → Backend: new key? charge and save result. Known key? return saved result
  → UI: success/decline → remove key | unknown → keep key and retry

More Details:

Get all articles related to system design
Hashtag: SystemDesignWithZeeshanAli

systemdesignwithzeeshanali

Git: https://github.com/ZeeshanAli-0704/front-end-system-design

⬆ Back to Top