tejaswipandava

Posted on Oct 6

Idempotency - "The customer clicked 'Pay' three times. How do you ensure they're charged only once?"

#techlettersbyteja #systemdesign #backend

Most know what idempotency is; very few can clearly explain why it's needed and how its actually implemented.

🤔 The Problem

Imagine a payment system. The user clicks "Pay." The request reaches the server and is
processed successfully — but the response never reaches the user because of a network timeout,
a dropped connection, or a slow proxy.

What does the user do? They click "Pay" again.

Without idempotency, this retry can cause the API to:

Create multiple orders
Charge the customer duplicate payments
Deduct inventory more than once
Produce a very angry customer

The core issue: the client cannot tell the difference between "my request never arrived" and "my
request succeeded but the response got lost." Both look identical from the client's side — a
timeout — but calling the API again is only safe in the first case.

💡 The Solution: Idempotency Keys

Generate a unique Idempotency Key for each logical request (not each HTTP call — the same
logical intent, e.g., "charge this cart," keeps the same key across retries).

First request with a given key → process it normally, then store the response alongside the key.
Retry with the same key → the server recognizes it's already been handled, and simply returns the previously stored response instead of processing the request again.
No duplicate processing, no matter how many times the client retries.

The core guarantee: Same request + same key = same result, every time.

Client                         Server
  │  POST /pay (Idempotency-Key: abc123)
  ├──────────────────────────────►
  │                               │  key "abc123" not seen before
  │                               │  → process payment
  │                               │  → store {key: abc123, response: {...}}
  │  ◄── 200 OK {charge_id: X}────┤
  │  (response lost in transit)   │
  │
  │  Retry: POST /pay (Idempotency-Key: abc123)
  ├──────────────────────────────►
  │                               │  key "abc123" already exists
  │                               │  → skip processing, return stored response
  │  ◄── 200 OK {charge_id: X}────┤   (same charge_id, no new charge)

🌍 Where Idempotency Is Used

Anywhere duplicate side effects are unacceptable:

Payment APIs
Order creation
Money transfers
Ticket booking
Inventory updates

🎯 Common Questions

1. What is idempotency?

An operation is idempotent if performing it multiple times has the same effect as performing it
once. In HTTP/API terms: retrying an idempotent request should never change the outcome beyond
what the first successful call already did.

2. Why is POST not idempotent by default?

By HTTP semantics, POST typically means "create a new resource" — calling it twice naturally
creates two resources (two orders, two charges), because the server has no built-in way to know
the second call is a retry of the first rather than a genuinely new request. GET, PUT, and
DELETE are idempotent by convention/spec (see Q6), but POST is not — which is exactly why
payment/order-creation APIs (which are inherently POST-based, since they create something) need
an explicit idempotency mechanism layered on top; it doesn't come for free from the HTTP verb.

3. How do idempotency keys work?

The client generates a unique key (typically a UUID) per logical operation — generated once and reused across all retries of that same operation, never regenerated on retry.
The client sends it with the request, usually as a header (e.g., Idempotency-Key: <uuid>).
The server checks a fast lookup store for that key:
- Not seen before → process the request, store {key → response}, return the response.
- Already seen, completed → skip reprocessing, return the stored response directly.
- Already seen, still in-flight → the request is currently being processed by a concurrent call; the server should make the retry wait (or return a 409 Conflict telling the client to retry after a short delay) rather than let both proceed and race each other.

4. Where should idempotency keys be stored?

A fast, shared, low-latency key-value store — Redis is the typical choice, since idempotency
checks sit directly on the hot path of every write request and need sub-millisecond lookups. Some
systems back this with a database unique constraint on the idempotency key column instead of
(or in addition to) Redis, trading a bit of latency for stronger durability guarantees (see the
Redis-crash question below for why this matters).

5. How long should they be retained?

Long enough to cover the realistic retry/timeout window of the client, but not indefinitely:

Too short → a legitimate late retry (e.g., a mobile client that was offline for a few minutes) is treated as a brand-new request, causing a duplicate.
Too long → the store grows unbounded, wasting memory/storage on keys nobody will ever reuse.

A common choice is 24 hours for payment-style APIs (covers virtually all realistic client
retry/backoff windows), implemented via a TTL on the stored key.

6. Which HTTP methods are idempotent?

Method	Idempotent?	Why
`GET`	✅ Yes	Read-only, no side effects regardless of call count
`PUT`	✅ Yes	Replaces a resource with a given representation — calling it N times leaves the same end state as calling it once
`DELETE`	✅ Yes	Deleting an already-deleted resource still results in "resource doesn't exist"
`POST`	❌ No (by default)	Semantically "create" — repeated calls create repeated resources unless an idempotency key is explicitly layered on top
`PATCH`	❌ Not guaranteed	Depends on the semantics of the patch — e.g., "increment balance by 10" is not idempotent, "set status to X" is

💡Tip

Don't say:

❌ "Idempotency prevents duplicate requests."

This is imprecise — idempotency doesn't prevent the client from sending duplicate requests at
all (they will, especially on timeout/retry); it prevents duplicate requests from causing duplicate
side effects.

Say instead:

✅ "Idempotency ensures the same request produces the same outcome. A common implementation uses
an Idempotency Key to identify retries and return the original response instead of processing
the request again."

This shows you understand both the problem (retries are inevitable and indistinguishable from
first attempts) and the solution mechanism (key-based deduplication with a stored response),
not just the buzzword.

💬 Follow-up: What if Redis (storing the idempotency keys) crashes?

This is the natural next question which tests your understanding of the trade-offs behind your own solution, not just the happy path.

Key considerations to raise:

In-memory Redis alone is a single point of failure for a correctness-critical mechanism.
If Redis crashes and loses data (no persistence configured), a request that was already
processed but not yet acknowledged to the client can be reprocessed on retry, since the
server no longer remembers seeing that key — defeating the entire point of idempotency at
exactly the moment it matters most.
Mitigations, roughly from cheapest to strongest:
- Redis persistence (AOF/RDB) + replication — reduces data loss on a crash, but doesn't eliminate the window between a write and it being durably persisted/replicated.
- Backing store with a durable, unique constraint — write the idempotency key to a relational DB (or DynamoDB/Cassandra with a unique key constraint) as the source of truth, using Redis only as a fast-path cache in front of it. If Redis is down or cold, fall back to the DB — slower, but still correct. A DB INSERT ... ON CONFLICT DO NOTHING (or equivalent unique constraint violation) gives you atomic "claim this key or fail" semantics even without Redis.
- Idempotency at the payment provider itself, as a second line of defense — many payment processors (Stripe, etc.) support their own idempotency keys passed through to them. Even if our own store fails and we accidentally call the provider twice with the same key, the provider's own deduplication prevents an actual double charge. This is the "defense in depth" answer: don't rely on a single layer being perfectly durable.
Fail-closed vs. fail-open trade-off: if the idempotency store is completely unavailable and
there's no fallback, should the request be rejected (fail-closed — safe, but hurts availability)
or allowed through with the retry risk accepted (fail-open — available, but risks a duplicate)?
For payments specifically, I'd lean fail-closed (reject with a clear retriable error) rather
than risk a duplicate charge — the cost of a duplicate payment is much higher than the cost of a
brief availability blip on the write path.

Strong answer framing: "I wouldn't rely on Redis alone for something this correctness-critical.
I'd use Redis as a fast-path cache, back it with a durable store with a unique constraint as the
real source of truth, and pass the same idempotency key through to the downstream payment provider
as a second independent layer of protection — so no single component failing causes an actual
double charge."

DEV Community