Why Your API Needs Idempotency Keys (And How to Implement Them in 10 Minutes)
Last month, a user of our payment API got charged three times for a single transaction. The culprit? A flaky mobile network that retried the same POST request three times before the client finally got a 200 response. The server processed all three.
If you've ever built an API that handles money, inventory, or anything stateful, you've probably encountered this problem. The solution is simpler than you think: idempotency keys.
What Is Idempotency?
An operation is idempotent if performing it multiple times produces the same result as doing it once. In HTTP terms:
- GET — naturally idempotent (fetching a resource doesn't change it)
- PUT — naturally idempotent (setting a resource to state X gives the same result every time)
- POST — not idempotent (creating a new resource each time)
- DELETE — technically idempotent (deleting something that's already deleted is still… deleted)
The problem lives in POST. Every time a client sends a POST, a new resource gets created. If the client retries because of a network timeout, you get duplicates.
The Idempotency Key Pattern
Here's the core idea: the client generates a unique key for each intended operation and sends it as a header. The server uses this key to detect duplicates.
Client → POST /payments
Idempotency-Key: 4f8b3a2c-1d9e-4e7f-a3b1-c6d8e9f0a1b2
Body: {"amount": 49.99, "currency": "USD"}
Server → 201 Created (first time)
Client → POST /payments (network retry — same key!)
Idempotency-Key: 4f8b3a2c-1d9e-4e7f-a3b1-c6d8e9f0a1b2
Body: {"amount": 49.99, "currency": "USD"}
Server → 200 OK (returns the original response, no duplicate!)
That's it. The magic is that the server remembers "I've already processed this key" and returns the cached response instead of creating a duplicate.
Implementation in 10 Minutes
Here's a production-ready implementation. We'll use Redis for storage since it's fast and has built-in TTL support.
Step 1: The Middleware (FastAPI)
import hashlib
import json
from fastapi import Request, HTTPException
import redis.asyncio as redis
from functools import wraps
redis_client = redis.Redis(host="localhost", port=6379, decode_responses=True)
IDEMPOTENCY_TTL = 86400 # 24 hours
def idempotent(func):
@wraps(func)
async def wrapper(*args, **kwargs):
request: Request = kwargs.get("request") or args[0]
idempotency_key = request.headers.get("Idempotency-Key")
if not idempotency_key:
raise HTTPException(
status_code=400,
detail="Idempotency-Key header is required"
)
# Check if we've seen this key before
cached = await redis_client.get(f"idem:{idempotency_key}")
if cached:
cached_data = json.loads(cached)
return JSONResponse(
content=cached_data["body"],
status_code=cached_data["status"],
headers=cached_data.get("headers", {})
)
# Key doesn't exist — process the request
response = await func(*args, **kwargs)
# Cache the response before returning
response_data = {
"status": response.status_code,
"body": json.loads(response.body.decode()),
"headers": dict(response.headers),
}
await redis_client.setex(
f"idem:{idempotency_key}",
IDEMPOTENCY_TTL,
json.dumps(response_data)
)
return response
return wrapper
Step 2: Use It on Your Endpoint
@app.post("/payments")
@idempotent
async def create_payment(request: Request):
body = await request.json()
payment = await process_payment(body["amount"], body["currency"])
return JSONResponse(
content={"id": payment.id, "status": payment.status},
status_code=201,
)
That's it. Every POST to /payments now requires an Idempotency-Key header, and duplicates return the exact same response.
Handling Concurrent Requests
There's one edge case: what if two requests with the same key arrive at the same time? Without a lock, both might pass the cache check, both process the payment, and both try to write the response.
The fix: atomic SET NX (set if not exists). Redis supports this natively:
# Replace the simple GET + SET with an atomic operation
lock_key = f"idem:lock:{idempotency_key}"
# Try to acquire the lock
acquired = await redis_client.setnx(lock_key, "processing")
if not acquired:
# Another request is processing this key — wait and retry
for _ in range(30): # 30 × 100ms = 3 second timeout
await asyncio.sleep(0.1)
cached = await redis_client.get(f"idem:{idempotency_key}")
if cached:
return json.loads(cached)["body"]
raise HTTPException(status_code=409, detail="Request is still processing")
try:
# Process the request...
pass
finally:
# Always clean up the lock
await redis_client.delete(lock_key)
What I Learned the Hard Way
I've implemented this pattern across three different systems. Here's what the docs don't tell you:
1. Key generation matters more than you think
Don't let clients generate random UUIDs and call it a day. The key should be derived from the intent of the request — a hash of (user_id + operation_type + idempotent_payload). If a user accidentally sends two different amounts with the same key, you want to reject the second one, not silently return the first.
# Better approach: validate that the payload matches
async def wrapper(*args, **kwargs):
...
body_hash = hashlib.sha256(
json.dumps(body, sort_keys=True).encode()
).hexdigest()
if cached:
if cached_data["body_hash"] != body_hash:
raise HTTPException(
status_code=422,
detail="Idempotency-Key reused with different payload"
)
2. 24 hours is not always the right TTL
Stripe uses 24 hours. Shopify uses 72. I use 24 hours for payments and 1 hour for everything else. The tradeoff: longer TTL = more Redis memory = better protection against very-late retries. Pick based on your domain.
3. Don't idempotent everything
Not every POST needs this. Read-only operations, analytics events (where duplicates are acceptable), and fire-and-forget notifications don't benefit from idempotency keys. Applying it everywhere adds latency and Redis overhead for no gain.
Use idempotency keys when:
- Money is involved (payments, refunds, credits)
- Inventory changes (stock decrements, reservation creation)
- State transitions (order status changes, account activation)
Skip idempotency keys for:
- Logging / analytics events
- Search queries
- Email/sms notifications (duplicates are annoying but not catastrophic)
The Stripe Standard
If you want to see this pattern in the wild, look at Stripe's API. They pioneered the Idempotency-Key header and their implementation is the gold standard:
- Keys expire after 24 hours
- POST requests with the same key and different body return a 422
- Their client libraries auto-generate keys from (method + path + body hash)
- They store the entire response including HTTP status code and headers
This is the pattern I recommend copying directly. It's battle-tested across billions of requests.
Quick Reference: The Copypaste Version
Here's everything you need in one place. Copy, paste, adapt:
Header: Idempotency-Key: <uuid>
Storage: Redis with 24h TTL
Strategy: GET key → if exists return cached → if not, SET NX lock → process → SET response → DELETE lock
Conflict: Same key + different body → 422
Missing: No key header → 400
Wrapping Up
Adding idempotency keys to your API takes about 10 minutes of coding and saves you from the kind of 3 AM production incident where a user got charged multiple times and your phone won't stop buzzing.
Start with your payment endpoints. Then expand to anything that mutates state in a way you'd regret seeing twice. Your future self (and your users) will thank you.
Have you implemented idempotency keys in your API? What storage backend did you use? I'd love to hear about alternatives to Redis — let me know in the comments.
Top comments (0)