Idempotency Keys: The One Pattern That Saves You From Duplicate API Requests

Network failures are inevitable. A payment request times out. Your client retries. The server processed the original — now your user just got charged twice.

Idempotency keys are the standard solution. They let clients safely retry any request while guaranteeing the server executes it exactly once. Every major payment API (Stripe, Adyen, PayPal) uses them. You should too.

How It Works

The client generates a unique key (typically a UUID) for each logical operation and attaches it to the request — usually as an Idempotency-Key header. The server stores the key alongside the result on first execution. On any retry with the same key, it returns the cached result without re-executing the operation.

POST /payments
Idempotency-Key: a8098c1a-f86e-11da-bd1a-00112444be1e
Content-Type: application/json

{
  "amount": 5000,
  "currency": "usd",
  "customer_id": "cus_abc123"
}

If the server returns a 200 OK with the payment result — great. If the network dies before the response arrives, the client retries with the same key and gets the same 200 OK without a second charge.

Implementing Idempotency on the Server Side

Here's a minimal Express implementation backed by Redis:

const express = require('express');
const redis = require('redis');

const app = express();
const client = redis.createClient();
app.use(express.json());

const IDEMPOTENCY_TTL = 86400; // 24 hours

async function idempotencyMiddleware(req, res, next) {
  const key = req.headers['idempotency-key'];

  if (!key) return next(); // Optional: enforce for mutating methods only

  const cacheKey = `idempotency:${key}`;
  const cached = await client.get(cacheKey);

  if (cached) {
    const { status, body } = JSON.parse(cached);
    return res.status(status).json(body);
  }

  // Intercept the response to cache it
  const originalJson = res.json.bind(res);
  res.json = async (body) => {
    if (res.statusCode < 500) {
      await client.setEx(
        cacheKey,
        IDEMPOTENCY_TTL,
        JSON.stringify({ status: res.statusCode, body })
      );
    }
    return originalJson(body);
  };

  next();
}

app.post('/payments', idempotencyMiddleware, async (req, res) => {
  // Your payment logic here
  const payment = await processPayment(req.body);
  res.status(201).json(payment);
});

Key decisions in this implementation:

• Only cache non-5xx responses. A 500 means something went wrong server-side — don't cache the failure, let the client retry and trigger a fresh attempt.
• Set a TTL. 24 hours is standard. After expiry, clients should generate a new key rather than retrying the original operation.
• Scope the cache key. Prefix with idempotency: (or even per-user: idempotency:{userId}:{key}) to avoid collisions.

Generating Keys on the Client Side

The client is responsible for generating a key per logical operation — not per HTTP request. This distinction matters:

import uuid
import httpx
import time

def charge_customer(amount: int, currency: str, customer_id: str) -> dict:
    # Generate once per operation, persist across retries
    idempotency_key = str(uuid.uuid4())

    for attempt in range(3):
        try:
            response = httpx.post(
                "https://api.example.com/payments",
                headers={"Idempotency-Key": idempotency_key},
                json={
                    "amount": amount,
                    "currency": currency,
                    "customer_id": customer_id
                },
                timeout=10.0
            )
            response.raise_for_status()
            return response.json()
        except (httpx.TimeoutException, httpx.NetworkError):
            if attempt == 2:
                raise
            time.sleep(2 ** attempt)  # Exponential backoff

The same idempotency_key is used across all three retry attempts. If you generated a new UUID on each retry, you'd defeat the purpose entirely.

Edge Cases to Handle

Concurrent requests with the same key. Two retries can arrive simultaneously before the first completes. Use a database lock or Redis SET NX (set-if-not-exists) to hold a "processing" placeholder:

// Atomic lock: only succeeds if key doesn't exist
const locked = await client.set(cacheKey, 'PROCESSING', {
  NX: true,          // Only set if not exists
  EX: 30             // Lock expires in 30s (safety net)
});

if (!locked) {
  // Another request is in-flight — return 409 or wait and retry
  return res.status(409).json({ error: 'Request in progress' });
}

Key conflicts across users. Scope your cache key to the user or API key to prevent one user's key colliding with another's. A UUID is practically collision-free, but defence-in-depth doesn't hurt.

Different request bodies, same key. Treat this as an error. If the stored fingerprint of the original request body doesn't match the retry, return 422 Unprocessable Entity. Some clients reuse keys accidentally; catching this early surfaces bugs.

What to Expose in Your API Docs

Document idempotency clearly:

• Which endpoints support (or require) Idempotency-Key
• How long keys are retained (the TTL)
• What happens on body mismatch
• The expected format (UUID v4 is convention)

Tools like APIKumo let you document these headers directly alongside your endpoint specs and test idempotent flows by replaying requests with the same key — useful for verifying your implementation behaves correctly on retries without spinning up a separate test harness.

The Short Version

1. Client generates a UUID per logical operation
2. Client sends it as Idempotency-Key on every attempt
3. Server stores key → result in Redis with a TTL
4. On retry, server returns the cached result without re-executing
5. Server-side errors (5xx) are never cached

It's a small addition that eliminates an entire class of data-integrity bugs. If your API handles payments, order creation, or anything with real-world side effects — add idempotency keys before you need them.