Idempotency Keys in Production: The Race Conditions, Expiration Traps, and Edge Cases Most Tutorials Skip

Every payment flow has a silent enemy: the network. Requests time out, connections drop, users panic and click "Pay" twice. If your backend isn't ready for that double-tap, someone gets charged twice — and you lose their trust forever.

Most tutorials explain idempotency keys in about three paragraphs: "generate a UUID, send it in a header, check it on the server." That's the hello-world version. It works until you have concurrent requests, distributed servers, or a payment that takes longer than your key expiration window.

This article covers what happens after the tutorial ends: the concurrency race condition that breaks most implementations, the difference between idempotency and caching (they're not the same), how to handle key expiration without creating duplicates, and a production-ready implementation you can actually ship.

---

Why Idempotency Keys Exist

The problem is simple. The solution is not.

Imagine a checkout flow:

1. User clicks "Pay"
2. The request hits your server
3. Your server calls Stripe
4. The network hangs — no response for 10 seconds
5. The user clicks "Pay" again
6. Now your server has two requests in flight for the same charge

Without idempotency, both requests hit Stripe. Both succeed. The user is charged twice.

This isn't a bug — it's normal behavior in distributed systems. Networks fail, load balancers retry, mobile apps auto-retry on reconnect, and webhook providers all re-deliver on failure because they assume you handle it.

Idempotency means: send the same request multiple times, but process it only once.

---

The Tutorial Version (And Why It Breaks)

// Server-side: the "simple" version
async function handleCharge(req, res) {
  const key = req.headers['idempotency-key'];

  // Check if we've seen this key before
  const existing = await db.getIdempotencyKey(key);
  if (existing) {
    return res.json(existing.result);
  }

  // Process the payment
  const result = await stripe.charges.create(req.body);

  // Store the result
  await db.setIdempotencyKey(key, result);
  return res.json(result);
}

Looks correct. It's not. Here are the four ways this breaks.

---

Trap #1: The Concurrency Race Condition

Two identical requests arrive at the same time:

Request A: check key → not found → proceed to charge
Request B: check key → not found → proceed to charge
Both charge the user. Both store results. Duplicate charge.

This is a classic check-then-act race condition. It doesn't matter how fast your database is — if two requests arrive within the same event loop tick (or on different server instances), both pass the check.

The Fix: Atomic Key Claims

You need an atomic "insert if not exists" operation. Only one request wins the race.

PostgreSQL:

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  response JSONB,
  status_code INT,
  business_signature TEXT,
  created_at TIMESTAMP DEFAULT NOW()
);

Then use INSERT ... ON CONFLICT to make the insert atomic:

async function handleCharge(req, res) {
  const key = req.headers['idempotency-key'];
  const payload = req.body;
  const signature = extractBusinessSignature(payload);

  // Attempt to claim the key atomically
  const inserted = await db.query(
    `INSERT INTO idempotency_keys (key, business_signature)
     VALUES ($1, $2)
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, signature]
  );

  if (inserted.rowCount === 0) {
    // Key already claimed — fetch and return the stored result
    const existing = await db.query(
      `SELECT status_code, response FROM idempotency_keys WHERE key = $1`,
      [key]
    );
    return res.status(existing.rows[0].status_code).json(existing.rows[0].response);
  }

  // We won the race — process the payment
  try {
    const result = await stripe.charges.create(payload);
    await db.query(
      `UPDATE idempotency_keys SET status_code = 200, response = $1 WHERE key = $2`,
      [JSON.stringify(result), key]
    );
    return res.json(result);
  } catch (err) {
    await db.query(
      `UPDATE idempotency_keys SET status_code = $1, response = $2 WHERE key = $3`,
      [err.statusCode || 500, JSON.stringify({ error: err.message }), key]
    );
    return res.status(err.statusCode || 500).json({ error: err.message });
  }
}

Whatever you use, the operation must be atomic. A check-then-insert pattern will always have a race condition. Other options:

• Redis: SET key value NX EX 86400 — atomic create-if-not-exists with TTL
• MongoDB: unique index + updateOne with upsert
• DynamoDB: PutItem with attribute_not_exists(idempotencyKey)

---

Trap #2: Idempotency ≠ Caching

It's tempting to think: "If a request failed, just return the same failure on retry." But that's wrong.

Consider these scenarios:

Scenario	What happened	What retry should do
Request never reached your server	Network dropped before arrival	Process normally — it's a genuinely new request
Request reached server, processing started but failed	Payment provider returned 402 (card declined)	Return stored 402 — the charge was attempted
Request reached server, outcome unknown	Network timeout to payment provider	Retry with same key — check provider state first

The critical distinction: store every deterministic response (success or known failure), but don't store unknown outcomes. If the payment provider timed out, you don't know whether the charge went through. A retry must check the provider's state — not assume it failed.

// After calling the payment provider
if (result.status === 'succeeded' || result.status === 'declined') {
  // Known outcome — store it
  await storeResult(key, result);
} else if (result.status === 'timeout' || result.status === 'unknown') {
  // Unknown outcome — DON'T store, let retry re-check
  // The retry will hit the provider's status endpoint
  const providerResult = await stripe.charges.retrieve(chargeId);
  await storeResult(key, providerResult);
}

---

Trap #3: Key Reuse With Different Payloads

If someone sends the same key with different payment details, you must reject it. A business signature captures only the immutable fields — not timestamps, request IDs, or metadata:

function extractBusinessSignature(payload) {
  // Only include immutable business fields
  return JSON.stringify({
    amount: payload.amount,
    currency: payload.currency,
    customerId: payload.customerId,
    description: payload.description,
  });
}

When a key is reused, compare the signature:

if (existing && existing.businessSignature !== currentSignature) {
  return res.status(409).json({
    error: 'Idempotency key reused with different payment details'
  });
}

This prevents silent mismatches. The 409 tells the client: "You've already used this key for a different payment. Generate a new one."

---

Trap #4: Key Expiration Creates Duplicates

Idempotency keys shouldn't live forever, but if a key expires while a payment is still processing, a retry creates a brand-new charge. Duplicate.

The Fix: Expiration Must Outlive the Transaction

Set your TTL long enough to cover payment processing time, the client retry window, and any provider-side settlement. 24 hours is a safe default.

// Redis with 24-hour TTL
await redis.set(`idem:${key}`, JSON.stringify(result), 'NX', 'EX', 86400);

And never delete a key just because the checkout session ended. Let the TTL handle it. Manual deletion is how you create exactly the bug you're trying to prevent.

---

The Client Side: One Key Per Intent

// ❌ WRONG: New key on every click
const handlePay = () => {
  const key = crypto.randomUUID(); // generates a new key every time!
  fetch('/api/charge', {
    method: 'POST',
    headers: { 'Idempotency-Key': key },
    body: JSON.stringify(payload)
  });
};

// ✅ RIGHT: One key per checkout session
function CheckoutForm() {
  const idempotencyKeyRef = useRef(crypto.randomUUID()); // generated once

  const handlePay = () => {
    fetch('/api/charge', {
      method: 'POST',
      headers: { 'Idempotency-Key': idempotencyKeyRef.current },
      body: JSON.stringify(payload)
    });
  };
}

Where the key lives depends on your flow:

Storage	Use When
React state/ref	Single-page checkout, no reloads expected
sessionStorage	User might reload the page mid-checkout
Server-generated key	Most robust — key lives in your checkout session DB

Server-generated keys are the gold standard because they survive tab closes, browser crashes, and back-button navigation. Your server creates the key when the checkout session starts, and the client just reuses it.

---

The Full Production Flow

CHECKOUT LOADS
  ↓
Generate idempotency key (once, server or client)
  ↓
User clicks "Pay"
  ↓
POST /api/charge (Idempotency-Key header)
  ↓
SERVER: Atomic insert key into DB
  ├─ Insert succeeded → process payment → store result → return
  └─ Insert failed (key exists) → fetch stored result → return
  ↓
If network timeout: retry with SAME key
  ↓
Result: exactly one charge, regardless of retries

---

How to Test It

Don't just hope it works. Prove it. The test is simple: send the same key twice and verify your payment provider is only called once.

const mockCharge = jest.fn().mockResolvedValue({ id: 'ch_123', status: 'succeeded' });

test('idempotent charge: second request returns cached result', async () => {
  const key = 'test_order_123';
  const payload = { amount: 2000, currency: 'USD', customerId: 'cus_abc' };

  // First request — processes payment
  const res1 = await request(app)
    .post('/charge')
    .set('Idempotency-Key', key)
    .send(payload);

  // Second request — should return cached result
  const res2 = await request(app)
    .post('/charge')
    .set('Idempotency-Key', key)
    .send(payload);

  expect(res2.body).toEqual(res1.body);
  expect(mockCharge).toHaveBeenCalledTimes(1); // 🔑 the critical check
});

If your mock gets called twice, your idempotency is leaking. Fix it before production.

Test concurrency too — fire two requests simultaneously and verify only one charge is created:

test('concurrent requests: only one charge created', async () => {
  const key = 'concurrent_test_456';
  const payload = { amount: 5000, currency: 'USD', customerId: 'cus_xyz' };

  const [res1, res2] = await Promise.all([
    request(app).post('/charge').set('Idempotency-Key', key).send(payload),
    request(app).post('/charge').set('Idempotency-Key', key).send(payload),
  ]);

  expect(mockCharge).toHaveBeenCalledTimes(1);
  expect(res1.body).toEqual(res2.body);
});

---

Troubleshooting FAQ

Q: I'm getting duplicate charges even with idempotency keys. What's wrong?

Check three things: (1) Is your key generated once per intent, not per click? (2) Does your database enforce a unique constraint on the key? (3) Is your key store shared across all server instances? If you're using in-memory storage in a multi-instance deployment, each instance has its own key store — and duplicates slip through.

Q: Should I store failed payment results too?

Yes. If the payment provider returns a deterministic failure (card declined, insufficient funds), store that result. A retry with the same key should return the same failure — not attempt a new charge. The only exception: unknown outcomes (timeouts). Those should let the retry re-check the provider state.

Q: How long should idempotency keys live?

24 hours is the standard. It covers processing time, the user's retry window, and any delayed settlement. Shorter TTLs risk creating duplicates on retries; longer TTLs waste storage. For non-payment operations (webhooks, email), 1 hour is usually sufficient.

Q: What if my payment provider already supports idempotency (like Stripe)?

Stripe's idempotency only protects you at the Stripe level. If your server calls Stripe twice with two different keys, Stripe processes both charges. You still need your own idempotency layer to prevent duplicate requests from reaching Stripe.

Q: Can I use the same key across different API endpoints?

No. Scope keys to specific operations. Use prefixes: pay_ for charges, ref_ for refunds, email_ for sends.

Q: What about security — can keys leak information?

Yes. Always send keys over HTTPS, avoid logging full keys in plaintext, and hash or truncate them in observability tools.

---

The Checklist

Before shipping any state-changing endpoint:

• [ ] Key generated once per intent, not per click
• [ ] Key persisted across retries (not lost on page reload)
• [ ] Backend checks key before processing
• [ ] Response stored and reused on duplicate requests
• [ ] Business signature validates payload consistency
• [ ] Database enforces a unique constraint (not just a check)
• [ ] Key store is shared across all server instances
• [ ] Expiration policy defined (24h default)
• [ ] Unknown outcomes (timeouts) trigger re-checks, not cached errors
• [ ] Concurrency tested — two simultaneous requests, one charge

Miss one of these? You don't have idempotency. You have hope.

The double-tap is coming. Be ready for it.