Your API will be called twice. Here's how to make it run once.

The bug report that has no bug

Someone got charged twice for one order. The report lands with a screenshot and an angry emoji, and you start pulling threads expecting to find the broken line of code that did it.

You won't find it.

The payment service is fine. The tests pass. The review was thorough. What actually happened is almost boring: the client sent a charge, the response got eaten by a timeout, and something — a retry library, a proxy, an impatient user mashing the button — sent it again. Your correct, reviewed, well-tested service did exactly what it was told. Twice.

Here's the part that reframes everything: nobody made a mistake. The timeout was correct — the connection really did hang. The retry was correct — the alternative is dropping real requests every time the network sneezes. Executing a valid request was correct. The double charge didn't come from a bug. It came from three correct behaviors composing into one wrong outcome. That composition is the native failure mode of distributed systems, and idempotency is the thing that defuses it.

TL;DR — In any system that retries (so: every reliable system), duplicates aren't a bug you fix once, they're a delivery guarantee you handle forever. There are three ways to handle them: reshape the operation so repeats are harmless, hand the caller a key so you can detect and replay, or let the storage layer reject the repeat. None of them cover partial failures, zombie workers, or someone double-submitting from two tabs — so know which tool owns each.

Why duplicates are a law, not a bug

Strip the story down and you hit a hard limit that has nothing to do with your code. When a caller sends a request and gets nothing back, it cannot tell which of three worlds it's in:

1. The request never arrived.
2. The request arrived, and failed.
3. The request arrived, succeeded, and the response got lost on the way home.

Worlds 1 and 2 say "resend." World 3 says "whatever you do, don't." And the caller has no way to know which one it's living in. Same silence, three different correct reactions.

Given that, a caller has exactly two options: never retry (and drop real work every time a network blips — unacceptable for anything that matters), or retry and accept that sometimes it's re-sending something that already happened. Every serious system picks retries. That's why the whole industry's messaging guarantees bottom out at at-least-once delivery. "At-least-once" is just a polite way of saying duplicates are part of the contract.

Once duplicates are in the contract, handling them stops being defensive paranoia and becomes half of the protocol — your half, the receiver's half. Which turns one question into the most useful thing you can ask in a design review:

Not "could this get duplicated?" — everything can. The question is "what happens when it does?" And every state-changing endpoint needs a written answer.

The three roads to idempotency

Every practical fix reduces to one of three mechanisms. Roughly in order of how much you'll thank yourself later:

Prefer them left to right. Natural idempotency has no moving parts to run and no edge cases to debug. Keys add a dedupe store and a lifecycle. Conditional writes add precondition bookkeeping. But most real systems use all three, in different places — the actual skill is matching the road to the operation.

Road 1 — Make duplicates boring by design

Some operations are idempotent for free. Setting a value lands the same no matter how many times it runs. Deleting by ID is a no-op the second time. Inserting with a unique key just bounces off the constraint.

The skill is reshaping the operations that aren't. The move that matters most: increments become facts.

-- ❌ A duplicate literally doubles the money.
UPDATE accounts SET balance = balance + 50 WHERE id = 'acct_123';

-- ✅ A duplicate hits the unique constraint and no-ops.
INSERT INTO ledger_entries (id, account_id, amount)
VALUES ('txn_abc', 'acct_123', 50)
ON CONFLICT (id) DO NOTHING;
-- balance is now SUM(amount) over a set of de-duplicated facts.

That second version is why accountants invented the ledger centuries before we rediscovered it: don't store the running total, store the immutable events and derive the total. A replayed event is already in the set, so it changes nothing.

The same trick generalizes. "Append to a list" becomes "ensure this member is in the set." "Send a notification" becomes "record the intent, deliver from the record" — the insert of a unique-keyed intent row collapses duplicates, and a separate worker does the actual send once per row. The pattern underneath all of these is one sentence:

Split "decide" from "do." The decision becomes an idempotent fact you write down; the doing becomes a worker draining those facts. Most "un-idempotent-able" operations quietly surrender to this split.

Road 2 — Idempotency keys, done properly

Some things can't be reshaped into a set-and-forget. A charge, an order, a booking — these are requests with effects, and they need to stay that way. The standard tool here is the idempotency key: the caller generates a unique ID for the logical operation and sends it with every attempt. Stripe's API made this pattern famous, and the gap between "we have idempotency keys" and "our idempotency keys actually work" is entirely in the details.

Here's the shape of it — atomic claim, in-progress handling, stored outcome, payload binding, all in one handler:

async function charge(req, res) {
  const key = req.header("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });

  const fingerprint = sha256(canonicalize(req.body));

  // 1. Claim the key atomically. Whoever wins the INSERT does the work.
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, fingerprint, status)
     VALUES ($1, $2, 'in_progress')
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, fingerprint]
  );

  if (claimed.rowCount === 0) {
    // The key already exists → this is a retry (or an abuse).
    const prior = await db.one(
      `SELECT status, fingerprint, response FROM idempotency_keys WHERE key = $1`,
      [key]
    );
    if (prior.fingerprint !== fingerprint)
      return res.status(422).json({ error: "Idempotency-Key reused with a different body" });
    if (prior.status === "in_progress")
      return res.status(409).json({ error: "Original request still in flight — retry shortly" });
    return res.status(prior.response.status).json(prior.response.body); // replay the original
  }

  // 2. We own the key. Do the real work exactly once.
  //    Pass `key` downstream too, so the provider dedupes on its side.
  const result = await payments.charge({ ...req.body, idempotencyKey: key });

  // 3. Record the outcome so the retry can replay it verbatim.
  await db.query(
    `UPDATE idempotency_keys SET status = 'completed', response = $2 WHERE key = $1`,
    [key, { status: 201, body: result }]
  );
  return res.status(201).json(result);
}

Five things in that handler separate "correct" from "almost":

• The claim has to be atomic. Check-then-execute is a race: two concurrent retries both see the key missing and both charge. The atomic insert-if-absent (ON CONFLICT DO NOTHING, Redis SET NX, a DynamoDB conditional put) is what makes exactly one attempt the winner.
• Store the outcome, not just the fact. A dedupe store that only remembers "seen" leaves the retry with "yeah, something happened, no idea what." The caller needs the created order's ID, the charge confirmation — the actual result, replayed.
• Handle "in progress" on purpose. The first attempt is often still running when the retry shows up — that's literally why the retry showed up. You refuse the concurrent execution (409), you don't queue up a second one. (And a claim stuck in-progress because the worker died needs a lease with an expiry, after which you inspect real state before touching anything.)
• Bind the key to the payload. Same key, different body is either a client bug or an attack. Store a fingerprint of the request and reject mismatches loudly — don't cheerfully replay an unrelated response.
• Generate the key at the intent, not per HTTP call. The key identifies the logical action ("this checkout"), so it's minted where the intent is born — the user's session, the job row — and reused across every retry. Mint it inside your retry wrapper and you get a fresh key per attempt, which defeats the entire mechanism while looking like you implemented it.

One more that costs people real money: set the TTL by your business window, not your Redis bill. Keys have to outlive the longest realistic retry horizon — offline mobile clients, dead-letter replays — which is usually hours to days, not the five minutes that keeps the memory graph pretty. Expired-key duplicates are rare and genuinely miserable to debug. Err long.

Road 3 — Let the database say no

The third road pushes the rejection down into the storage layer, and it comes with a bonus: it's also the tool for idempotency's evil twin, the stale actor.

Conditional / versioned writes. Every mutation carries the version it read. A duplicate — or a concurrent writer — finds the version has moved and updates zero rows:

UPDATE documents SET body = $1, version = version + 1
WHERE id = $2 AND version = 41;
-- 0 rows? Someone (maybe you, a second time) already moved past 41.

State-machine guards. Encode the legal transition into the WHERE clause and the business state becomes the dedupe state:

UPDATE orders SET status = 'paid', paid_at = now()
WHERE id = $1 AND status = 'awaiting_payment';
-- 0 rows updated → already paid (or cancelled). Don't re-charge. You're done.

This is quietly better than a generic dedupe table for anything workflow-shaped, because it rejects not just exact duplicates but any out-of-order or stale transition, and it needs no extra store.

Fencing tokens are the one that saves you from the zombie. A worker holds a lease, pauses for a GC or a network partition, loses the lease to a successor... and then wakes up and keeps writing like nothing happened. Its writes aren't duplicates — they're stale originals, so idempotency keys are blind to them. The fix is a monotonically increasing token issued with the lease; every downstream write carries it, and the receiver rejects any token older than the highest it's seen. The zombie's writes carry an old token and get bounced.

The consumer side, and the "exactly-once" fairy tale

Event-driven systems put the exact same problem on the consumer. Brokers deliver at-least-once, so every consumer eats duplicates as a matter of routine: redeliveries after an unacked crash, rebalances, dead-letter replays.

And that shiny "exactly-once delivery" on the marketing page? Every time, under inspection, it turns out to be at-least-once delivery plus de-duplicated processing. The dedupe is either the platform's — inside its own transactional boundary, like Kafka transactions covering a read-process-write that stays inside Kafka — or, for any effect that leaves the platform (a database your transaction doesn't span, an email, an HTTP call), it's yours. There is no magic that makes a side effect on someone else's system exactly-once for free.

The workhorse pattern is the idempotent consumer: record the processed event ID in the same transaction as the state change it causes.

BEGIN;
  INSERT INTO processed_events (event_id) VALUES ($1);
  -- unique_violation? Already handled → ROLLBACK, ack the message, move on.
  UPDATE inventory SET reserved = reserved + 1 WHERE sku = $2;
COMMIT;

The load-bearing words are same transaction. Track processed IDs in Redis while you write state to Postgres and congratulations, you've rebuilt the dual-write bug inside your dedupe mechanism: crash between the two and the event is marked done but never applied, or applied but never marked. The processed-set lives next to the state it protects, atomically, or it's decoration.

(If the consumer's effect is itself an external call, you combine roads: record intent transactionally, pass an idempotency key downstream, let the downstream dedupe. It's roads 1 and 2 shaking hands.)

The edge cases that actually page you

You wire all this up, ship it, and then get paged anyway. This is usually where:

Partial failure inside the operation. Your handler does three things — write the DB, call the payment provider, emit an event — and dies after step two. The retry, deduped at the top-level key, tries to replay a stored response that never got stored, because the operation never finished. Recovery has to resume or roll back the partial work, which means the operation needs recorded progress — an outbox, a saga log — not just an all-or-nothing wrapper around the outside. Keys protect the entry; they don't make the insides atomic.

A downstream with no idempotency. You dedupe flawlessly, then call a vendor API that has never heard of idempotency keys. Your retry of a timed-out call to them is now the double charge, one hop over. The mitigations aren't pretty: use their mechanism if they have one, otherwise query-before-retry ("does a charge with my reference already exist?"), otherwise reconcile asynchronously and alert. Your system is only as idempotent as its most naive dependency. Audit them.

Semantic duplicates. Two requests, two different keys, same human intent — the user submitted from two tabs, each minting its own key. Key-based dedupe can't see this by design. The defense lives in the business layer: a uniqueness constraint on a natural key ("one active order per cart"), a short-window check, or UX that funnels the intent to a single path. Just know the infra layer provably can't catch this one.

Time-variant operations. "Apply this month's discount," retried across a month boundary, computes a different answer on replay. The fix is to capture the decision inputs at first attempt — the rate, the price — into the recorded intent, so a replay re-emits the original decision instead of re-deciding in a world that moved.

A checklist you can actually use

• Write the "what happens when this runs twice" answer for every state-changing endpoint and consumer, in the design doc. Make it a required review field.
• Reshape toward natural idempotency first: sets over increments, ledgers over balances, recorded-intent over fire-and-forget.
• Implement keys with all five pieces: atomic claim, stored outcome, payload binding, in-progress handling, business-window TTL. Four out of five is a latent incident.
• Put a version column or a state guard on every workflow table. It's idempotency and concurrency control in one line.
• Keep processed-event tracking in the same transaction as the state it guards. Different systems = the dual-write bug wearing a safety vest.
• Don't trust a platform's "exactly-once" past its own boundary. Everything external is your dedupe.
• Test it on purpose: a proxy that replays every Nth request in staging finds in an afternoon what production finds on a Saturday night.

The one thing to remember

Idempotency has a boring reputation — a checklist item, a header on a payments API. But it sits at the exact center of what makes a distributed system trustworthy. The network will lose responses. Callers will retry. Brokers will redeliver. Every one of those is correct behavior, and their composition will double-execute your operations unless every state-changing surface has a considered answer to the second arrival.

The craft is picking the right road per operation: reshape what you can, key what has to stay a request, guard what the storage layer can guard — and respect the edges each one leaves uncovered. Do it consistently and the double-charge screenshot never gets taken. Skip it in one place, and that's exactly where the timeout will land.

The network always finds the answer you didn't write down.

---

If you want the mental model underneath this — delivery semantics, replication, ordering, and why these patterns fall out of them rather than needing to be memorized — that's the through-line of distributed systems track. It builds from first principles up to exactly the production patterns above.

Your turn: what's the gnarliest duplicate you've had to hunt down — a double charge, a zombie worker, a two-tab double submit? And which road would've caught it? Drop it in the comments; the war stories are the best part.