DEV Community

Valipireddy Kowshik
Valipireddy Kowshik

Posted on

Retries Are a Distributed Systems Bug Waiting to Happen: Designing Idempotent APIs

#api #architecture #backend #systemdesign

Every network call can fail in three ways: it fails, it succeeds, or it succeeds and you never find out. The third case is where most production incidents live.

Junior instinct: "Request timed out, retry it."
Senior question: "What if the first one actually went through?"

The core problem 

Client                 Server                  DB
  | --- POST /charge --> |                      |
  |                      | --- INSERT charge -> |
  |                      | <------- OK -------- |
  |   X (response lost)  |                      |
  | --- POST /charge --> |   (retry)            |
  |                      | --- INSERT charge -> |   <- duplicate
Enter fullscreen mode Exit fullscreen mode

The client can't tell "never arrived" from "arrived, reply lost." Retrying blindly means double charges, duplicate orders, double emails.

Fix 1: Idempotency keys

The client generates a unique key per logical operation and sends it on every attempt. The server stores the result and replays it.

Client                       Server
  | POST /charge                |
  | Idempotency-Key: abc-123 -> | not seen: process, store result
  |  X (response lost)          |
  | POST /charge                |
  | Idempotency-Key: abc-123 -> | seen: return stored result, no new charge
Enter fullscreen mode Exit fullscreen mode 
app.post("/charge", async (req, res) => {
  const key = req.get("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });

  // Atomic claim: unique constraint on key
  const claimed = await db.idempotency.insertIfAbsent({
    key, status: "in_progress", requestHash: hash(req.body),
  });

  if (!claimed) {
    const existing = await db.idempotency.get(key);
    if (existing.requestHash !== hash(req.body))
      return res.status(422).json({ error: "Key reused with different payload" });
    if (existing.status === "in_progress")
      return res.status(409).json({ error: "Request in progress" });
    return res.status(existing.code).json(existing.response); // replay
  }

  const result = await chargeCard(req.body);
  await db.idempotency.complete(key, 200, result);
  res.json(result);
});
Enter fullscreen mode Exit fullscreen mode

Details that matter:

  • Make the claim atomic (unique constraint, not check-then-insert).
  • Hash the payload. Same key with a different body is a client bug, so reject it.
  • Handle the in-progress state so concurrent retries don't race.
  • Set a TTL (24-72h is common).

Fix 2: Retry correctly

Retries without backoff turn a blip into an outage (the retry storm).

Attempt:  1    2      3         4              5
Wait:     0 -> 1s -> 2s ------> 4s -----------> 8s   (exponential)
               +-jitter: random within each window
Enter fullscreen mode Exit fullscreen mode 
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function retry(fn, { retries = 5, base = 200, cap = 10_000 } = {}) {
  for (let i = 0; ; i++) {
    try { return await fn(); }
    catch (err) {
      if (i >= retries || !isRetryable(err)) throw err;
      const backoff = Math.min(cap, base * 2 ** i);
      await sleep(Math.random() * backoff); // full jitter
    }
  }
}
Enter fullscreen mode Exit fullscreen mode
  • Jitter is not optional. Without it, every client retries in lockstep.
  • Only retry transient errors (timeouts, 429, 503), never 400/401/404.
  • Respect Retry-After. Put a retry budget at the service level.

Fix 3: The dual-write problem 

await db.save(order);        // succeeds
await queue.publish(event);  // crashes here -> DB and queue disagree
Enter fullscreen mode Exit fullscreen mode

You can't make two systems commit atomically. Use the transactional outbox:

+---------- one DB transaction ----------+
|  INSERT order       INSERT outbox_event |
+-----------------------------------------+
              |
     relay polls outbox -> publishes to queue -> marks sent
Enter fullscreen mode Exit fullscreen mode

This gives at-least-once delivery, so consumers must be idempotent too (dedupe by event ID).

What I'd standardize on

Layer Rule
Write endpoints Require Idempotency-Key
Clients Exponential backoff + full jitter + cap
Messaging Outbox + idempotent consumers
Semantics Assume at-least-once, design for it

Takeaway

"Exactly-once" is mostly a myth. What you actually build is at-least-once delivery plus idempotent processing, which behaves like exactly-once from the outside.

Design every write as if it will be sent twice, because eventually it will be.

How does your team handle idempotency? Keys, natural dedupe, or hope? Let me know in the comments. 👇

Top comments (1)

Collapse
 
launchgatecheck profile image
Launch Gate •

The in-progress state is the part most idempotency writeups skip, and it matters. One gap in the handler as written: if the process dies after the claim but before complete(), the key stays in_progress and every retry gets 409 until the TTL. A lease timestamp on the claim, so a stale in-progress row can be taken over after a bounded time, closes it. Also worth deciding whether a failed charge stores its error for replay or releases the key, since retrying a deterministic failure with the same key should not behave differently from a transient one.