Idempotency: Why "Just Retry It" Breaks More Than It Fixes

Ifeoluwa Obayemi — Wed, 15 Jul 2026 16:48:01 +0000

A payment request goes out. The network hiccups. The client never sees a response, so it does the reasonable thing and retries. Thirty seconds later, a customer gets charged twice for one order, and someone on support is trying to explain to them why.

Nobody wrote a bug here. The client behaved exactly the way retry logic is supposed to behave. The problem is one layer deeper: the system let a "did this actually happen" question get answered twice, and each answer was "yes, charge them."

That's idempotency, or rather, the absence of it. It's one of those concepts that's simple to state and easy to get wrong in practice, and once you've been burned by it once, you start seeing the gap everywhere.

Why retries are unavoidable, not optional

In a distributed system, a timeout doesn't tell you the request failed. It tells you that you didn't get a response in time. The request might have never arrived. It might have arrived, succeeded, and the response got lost on the way back. From the caller's side, those two situations are indistinguishable, and the only reasonable move is to retry.

That's not a design flaw you can engineer away. Networks drop packets, servers restart mid-request, load balancers time out connections. Any system that talks to another system over a network will eventually face this exact ambiguity. Which means the real question isn't "should we retry." It's "what happens when we do."

What idempotency actually means

An operation is idempotent if doing it once and doing it five times leave the system in the same state. SET balance = 100 is idempotent. Run it once or five times, the balance ends up at 100 either way. ADD 100 to balance is not. Run that five times and you've added 500, which is exactly the double-charge scenario from the top of this article, just with extra zeros.

Notice these two operations can express the same intent ("this balance should reflect a $100 deposit") in a way that's either safe to retry or actively dangerous to retry, depending purely on how you phrased it. That's the part that catches people off guard. Idempotency isn't really about retries as a mechanism. It's a property of how you designed the operation in the first place.

Where this shows up whether you notice it or not

HTTP. The spec has opinions here. GET, PUT, and DELETE are supposed to be idempotent. POST is not, which is exactly why "just retry the POST" is the instinct that causes duplicate orders, duplicate emails, and duplicate charges. If your API exposes a POST /charges endpoint with no other protection, every client retry is a coin flip on whether the customer gets billed twice.

Message queues. Most queues (SQS, Kafka, RabbitMQ, pick one) give you at-least-once delivery, not exactly-once, because exactly-once across a network is a much harder guarantee than it sounds like and most systems don't actually need it if they handle duplicates correctly downstream. That "at-least-once" part means your consumer will, eventually, see the same message twice. If processing that message means charging a card or sending an email, your consumer needs to be the one enforcing idempotency, because the queue isn't going to do it for you.

Database writes. An INSERT fails if you run it twice against a unique key, which is annoying until you realize that's actually a gift: it turned an ambiguous "did this insert happen" question into a clear error you can catch and treat as "yes, it already happened." An UPSERT goes further and just makes the operation naturally idempotent from the start.

Making an operation idempotent on purpose

The standard fix is the idempotency key: the client generates a unique ID for a logical attempt (a UUID is fine) and sends it along with the request. The server keeps a record of keys it's already processed. Before doing any real work, it checks: have I seen this key before? If yes, return the same result as last time without processing anything twice. If no, do the work and record the key as part of the same operation.

That last part, "as part of the same operation," is where most naive implementations quietly break. If you process the charge and then write the idempotency key in a separate step, you've just built a smaller version of the same problem: a crash between those two steps means a retry sees no record of the key and processes the charge again. The fix is to make the write and the key-check atomic, typically by writing both inside the same database transaction, or by using the database's own uniqueness constraints to reject the duplicate outright.

A rough shape of the check, language aside:

begin transaction
  if idempotency_key exists in processed_requests:
      return stored_response for that key
  else:
      result = do_the_actual_work()
      insert (idempotency_key, result) into processed_requests
      return result
commit transaction

The specifics vary by stack, but the shape doesn't: check and record have to succeed or fail together, or you haven't actually solved anything.

The gotchas that show up later

Scope and expiration. An idempotency key needs a defined lifetime and a defined scope (per customer, per endpoint, whatever fits). Keep keys around forever and your dedupe table grows without bound. Expire them too aggressively and a slow retry (say, a client that waited ten minutes to retry after some other failure) sails right past the check and processes twice anyway.

Partial success. Sometimes the original request actually succeeded, but something else downstream failed, like the confirmation email. A well-meaning retry of "the whole operation" might redo the parts that worked fine the first time. This is usually the sign that a single "operation" needs to be broken into smaller idempotent steps, each individually safe to retry, rather than treated as one big all-or-nothing block.

Client-generated keys done wrong. If the client regenerates a new key every time it retries (instead of reusing the same key for the same logical attempt), you've disabled the entire mechanism without any errors to tell you so. This is a surprisingly common bug, usually introduced by whatever auto-retry wrapper the client's HTTP library uses by default.

The actual takeaway

Idempotency isn't a resilience feature you bolt on after the fact. It's a decision that belongs in the API contract, right next to the request and response shapes. Every endpoint that can plausibly cause a side effect (money moving, an email sending, a record getting created) is a candidate for "what happens if this gets called twice," and that question is a lot cheaper to answer at design time than at 3 AM when support is asking why a customer got charged twice for one order.

DEV Community: Ifeoluwa Obayemi

Idempotency: Why "Just Retry It" Breaks More Than It Fixes