Distributed systems fail in the most ordinary ways: a packet drops, a mobile connection flips, a load balancer times out, or a queue redelivers a message because your consumer was slow. In those moments, “retry” becomes the default survival mechanism. If you want a quick refresher, a compact primer on idempotency is a handy starting point, but the real work begins when you treat idempotency as a business-level contract: “This action must happen at most once, even if it is requested many times.”
Idempotency is often explained with HTTP verbs (GET is safe, PUT is idempotent, POST is not), but that framing is too small. In production, the problem is not HTTP—it’s duplicates. Duplicates come from timeouts, retries, at-least-once delivery, client bugs, race conditions, and human behavior (double taps, refreshes, back buttons). Your job isn’t to “avoid retries.” Your job is to build systems where retries are harmless.
Why duplicates are inevitable (even in “reliable” stacks)
Teams love to say they want “exactly-once.” In reality, exactly-once delivery is rare, expensive, and usually limited to narrow scenarios. Across boundaries—microservices, networks, payment processors, third-party APIs—what you get is uncertainty. A client sends a request. The server processes it. The client doesn’t receive the response. Did it succeed? The client cannot know. So the client retries. Now you have two requests that represent the same intent.
Even worse, modern platforms amplify duplicates:
Queues typically favor availability and will redeliver when acknowledgements are late.
Serverless runtimes can retry executions automatically under certain failure modes.
Client libraries can retry under the hood.
Reverse proxies may replay requests when connections reset.
In other words: duplicates aren’t an edge case. They are a normal operating condition. If your system treats duplicates as “shouldn’t happen,” you’re building a latent incident.
Idempotency vs deduplication vs “same result”
Here’s where teams get confused: idempotency isn’t “do nothing on retry.” It’s “repeating the same intent doesn’t produce additional side effects.”
That gives you three practical interpretations:
1) Idempotent outcome: The system ends in the same state, regardless of repeats.
Example: “Set subscription plan to Pro” can be applied multiple times without changing the final state.
2) Idempotent side effects: External actions happen once (charge once, ship once, email once).
Example: Payment capture must happen at most once for the same order intent.
3) Idempotent response: The client gets a stable answer for the same intent.
Example: The system returns the same order ID, even if the request was retried five times.
Deduplication is a tool you can use to implement idempotency (store request IDs, ignore repeats), but it’s not the goal. The goal is the business promise: “This user won’t be charged twice.”
The core pattern: identify intent, then make it safe
Idempotency becomes straightforward when you do two things consistently:
A) Give each business intent a unique identifier.
This can be an idempotency key generated by the client, a deterministic key derived from business data (order_id + action + amount), or a server-issued token tied to a workflow step.
B) Bind the side effects to that identifier using atomicity.
That means you must ensure the system can say: “I have already processed this intent,” in a way that remains true under concurrency and failures.
A highly readable pattern write-up is Martin Fowler’s explanation of the Idempotent Receiver idea, which focuses on uniquely identifying requests so duplicates can be safely ignored: Idempotent Receiver.
The trap is implementing this “logically” but not atomically. If you check “have I processed this?” and then later write “processed,” you can still double-execute under race conditions. The check and the claim must be one indivisible operation (transaction, conditional write, unique constraint, compare-and-set).
Where idempotency usually breaks in real products
Most incidents don’t come from “we forgot idempotency entirely.” They come from partial idempotency:
Payment or billing flows: You create the charge twice because you treat the gateway call as the only side effect, but you also create ledger entries, invoices, emails, or account credits separately.
Order creation + fulfillment: You dedupe “create order,” but the shipping label generation is triggered by an event that can be redelivered.
Async workflows: You make the HTTP endpoint idempotent, but the downstream consumer isn’t, so repeats reapply state transitions.
Compensations: You attempt to “undo” duplicates after the fact. That’s risky because compensation is often not symmetric (you can refund a charge, but you can’t undo a leaked email or a triggered KYC review cleanly).
This is why idempotency must be designed at the workflow level, not only at the API edge.
A practical checklist that prevents double side effects
- Define the intent boundary. Decide what “same request” means in business terms (same user action, same cart, same invoice, same amount, same merchant reference).
- Choose an idempotency key strategy. Client-generated keys are great for retries; deterministic keys are great for natural dedupe; server-issued tokens are great for multi-step flows.
- Make the “claim” atomic. Use a transaction, conditional write, or unique constraint so two concurrent attempts cannot both win.
- Store the result and return it on retry. Don’t just drop duplicates—return the prior order ID / status / response so the client can proceed without guessing.
- Set a sensible dedupe window. Keep keys long enough to cover real retry behavior (mobile networks, delayed queues), but not forever unless the domain demands it.
- Instrument duplicates as a signal. Count how often you receive the same key, where retries happen, and which clients are noisy—duplicates are telemetry, not just annoyance.
Notice what’s missing: “Tell clients not to retry.” In reliable systems, retries are assumed. The system is built to handle them.
Why “idempotent APIs” are how big systems survive retries
At scale, retries aren’t a minor implementation detail—they’re a fundamental mechanism to achieve availability. The engineering question becomes: how do you get the benefits of retry behavior without the risk of repeated side effects?
A clear, production-oriented explanation is AWS’s discussion of designing idempotent APIs specifically to make retries safe: Making retries safe with idempotent APIs. The core message aligns with hard-earned industry practice: if you assume retries, you must assume duplicates, and the API must define what happens when the same intent is submitted multiple times.
This matters because, in the real world, you cannot reliably distinguish “client retried because the first request failed” from “client retried because the response was lost.” Idempotency is what turns that uncertainty into predictable behavior.
The future-proof way to think about idempotency
If you’re building anything that involves money, inventory, permissions, identity, or user trust, idempotency is not optional polish. It is part of the product’s safety model. The most useful mindset is this:
Every operation that has irreversible or user-visible side effects must be safely repeatable.
That doesn’t mean every operation must be idempotent in the mathematical sense. It means the system must be able to recognize repeated intent and ensure the business outcome stays correct.
If you do this well, an entire category of incidents becomes boring. A timeout becomes “retry and move on,” not “run a war-room and reconcile broken state.” And that boring reliability is exactly what lets your product scale without turning every transient glitch into a customer-facing disaster.
Top comments (0)