Network calls fail in ways that are fundamentally ambiguous to the caller. A request can fail before it reaches the server, in which case retrying is obviously safe. It can also fail after the server has fully processed it but before the response makes it back to the client, in which case the caller has no way to distinguish that outcome from a request that never arrived at all. Idempotency is what makes it safe to retry in the second case without causing duplicate effects, and designing for it deliberately, rather than assuming retries are inherently safe, is what separates APIs that hold up under real network conditions from ones that quietly create duplicate charges, duplicate orders, or duplicate records whenever a client retries after an ambiguous failure.
Why "the operation looks safe to retry" is often a false assumption
A request that reads data is naturally idempotent, retrying a read has no side effect beyond returning the same data again. The risk concentrates specifically in requests that create or modify state, submitting a payment, creating a record, sending a notification. For these operations, a naive retry strategy, simply resending the exact same request if no response was received, can result in the operation executing twice if the original request actually succeeded server-side but the response was lost in transit.
This failure mode is easy to underestimate because it requires a specific, if not rare, sequence of events, and doesn't show up in normal testing where network conditions are typically clean. It shows up in production, often intermittently, under real network conditions where timeouts and dropped connections are a normal, if infrequent, occurrence.
Idempotency keys are the standard solution, and they need to be designed carefully
The common pattern for solving this is having the client generate a unique idempotency key for each logical operation, and including that key with the request. The server checks whether it has already processed a request with that specific key and, if so, returns the original result rather than executing the operation again. This shifts the responsibility for expressing "this is the same logical operation, even if it's arriving as a second network request" explicitly into the request itself, rather than relying on the server trying to infer sameness from the request payload alone.
A few design details matter more than they might initially appear to. The key needs to be generated once per logical operation and reused across retries of that specific operation, not regenerated on each retry attempt, since a new key each time defeats the entire purpose. The server needs to store enough information about a processed key to return the original result on a duplicate request, not just a flag indicating the key was seen, since simply rejecting the duplicate without returning the original result leaves the client without the information it actually needs. And the stored key records need a reasonable expiration policy, since retaining every idempotency key indefinitely is rarely necessary and can become a meaningful storage cost at scale.
Database-level idempotency versus application-level idempotency
Idempotency can be enforced at different layers, and where it's enforced affects how reliable the guarantee actually is under concurrent conditions. Checking for a duplicate key at the application level, in code, before performing the underlying database operation, is vulnerable to a race condition if two retries of the same request happen to arrive at nearly the same moment, both checks might complete before either request has finished writing its result, leading both to proceed as if it were the first attempt.
A more robust approach enforces uniqueness at the database level itself, typically through a unique constraint on the idempotency key column, so that even under concurrent retries, only one request can successfully insert the record, and the other reliably fails with a constraint violation that the application layer can catch and handle by returning the original result. This database-level guarantee is significantly more reliable than an application-level check alone under genuine concurrent access.
Not every operation needs a client-generated key
For operations that are naturally idempotent based on their own semantics, setting a resource to a specific state rather than incrementing it, for example, explicit idempotency keys are often unnecessary, since the operation produces the same end state regardless of how many times it's applied. The client-generated idempotency key pattern is specifically valuable for operations that would otherwise have a cumulative or creation-based effect if repeated, where the operation's natural semantics don't already provide idempotency on their own.
Recognizing which category a given operation falls into avoids over-engineering idempotency handling for operations that don't structurally need it, while correctly identifying and protecting the operations that genuinely do.
Idempotency needs to extend through the full chain of downstream effects
An operation that triggers downstream side effects, sending an email notification, triggering a webhook to another system, needs those downstream effects considered as part of the idempotency design, not just the primary database write. An idempotency key that correctly prevents a duplicate database record but still triggers a duplicate notification on retry has only solved part of the problem, since the user experience of receiving the same email twice is itself a real, visible failure even if the underlying data is correct.
This means idempotency handling often needs to wrap the entire logical operation, including its side effects, rather than being scoped narrowly to just the primary write, which requires deliberately thinking through what the complete set of effects a given operation triggers actually includes.
The underlying discipline
Idempotent design isn't primarily about clever technical implementation, the actual mechanisms, unique keys, database constraints, are relatively well understood. The discipline that actually matters is systematically identifying which operations in an API are state-changing and therefore need explicit idempotency protection, and consistently applying that protection rather than assuming network retries are rare enough to not warrant the design effort. They aren't rare in any system operating at meaningful scale over a long enough period, which means the question isn't really whether an unprotected retry will eventually cause a duplicate effect, but when.
Top comments (0)