A network blip happens mid-checkout. The client never receives the response, so it retries the request, doing exactly what good client code is supposed to do. The server, meanwhile, already processed the first request successfully, it just never got to say so. Now the customer has been charged twice, and nobody wrote a single line of buggy code to make that happen. The system worked exactly as designed. The design just never accounted for this case.
This is one of the most common, quietly expensive gaps in API design, and the fix has a name most developers have heard of but far fewer have actually implemented correctly: idempotency keys. Getting this right consistently is exactly the kind of thing that comes up constantly in real API development and custom API development work, since it's rarely obvious until a customer notices a duplicate charge on their statement.
This gap shows up most visibly in payment processing, but it's just as relevant across API integration work generally, anywhere a client and server communicate over a network that can fail mid-request. It's also a recurring theme in backend development and cloud infrastructure work, since distributed systems with multiple service hops multiply the number of places a retry can originate from.
Teams building AI-powered features are running into this too, particularly around AI API integration and AI API development, where a single user action can trigger a chain of API calls to external model providers, and a dropped connection partway through that chain creates exactly the same ambiguity described above, just with a more expensive retry if it re-triggers a paid inference call. This overlaps with broader AI infrastructure and enterprise AI architecture work as well, since reliable request handling becomes a foundational requirement the moment AI features touch anything with a real cost or side effect attached.
None of this is exotic distributed-systems trivia reserved for large platforms. It's foundational software architecture work, the kind that shows up in fintech software development and healthcare software projects just as often as in a standard SaaS product, anywhere a request changing real state might ever be retried.
Here's what's actually happening under the hood, why it's so easy to miss, and how to build the fix properly.
Why This Happens More Often Than You'd Think
Network failures aren't rare edge cases, they're a routine part of how distributed systems actually behave. A request can fail in ways that leave genuine ambiguity about what actually happened.
- The request never reached the server at all. Retrying is completely safe here
- The request reached the server, was processed successfully, but the response never made it back to the client. Retrying here re-executes the operation, which is exactly the dangerous case
- The request reached the server and failed partway through processing. Retrying might be safe, or might not, depending on what state was left behind
From the client's perspective, all three of these look identical: a timeout with no response. Without a mechanism to distinguish "this never happened" from "this already happened, you just didn't hear back," a naive retry can't tell the difference, and it will happily re-execute an operation that should only ever happen once.
What an Idempotency Key Actually Does
An idempotency key is a unique identifier the client generates and attaches to a request, ensuring the server can safely deduplicate that request no matter how many times it's retried.
POST /api/charges
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7
{
"amount": 4999,
"currency": "usd",
"customer_id": "cus_abc123"
}
The server's job is simple in concept: the first time it sees a given idempotency key, it processes the request normally and stores the result against that key. Every subsequent request carrying the same key returns the stored result from the first attempt, without re-executing the underlying operation at all.
Building This Properly, Step by Step
Step one: generate the key on the client, not the server
The key needs to represent "this specific attempt at this specific operation" from the client's perspective. A UUID generated once per logical operation, stored locally until the operation is confirmed complete, works well. Generating it server-side defeats the purpose entirely, since the server can't distinguish a retry from a new request if it's the one assigning the identifier.
Step two: store the key alongside the result, not just a boolean flag
CREATE TABLE idempotency_keys (
key VARCHAR(255) PRIMARY KEY,
request_hash VARCHAR(64) NOT NULL,
response_status INT,
response_body JSONB,
created_at TIMESTAMP DEFAULT NOW(),
expires_at TIMESTAMP
);
Storing the actual response, not just "this key was seen before," matters because a retry needs to return the same result the original request would have, not just a generic "already processed" message.
Step three: hash the request payload alongside the key
A subtle but important detail: if a client reuses an idempotency key with a genuinely different request body, that's not a safe retry, it's a client bug or a conflict that needs a clear error, not a silently wrong cached response.
def handle_request(idempotency_key, request_body):
request_hash = hash(request_body)
existing = lookup_key(idempotency_key)
if existing:
if existing.request_hash != request_hash:
return error_response(422, "Idempotency key reused with different payload")
return existing.response_body, existing.response_status
result = process_request(request_body)
store_key(idempotency_key, request_hash, result)
return result
Step four: handle the in-flight case explicitly
A retry can arrive while the original request is still being processed, not yet complete. Without handling this, a fast enough retry can still slip through and execute twice.
def handle_request(idempotency_key, request_body):
lock = acquire_lock(idempotency_key)
if not lock:
return error_response(409, "Request with this key is already being processed")
try:
# existing lookup and processing logic here
...
finally:
release_lock(idempotency_key)
Step five: set a sensible expiration on stored keys
Keeping idempotency records forever isn't necessary and adds unbounded storage growth. A window of 24 hours to a few days is typical, long enough to cover realistic retry scenarios, short enough to keep storage manageable.
Where This Matters Most
- Payment processing. The canonical example, and the one with the most visible, painful consequences when it's missing
- Order creation and inventory reservation. A duplicate order or a double inventory deduction causes real operational headaches downstream
- Any operation with a side effect that isn't naturally idempotent. Sending an email, triggering a webhook, creating a resource, all of these can go wrong on an unintended retry without this protection
- Distributed systems with retry logic at multiple layers. The more hops a request takes, the more places a retry can originate from, and the more important it becomes that the final effect only happens once
- AI-powered features calling paid external APIs. A retried request that re-triggers a billed inference call costs real money on top of the reliability problem itself
Common Mistakes Worth Avoiding
- Generating the idempotency key on the server instead of the client, which defeats its entire purpose
- Storing only a "seen before" flag instead of the actual response, so retries can't return a correct, consistent result
- Forgetting to handle the case where a retry arrives while the original request is still mid-flight
- Never expiring stored keys, leading to unbounded storage growth over time
- Assuming idempotency keys are only relevant for payments, when any operation with a meaningful side effect benefits from the same protection
A Quick Self-Check
- Does your API currently distinguish between "this request is new" and "this request is a retry of something already processed"?
- If a client's network dropped right after your server successfully processed a request, what would happen on retry today?
- Are your idempotency keys, if you have them, generated by the client or the server?
- Do you have a defined, tested behavior for a retry that arrives while the original request is still being processed?
If any of these leave you uncertain, that's usually a sign this protection isn't fully in place yet, and it's worth testing deliberately rather than finding out the hard way in production.
The Takeaway
This isn't an exotic edge case reserved for large-scale distributed systems. Any API that changes state and might ever be called over an unreliable network, which is every API, benefits from this protection. The failure mode is quiet, it doesn't crash anything, it doesn't throw an obvious error, it just silently does something twice that should have happened once. That quietness is exactly what makes it worth building in deliberately rather than discovering after a customer notices a duplicate charge on their statement.
Does your API currently handle retries safely, or is this a gap you're realizing you should actually test for? Curious how many people have actually verified this rather than just assumed it works.
Top comments (0)