A client of mine kept seeing duplicate merged PDFs show up in a customer's output bucket, always in pairs, always a few seconds apart. The cause turned out to be exactly what it usually is: a request that timed out on the client side but had actually succeeded on the server side, followed by a naive retry that ran the whole operation again from scratch.
Why this happens more with PDF jobs than with typical API calls
A lot of API calls are fast enough that retry-on-timeout rarely causes visible duplication, the operation either genuinely fails or genuinely completes well within any reasonable timeout window. PDF operations don't always fit that pattern. A large merge or a heavy compression job can legitimately take a few seconds, long enough that a client-side timeout set for typical requests can fire on a request that's actually still running, or has already finished and is just slow to respond. The client sees a timeout, assumes failure, and retries. The server sees two full requests for the same logical operation.
What an idempotency key actually does
The fix is a pattern, not a specific PDF-related trick: the caller generates a unique key for each logical operation, not each HTTP request, and sends it with every attempt at that operation, retries included. The server checks that key before doing any work. If it's seen the key before and the operation completed, it returns the same result again without re-running anything. If the key is new, it runs the operation and stores the result against that key for some retention window.
import uuid
def merge_with_retry(files, max_attempts=3):
idempotency_key = str(uuid.uuid4()) # one key for this logical operation
for attempt in range(max_attempts):
try:
return pdf_api.run({
"action": "merge",
"files": files,
"idempotency_key": idempotency_key,
}, timeout=8)
except TimeoutError:
if attempt == max_attempts - 1:
raise
continue # same key on every retry
The key line is that the key is generated once, outside the retry loop, not regenerated on each attempt. Generating a new key per attempt defeats the entire mechanism, since the server has no way to recognize a retry as the same logical operation if every attempt looks like a brand new one.
What this actually saves you
Two things, and they're both worth naming separately. The obvious one is duplicate output: without idempotency, a retried merge produces two merged files instead of one, and now something downstream has to figure out which one is canonical, or worse, processes both. The less obvious one is duplicate cost. On a pay-per-successful-result pricing model, a retried operation that re-runs from scratch gets charged twice for one logical job, and at any real volume, that adds up to a bill that doesn't match what actually happened from the caller's point of view.
There's a third benefit that only shows up under load, and it's the one that convinced me this was worth doing everywhere rather than just on the operations that had already caused a visible problem: idempotency keys make it safe to retry aggressively. A client that isn't sure whether a request landed has two bad options without idempotency, wait indefinitely and risk hanging forever on a request that actually failed, or retry and risk duplicating a request that actually succeeded. With idempotency in place, that dilemma disappears, retrying promptly on any uncertain outcome becomes the correct default instead of a risky one, which in turn makes the whole system more resilient to ordinary network flakiness instead of more fragile.
Testing that the behavior actually holds
It's worth writing an explicit test for this rather than trusting that it works because the code looks right. The test that caught the most in practice was firing two concurrent requests with the same idempotency key and asserting that exactly one merge actually ran, not two requests that happened to return the same-looking result by coincidence:
def test_concurrent_requests_with_same_key_run_once():
key = str(uuid.uuid4())
results = run_concurrently([
lambda: pdf_api.run({"action": "merge", "files": FILES, "idempotency_key": key}),
lambda: pdf_api.run({"action": "merge", "files": FILES, "idempotency_key": key}),
])
assert results[0].output_url == results[1].output_url
assert count_merge_operations_logged(key) == 1
Sequential retries are the easy case to get right. Concurrent requests racing on the same key, which happens in practice whenever a client fires a retry before the original request has actually finished, is where a naive implementation, check-then-act without proper locking, tends to fall over.
Where the retention window matters
An idempotency key that's remembered forever is unnecessary overhead, and one that's forgotten too quickly reopens the exact problem it was meant to solve. A retry that arrives six minutes after the original request, because of a slow network or a client that backed off aggressively, needs the server to still recognize the key. A reasonable retention window, long enough to cover realistic retry delays, short enough not to accumulate indefinitely, is usually somewhere in the range of minutes to a couple of hours depending on how aggressive your retry logic is allowed to be.
The failure mode idempotency doesn't fix
Idempotency keys prevent duplicate execution of the same logical operation. They don't prevent a caller from treating two genuinely different operations as the same one by reusing a key across unrelated requests, which is a caller-side bug, not something the server can meaningfully protect against. The discipline has to live on both sides: the server needs to honor the key correctly, and the caller needs to generate a fresh key per logical operation and hold onto it correctly across retries of that specific operation, not longer and not for anything else.
Building this on top of the API
None of this requires the caller to build a distributed cache or a job-status database. The idempotency key gets passed straight through to an idempotent PDF API that handles merge, split, compress, rotate, watermark, and convert, and the server-side deduplication is the API's problem to solve, not the caller's. If your client code retries on timeout, and it should, check whether idempotency keys are part of that retry path before duplicate files and duplicate charges tell you they weren't.
Top comments (0)