Idempotency is one of those concepts that sounds simple until you try to implement it under load. The idea — "running the same command twice has the same effect as running it once" — is easy to state. The edge cases are where systems break.
This post catalogues five common ways teams try to implement idempotency without actually solving the problem. Each one looks reasonable in isolation. Each one fails under a real failure mode.
What you will learn:
- Why client-side logs cannot prevent duplicates.
- Why upserts without payload digests silently corrupt results.
- Why locks and caches are not enough under concurrency.
- Why idempotency keys must detect changed intent.
- Why reusing the same key across retries is the whole point.
Repository: github.com/manjunath-hanmantgad/durable-ai-workflow-lab
1. The client checks its own log before retrying
The most intuitive fix is also the most wrong.
# Anti-pattern
if my_local_log.has(request_id):
return # assume it already succeeded
response = api.create_ticket(payload)
my_local_log.add(request_id, response)
Why it fails
The write and the acknowledgement are independent events. The API may have committed the ticket and then crashed, dropped the response, or lost the network before the client received it. The client's log only records what the client saw.
client -> API: create ticket
API -> database: INSERT
API -> client: [response lost]
client: "I never got a response, so it must have failed"
client -> API: create ticket -- duplicate!
A local log can never know whether a command was executed on the other side of a network partition. Only the receiver has that information.
2. Database upsert without a payload digest
The next attempt stores the idempotency key, but ignores the payload.
# Anti-pattern
INSERT INTO commands (command_key, result)
VALUES (?, ?)
ON CONFLICT (command_key) DO UPDATE SET result = EXCLUDED.result;
Why it fails
If the client reuses a command_key with a different payload, the upsert silently overwrites the previous result. The system now returns the result of a different command.
This is dangerous because it hides a client bug. A reused key with changed data should be a conflict, not an update. The receiver must verify that the payload matches before replaying a stored result.
The correct contract
same key + same payload -> return stored result
same key + different payload -> conflict
3. Application-level locking only
A common pattern in single-process applications is to guard the idempotency check with a mutex or lock.
# Anti-pattern
with lock:
if key in cache:
return cache[key]
result = api.create_ticket(payload)
cache[key] = result
return result
Why it fails
A lock protects one process. It does not protect against:
- Multiple worker processes on the same machine.
- Multiple machines behind a load balancer.
- A process crash between the cache write and the response.
- A restart that wipes the in-memory cache.
The only serialization primitive that survives all of these is a database constraint.
CREATE TABLE tickets (
command_key TEXT PRIMARY KEY,
payload_digest TEXT NOT NULL,
ticket_id TEXT NOT NULL
);
The PRIMARY KEY constraint is what makes ten concurrent retries race safely.
4. Idempotency keys without conflict detection
This is close to correct, but it misses one detail.
# Anti-pattern
if key_exists(key):
return get_result(key)
result = execute(payload)
store(key, result)
return result
Why it fails
The code checks for the key, but does not compare the stored payload with the incoming payload. If the client accidentally reuses a key with different data, the system returns the old result for the new intent.
This is especially bad in payment systems. A customer might reuse an old form submission ID, or a frontend bug might attach the wrong payload to a stable key. The receiver should detect the mismatch and reject the call.
The correct contract
stored = get(key)
if stored:
if stored.digest != incoming.digest:
raise ConflictError("key reused with different payload")
return stored.result
result = execute(payload)
store(key, incoming.digest, result)
return result
5. Retrying with a new key every time
Sometimes the "fix" is to generate a fresh idempotency key on every retry.
# Anti-pattern
for attempt in range(3):
key = str(uuid.uuid4()) # new key every attempt!
response = api.create_ticket(payload, idempotency_key=key)
Why it fails
This defeats the entire mechanism. Each retry looks like a new command to the receiver, so the receiver executes it again. The whole point of an idempotency key is that it identifies the logical operation across attempts.
The key must be generated once per logical operation and reused on every retry.
# Correct
key = str(uuid.uuid4()) # once per operation
for attempt in range(3):
response = api.create_ticket(payload, idempotency_key=key)
Summary table
| Anti-pattern | Fails when | Correct behavior |
|---|---|---|
| Client checks its own log | Response lost after commit | Receiver stores the authoritative result |
| Upsert without digest | Key reused with different payload | Compare digest, reject conflict |
| Application-level lock | Concurrent processes/machines/restarts | Database UNIQUE constraint |
| Keys without conflict detection | Key reused with different payload | Compare and reject mismatches |
| New key per retry | Any retry happens | Reuse the same key across retries |
The reference implementation
The durable-ai-workflow-lab repository implements the correct contract in ~200 lines of Python:
-
Command(key, payload)with a deterministic SHA-256 digest. -
MockReceiverbacked by SQLite withUNIQUE(command_key). - Five pytest proofs: lost-ack retry, concurrent duplicates, changed payload conflict, restart survival, and digest stability.
uv sync --frozen
uv run --frozen pytest -q
The full test suite runs in ~0.05 s.
When to apply this
Use receiver-side idempotency whenever:
- A command crosses a process or network boundary.
- The command has a side effect that is expensive or impossible to undo.
- The client will retry on timeout or error.
Common domains: payments, inventory, shipments, support tickets, subscriptions, bookings, webhook handlers.
Takeaway
Idempotency is not a client-side setting, a framework feature, or a nice-to-have. It is a contract between the client and the receiver. The client must supply a stable key. The receiver must atomically store the key, the payload digest, and the result. Anything less leaves a window for duplicates.
Which anti-pattern have you seen?
Most production systems I have seen hit at least two of these before they centralize the contract. If you have a war story about duplicate charges, double shipments, or corrupted retries, drop it in the comments.
Reference implementation: github.com/manjunath-hanmantgad/durable-ai-workflow-lab
Continue reading: Closing the Write/Ack Gap with Receiver-Side Idempotency

Top comments (0)