DEV Community

Manjunath Hanamantgad
Manjunath Hanamantgad

Posted on Originally published at dev.to

5 Ways Idempotency Fails Without Receiver-Side Deduplication

Idempotency is one of those concepts that sounds simple until you try to implement it under load. The idea — "running the same command twice has the same effect as running it once" — is easy to state. The edge cases are where systems break.

This post catalogues five common ways teams try to implement idempotency without actually solving the problem. Each one looks reasonable in isolation. Each one fails under a real failure mode.

Receiver decision flow

What you will learn:

  • Why client-side logs cannot prevent duplicates.
  • Why upserts without payload digests silently corrupt results.
  • Why locks and caches are not enough under concurrency.
  • Why idempotency keys must detect changed intent.
  • Why reusing the same key across retries is the whole point.

Repository: github.com/manjunath-hanmantgad/durable-ai-workflow-lab

1. The client checks its own log before retrying

The most intuitive fix is also the most wrong.

# Anti-pattern
if my_local_log.has(request_id):
    return  # assume it already succeeded

response = api.create_ticket(payload)
my_local_log.add(request_id, response)
Enter fullscreen mode Exit fullscreen mode

Why it fails

The write and the acknowledgement are independent events. The API may have committed the ticket and then crashed, dropped the response, or lost the network before the client received it. The client's log only records what the client saw.

client -> API: create ticket
API -> database: INSERT
API -> client: [response lost]
client: "I never got a response, so it must have failed"
client -> API: create ticket  -- duplicate!
Enter fullscreen mode Exit fullscreen mode

A local log can never know whether a command was executed on the other side of a network partition. Only the receiver has that information.

2. Database upsert without a payload digest

The next attempt stores the idempotency key, but ignores the payload.

# Anti-pattern
INSERT INTO commands (command_key, result)
VALUES (?, ?)
ON CONFLICT (command_key) DO UPDATE SET result = EXCLUDED.result;
Enter fullscreen mode Exit fullscreen mode

Why it fails

If the client reuses a command_key with a different payload, the upsert silently overwrites the previous result. The system now returns the result of a different command.

This is dangerous because it hides a client bug. A reused key with changed data should be a conflict, not an update. The receiver must verify that the payload matches before replaying a stored result.

The correct contract

same key + same payload -> return stored result
same key + different payload -> conflict
Enter fullscreen mode Exit fullscreen mode

3. Application-level locking only

A common pattern in single-process applications is to guard the idempotency check with a mutex or lock.

# Anti-pattern
with lock:
    if key in cache:
        return cache[key]
    result = api.create_ticket(payload)
    cache[key] = result
    return result
Enter fullscreen mode Exit fullscreen mode

Why it fails

A lock protects one process. It does not protect against:

  • Multiple worker processes on the same machine.
  • Multiple machines behind a load balancer.
  • A process crash between the cache write and the response.
  • A restart that wipes the in-memory cache.

The only serialization primitive that survives all of these is a database constraint.

CREATE TABLE tickets (
    command_key TEXT PRIMARY KEY,
    payload_digest TEXT NOT NULL,
    ticket_id TEXT NOT NULL
);
Enter fullscreen mode Exit fullscreen mode

The PRIMARY KEY constraint is what makes ten concurrent retries race safely.

4. Idempotency keys without conflict detection

This is close to correct, but it misses one detail.

# Anti-pattern
if key_exists(key):
    return get_result(key)

result = execute(payload)
store(key, result)
return result
Enter fullscreen mode Exit fullscreen mode

Why it fails

The code checks for the key, but does not compare the stored payload with the incoming payload. If the client accidentally reuses a key with different data, the system returns the old result for the new intent.

This is especially bad in payment systems. A customer might reuse an old form submission ID, or a frontend bug might attach the wrong payload to a stable key. The receiver should detect the mismatch and reject the call.

The correct contract

stored = get(key)
if stored:
    if stored.digest != incoming.digest:
        raise ConflictError("key reused with different payload")
    return stored.result

result = execute(payload)
store(key, incoming.digest, result)
return result
Enter fullscreen mode Exit fullscreen mode

5. Retrying with a new key every time

Sometimes the "fix" is to generate a fresh idempotency key on every retry.

# Anti-pattern
for attempt in range(3):
    key = str(uuid.uuid4())  # new key every attempt!
    response = api.create_ticket(payload, idempotency_key=key)
Enter fullscreen mode Exit fullscreen mode

Why it fails

This defeats the entire mechanism. Each retry looks like a new command to the receiver, so the receiver executes it again. The whole point of an idempotency key is that it identifies the logical operation across attempts.

The key must be generated once per logical operation and reused on every retry.

# Correct
key = str(uuid.uuid4())  # once per operation
for attempt in range(3):
    response = api.create_ticket(payload, idempotency_key=key)
Enter fullscreen mode Exit fullscreen mode

Summary table

Anti-pattern Fails when Correct behavior
Client checks its own log Response lost after commit Receiver stores the authoritative result
Upsert without digest Key reused with different payload Compare digest, reject conflict
Application-level lock Concurrent processes/machines/restarts Database UNIQUE constraint
Keys without conflict detection Key reused with different payload Compare and reject mismatches
New key per retry Any retry happens Reuse the same key across retries

The reference implementation

The durable-ai-workflow-lab repository implements the correct contract in ~200 lines of Python:

  • Command(key, payload) with a deterministic SHA-256 digest.
  • MockReceiver backed by SQLite with UNIQUE(command_key).
  • Five pytest proofs: lost-ack retry, concurrent duplicates, changed payload conflict, restart survival, and digest stability.
uv sync --frozen
uv run --frozen pytest -q
Enter fullscreen mode Exit fullscreen mode

The full test suite runs in ~0.05 s.

When to apply this

Use receiver-side idempotency whenever:

  • A command crosses a process or network boundary.
  • The command has a side effect that is expensive or impossible to undo.
  • The client will retry on timeout or error.

Common domains: payments, inventory, shipments, support tickets, subscriptions, bookings, webhook handlers.

Takeaway

Idempotency is not a client-side setting, a framework feature, or a nice-to-have. It is a contract between the client and the receiver. The client must supply a stable key. The receiver must atomically store the key, the payload digest, and the result. Anything less leaves a window for duplicates.

Which anti-pattern have you seen?

Most production systems I have seen hit at least two of these before they centralize the contract. If you have a war story about duplicate charges, double shipments, or corrupted retries, drop it in the comments.

Reference implementation: github.com/manjunath-hanmantgad/durable-ai-workflow-lab


Continue reading: Closing the Write/Ack Gap with Receiver-Side Idempotency

Top comments (0)