Closing the Write/Ack Gap with Receiver-Side Idempotency
When a client sends a command to an external service, the response can be lost while the write succeeds. The client retries. Without receiver-side idempotency, that retry creates a duplicate charge, ticket, shipment, or booking.
This post is a complete walkthrough of a minimal, tested reference implementation that closes the gap at the receiver. The code is ~200 lines of Python, the full test suite runs in ~0.05 s, and the pattern applies to payments, inventory, support systems, and any other consequential external command.
Repository: github.com/manjunath-hanmantgad/durable-ai-workflow-lab
What problem this solves
The failure mode is simple but expensive:
client -> receiver: create ticket (key = "req-1")
receiver -> database: INSERT ticket
receiver -> client: [response lost]
client -> receiver: create ticket (key = "req-1") -- retry
After the first request, the client enters an ambiguous state. It cannot distinguish three possibilities:
- Request never arrived. A retry is safe and necessary.
- Request arrived but failed before writing. A retry is safe and necessary.
- Request was written, but the acknowledgement was lost. A retry is dangerous.
A client-side log, checkpoint, or "retry id" cannot distinguish case 3 from cases 1 and 2. The only system that knows whether the write happened is the receiver, because only the receiver saw the database commit.
Where this matters: real-world use cases
Any system that issues consequential commands to an external service needs this property.
| Domain | Command | Cost of duplicate |
|---|---|---|
| Payments | Charge customer | Double charge; refunds; compliance risk |
| Support platforms | Create ticket | Two agents work the same issue; customer confusion |
| Inventory | Reserve item | Oversold stock; fulfillment failure |
| Logistics | Book shipment | Two trucks for one load; billing disputes |
| Subscriptions | Activate plan | Duplicate billing; support tickets |
| Healthcare | Schedule procedure | Double booking; patient safety risk |
| Webhooks | Process delivery event | Duplicate side effects downstream |
In each case, the command is not safe to retry unless the receiver can recognize that it has already been executed.
Where the receiver lives in the architecture
The receiver sits at the boundary between your process and the external service. It is the last place where you can prevent duplicate external effects before they become visible to customers, finance, or operations.
┌─────────────┐ ┌─────────────────────┐ ┌─────────────────┐
│ Client │ ──▶ │ Durable boundary │ ──▶ │ External API │
│ (worker, │ │ (this receiver) │ │ (ticket vendor,│
│ service) │ ◀── │ │ ◀── │ payment proc, │
└─────────────┘ └─────────────────────┘ │ inventory) │
└─────────────────┘
The client owns the retry policy. The receiver owns the deduplication contract. Separating these concerns makes the system easier to reason about and test.
The receiver contract
Receiver-side idempotency stores three things atomically:
- a stable
command_key, - a digest of the payload,
- the business result.
On retry:
- Same key + same digest → return the stored result.
- Same key + different digest → reject as a conflict.
The uniqueness of the key must be enforced by the database, not by application-level locking. That is what makes concurrent retries race safely.
The implementation
The full repository is public: durable-ai-workflow-lab.
Core types
# src/p01/contracts.py
import hashlib
import json
from dataclasses import dataclass
class ContractError(Exception):
"""A domain rule violation."""
@dataclass(frozen=True)
class Command:
"""A stable intent to create an external ticket."""
key: str
payload: dict
def payload_digest(self) -> str:
encoded = json.dumps(
self.payload,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
)
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
@dataclass(frozen=True)
class Ticket:
"""The stored result of one successful receiver write."""
ticket_id: str
command_key: str
payload_digest: str
The Command combines a stable key with a deterministic digest of the payload. The digest is what lets us detect a changed intent under a reused key.
The receiver
# src/p01/receiver.py
import json
import sqlite3
import threading
from contextlib import contextmanager
from pathlib import Path
from p01.contracts import Command, ContractError, Ticket
class ConflictError(ContractError):
"""Raised when a key is reused with a different payload."""
class MockReceiver:
def __init__(self, db_path: str | Path):
self._db_path = str(db_path)
self._local = threading.local()
self._ensure_schema()
@contextmanager
def _connection(self):
conn = getattr(self._local, "conn", None)
own = conn is None
if own:
conn = sqlite3.connect(self._db_path, check_same_thread=False)
conn.execute("PRAGMA foreign_keys = ON")
self._local.conn = conn
try:
yield conn
finally:
if own:
conn.close()
self._local.conn = None
def _ensure_schema(self) -> None:
with self._connection() as conn:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS tickets (
command_key TEXT PRIMARY KEY,
payload_digest TEXT NOT NULL,
ticket_id TEXT NOT NULL,
result_json TEXT NOT NULL
)
"""
)
conn.commit()
def create_ticket(self, command: Command) -> Ticket:
digest = command.payload_digest()
ticket_id = f"TICKET-{digest[:16]}"
result_json = json.dumps(
{"ticket_id": ticket_id, "command_key": command.key},
sort_keys=True,
)
with self._connection() as conn:
try:
with conn:
conn.execute(
"""
INSERT INTO tickets (command_key, payload_digest, ticket_id, result_json)
VALUES (?, ?, ?, ?)
""",
(command.key, digest, ticket_id, result_json),
)
return Ticket(ticket_id, command.key, digest)
except sqlite3.IntegrityError:
pass
row = conn.execute(
"SELECT payload_digest, ticket_id FROM tickets WHERE command_key = ?",
(command.key,),
).fetchone()
stored_digest, stored_ticket_id = row
if stored_digest != digest:
raise ConflictError(
f"key {command.key!r} exists with different payload"
)
return Ticket(stored_ticket_id, command.key, stored_digest)
The UNIQUE(command_key) constraint is the critical line. It serializes races so that ten concurrent retries still produce exactly one ticket.
Tests that prove it
# tests/test_gap.py
import threading
from pathlib import Path
import pytest
from p01.contracts import Command, canonical_digest
from p01.receiver import ConflictError, MockReceiver
def test_lost_ack_retry_returns_same_ticket(tmp_path: Path) -> None:
receiver = MockReceiver(tmp_path / "tickets.db")
command = Command(key="req-1", payload={"title": "Fix login"})
first = receiver.create_ticket(command)
second = receiver.create_ticket(command)
assert first.ticket_id == second.ticket_id
assert receiver.count_tickets() == 1
def test_concurrent_duplicates_create_one_ticket(tmp_path: Path) -> None:
db_path = tmp_path / "tickets.db"
command = Command(key="concurrent-1", payload={"title": "Concurrent"})
def attempt():
MockReceiver(db_path).create_ticket(command)
threads = [threading.Thread(target=attempt) for _ in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
assert MockReceiver(db_path).count_tickets() == 1
def test_changed_payload_conflicts(tmp_path: Path) -> None:
receiver = MockReceiver(tmp_path / "tickets.db")
key = "mutable-1"
receiver.create_ticket(Command(key=key, payload={"title": "Original"}))
with pytest.raises(ConflictError):
receiver.create_ticket(Command(key=key, payload={"title": "Changed"}))
assert receiver.count_tickets() == 1
def test_survives_receiver_restart(tmp_path: Path) -> None:
db_path = tmp_path / "tickets.db"
command = Command(key="survive-1", payload={"title": "Survive restart"})
before = MockReceiver(db_path).create_ticket(command)
after = MockReceiver(db_path).create_ticket(command)
assert before.ticket_id == after.ticket_id
assert MockReceiver(db_path).count_tickets() == 1
Run the suite:
uv sync --frozen
uv run --frozen pytest -q
Key design decisions
1. Client generates the key
The server cannot tell a retry from a new request without a stable identifier. A UUID v4 generated by the client at submission time, and reused across retries, works well.
2. Hash the payload canonically
Sorting JSON keys before hashing means key order does not matter, but value changes do. This prevents a retry from silently overwriting a previous intent.
3. Let the database enforce uniqueness
A Python lock only protects one process. A UNIQUE constraint protects against concurrent processes, threads, and replicas.
4. Return the stored result, not just success
If the first attempt returned a 201 with a ticket ID, the retry must return the same 201 and ID. Returning a generic success flag breaks downstream systems that need the original identifier.
5. Conflict on changed payload
Reusing a key with different data is a client bug. Detect it and reject instead of silently returning the old result or overwriting it.
Failure modes handled
| Scenario | Behavior |
|---|---|
| Lost acknowledgement | Retry returns the original result. |
| Concurrent retries | Database constraint serializes; one ticket created. |
| Changed intent under same key |
ConflictError; existing result preserved. |
| Receiver restart before acknowledgement | New process reads the same SQLite file and returns stored result. |
| Crash between commit and response | Next retry reads the committed row. |
From SQLite to production
This implementation uses SQLite to keep the proof minimal. In production, the contract stays the same; only the storage changes:
- Use PostgreSQL with a unique index on
(command_key). - Wrap insert and select in one transaction.
- Scope keys per tenant or endpoint to avoid accidental collisions.
- Set a TTL on idempotency records so the table does not grow forever.
- Use the vendor's own idempotency mechanism if available (e.g., Stripe idempotency keys), but understand that the contract is identical.
What this is not
This is a bounded primitive, not a full workflow engine. It intentionally does not include HTTP, auth, multi-tenancy, LangGraph, distributed workers, real vendors, or performance benchmarks. Use it as a foundation and add layers only when your workload needs them.
Takeaway
A local ledger cannot prevent duplicate external writes. Only the receiver can, by atomically storing the command key and the result it produced. If you are building anything that retries commands across a network, make the receiver idempotent before you add retries.
Try it
git clone https://github.com/manjunath-hanmantgad/durable-ai-workflow-lab.git
cd durable-ai-workflow-lab
uv sync --frozen
uv run --frozen pytest -q
No Docker, no external services, no secrets. The five proofs run in ~0.05 s.
What would you add next?
The repository is intentionally a bounded primitive. Common next steps are HTTP ingress, tenant-scoped auth, PostgreSQL persistence, and LangGraph orchestration. Each layer should be tested independently before it is added.
If this helped you think about retries differently, share the repo or leave a comment with the system in your stack that would break if a command ran twice.
Code, diagrams, and tests: github.com/manjunath-hanmantgad/durable-ai-workflow-lab


Top comments (0)