DEV Community

Manjunath Hanamantgad
Manjunath Hanamantgad

Posted on

Closing the Write/Ack Gap with Receiver-Side Idempotency

Closing the Write/Ack Gap with Receiver-Side Idempotency

When a client sends a command to an external service, the response can be lost while the write succeeds. The client retries. Without receiver-side idempotency, that retry creates a duplicate charge, ticket, shipment, or booking.

This post is a complete walkthrough of a minimal, tested reference implementation that closes the gap at the receiver. The code is ~200 lines of Python, the full test suite runs in ~0.05 s, and the pattern applies to payments, inventory, support systems, and any other consequential external command.

Repository: github.com/manjunath-hanmantgad/durable-ai-workflow-lab

What problem this solves

The failure mode is simple but expensive:

client -> receiver: create ticket (key = "req-1")
receiver -> database: INSERT ticket
receiver -> client: [response lost]
client -> receiver: create ticket (key = "req-1")  -- retry
Enter fullscreen mode Exit fullscreen mode

After the first request, the client enters an ambiguous state. It cannot distinguish three possibilities:

  1. Request never arrived. A retry is safe and necessary.
  2. Request arrived but failed before writing. A retry is safe and necessary.
  3. Request was written, but the acknowledgement was lost. A retry is dangerous.

A client-side log, checkpoint, or "retry id" cannot distinguish case 3 from cases 1 and 2. The only system that knows whether the write happened is the receiver, because only the receiver saw the database commit.

Write/ack gap sequence diagram

Where this matters: real-world use cases

Any system that issues consequential commands to an external service needs this property.

Domain Command Cost of duplicate
Payments Charge customer Double charge; refunds; compliance risk
Support platforms Create ticket Two agents work the same issue; customer confusion
Inventory Reserve item Oversold stock; fulfillment failure
Logistics Book shipment Two trucks for one load; billing disputes
Subscriptions Activate plan Duplicate billing; support tickets
Healthcare Schedule procedure Double booking; patient safety risk
Webhooks Process delivery event Duplicate side effects downstream

In each case, the command is not safe to retry unless the receiver can recognize that it has already been executed.

Where the receiver lives in the architecture

The receiver sits at the boundary between your process and the external service. It is the last place where you can prevent duplicate external effects before they become visible to customers, finance, or operations.

┌─────────────┐      ┌─────────────────────┐      ┌─────────────────┐
│   Client    │ ──▶  │  Durable boundary   │ ──▶  │  External API   │
│  (worker,   │      │  (this receiver)    │      │  (ticket vendor,│
│   service)  │ ◀──  │                     │ ◀──  │   payment proc, │
└─────────────┘      └─────────────────────┘      │   inventory)    │
                                                   └─────────────────┘
Enter fullscreen mode Exit fullscreen mode

Architecture diagram

The client owns the retry policy. The receiver owns the deduplication contract. Separating these concerns makes the system easier to reason about and test.

The receiver contract

Receiver-side idempotency stores three things atomically:

  1. a stable command_key,
  2. a digest of the payload,
  3. the business result.

On retry:

  • Same key + same digest → return the stored result.
  • Same key + different digest → reject as a conflict.

The uniqueness of the key must be enforced by the database, not by application-level locking. That is what makes concurrent retries race safely.

Receiver decision flow

The implementation

The full repository is public: durable-ai-workflow-lab.

Core types

# src/p01/contracts.py
import hashlib
import json
from dataclasses import dataclass


class ContractError(Exception):
    """A domain rule violation."""


@dataclass(frozen=True)
class Command:
    """A stable intent to create an external ticket."""

    key: str
    payload: dict

    def payload_digest(self) -> str:
        encoded = json.dumps(
            self.payload,
            sort_keys=True,
            separators=(",", ":"),
            ensure_ascii=False,
        )
        return hashlib.sha256(encoded.encode("utf-8")).hexdigest()


@dataclass(frozen=True)
class Ticket:
    """The stored result of one successful receiver write."""

    ticket_id: str
    command_key: str
    payload_digest: str
Enter fullscreen mode Exit fullscreen mode

The Command combines a stable key with a deterministic digest of the payload. The digest is what lets us detect a changed intent under a reused key.

The receiver

# src/p01/receiver.py
import json
import sqlite3
import threading
from contextlib import contextmanager
from pathlib import Path

from p01.contracts import Command, ContractError, Ticket


class ConflictError(ContractError):
    """Raised when a key is reused with a different payload."""


class MockReceiver:
    def __init__(self, db_path: str | Path):
        self._db_path = str(db_path)
        self._local = threading.local()
        self._ensure_schema()

    @contextmanager
    def _connection(self):
        conn = getattr(self._local, "conn", None)
        own = conn is None
        if own:
            conn = sqlite3.connect(self._db_path, check_same_thread=False)
            conn.execute("PRAGMA foreign_keys = ON")
            self._local.conn = conn
        try:
            yield conn
        finally:
            if own:
                conn.close()
                self._local.conn = None

    def _ensure_schema(self) -> None:
        with self._connection() as conn:
            conn.execute(
                """
                CREATE TABLE IF NOT EXISTS tickets (
                    command_key TEXT PRIMARY KEY,
                    payload_digest TEXT NOT NULL,
                    ticket_id TEXT NOT NULL,
                    result_json TEXT NOT NULL
                )
                """
            )
            conn.commit()

    def create_ticket(self, command: Command) -> Ticket:
        digest = command.payload_digest()
        ticket_id = f"TICKET-{digest[:16]}"
        result_json = json.dumps(
            {"ticket_id": ticket_id, "command_key": command.key},
            sort_keys=True,
        )

        with self._connection() as conn:
            try:
                with conn:
                    conn.execute(
                        """
                        INSERT INTO tickets (command_key, payload_digest, ticket_id, result_json)
                        VALUES (?, ?, ?, ?)
                        """,
                        (command.key, digest, ticket_id, result_json),
                    )
                return Ticket(ticket_id, command.key, digest)
            except sqlite3.IntegrityError:
                pass

            row = conn.execute(
                "SELECT payload_digest, ticket_id FROM tickets WHERE command_key = ?",
                (command.key,),
            ).fetchone()

            stored_digest, stored_ticket_id = row
            if stored_digest != digest:
                raise ConflictError(
                    f"key {command.key!r} exists with different payload"
                )

            return Ticket(stored_ticket_id, command.key, stored_digest)
Enter fullscreen mode Exit fullscreen mode

The UNIQUE(command_key) constraint is the critical line. It serializes races so that ten concurrent retries still produce exactly one ticket.

Tests that prove it

# tests/test_gap.py
import threading
from pathlib import Path

import pytest

from p01.contracts import Command, canonical_digest
from p01.receiver import ConflictError, MockReceiver


def test_lost_ack_retry_returns_same_ticket(tmp_path: Path) -> None:
    receiver = MockReceiver(tmp_path / "tickets.db")
    command = Command(key="req-1", payload={"title": "Fix login"})

    first = receiver.create_ticket(command)
    second = receiver.create_ticket(command)

    assert first.ticket_id == second.ticket_id
    assert receiver.count_tickets() == 1


def test_concurrent_duplicates_create_one_ticket(tmp_path: Path) -> None:
    db_path = tmp_path / "tickets.db"
    command = Command(key="concurrent-1", payload={"title": "Concurrent"})

    def attempt():
        MockReceiver(db_path).create_ticket(command)

    threads = [threading.Thread(target=attempt) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert MockReceiver(db_path).count_tickets() == 1


def test_changed_payload_conflicts(tmp_path: Path) -> None:
    receiver = MockReceiver(tmp_path / "tickets.db")
    key = "mutable-1"

    receiver.create_ticket(Command(key=key, payload={"title": "Original"}))

    with pytest.raises(ConflictError):
        receiver.create_ticket(Command(key=key, payload={"title": "Changed"}))

    assert receiver.count_tickets() == 1


def test_survives_receiver_restart(tmp_path: Path) -> None:
    db_path = tmp_path / "tickets.db"
    command = Command(key="survive-1", payload={"title": "Survive restart"})

    before = MockReceiver(db_path).create_ticket(command)
    after = MockReceiver(db_path).create_ticket(command)

    assert before.ticket_id == after.ticket_id
    assert MockReceiver(db_path).count_tickets() == 1
Enter fullscreen mode Exit fullscreen mode

Run the suite:

uv sync --frozen
uv run --frozen pytest -q
Enter fullscreen mode Exit fullscreen mode

Key design decisions

1. Client generates the key

The server cannot tell a retry from a new request without a stable identifier. A UUID v4 generated by the client at submission time, and reused across retries, works well.

2. Hash the payload canonically

Sorting JSON keys before hashing means key order does not matter, but value changes do. This prevents a retry from silently overwriting a previous intent.

3. Let the database enforce uniqueness

A Python lock only protects one process. A UNIQUE constraint protects against concurrent processes, threads, and replicas.

4. Return the stored result, not just success

If the first attempt returned a 201 with a ticket ID, the retry must return the same 201 and ID. Returning a generic success flag breaks downstream systems that need the original identifier.

5. Conflict on changed payload

Reusing a key with different data is a client bug. Detect it and reject instead of silently returning the old result or overwriting it.

Failure modes handled

Scenario Behavior
Lost acknowledgement Retry returns the original result.
Concurrent retries Database constraint serializes; one ticket created.
Changed intent under same key ConflictError; existing result preserved.
Receiver restart before acknowledgement New process reads the same SQLite file and returns stored result.
Crash between commit and response Next retry reads the committed row.

From SQLite to production

This implementation uses SQLite to keep the proof minimal. In production, the contract stays the same; only the storage changes:

  • Use PostgreSQL with a unique index on (command_key).
  • Wrap insert and select in one transaction.
  • Scope keys per tenant or endpoint to avoid accidental collisions.
  • Set a TTL on idempotency records so the table does not grow forever.
  • Use the vendor's own idempotency mechanism if available (e.g., Stripe idempotency keys), but understand that the contract is identical.

What this is not

This is a bounded primitive, not a full workflow engine. It intentionally does not include HTTP, auth, multi-tenancy, LangGraph, distributed workers, real vendors, or performance benchmarks. Use it as a foundation and add layers only when your workload needs them.

Takeaway

A local ledger cannot prevent duplicate external writes. Only the receiver can, by atomically storing the command key and the result it produced. If you are building anything that retries commands across a network, make the receiver idempotent before you add retries.

Try it

git clone https://github.com/manjunath-hanmantgad/durable-ai-workflow-lab.git
cd durable-ai-workflow-lab
uv sync --frozen
uv run --frozen pytest -q
Enter fullscreen mode Exit fullscreen mode

No Docker, no external services, no secrets. The five proofs run in ~0.05 s.

What would you add next?

The repository is intentionally a bounded primitive. Common next steps are HTTP ingress, tenant-scoped auth, PostgreSQL persistence, and LangGraph orchestration. Each layer should be tested independently before it is added.


If this helped you think about retries differently, share the repo or leave a comment with the system in your stack that would break if a command ran twice.

Code, diagrams, and tests: github.com/manjunath-hanmantgad/durable-ai-workflow-lab

Top comments (0)