The Idempotency Pattern: Making Retries Safe in Distributed Systems

Introduction

Networks fail. Timeouts happen. Clients retry. Message brokers may redeliver messages.In distributed systems, you often can't know whether a request actually succeeded. The server may have completed the operation, but the response could have been lost.

The obvious solution is to retry. But retries are safe only when an operation is idempotent—running it multiple times produces the same result as running it once.

The Idempotency Pattern makes retries safe and helps build reliable distributed systems. This article explains the problem, common failure scenarios, how the pattern works, and how it supports safe scaling and loose coupling.

---

The Problem Statement

The "At-Least-Once" Reality

Most distributed systems deliver messages or requests at least once, not exactly once.

Retries, message broker redelivery, Outbox relays, and failover mechanisms can all cause the same request to be processed more than once. This is a practical trade-off because guaranteeing exactly-once delivery across a network is extremely difficult.

So the responsibility often moves to the receiver: it must safely handle duplicate requests.

If a service isn't designed for duplicates, a single retry can accidentally perform the same operation twice—with serious consequences such as duplicate payments, orders, or updates.

sequenceDiagram
    participant Client
    participant PaymentService
    participant Database

    Client->>PaymentService: POST /charge ($100)
    PaymentService->>Database: charge card, deduct balance
    Database-->>PaymentService: OK
    Note over PaymentService,Client: Response lost on the way back<br/>(timeout, network blip, crash)
    Client--xClient: Times out, assumes failure
    Client->>PaymentService: Retry: POST /charge ($100)
    PaymentService->>Database: charge card, deduct balance AGAIN
    Database-->>PaymentService: OK
    PaymentService-->>Client: 200 OK

The customer is charged twice for a single purchase—not because the payment failed, but because the client retried the request after not receiving the original response.

---

Why "Just Don't Retry" Isn't an Option

You might think: "If retries cause duplicates, why not avoid retries?"
Because that simply creates a different problem.

Many failures are temporary— a brief network interruption, a service restart, or a load balancer hiccup. Without retries, these temporary failures become permanent errors for the user, leading to failed requests and lost business.

So retries are an important part of building resilient distributed systems.

The real solution isn't to avoid retries. It's to make operations safe to repeat.

That's exactly what idempotency provides: the same request can be processed multiple times without causing unintended side effects.

flowchart LR
    A[Request sent] --> B{Response received?}
    B -->|Yes| C[Done]
    B -->|No| D[Did the operation execute?]
    D -->|Unknown| E[Retry]
    E --> F{Idempotent?}
    F -->|Yes| G[Safe to retry]
    F -->|No| H[Duplicate side effect]

---

Where This Bites in Practice

Idempotency becomes important anywhere a request can be retried, redelivered, or reprocessed.

Scenario	What can go wrong without idempotency
Payment processing	A timed-out payment request is retried → the customer is charged twice.
Order placement	A mobile app retries after a network failure → two identical orders are created.
Message consumers (Kafka/SQS/RabbitMQ)	A broker redelivers a message → the same event is processed twice, such as deducting inventory twice.
Outbox relay	The relay publishes an event but crashes before marking it as published → the event is published again after restart.
Email/notifications	A retry sends the same notification multiple times.
Distributed sagas	A saga step is retried after a coordinator failure → an action such as a refund may be applied twice.
API gateways/load balancers	A gateway retries a timed-out request while the original request is still running → the operation executes twice.
Batch/ETL jobs	A failed job is restarted from the beginning → records already processed may be inserted or updated again.

The Common Problem

All of these scenarios have the same root cause:

The same operation can be executed more than once, and the system cannot tell whether it is a new request or a retry of an operation that already succeeded.

This is why idempotency matters. It allows a system to safely retry or reprocess an operation without creating unintended side effects.

---

The Idempotency Pattern — The Solution

Core Idea

Attach a unique idempotency key to every request that can cause a side effect.

The server stores the key along with the result of the operation. If the same key is received again, the server knows it is a retry.

Instead of executing the operation again, it returns the original result.

In simple terms:

Same idempotency key = same operation → execute once, return the same result on retries.

flowchart LR
    A[Client generates<br/>idempotency key] --> B[Send request<br/>+ Idempotency-Key header]
    B --> C{Server: key seen before?}
    C -->|No| D[Execute operation]
    D --> E[Store key + result]
    E --> F[Return result]
    C -->|Yes| G[Skip execution]
    G --> H[Return stored result]

The key insight is that the client decides whether a request represents a new operation or a retry because the client knows its own intent.

The client generates an idempotency key once for each logical operation— for example, when the user clicks "Place Order".

If the request needs to be retried, the client sends the same key again. The server uses that key to recognize the request as a retry and returns the original result instead of executing the operation again.

---

Sequence Diagram: Idempotent Payment

sequenceDiagram
    participant Client
    participant PaymentService
    participant IdempotencyStore as Idempotency Store
    participant Database

    Client->>PaymentService: POST /charge<br/>Idempotency-Key: abc-123
    PaymentService->>IdempotencyStore: has key "abc-123"?
    IdempotencyStore-->>PaymentService: not found
    PaymentService->>Database: BEGIN TX
    PaymentService->>Database: charge card, deduct balance
    PaymentService->>IdempotencyStore: store key "abc-123" + result (same TX)
    PaymentService->>Database: COMMIT
    PaymentService-->>Client: 200 OK (charge succeeded)

    Note over Client: Response lost / times out
    Client->>PaymentService: Retry: POST /charge<br/>Idempotency-Key: abc-123

    PaymentService->>IdempotencyStore: has key "abc-123"?
    IdempotencyStore-->>PaymentService: found — return stored result
    PaymentService-->>Client: 200 OK (same result, no re-charge)

Notice that the idempotency check, business operation, and idempotency record should be handled in the same local transaction. This is important.

If the idempotency check and business write happen in separate transactions, you introduce a dual-write problem— the same kind of consistency problem that the Outbox Pattern addresses.

For a database-backed implementation, the idempotency record and the business side effect should be committed atomically.

Either both are committed, or neither is.

---

The Idempotency Store

The idempotency store keeps track of requests that have already been processed. It can be implemented using a database table, Redis, or another durable store, depending on the system's requirements.

A simple database-backed implementation might look like this:

idempotency_key	status	response_body	created_at	expires_at
abc-123	completed	{"chargeId":"ch_1"}	2026-09-10T10:00:00Z	2026-09-17T10:00:00Z

Typical fields include:

• idempotency_key — A client-generated unique key representing one logical operation. The same key is reused for all retries of that operation.
• status — Tracks the processing state, such as in_progress or completed. Some systems may also store failed, depending on how failed requests should be handled.
• response_body — Stores the result that can be returned when the same request is retried.
• created_at — Records when the idempotency entry was created.
• expires_at — Defines how long the key should be retained. Idempotency records are usually kept for a bounded period rather than forever.

The exact fields and storage mechanism depend on the system. The important requirement is that the store can reliably recognize a previously processed key and return the appropriate result without repeating the side effect.

---

Handling the "In-Flight" Race

What happens if a retry arrives while the original request is still being processed?

Without proper coordination, both requests could see that the idempotency key does not exist and execute the business operation at the same time.

A common solution is to enforce a unique constraint on the idempotency key:

flowchart LR
    %% Request A (Primary Winner)
    subgraph ReqA ["Request A (Primary)"]
        A1["Incoming Request"] --> A2["Try INSERT idempotency key"]
        A2 -->|Success| A3["Mark status = in_progress"]
        A3 --> A4["Execute Business Logic"]
        A4 --> A5["Update status = completed"]
        A5 --> A6["Persist Response"]
    end

    %% Request B (Concurrent)
    subgraph ReqB ["Request B (Concurrent)"]
        B1["Incoming Request"] --> B2["Try INSERT idempotency key"]
        B2 -->|Constraint Violation| B3["Lookup existing key in Idempotency Table"]
        B3 --> B4{"Status Check"}
        B4 -->|Completed| B5["Return Cached Response"]
        B4 -->|In Progress| B6["Retry with Backoff / Return 409"]
    end

    %% Race condition visualization
    A2 -.->|Unique Key Conflict| B2

    %% Styling
    style A1 fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,color:#0c4a6e
    style A2 fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    style A3 fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    style A4 fill:#f3e8ff,stroke:#7e22ce,stroke-width:2px,color:#3b0764
    style A5 fill:#dcfce7,stroke:#15803d,stroke-width:2px,color:#14532d
    style A6 fill:#dcfce7,stroke:#15803d,stroke-width:2px,color:#14532d

    style B1 fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,color:#0c4a6e
    style B2 fill:#fee2e2,stroke:#b91c1c,stroke-width:2px,color:#7f1d1d
    style B3 fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    style B4 fill:#f3e8ff,stroke:#7e22ce,stroke-width:2px,color:#3b0764
    style B5 fill:#dcfce7,stroke:#15803d,stroke-width:2px,color:#14532d
    style B6 fill:#fee2e2,stroke:#b91c1c,stroke-width:2px,color:#7f1d1d

The first request successfully creates the idempotency record and becomes the owner of the operation.

If another request arrives with the same key while the operation is still running, the unique constraint prevents it from creating another record. The second request must not execute the business operation. Depending on the system, it can wait, poll for the result, or return an in_progress/conflict response.

Once the first request completes, the record is updated with the final status and response.

One key → one active operation → no parallel execution of the same logical request.

---

Different Layers Where Idempotency Applies

Idempotency isn't a single technique — it shows up differently depending on where in the stack you apply it:

1. HTTP API level (client-supplied key)
Client sends an Idempotency-Key header (this is literally how Stripe's and PayPal's payment APIs work). Best for client-initiated actions like charges, orders, and transfers.

2. Message consumer level (message ID / offset)
When consuming from Kafka, SQS, or RabbitMQ, use the message's unique ID (or a business key embedded in the payload) to check "have I already processed this message?" before applying its side effects — critical because brokers guarantee at-least-once delivery by design.

flowchart LR
    Broker((Broker)) --> Consumer[Consumer Service]
    Consumer --> Check{"Seen in<br/>processed_messages?"}

    Check -->|No| Process["1. Apply Side Effect<br/>2. Record message_id (Same TX)"]
    Check -->|Yes| Skip[Skip — Duplicate Message]

    style Broker fill:#f3e8ff,stroke:#7e22ce,stroke-width:2px,color:#3b0764
    style Consumer fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,color:#0c4a6e
    style Check fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    style Process fill:#dcfce7,stroke:#15803d,stroke-width:2px,color:#14532d
    style Skip fill:#fee2e2,stroke:#b91c1c,stroke-width:2px,color:#7f1d1d

3. Natural idempotency via design (idempotent-by-construction)
Some operations are inherently safe to repeat without requiring an external deduplication store, explicit state tracking, or distributed locks:

• Absolute vs. relative state changes: An absolute update like UPDATE accounts SET status = 'active' is idempotent because applying it multiple times leaves the row in the exact same state. Conversely, a relative update like UPDATE accounts SET balance = balance - 100 is non-idempotent because each execution subtracts more funds. (Note: While absolute updates are mathematically idempotent, ensure they don't blindly overwrite concurrent state changes made by other transactions).
• Database constraints and upserts: Using clauses like INSERT ... ON CONFLICT DO NOTHING or ON CONFLICT DO UPDATE anchored to a unique business key (such as an order_number) turns duplicate write attempts into safe, predictable outcomes.
• RESTful HTTP semantics: Designing APIs around PUT /resource/{id} (which replaces the target resource entirely with the provided payload) ensures idempotency because sending the exact same payload twice yields the identical end state. In contrast, POST /resource is designed for creation and is non-idempotent, spawning a new resource with a new identifier on every call.

---

How This Achieves Loose Coupling

Idempotency reduces the amount of coordination required between components in a distributed system.

• Clients and servers can evolve independently. A client can safely retry after a timeout, connection failure, or 5xx response without knowing whether the server already processed the request. The server guarantees that repeating the same operation won't create another side effect.
• Producers and consumers don't need exactly-once delivery. In event-driven systems, the broker can use simple at-least-once delivery and redeliver messages when necessary. The consumer handles duplicates using its own idempotency mechanism.
• Failures become easier to recover from. A service, relay, or workflow can retry an operation after a crash without needing to know exactly where the previous attempt stopped. Idempotency makes repeating the operation safe.

In short:

At-least-once delivery + idempotent processing = reliable systems with less coordination.

Each component can retry independently, while the receiver ensures that repeated requests or messages don't produce repeated side effects.

---

How This Achieves Scaling

Idempotency makes it easier to scale distributed systems because requests and messages can be retried or processed by different instances without repeating their side effects.

• Safe horizontal scaling and failover. A request can be retried against any healthy service instance. The idempotency key ensures that another instance doesn't perform the same operation again.
• Simpler retry handling. Clients and services can use retries with exponential backoff without worrying that a retry will automatically create another side effect. Retry limits and backoff are still important to avoid overwhelming the system.
• At-least-once messaging becomes practical. Brokers such as Kafka, SQS, and RabbitMQ can use at-least-once delivery and redeliver messages when necessary. Idempotent consumers prevent those redeliveries from causing duplicate effects.
• Parallel processing with less coordination. Multiple consumer instances can process messages concurrently. If a message is delivered more than once—for example, during a consumer restart or rebalance—the idempotency check prevents the side effect from being applied twice.
• Reduces the impact of retries during failures. Under heavy load, timeouts and retries can increase. Without idempotency, those retries may create additional work or duplicate side effects. Idempotency prevents the same operation from being applied repeatedly, helping the system recover more safely.

The key idea is:

Idempotency allows the system to scale and recover using retries and at-least-once delivery without requiring every component to coordinate exactly-once execution.

---

Trade-offs to Be Aware Of

• Extra storage and lookup cost. Idempotency requires storing and checking a key, which adds a small amount of latency, database/storage usage, and operational overhead.
• Key expiration and retention. Idempotency records cannot be kept forever. You need a retention period that is long enough to cover realistic retry delays while preventing unbounded storage growth.
• Client responsibility. The client must generate the key once for each logical operation and reuse the same key for all retries. Generating a new key for every retry makes the server treat each retry as a new operation.
• Not automatic for multi-step workflows. Idempotency protects an individual operation; it does not automatically make an entire workflow idempotent. In a saga or multi-step process, each side-effecting step should be designed to handle retries safely, along with the workflow's coordination and recovery logic.
• External side effects require additional handling. A local idempotency record cannot be committed atomically with an external API call. If the external system supports idempotency, the same key should be propagated downstream. Otherwise, reconciliation or another coordination mechanism may be required.
• Concurrency requires careful implementation. A naive check → process → save approach is vulnerable to race conditions when the same key arrives concurrently. Use an atomic claim, unique constraint, or another concurrency-control mechanism to ensure that only one request can own the operation.

---

Idempotency + Outbox: Natural Partners

These two patterns are frequently used together because they solve two closely related reliability problems:

• Idempotency prevents the same operation from producing duplicate side effects when requests or messages are retried.
• Outbox ensures that an event generated by a successful database transaction is reliably published to a message broker.

Together, they provide a robust approach for building systems that can safely retry operations and reliably propagate the resulting events.

flowchart TD
    A[Business Operation] --> B["Outbox Pattern:<br/>Atomically write state + event locally"]
    B --> C["Relay Worker:<br/>Publishes event (at-least-once)"]
    C --> D["Idempotent Consumer:<br/>Dedupes by message_id"]
    D --> E["Reliable Exactly-Once Effect<br/>(without distributed transactions)"]

    style A fill:#f3e8ff,stroke:#7e22ce,stroke-width:2px,color:#3b0764
    style B fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,color:#0c4a6e
    style C fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    style D fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,color:#0c4a6e
    style E fill:#dcfce7,stroke:#15803d,stroke-width:2px,color:#14532d

Outbox guarantees an event is never lost. Idempotency guarantees that even if the event (or the retry of a request) arrives more than once, the end effect is as if it arrived exactly once. Together, they turn "at-least-once, unreliable network" into "reliable, correct, exactly-once-effect" system behavior — without needing expensive distributed transactions anywhere.

---

Summary

Without Idempotency	With Idempotency
Retries can cause duplicate side effects such as double charges or duplicate orders	Retries with the same key can safely return the original result without repeating the side effect
Avoiding retries may seem safer, but reduces resilience	Retries can be used safely with appropriate limits, backoff, and jitter
Systems may try to achieve exactly-once processing across unreliable networks	At-least-once delivery + idempotent processing provides reliable effective behavior
Client and server need tighter coordination around retry outcomes	Client can retry independently; the server handles duplicate requests
Retries during failures can amplify load and duplicate side effects	Duplicate requests are absorbed, reducing the impact of retries and redelivery

The key idea behind the Idempotency Pattern is to avoid making the network guarantee exactly-once execution. Instead, the system accepts that requests or messages may be delivered more than once and makes the processing itself safe to repeat.

In practice:

At-least-once delivery + idempotent processing = reliable retry behavior

This simple shift makes retries, message redelivery, failover, and horizontal scaling much easier to design. Instead of trying to prevent every duplicate delivery, the system ensures that duplicates do not produce duplicate side effects.