DEV Community

Ayi NEDJIMI
Ayi NEDJIMI

Posted on

Implementing idempotency in distributed systems: patterns and pitfalls

Distributed systems fail in partial ways. A network timeout does not tell you whether the request arrived, executed, and was lost on the way back — or never arrived at all. Without idempotency, retrying that request can double-charge a customer, send the same email twice, or insert duplicate records. This article covers the practical patterns you need and the traps most teams fall into.

What idempotency actually means

An operation is idempotent if applying it multiple times produces the same result as applying it once. DELETE /users/42 is naturally idempotent: deleting an already-deleted user returns 404 but leaves the system in the same state. POST /payments is not: a naive implementation creates a new payment on each call.

The goal is not to prevent duplicate requests — networks are unreliable and clients will retry. The goal is to make duplicates safe.

Pattern 1: Idempotency keys

The most common approach for POST endpoints is the idempotency key: the client generates a unique key (usually UUID v4), sends it as a header, and the server deduplicates on it.

Here is a minimal implementation in Go using PostgreSQL:

package main

import (
    "crypto/sha256"
    "database/sql"
    "encoding/hex"
    "encoding/json"
    "net/http"
    "time"
)

type PaymentRequest struct {
    Amount   int    `json:"amount"`
    Currency string `json:"currency"`
    To       string `json:"to"`
}

type IdempotencyRecord struct {
    StatusCode int
    Body       []byte
}

func createPayment(db *sql.DB, w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        http.Error(w, "Idempotency-Key header required", http.StatusBadRequest)
        return
    }

    // Hash the key to a fixed length — prevents inflated storage from arbitrary inputs
    h := sha256.Sum256([]byte(key))
    keyHash := hex.EncodeToString(h[:])

    var rec IdempotencyRecord
    err := db.QueryRow(
        `SELECT status_code, body FROM idempotency_records WHERE key_hash = $1`,
        keyHash,
    ).Scan(&rec.StatusCode, &rec.Body)

    if err == nil {
        // Replay stored response byte-for-byte
        w.Header().Set("Content-Type", "application/json")
        w.Header().Set("Idempotent-Replayed", "true")
        w.WriteHeader(rec.StatusCode)
        w.Write(rec.Body)
        return
    }

    var req PaymentRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "invalid body", http.StatusBadRequest)
        return
    }

    paymentID, procErr := processPayment(db, req)

    var statusCode int
    var responseBody []byte
    if procErr != nil {
        statusCode = http.StatusInternalServerError
        responseBody, _ = json.Marshal(map[string]string{"error": procErr.Error()})
    } else {
        statusCode = http.StatusCreated
        responseBody, _ = json.Marshal(map[string]string{"id": paymentID})
    }

    // ON CONFLICT DO NOTHING handles concurrent duplicate requests atomically
    db.Exec(
        `INSERT INTO idempotency_records (key_hash, status_code, body, created_at)
         VALUES ($1, $2, $3, NOW())
         ON CONFLICT (key_hash) DO NOTHING`,
        keyHash, statusCode, responseBody,
    )

    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(statusCode)
    w.Write(responseBody)
}
Enter fullscreen mode Exit fullscreen mode

Two things worth noting: store the full response — status code and body — not just a "done" flag. Replaying the exact original response is what makes the client treat the duplicate as a no-op. Also store error responses; if the card was declined, the client needs to know that, not receive a silent 200 on retry.

Pattern 2: Deterministic IDs

For some operations you can skip a separate idempotency store entirely. If a payment's ID is derived from hash(user_id + date + amount), inserting the same payment twice hits the same primary key and fails gracefully.

import hashlib
import psycopg2

def create_payment(conn, user_id: str, amount: int, currency: str, date: str) -> str:
    raw = f"{user_id}:{amount}:{currency}:{date}"
    payment_id = hashlib.sha256(raw.encode()).hexdigest()[:32]

    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO payments (id, user_id, amount, currency, date, status)
            VALUES (%s, %s, %s, %s, %s, 'pending')
            ON CONFLICT (id) DO NOTHING
            """,
            (payment_id, user_id, amount, currency, date)
        )
        conn.commit()

    # Same value whether the row was inserted or already existed
    return payment_id
Enter fullscreen mode Exit fullscreen mode

This works well for batch imports and event processing. It does not work when the same operation can legitimately recur (charging the same amount on two different days).

Pattern 3: Outbox pattern for event publishing

Idempotency collapses when you need to write to a database and publish an event. The classic failure: you commit the DB write, then crash before publishing — or you publish, then the write fails.

The outbox pattern keeps everything in one transaction:

  1. In the same transaction: insert your domain record + a row in an outbox table.
  2. A background process polls the outbox, publishes the event, marks the row processed.
  3. The consumer deduplicates on event ID.

The DB write is atomic. The publish is at-least-once, so consumer deduplication is still required — but at least you will never lose an event.

Common pitfalls

Checking idempotency after the side effect. The check must happen before any external call. If you call your payment processor first and then check the idempotency store, you have already charged the card before deduplicating.

Not purging old records. An idempotency store that grows forever is a liability. Stripe retains keys for 24 hours. Set a TTL that covers your retry window and schedule a cleanup:

DELETE FROM idempotency_records
WHERE created_at < NOW() - INTERVAL '24 hours';
Enter fullscreen mode Exit fullscreen mode

Assuming your dependencies are idempotent. Each layer of a distributed call chain needs its own idempotency key, passed through. An idempotent API can still trigger non-idempotent downstream calls.

Using timestamps or client IPs as keys. Timestamps collide under load; IPs are shared. Keys must be client-generated UUIDs with enough entropy — typically UUID v4.

Treating Kafka producer idempotency as enough. enable.idempotence=true prevents duplicate messages within one producer session but does not survive restarts. Consumer-side deduplication is still required.

The takeaway

Idempotency is not an edge case — it is a design requirement for every distributed write that might be retried, which is all of them. The pattern is simple: client-generated key, server-side result storage, exact response replay. The complexity comes from TTLs, error handling, ordering of checks, and propagating keys through call chains. Getting this right before production is cheaper than debugging duplicate payments after.

If you are running an audit of your API's resilience posture, the free architecture and security checklists we publish cover idempotency alongside authentication, rate limiting, and input validation — in a format you can hand to a team.


I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Top comments (0)