Your client sends a POST /products. The server creates the product, then the load balancer kills the connection before the response gets out. The client sees a timeout. What does every reasonable HTTP client do on a timeout? It retries. Now you have two products.
This is not an exotic failure mode. It's Tuesday. Mobile clients on flaky networks, Kubernetes rolling deploys, aggressive proxy timeouts — anything that separates "the server did the work" from "the client learned about it" will eventually duplicate your writes. The standard fix is an idempotency key: the client generates a unique key per logical operation, sends it with the request, and the server guarantees that the same key never executes the operation twice.
The concept is simple. The implementation is where I got burned, so let me walk through the version I actually ship now, including the parts I got wrong the first time.
The naive version (which I shipped)
Here's the implementation almost everyone writes first, me included:
func (s *ProductService) CreateProduct(ctx context.Context, cmd *CreateProductCommand) (*Result, error) {
existing, err := s.idempotencyRepo.FindByKey(ctx, cmd.IdempotencyKey)
if err != nil {
return nil, err
}
if existing != nil {
return unmarshalCachedResponse(existing)
}
result, err := s.doCreateProduct(ctx, cmd)
if err != nil {
return nil, err
}
s.idempotencyRepo.Store(ctx, cmd.IdempotencyKey, result)
return result, nil
}
Check, execute, store. It reads correctly. It passes every test you'll write for it, because sequential tests can't see the bug.
The bug: two requests with the same key arrive at the same time. Both call FindByKey. Both get nil — the key hasn't been stored yet, because storing happens after execution. Both execute. Two products, one idempotency key, and the whole mechanism silently did nothing exactly when it mattered.
Concurrent duplicates are not a corner case, they're the main case. Retries fire because something is slow, and when something is slow the original request is often still running. The naive check-then-act version protects you against the retry that arrives after the first request finished — the one scenario that was never dangerous to begin with.
I ran this bug in production for a while before a payment-adjacent endpoint made it visible. Ask me how I know duplicate writes are hard to clean up.
Fix one: reserve the key atomically
The check and the claim have to be one atomic operation. In Postgres that's a unique constraint plus ON CONFLICT DO NOTHING:
-- name: ReserveIdempotencyKey :execrows
-- Atomically claims the key. Zero rows means another request already holds it.
INSERT INTO idempotency_records (id, key, request, response, status_code, created_at)
VALUES ($1, $2, $3, '', 0, $4)
ON CONFLICT (key) DO NOTHING;
The :execrows annotation is sqlc for "give me the affected row count". That count is the whole trick: 1 means you won the race and may execute; 0 means somebody else holds the key and you must not.
func (r *SqlcIdempotencyRepository) Reserve(ctx context.Context, record *entities.IdempotencyRecord) (bool, error) {
rows, err := r.queries.ReserveIdempotencyKey(ctx, db.ReserveIdempotencyKeyParams{
ID: record.Id,
Key: record.Key,
Request: record.Request,
CreatedAt: timestamptzFromTime(record.CreatedAt),
})
if err != nil {
return false, err
}
return rows > 0, nil
}
Note what changed structurally: the row is inserted before execution, as a reservation, not after execution as a cache entry. response = '' and status_code = 0 mean "in flight". Postgres's unique index is now the arbiter of who executes — no advisory locks, no Redis, no distributed-lock library. Under concurrency, exactly one INSERT wins. That guarantee comes from the database, not from my code, which is exactly where I want it.
The losers need answers too
If you lost the reservation race, what do you tell the client? Depends on what the winner is doing:
- Winner already finished → serve its stored response. The retry gets the same
201body the original would have gotten. - Winner still running →
409 Conflict. The client should back off and retry; on the retry it'll hit the cached response. - Same key, different payload →
422 Unprocessable Entity. This one is easy to skip and dangerous to skip. If a client bug reuses a key across different requests, silently returning the first request's cached response means the caller thinks it created product B while you hand it product A. Fail loudly instead.
The whole flow lives in one generic decorator that every command-side service method wraps around its work. Trimmed to the interesting part:
reserved := false
for attempt := 0; attempt < 3 && !reserved; attempt++ {
reserved, err = repo.Reserve(ctx, record)
if err != nil {
return nil, err
}
if reserved {
break
}
existing, err := repo.FindByKey(ctx, key)
if err != nil {
return nil, err
}
if existing == nil {
// Released between Reserve and FindByKey; try again.
continue
}
if existing.Request != string(requestJSON) {
return nil, ErrIdempotencyKeyReuse
}
if existing.IsCompleted() {
var result T
if err := json.Unmarshal([]byte(existing.Response), &result); err != nil {
return nil, fmt.Errorf("unmarshal cached idempotency response: %w", err)
}
return &result, nil
}
if time.Since(existing.CreatedAt) < reservationTTL {
return nil, ErrRequestInFlight
}
// Stale reservation: the previous holder crashed before completing.
// Release it and retry the reservation.
if err := repo.Delete(ctx, key); err != nil {
return nil, err
}
}
Generics earn their keep here. withIdempotency[T any] wraps any command result type, so CreateProduct, UpdateSeller and friends all get the same behavior from one call:
return withIdempotency(ctx, s.idempotencyRepo, cmd.IdempotencyKey, cmd,
func() (*command.CreateProductCommandResult, error) {
// actual business logic
})
Payload comparison is a straight string compare of the marshaled command. A SHA-256 of the request would work too and keeps rows smaller; storing the full request JSON costs a bit more but is worth it when you're debugging at 2am and want to see what the original request actually was. Pick one, but pick one — skipping the comparison entirely is the real mistake.
The HTTP mapping is boring on purpose:
case errors.Is(err, services.ErrRequestInFlight):
return c.JSON(http.StatusConflict, map[string]string{"error": err.Error()})
case errors.Is(err, services.ErrIdempotencyKeyReuse):
return c.JSON(http.StatusUnprocessableEntity, map[string]string{"error": err.Error()})
Fix two: crashes must not brick a key
Reservation-before-execution creates a new failure mode: reserve the key, then the process dies — OOM kill, deploy, kill -9 — before storing a response. The row now says "in flight" forever, and every retry gets 409 until a human deletes it. Congratulations, you've traded duplicate writes for a permanently wedged operation.
That's what reservationTTL in the loop above is for. A reservation older than the TTL with no stored response means the original holder is dead. The next retry deletes the stale row and re-reserves. And because re-reserving is the same atomic INSERT ... ON CONFLICT, two retries racing to take over a stale key still resolve to exactly one winner.
Sizing the TTL is a judgment call: it must be comfortably longer than your slowest legitimate execution of the wrapped operation, otherwise you'll take over reservations that are merely slow, not dead — and you're back to duplicates. One minute is generous for a CRUD endpoint; a batch operation needs more, or a different mechanism.
Fix three: release on failure — with a detached context
If execution fails, the reservation must be released so the client's retry can actually run instead of hitting 409 until the TTL expires. Easy. Except for one detail that bit me: why did execution fail?
Often because the client disconnected and the request context got cancelled. If you then call repo.Delete(ctx, key) with that same cancelled context, the DELETE never reaches Postgres. The reservation leaks, and the disconnecting client — the exact client most likely to retry — is locked out for a full TTL. The cleanup path fails precisely in the scenario it exists for.
context.WithoutCancel (Go 1.21+) is the fix. It keeps the context's values — trace IDs, loggers — but detaches it from cancellation:
result, err := execute()
if err != nil {
// Detached context: the failure may be a cancelled request context,
// and the release must still reach the database.
if deleteErr := repo.Delete(context.WithoutCancel(ctx), key); deleteErr != nil {
slog.WarnContext(ctx, "failed to release idempotency key",
slog.String("idempotency_key", key), slog.Any("error", deleteErr))
}
return nil, err
}
storeResponse(context.WithoutCancel(ctx), repo, key, result)
Same reasoning for storeResponse: the business operation already committed, so persisting the cached response must not be at the mercy of the request context either. And note it's best-effort — if storing the response fails after the operation succeeded, failing the whole request would be lying to the client about work that's already done. Log it and move on; the TTL takeover covers the wedged in-flight row.
Proving it under concurrency
Sequential tests can't catch the original bug, so the test suite fires actual concurrent goroutines at one key:
const callers = 8
var wg sync.WaitGroup
for i := 0; i < callers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
_, err := withIdempotency(context.Background(), repo, "shared-key", "cmd",
func() (*testResult, error) {
mu.Lock()
executions++
mu.Unlock()
return &testResult{Value: "fresh"}, nil
})
// tally: nil error, ErrRequestInFlight, or fail the test
}()
}
wg.Wait()
assert.Equal(t, 1, executions, "exactly one caller may execute")
assert.Equal(t, callers, successes+inFlight)
The assertion that matters is executions == 1. Successes can be more than one — a loser that arrives after the winner finished legitimately gets the cached response — but the business logic runs exactly once, always. Run it with -race; it's the kind of test that catches you when someone "simplifies" the reservation later.
Against a live server, the behavior looks like this:
for i in $(seq 5); do
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8080/api/v1/products \
-H 'Content-Type: application/json' \
-d '{"idempotency_key":"k-42","name":"Widget","price_cents":999,"currency":"EUR","seller_id":"..."}' &
done; wait
One 201, four 409, one row in the database. Retry any of the 409s a moment later and you get the winner's 201 body back, byte for byte. Send the same key with "name":"Gadget" and you get 422.
What I'd tell past me
The one-sentence version: idempotency is a write-side claim, not a read-side check. If your implementation reads before it writes, it has the race, full stop — the fix is to make the database's unique constraint do the deciding and branch on rows-affected. Everything else here (TTL takeover, payload comparison, detached-context cleanup) is consequences of taking that seriously.
This implementation lives in my Go DDD template at https://github.com/sklinkert/go-ddd — alongside the rest of a working domain-driven setup: entity validation at construction, a Money value object, domain events with a transactional outbox, sqlc repositories, and testcontainers-based integration tests.
Top comments (0)