Idempotency keys in Go: the check-then-act race nobody tests for

A request comes in to charge a customer. It times out on the way back. The client retries. Now you've charged them twice — not because your code is wrong, but because "send it again" is the default behaviour of every HTTP client, load balancer, and job queue between you and the caller.

Idempotency keys are the standard answer: the caller attaches a unique key, you remember it, and a second request with the same key returns the first result instead of doing the work again. Almost every explanation stops there. The part that actually bites is how you remember the key, because the obvious way has a race that never shows up in a single-threaded test.

The version that looks right

// do() runs the side effect and returns the serialized response to cache.
func Charge(ctx context.Context, db *pgxpool.Pool, key string, do func() ([]byte, error)) ([]byte, error) {
    // 1. Have we seen this key before?
    var stored []byte
    err := db.QueryRow(ctx, `SELECT result FROM idempotency WHERE key = $1`, key).Scan(&stored)
    if err == nil {
        return stored, nil // already done — return the first result
    }
    if err != pgx.ErrNoRows {
        return nil, err
    }

    // 2. New key — do the work and remember it.
    result, err := do()
    if err != nil {
        return nil, err
    }
    _, err = db.Exec(ctx, `INSERT INTO idempotency (key, result) VALUES ($1, $2)`, key, result)
    return result, err
}

Read it top to bottom and it's fine. Run it under two concurrent retries of the same key and it isn't.

Both requests hit step 1 at the same time. Neither key exists yet, so both get ErrNoRows. Both fall through to step 2. Both call do(). You've charged the customer twice. One of the two INSERTs will fail on the primary key — but only after both side effects already ran. The unique constraint caught the symptom, not the cause.

This is the failure mode that survives every local test: the test calls the function once, sees one charge, and passes. The bug only exists when two copies of the same request overlap in time, which is exactly what a retry storm produces.

The fix: claim the key before you do the work

The problem is that "check" and "act" are two separate statements with a gap between them. Close the gap by making the claim the first thing that happens, atomically, using the unique constraint as the lock instead of as an afterthought:

func Charge(ctx context.Context, db *pgxpool.Pool, key string, do func() ([]byte, error)) ([]byte, error) {
    // Claim the key. ON CONFLICT DO NOTHING means: if it already exists,
    // insert nothing and return no rows.
    var claimedKey string
    err := db.QueryRow(ctx,
        `INSERT INTO idempotency (key) VALUES ($1)
         ON CONFLICT (key) DO NOTHING
         RETURNING key`,
        key,
    ).Scan(&claimedKey)

    if err == pgx.ErrNoRows {
        // Someone else already claimed this key — they are doing (or did) the work.
        return waitForResult(ctx, db, key)
    }
    if err != nil {
        return nil, err
    }

    // We won the claim. We are the only caller allowed to run the side effect.
    result, err := do()
    if err != nil {
        // Release the claim so a later retry can start cleanly.
        _, _ = db.Exec(ctx, `DELETE FROM idempotency WHERE key = $1`, key)
        return nil, err
    }

    _, err = db.Exec(ctx, `UPDATE idempotency SET result = $2 WHERE key = $1`, key, result)
    return result, err
}

Now the ordering is claim, then act. INSERT ... ON CONFLICT DO NOTHING is atomic: out of any number of concurrent requests, exactly one gets a row back from RETURNING, and every other one gets ErrNoRows. The winner runs do() once; the losers never touch the side effect.

Two things about that RETURNING are worth pinning down, because they're the parts people get wrong:

• ON CONFLICT DO NOTHING together with RETURNING returns no rows on conflict — not the existing row. That's why the loser branch checks for ErrNoRows instead of reading a value. If you genuinely want the existing row back in a single statement, you have to write ON CONFLICT (key) DO UPDATE SET key = EXCLUDED.key RETURNING ..., a no-op write whose only job is to make RETURNING fire. Usually not worth it.
• The claim and the result live in the same row but are written at different times: an empty claim first, the result once do() finishes. Which means a second request can arrive while the first is still running.

The in-progress case

That last point is the one most idempotency write-ups skip. Between "claim the key" and "store the result," there is a window where the row exists but result is still null. A retry that lands in that window can't return the result — it isn't there yet — and it must not run the side effect either.

There's no clever trick here; you pick a policy:

• Wait and poll. The loser re-reads the row until result is non-null, with a timeout. Simple, correct, costs you a held request. That's what waitForResult does above.
• Fail fast. Return a 409 — "a request with this key is already in flight" — and let the client retry later. Cheaper, pushes the wait to the caller.

Either way, the point is that "the key exists" has three states, not two: done, in progress, and failed-and-released. Handle "in progress" as its own case and the race is genuinely closed.

One honest caveat that comes with the claim-first approach: if the winner crashes between claiming the key and writing the result, the row is stuck in "in progress" forever, and every retry waits on a result that will never arrive. So the claim needs a claimed_at timestamp and a sweeper (or a check in the loser branch) that treats a claim older than your side effect's max runtime as abandoned and reclaimable. Without it you've traded a double-charge for a deadlock.

What you actually need

Strip it down and the whole thing is:

• one table with a unique key,
• INSERT ... ON CONFLICT DO NOTHING RETURNING to claim it atomically,
• the side effect gated behind winning the claim,
• a policy for the in-flight window, and a timeout for abandoned claims.

No distributed lock, no coordination service, no exactly-once delivery (which doesn't exist over a network anyway). The database's unique constraint is the only lock you need, and it's atomic by construction. The trick is using it to gate the work, not just to detect a duplicate after the work already happened.