DEV Community: Simon Klinkert

Race-Safe Idempotent POST Endpoints in Go, the Hard Way

Simon Klinkert — Thu, 16 Jul 2026 22:35:02 +0000

Your client sends a POST /products. The server creates the product, then the load balancer kills the connection before the response gets out. The client sees a timeout. What does every reasonable HTTP client do on a timeout? It retries. Now you have two products.

This is not an exotic failure mode. It's Tuesday. Mobile clients on flaky networks, Kubernetes rolling deploys, aggressive proxy timeouts — anything that separates "the server did the work" from "the client learned about it" will eventually duplicate your writes. The standard fix is an idempotency key: the client generates a unique key per logical operation, sends it with the request, and the server guarantees that the same key never executes the operation twice.

The concept is simple. The implementation is where I got burned, so let me walk through the version I actually ship now, including the parts I got wrong the first time.

The naive version (which I shipped)

Here's the implementation almost everyone writes first, me included:

func (s *ProductService) CreateProduct(ctx context.Context, cmd *CreateProductCommand) (*Result, error) {
    existing, err := s.idempotencyRepo.FindByKey(ctx, cmd.IdempotencyKey)
    if err != nil {
        return nil, err
    }
    if existing != nil {
        return unmarshalCachedResponse(existing)
    }

    result, err := s.doCreateProduct(ctx, cmd)
    if err != nil {
        return nil, err
    }

    s.idempotencyRepo.Store(ctx, cmd.IdempotencyKey, result)
    return result, nil
}

Check, execute, store. It reads correctly. It passes every test you'll write for it, because sequential tests can't see the bug.

The bug: two requests with the same key arrive at the same time. Both call FindByKey. Both get nil — the key hasn't been stored yet, because storing happens after execution. Both execute. Two products, one idempotency key, and the whole mechanism silently did nothing exactly when it mattered.

Concurrent duplicates are not a corner case, they're the main case. Retries fire because something is slow, and when something is slow the original request is often still running. The naive check-then-act version protects you against the retry that arrives after the first request finished — the one scenario that was never dangerous to begin with.

I ran this bug in production for a while before a payment-adjacent endpoint made it visible. Ask me how I know duplicate writes are hard to clean up.

Fix one: reserve the key atomically

The check and the claim have to be one atomic operation. In Postgres that's a unique constraint plus ON CONFLICT DO NOTHING:

-- name: ReserveIdempotencyKey :execrows
-- Atomically claims the key. Zero rows means another request already holds it.
INSERT INTO idempotency_records (id, key, request, response, status_code, created_at)
VALUES ($1, $2, $3, '', 0, $4)
ON CONFLICT (key) DO NOTHING;

The :execrows annotation is sqlc for "give me the affected row count". That count is the whole trick: 1 means you won the race and may execute; 0 means somebody else holds the key and you must not.

func (r *SqlcIdempotencyRepository) Reserve(ctx context.Context, record *entities.IdempotencyRecord) (bool, error) {
    rows, err := r.queries.ReserveIdempotencyKey(ctx, db.ReserveIdempotencyKeyParams{
        ID:        record.Id,
        Key:       record.Key,
        Request:   record.Request,
        CreatedAt: timestamptzFromTime(record.CreatedAt),
    })
    if err != nil {
        return false, err
    }
    return rows > 0, nil
}

Note what changed structurally: the row is inserted before execution, as a reservation, not after execution as a cache entry. response = '' and status_code = 0 mean "in flight". Postgres's unique index is now the arbiter of who executes — no advisory locks, no Redis, no distributed-lock library. Under concurrency, exactly one INSERT wins. That guarantee comes from the database, not from my code, which is exactly where I want it.

The losers need answers too

If you lost the reservation race, what do you tell the client? Depends on what the winner is doing:

Winner already finished → serve its stored response. The retry gets the same 201 body the original would have gotten.
Winner still running → 409 Conflict. The client should back off and retry; on the retry it'll hit the cached response.
Same key, different payload → 422 Unprocessable Entity. This one is easy to skip and dangerous to skip. If a client bug reuses a key across different requests, silently returning the first request's cached response means the caller thinks it created product B while you hand it product A. Fail loudly instead.

The whole flow lives in one generic decorator that every command-side service method wraps around its work. Trimmed to the interesting part:

reserved := false
for attempt := 0; attempt < 3 && !reserved; attempt++ {
    reserved, err = repo.Reserve(ctx, record)
    if err != nil {
        return nil, err
    }
    if reserved {
        break
    }

    existing, err := repo.FindByKey(ctx, key)
    if err != nil {
        return nil, err
    }
    if existing == nil {
        // Released between Reserve and FindByKey; try again.
        continue
    }

    if existing.Request != string(requestJSON) {
        return nil, ErrIdempotencyKeyReuse
    }

    if existing.IsCompleted() {
        var result T
        if err := json.Unmarshal([]byte(existing.Response), &result); err != nil {
            return nil, fmt.Errorf("unmarshal cached idempotency response: %w", err)
        }
        return &result, nil
    }

    if time.Since(existing.CreatedAt) < reservationTTL {
        return nil, ErrRequestInFlight
    }

    // Stale reservation: the previous holder crashed before completing.
    // Release it and retry the reservation.
    if err := repo.Delete(ctx, key); err != nil {
        return nil, err
    }
}

Generics earn their keep here. withIdempotency[T any] wraps any command result type, so CreateProduct, UpdateSeller and friends all get the same behavior from one call:

return withIdempotency(ctx, s.idempotencyRepo, cmd.IdempotencyKey, cmd,
    func() (*command.CreateProductCommandResult, error) {
        // actual business logic
    })

Payload comparison is a straight string compare of the marshaled command. A SHA-256 of the request would work too and keeps rows smaller; storing the full request JSON costs a bit more but is worth it when you're debugging at 2am and want to see what the original request actually was. Pick one, but pick one — skipping the comparison entirely is the real mistake.

The HTTP mapping is boring on purpose:

case errors.Is(err, services.ErrRequestInFlight):
    return c.JSON(http.StatusConflict, map[string]string{"error": err.Error()})
case errors.Is(err, services.ErrIdempotencyKeyReuse):
    return c.JSON(http.StatusUnprocessableEntity, map[string]string{"error": err.Error()})

Fix two: crashes must not brick a key

Reservation-before-execution creates a new failure mode: reserve the key, then the process dies — OOM kill, deploy, kill -9 — before storing a response. The row now says "in flight" forever, and every retry gets 409 until a human deletes it. Congratulations, you've traded duplicate writes for a permanently wedged operation.

That's what reservationTTL in the loop above is for. A reservation older than the TTL with no stored response means the original holder is dead. The next retry deletes the stale row and re-reserves. And because re-reserving is the same atomic INSERT ... ON CONFLICT, two retries racing to take over a stale key still resolve to exactly one winner.

Sizing the TTL is a judgment call: it must be comfortably longer than your slowest legitimate execution of the wrapped operation, otherwise you'll take over reservations that are merely slow, not dead — and you're back to duplicates. One minute is generous for a CRUD endpoint; a batch operation needs more, or a different mechanism.

Fix three: release on failure — with a detached context

If execution fails, the reservation must be released so the client's retry can actually run instead of hitting 409 until the TTL expires. Easy. Except for one detail that bit me: why did execution fail?

Often because the client disconnected and the request context got cancelled. If you then call repo.Delete(ctx, key) with that same cancelled context, the DELETE never reaches Postgres. The reservation leaks, and the disconnecting client — the exact client most likely to retry — is locked out for a full TTL. The cleanup path fails precisely in the scenario it exists for.

context.WithoutCancel (Go 1.21+) is the fix. It keeps the context's values — trace IDs, loggers — but detaches it from cancellation:

result, err := execute()
if err != nil {
    // Detached context: the failure may be a cancelled request context,
    // and the release must still reach the database.
    if deleteErr := repo.Delete(context.WithoutCancel(ctx), key); deleteErr != nil {
        slog.WarnContext(ctx, "failed to release idempotency key",
            slog.String("idempotency_key", key), slog.Any("error", deleteErr))
    }
    return nil, err
}

storeResponse(context.WithoutCancel(ctx), repo, key, result)

Same reasoning for storeResponse: the business operation already committed, so persisting the cached response must not be at the mercy of the request context either. And note it's best-effort — if storing the response fails after the operation succeeded, failing the whole request would be lying to the client about work that's already done. Log it and move on; the TTL takeover covers the wedged in-flight row.

Proving it under concurrency

Sequential tests can't catch the original bug, so the test suite fires actual concurrent goroutines at one key:

const callers = 8
var wg sync.WaitGroup
for i := 0; i < callers; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        _, err := withIdempotency(context.Background(), repo, "shared-key", "cmd",
            func() (*testResult, error) {
                mu.Lock()
                executions++
                mu.Unlock()
                return &testResult{Value: "fresh"}, nil
            })
        // tally: nil error, ErrRequestInFlight, or fail the test
    }()
}
wg.Wait()

assert.Equal(t, 1, executions, "exactly one caller may execute")
assert.Equal(t, callers, successes+inFlight)

The assertion that matters is executions == 1. Successes can be more than one — a loser that arrives after the winner finished legitimately gets the cached response — but the business logic runs exactly once, always. Run it with -race; it's the kind of test that catches you when someone "simplifies" the reservation later.

Against a live server, the behavior looks like this:

for i in $(seq 5); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8080/api/v1/products \
    -H 'Content-Type: application/json' \
    -d '{"idempotency_key":"k-42","name":"Widget","price_cents":999,"currency":"EUR","seller_id":"..."}' &
done; wait

One 201, four 409, one row in the database. Retry any of the 409s a moment later and you get the winner's 201 body back, byte for byte. Send the same key with "name":"Gadget" and you get 422.

What I'd tell past me

The one-sentence version: idempotency is a write-side claim, not a read-side check. If your implementation reads before it writes, it has the race, full stop — the fix is to make the database's unique constraint do the deciding and branch on rows-affected. Everything else here (TTL takeover, payload comparison, detached-context cleanup) is consequences of taking that seriously.

This implementation lives in my Go DDD template at https://github.com/sklinkert/go-ddd — alongside the rest of a working domain-driven setup: entity validation at construction, a Money value object, domain events with a transactional outbox, sqlc repositories, and testcontainers-based integration tests.

Stop Storing Money in float64

Simon Klinkert — Tue, 14 Jul 2026 20:31:02 +0000

I've been writing Go backends for over a decade, and money handling is the one topic where I keep seeing the same bug in codebase after codebase: a Price float64 field somewhere in a struct, a DECIMAL column mapped to float64 by the ORM, and a team that swears the numbers "look fine" — right up until finance asks why the invoice totals are off by a cent.

I got this wrong myself for years. This post is the fix I finally settled on and now bake into every project: a Money value object built on integer minor units, with the currency attached, and validation that makes invalid money literally unconstructible. All code below is real code from my go-ddd template repo (just tagged v1.0.0).

Why float64 money is broken

Not "risky in theory". Broken. Binary floating point cannot represent most decimal fractions exactly. 0.1 doesn't exist in float64 — you get the closest representable value, which is 0.1000000000000000055511151231257827021181583404541015625.

You've seen this one:

package main

import "fmt"

func main() {
    fmt.Println(0.1 + 0.2)                 // 0.30000000000000004
    fmt.Println(0.1+0.2 == 0.3)           // false
}

Cute party trick, easy to dismiss. The version that actually hurts is drift under accumulation, because real systems don't add two prices — they sum line items, apply percentages, aggregate daily revenue:

package main

import "fmt"

func main() {
    var total float64
    for i := 0; i < 1000; i++ {
        total += 0.10 // a 10-cent fee, a thousand times
    }
    fmt.Println(total)        // 99.9999999999986
    fmt.Println(total == 100) // false
}

A thousand 10-cent charges and you've already lost the exact comparison. Now imagine that total feeding a >= threshold check, or a reconciliation job diffing against the payment provider's report. The diff is never zero, someone writes an epsilon comparison, the epsilon is wrong for large amounts, and you're in whack-a-mole territory.

And float64 has a second, quieter problem that has nothing to do with binary representation: it's a bare number. 19.99 of what? EUR? USD? Cents already? I've debugged a production incident where one service sent euros and another interpreted the same field as cents. The type system happily let a 100x pricing error through, because float64 == float64.

So the fix has to solve both: exact arithmetic and an amount that can't be separated from its currency.

The Money value object

The whole thing fits in one small file. Amount in minor units (cents) as int64, currency as a typed string, both fields unexported:

// Currency is the ISO 4217 code of a Money value.
type Currency string

const (
    EUR Currency = "EUR"
    USD Currency = "USD"
)

var supportedCurrencies = map[Currency]struct{}{
    EUR: {},
    USD: {},
}

// Money is an immutable value object storing an amount in minor units
// (cents) to avoid floating-point rounding errors.
type Money struct {
    cents    int64
    currency Currency
}

func NewMoney(cents int64, currency Currency) (Money, error) {
    if cents < 0 {
        return Money{}, fmt.Errorf("%w: amount must not be negative", ErrValidation)
    }
    if _, ok := supportedCurrencies[currency]; !ok {
        return Money{}, fmt.Errorf("%w: unsupported currency %q", ErrValidation, currency)
    }

    return Money{cents: cents, currency: currency}, nil
}

func (m Money) Cents() int64      { return m.cents }
func (m Money) Currency() Currency { return m.currency }

func (m Money) String() string {
    return fmt.Sprintf("%d.%02d %s", m.cents/100, m.cents%100, m.currency)
}

Design decisions worth spelling out:

Unexported fields, constructor with validation. Because cents and currency are unexported, the only way to build a Money outside the package is NewMoney. That means every Money in the system passed the negative check and the currency whitelist. There's no "construct it raw and validate later" path to forget. This is the value-object idea from DDD in its most useful form: make invalid states unrepresentable, then stop re-validating everywhere else.

Integer cents, not big.Rat or a decimal library. int64 cents gives exact addition and comparison for free, is trivially indexable and sortable in the DB, and covers roughly ±92 quadrillion dollars. If you need arbitrary precision (FX rates, interest calc), reach for a decimal type — but for prices and balances, minor units are the boring answer that works.

Currency is part of equality. Money is a comparable struct, so == compares amount and currency. NewMoney(1000, EUR) != NewMoney(1000, USD). The euros-vs-cents-vs-dollars class of bug now fails at the type level instead of on an accountant's spreadsheet.

A whitelist of currencies. Two entries look restrictive, and that's the point. The domain says which currencies the business actually supports. Adding one is a one-line change and forces you to think about it — which is exactly what you want when the alternative is silently accepting "BTC" or "EURO" from a client.

Using it in the domain

The Product entity takes a Money, not a number:

type Product struct {
    Id        uuid.UUID
    CreatedAt time.Time
    UpdatedAt time.Time
    Name      string
    Price     Money
    SellerId  uuid.UUID
}

func (p *Product) validate() error {
    if p.Name == "" {
        return fmt.Errorf("%w: name must not be empty", ErrValidation)
    }
    if p.Price.Cents() == 0 {
        return fmt.Errorf("%w: price must be greater than 0", ErrValidation)
    }
    // ...
}

Notice what the entity does not have to check: negative prices, garbage currencies. NewMoney already refused those. The entity only adds the rule that belongs to products — a price of zero makes no sense for a product, but zero is a perfectly fine Money elsewhere (a discount that bottomed out, an empty balance). Each layer validates its own invariant, once.

Surviving the round trips: DB and JSON

A value object is only as good as its edges. Money enters and leaves your process constantly — database, API, message queue — and every crossing is a chance to smuggle in an invalid value.

Database. The schema mirrors the value object: a BIGINT for cents and a TEXT for the currency. Here's the actual migration from the repo, moving off the old DECIMAL column:

-- Store money as integer minor units plus an ISO 4217 currency code.
-- Floating point (and implicit currency) is how money bugs are born.
ALTER TABLE products ADD COLUMN price_cents BIGINT;
UPDATE products SET price_cents = ROUND(price * 100);
ALTER TABLE products ALTER COLUMN price_cents SET NOT NULL;
ALTER TABLE products DROP COLUMN price;

ALTER TABLE products ADD COLUMN currency TEXT;
UPDATE products SET currency = 'USD';
ALTER TABLE products ALTER COLUMN currency SET NOT NULL;

The UPDATE ... SET currency = 'USD' line is the honest part of this migration: the old schema had no idea what currency those decimals were in. We were only able to backfill because the system happened to be single-currency. If it hadn't been, this migration would have been a data archaeology project. Attach the currency before you need to.

JSON. This is the part most people skip. encoding/json bypasses your constructor — it writes straight into struct fields via reflection. With unexported fields it can't, so we implement the interfaces ourselves and route unmarshaling back through NewMoney:

type moneyJSON struct {
    Cents    int64    `json:"cents"`
    Currency Currency `json:"currency"`
}

func (m Money) MarshalJSON() ([]byte, error) {
    return json.Marshal(moneyJSON{Cents: m.cents, Currency: m.currency})
}

// UnmarshalJSON goes through NewMoney so a Money can never be deserialized
// into an invalid state.
func (m *Money) UnmarshalJSON(data []byte) error {
    var raw moneyJSON
    if err := json.Unmarshal(data, &raw); err != nil {
        return err
    }

    money, err := NewMoney(raw.Cents, raw.Currency)
    if err != nil {
        return err
    }

    *m = money
    return nil
}

Why bother, when the value was valid when we serialized it? Because JSON doesn't only come from us. It comes from an outbox table written by last year's code, a queue message from another service, a fixture someone hand-edited. Revalidating on the way in costs two comparisons and closes the whole category.

The API contract

At the REST edge I don't expose the domain type — DTOs carry the same shape explicitly:

type CreateProductRequest struct {
    IdempotencyKey string `json:"idempotency_key"`
    Name           string `json:"name"`
    PriceCents     int64  `json:"price_cents"`
    Currency       string `json:"currency"`
    SellerId       string `json:"seller_id"`
}

The field is named price_cents, not price. That naming does real work: no client developer will ever wonder whether to send 19.99 or 1999. The response mirrors it (PriceCents int64, Currency string), and the app layer converts the raw DTO values through NewMoney, so a request with "currency": "GBP" or a negative amount gets a 400 before it ever touches an entity.

Yes, that means the number 1999 on the wire instead of a pretty "19.99". Formatting is a display concern; let the frontend format for locale. Wire formats should be exact.

What this doesn't solve

I'd rather tell you the sharp edges myself:

Display formatting is on you. My String() does cents/100 with a two-digit remainder, which is correct for EUR and USD and wrong for JPY (0 decimal places) or KWD (3). If you go properly multi-currency, you need per-currency exponents — steal them from ISO 4217 or use a library.
Cross-currency arithmetic needs a decision. The moment you add Add() or Sub(), define what EUR + USD does. My answer: it returns an error, and conversion is an explicit domain operation with a rate and a timestamp. Never implicit.
Division and percentages still round. Integer cents make addition exact, but 100 cents split three ways is still 33+33+34. You need an allocation strategy (largest-remainder works fine) — no representation saves you from that.
DECIMAL in the DB is fine, actually — if you read it into a decimal type, not a float. If your reporting team lives in SQL and wants SUM(price) to look like money, NUMERIC(12,2) plus a currency column is a legitimate choice. What's not legitimate is DECIMAL in Postgres scanned into float64 in Go, which is the worst of both worlds and, in my experience, the most common setup out there.

The pattern's core is small enough to remember: integer minor units, currency attached, one constructor, and every deserialization path goes back through it. Everything else — the migration, the DTO naming, the entity validation — falls out of that.

The full implementation lives in my DDD template at https://github.com/sklinkert/go-ddd — alongside the rest of the patterns I keep rebuilding: validated entities, a transactional outbox for domain events, race-safe idempotency keys, and testcontainers-based integration tests.