DEV Community

lukman lukman
lukman lukman

Posted on

Making POST Requests Safe to Retry with Idempotency Keys

#softwaredevelopment #softwareengineering #distributedsystems #go

A practical guide to duplicate execution, request fingerprints, concurrency, and safe retries.

A request timeout does not mean the operation failed.

Sometimes the server has already completed the operation, but the response never reaches the client.

Consider:

POST /orders/order-123/pay
Enter fullscreen mode Exit fullscreen mode

with:

{
  "amount": 500000
}
Enter fullscreen mode Exit fullscreen mode

The timeline may look like this:

Client -> POST /pay
Server -> payment succeeds
Server -> response lost
Client -> timeout
Client -> retry
Server -> payment succeeds again
Enter fullscreen mode Exit fullscreen mode

The problem is not the duplicate HTTP request itself. The problem is duplicate execution.

The target invariant

For a mutation API, the goal should be:

1 logical operation
=
1 effective side effect
Enter fullscreen mode Exit fullscreen mode

even if the request is delivered multiple times.

Introduce an Idempotency Key

The client generates an identifier for one logical operation:

Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Enter fullscreen mode Exit fullscreen mode

The key must remain the same across retries.

same logical operation -> same key
new logical operation  -> new key
Enter fullscreen mode Exit fullscreen mode

The basic flow becomes:

Request
  ↓
Check Idempotency-Key
  ↓
New key?
  ↓
Process operation
  ↓
Store result
  ↓
Return response
Enter fullscreen mode Exit fullscreen mode

On retry:

Same key
  ↓
Existing completed result
  ↓
Replay response
  ↓
Do not execute side effect again
Enter fullscreen mode Exit fullscreen mode

Why check-then-act fails

This is not enough:

if key does not exist:
    process payment
Enter fullscreen mode Exit fullscreen mode

Two concurrent requests may both observe the key as missing:

A -> key not found
B -> key not found

A -> process
B -> process
Enter fullscreen mode Exit fullscreen mode

This is a race condition. The uniqueness decision needs to be atomic.

For a relational database, that normally means enforcing the invariant in storage:

CREATE UNIQUE INDEX idx_idempotency_key
ON payments(idempotency_key);
Enter fullscreen mode Exit fullscreen mode

Application checks are useful. The unique constraint is the final guard.

Same key, different payload

An idempotency key must represent exactly one logical operation.

This should be accepted:

key: abc
amount: 500000

retry

key: abc
amount: 500000
Enter fullscreen mode Exit fullscreen mode

This should not:

key: abc
amount: 500000

retry

key: abc
amount: 800000
Enter fullscreen mode Exit fullscreen mode

The lab uses a request fingerprint based on SHA-256:

func hashRequest(data []byte) string {
    sum := sha256.Sum256(data)
    return hex.EncodeToString(sum[:])
}
Enter fullscreen mode Exit fullscreen mode

Then:

same key + same fingerprint
-> replay safely

same key + different fingerprint
-> 409 Conflict
Enter fullscreen mode Exit fullscreen mode

PROCESSING vs COMPLETED

The safe flow uses two important states:

PROCESSING
COMPLETED
Enter fullscreen mode Exit fullscreen mode

When a second request arrives:

PROCESSING
-> return 409
-> do not execute payment again
Enter fullscreen mode Exit fullscreen mode

When the operation is already finished:

COMPLETED
-> load stored response
-> replay it
Enter fullscreen mode Exit fullscreen mode

Idempotency is not a transaction

A database transaction provides atomicity inside one execution.

Idempotency protects a logical operation across multiple execution attempts.

These can both succeed:

Request A
BEGIN
create payment
COMMIT

Request B
BEGIN
create payment
COMMIT
Enter fullscreen mode Exit fullscreen mode

The database is consistent. The business result is not. The customer was charged twice.

External side effects have a different boundary

Consider:

Call payment provider
↓
Provider SUCCESS
↓
Local commit fails
↓
ROLLBACK
Enter fullscreen mode Exit fullscreen mode

The local rollback does not undo the external charge.

local rollback
!=
external rollback
Enter fullscreen mode Exit fullscreen mode

If the provider supports provider-side idempotency, the backend should reuse a stable key for that external request too.

Lab implementation

The lab intentionally keeps the infrastructure small.

Unsafe:

func ProcessPayment(req PaymentRequest) (PaymentResult, error) {
    return gateway.Charge(req)
}
Enter fullscreen mode Exit fullscreen mode

Safe:

validate key
↓
calculate fingerprint
↓
reserve operation
↓
PROCESSING
↓
execute payment
↓
store result
↓
COMPLETED
Enter fullscreen mode Exit fullscreen mode

The storage implementation uses:

map + sync.RWMutex
Enter fullscreen mode Exit fullscreen mode

This simulates atomic uniqueness without introducing a real database into the lab.

Run both versions:

go test ./labs/01-idempotency/unsafe/... -v -count=1

go test ./labs/01-idempotency/safe/... -v -count=1
Enter fullscreen mode Exit fullscreen mode

Final mental model 

Retry is normal.
Duplicate execution is the problem.
Idempotency makes retry safe.
Enter fullscreen mode Exit fullscreen mode

Source code:

https://github.com/lukman-ss/software-engineering-lab/tree/main/labs/01-idempotency

Top comments (2)

Collapse
 
raknaos profile image
Raknaos •

The distinction that takes the longest to internalise is your "idempotency is not a transaction" point. A transaction guarantees atomicity inside one execution; an idempotency key guarantees a single effect across executions, and those two can both be satisfied by a system that charges the customer twice. The unique index is the right final guard because it fails at the only layer that can't lie to you.

One case the guide could go further on: the request that arrives with the same key while the first one is still mid-flight. Returning 409 on PROCESSING is safe but it pushes the retry decision back onto the client, which usually retries with a new key after a timeout and defeats the whole mechanism. Storing the fingerprint and rejecting on mismatch, plus a bounded wait-and-replay on match, is what made it behave in my setup. How do you handle the key expiring before the stored response is still useful?

Collapse
 
lukman-ss profile image
lukman lukman •

Great point. I think the key distinction is that the idempotency record and the cached HTTP response do not need to have the same lifetime.

I usually treat them separately: the deduplication record keeps the key, fingerprint, final status, and resource reference, while the cached response is only there for convenient replay.

The response can expire earlier, but the dedupe record should live for at least as long as the operation can realistically be retried. If the response is gone but the record still exists, the server should not execute the operation again; it can reconstruct the response from the resource or return a deterministic “already completed” result.

And agreed on PROCESSING: a bounded wait-and-replay is better than a plain 409 when the fingerprint matches. The client should also keep retrying with the same key rather than generating a new one after a timeout.

This is probably worth adding to the guide: idempotency-key TTL, response-cache TTL, and business-operation lifetime are separate concerns.