Modern applications rely heavily on APIs. But networks are unreliable - requests fail, time out, or get retried.
Now imagine this scenario:
- User clicks “Pay Now”
- Request sent to server
- Payment processed successfully
- But response is lost due to network issue
- Client retries the request
💥 Payment gets processed again. User is charged twice.
This is one of the most dangerous and common problems in distributed systems.
The solution? Idempotency.
What is Idempotency?
An operation is idempotent if performing it multiple times produces the same result as performing it once.
In simple terms:
Same request → same effect → no duplicate side effects
Example:
Create Order (without idempotency):
Request sent twice → 2 orders created ❌
Create Order (with idempotency):
Request sent twice → only 1 order created ✅
Idempotency ensures system correctness even when retries happen.
Why Do Retries Happen?
Retries are extremely common due to:
- Network timeouts
- Server crashes
- Client retries
- Load balancers retrying requests
- Mobile networks instability
- API gateway retries
- Reverse proxy retries
Retries are normal. Duplicate side effects are not.
Which HTTP Methods Are Idempotent?
According to HTTP specification:
| Method | Idempotent | Explanation |
|---|---|---|
| GET | ✅ Yes | Fetch data |
| PUT | ✅ Yes | Replace resource |
| DELETE | ✅ Yes | Delete resource |
| POST | ❌ No | Create resource |
| PATCH | ❌ No | Partial update |
POST is NOT idempotent by default.
Example:
POST /orders
Calling twice → creates 2 orders.
This must be handled explicitly.
Real-World Example: Payment API Problem
Client sends:
POST /payments
{
"user_id": "123",
"amount": 100
}
If client retries due to timeout:
POST /payments
Without idempotency → duplicate payment ❌
With idempotency → single payment ✅
Solution: Idempotency Key
Client sends a unique key with request:
POST /payments
Idempotency-Key: abc123
Server behavior:
- If key is new → process request
- If key already exists → return previous result
This guarantees single execution.
How Idempotency Works Internally
Flow:
- Client generates unique key
- Sends request with key
- Server checks database/cache for key
- If key exists → return stored response
- If not exists → process request
- Store key and response
- Return response
Go Implementation Example
Let’s implement idempotent payment API using Go.
Database Table
idempotency_keys table:
key (primary key)
response
created_at
Go Example (Basic Version)
type IdempotencyRecord struct {
Key string
Response string
}
Handler Implementation
func PaymentHandler(w http.ResponseWriter, r *http.Request) {
key := r.Header.Get("Idempotency-Key")
if key == "" {
http.Error(w, "Idempotency-Key required", http.StatusBadRequest)
return
}
// Check if key exists
record, exists := getIdempotencyRecord(key)
if exists {
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(record.Response))
return
}
// Process payment
paymentID := processPayment()
response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)
// Store key and response
saveIdempotencyRecord(key, response)
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(response))
}
Example Flow
First request:
POST /payments
Idempotency-Key: abc123
Response:
{
"payment_id": "pay_001"
}
Retry request:
POST /payments
Idempotency-Key: abc123
Response returned from storage:
{
"payment_id": "pay_001"
}
No duplicate payment created.
Production Implementation Using Redis (Recommended)
Redis is ideal because:
- Extremely fast
- Supports expiration (TTL)
- Perfect for temporary idempotency storage
Redis Example in Go
func PaymentHandler(w http.ResponseWriter, r *http.Request) {
key := r.Header.Get("Idempotency-Key")
val, err := redisClient.Get(ctx, key).Result()
if err == nil {
w.Write([]byte(val))
return
}
paymentID := processPayment()
response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)
redisClient.Set(ctx, key, response, time.Hour*24)
w.Write([]byte(response))
}
Important: Handle Race Conditions
Two requests may arrive simultaneously with same key.
Solution: Use atomic operations.
Redis example:
SET key value NX
NX = set only if not exists
Important Best Practice: Store Full Response
Do not just store key.
Store:
- status code
- response body
- timestamp
Because retry must return exact same response.
Example Database Schema (PostgreSQL)
CREATE TABLE idempotency_keys (
key TEXT PRIMARY KEY,
response JSONB,
status_code INT,
created_at TIMESTAMP DEFAULT NOW()
);
Primary key ensures uniqueness.
Real-World Systems That Use Idempotency
Critical systems use idempotency extensively:
- Payment APIs
- Order creation APIs
- Financial transactions
- Booking systems
- Wallet transfers
- Stripe payments
- Banking APIs
Without idempotency, financial systems would break.
Common Mistake Developers Make
❌ Using retry without idempotency
retry 3 times on failure
This multiplies side effects.
Example:
Retry logic: 3 times
Result: 3 payments
Dangerous bug.
Idempotency vs Duplicate Prevention Using Unique Constraints
Example:
UNIQUE(user_id, order_id)
This prevents duplicates at DB level.
But idempotency is better because:
- Works at API level
- Handles retries cleanly
- Returns same response
Best approach: use BOTH.
When Should You Use Idempotency?
Use idempotency for operations that create side effects:
- Payment creation
- Order creation
- Wallet transfer
- Booking
- Resource creation
Not required for:
- GET requests
- Read-only operations
How Client Should Generate Idempotency Key
Use UUID:
Example:
550e8400-e29b-41d4-a716-446655440000
Go example:
uuid.New().String()
Unique per operation.
Idempotency Key Expiration Strategy
Keys should expire after some time.
Common TTL:
- 24 hours
- 48 hours
- 7 days (depending on business need)
Redis makes expiration easy.
Example Architecture
Client → API Gateway → Service → Redis → Database
Redis handles idempotency keys
Database handles actual data
Fast and reliable.
Real Production Scenario: Mobile Network Retry
Mobile user taps "Place Order"
Network slow → timeout
Client retries automatically
Without idempotency → duplicate orders
With idempotency → safe retry
Important Difference: Safe Retry vs Unsafe Retry
Unsafe retry:
POST /orders
Safe retry:
POST /orders
Idempotency-Key: unique-key
Idempotency vs Exactly-Once Execution
Exactly-once execution is impossible in distributed systems.
Idempotency provides practical solution.
It ensures same final state.
Best Practices Checklist
Always:
- Use idempotency keys for POST APIs
- Store keys in Redis or database
- Store full response
- Use expiration
- Handle race conditions
- Use UUID keys
- Combine with database constraints
Key Takeaway
Retries are normal.
Duplicate side effects are dangerous.
Idempotency ensures your APIs remain safe, reliable, and production-ready.
It is not optional for critical APIs - it is essential.
Final Thought
If your API handles payments, orders, bookings, or financial transactions - idempotency is mandatory.
Without it, retries can silently corrupt your system.
With it, your APIs become reliable, predictable, and safe.
Top comments (0)