Imagine you are ordering food online.
You tap “Place Order.”
The app starts processing your order, but the internet connection suddenly drops. You don't know whether the order was placed successfully.
So, you tap the button again.
A few seconds later, you discover that two orders have been created. 😬
Now imagine the same thing happening with a banking application. You tap “Transfer Money,” the request times out, and you retry it. Your account gets debited twice.
The real problem isn't the retry. The problem is that repeating the same operation can produce an unintended additional effect.
This is where idempotency becomes one of the most important concepts in backend engineering and system design.
Let's understand it step by step.
1. What Is Idempotency?
In simple words:
Idempotency means that performing the same operation multiple times has the same intended effect as performing it once.
Consider setting your account's notification preference to OFF.
- First request: Notifications are turned off.
- Second request: Notifications are still off.
- Third request: Notifications are still off.
The final state remains the same.
Now consider transferring $100 to a friend.
- First request: $100 is transferred.
- Second request: Another $100 is transferred.
- Third request: Yet another $100 is transferred.
This operation is not naturally idempotent.
The technical definition
In mathematics, an operation (f) is idempotent if:
[
f(f(x)) = f(x)
]
In backend engineering, an operation is idempotent when repeating it produces the same intended effect on the system as performing it once.
Important: This does not necessarily mean every response must be identical, nor does it mean the server executes the request only once.
For example, the first request might return 200 OK, while a repeated request returns 204 No Content. The responses differ, but the intended final state can remain unchanged.
2. Build Your Mental Model
Think of an elevator.
You press the button for the fifth floor. Then you press it again.
You still want to go to the fifth floor—not the tenth.
The operation “set destination to floor five” is idempotent.
Now imagine an elevator button that means “move up one floor.” Pressing it twice moves the elevator two floors.
That operation is not idempotent.
The difference is simple:
- Set something to a specific state: Often idempotent.
- Apply an additional change each time: Often non-idempotent.
This distinction will help you recognize idempotency in APIs, databases, payment systems, and distributed applications.
3. Why Do We Need Idempotency?
Because modern applications cannot guarantee that every request will be delivered and acknowledged perfectly.
A request can time out even after the server has completed its work.
Client Server
| |
|------ Create Order ----->|
| |
| Order created
| |
|<----- Response lost -----X
|
| "Did it work?"
|
|------ Retry Request ---->|
| |
| Another order?
The client cannot reliably distinguish between these situations just from a timeout:
- The server never received the request.
- The server received it but did not finish processing it.
- The server completed the operation, but the response was lost.
This uncertainty is why clients retry requests.
Retries improve reliability, but they can also create duplicate side effects.
Idempotency lets us make retries safer.
What problem does it solve?
It helps prevent unintended duplicate effects, such as:
- Charging a customer twice.
- Creating duplicate orders.
- Transferring money multiple times.
- Issuing duplicate refunds.
- Processing the same webhook repeatedly.
- Creating duplicate background jobs' business effects.
Idempotency does not prevent every failure. It provides a way to make repeated operations safe.
4. Which HTTP Methods Are Idempotent?
HTTP defines certain methods as idempotent based on their intended effect on the server.
| HTTP method | Idempotent by definition? | Example |
|---|---|---|
GET |
Yes | Retrieve an order |
HEAD |
Yes | Retrieve headers without the response body |
PUT |
Yes | Replace a resource with a specified representation |
DELETE |
Yes | Ensure a resource is deleted |
OPTIONS |
Yes | Discover supported communication options |
TRACE |
Yes | Perform a diagnostic trace |
POST |
No, not inherently | Create an order or initiate a payment |
PATCH |
Not guaranteed | Partially update a resource |
Let's focus on the methods you will most commonly encounter in backend interviews.
GET — Idempotent
GET /api/orders/101
Calling this endpoint repeatedly should not create additional orders or otherwise change the intended server state.
Best practice: GET requests should be safe to repeat and should not trigger business-changing side effects.
PUT — Idempotent
PUT /api/users/42/preferences
Content-Type: application/json
{
"notifications_enabled": false
}
The request specifies the desired representation of the resource.
Repeating it should leave the preferences in the same intended state.
DELETE — Idempotent
DELETE /api/orders/101
The first request might delete the order. A subsequent request might return 404 Not Found.
Does that mean DELETE is no longer idempotent?
No. The responses can differ. What matters is that repeating the deletion does not keep producing additional intended state changes.
POST — Not inherently idempotent
POST /api/orders
Content-Type: application/json
{
"product_id": 15,
"quantity": 1
}
Each successful execution could create another order.
However, POST can be designed to be idempotent using an idempotency key.
PATCH — Not guaranteed to be idempotent
Consider this request:
PATCH /api/accounts/42
Content-Type: application/json
{
"increment_balance_by": 100
}
Repeating the request could increase the balance multiple times.
Now compare it with:
PATCH /api/users/42
Content-Type: application/json
{
"display_name": "Obaid"
}
If the operation simply sets the display name to Obaid, repeating it can be idempotent.
Therefore, PATCH is not inherently idempotent, but a particular PATCH operation can be idempotent.
The distinction to remember:
HTTP method semantics tell you what is expected of the method. The actual implementation determines whether the business operation is safe to repeat.
5. How Do We Make POST Idempotent?
Let's return to our food-ordering application.
The frontend sends a request to create an order. If the response times out, the frontend retries with the same idempotency key.
POST /api/orders
Idempotency-Key: order-checkout-8f71
{
"product_id": 15,
"quantity": 1
}
The key is a unique identifier for one logical operation.
A logical operation means the user's original intention, such as placing one particular order, even if the application sends the request multiple times.
The server stores the key and associates it with the result of that operation.
First request
|
v
Idempotency key: order-checkout-8f71
|
v
Key not found?
|
v
Create order + persist result
|
v
Return order #501
Retry with the SAME key
|
v
Key already processed?
|
v
Return stored result for order #501
|
v
No second order created
If the user intentionally places another order, the application must generate a new key.
Same logical operation → same key.
New logical operation → new key.
A production-grade implementation needs more than a key
The backend should:
- Generate or validate the key.
- Scope it to the correct customer and operation.
- Store it durably, with a unique database constraint.
- Atomically associate it with the business operation.
- Return the original result when the same completed request is retried.
- Reject reuse of the same key with a different payload.
- Handle requests that arrive concurrently with the same key.
- Define how in-progress operations, failures, and key expiration work.
A simple in-memory hash is not sufficient for a production system: it can disappear after a restart and is not shared across multiple application servers.
6. Production Example: Ruby on Rails
Suppose our Rails application processes payments.
A simplified database design might look like this:
payments
--------
id
user_id
amount
status
idempotency_records
-------------------
id
user_id
key
request_fingerprint
payment_id
status
The request_fingerprint is a stable representation or hash of the relevant request data. It helps detect when a client reuses a key for a different operation.
A unique database index could be:
add_index :idempotency_records,
[:user_id, :key],
unique: true
The database constraint is critical because two application servers can receive the same request simultaneously.
Checking for an existing key in Ruby alone is not enough.
A simplified Rails service
class CreatePayment
def initialize(user:, amount:, idempotency_key:)
@user = user
@amount = amount
@idempotency_key = idempotency_key
end
def call
Payment.transaction do
record = IdempotencyRecord.create_or_find_by!(
user: @user,
key: @idempotency_key
) do |r|
r.request_fingerprint = fingerprint
r.status = "processing"
end
record.lock!
unless record.request_fingerprint == fingerprint
raise ArgumentError, "Idempotency key reused with different input"
end
if record.status == "completed"
return record.payment
end
payment = Payment.create!(
user: @user,
amount: @amount,
status: "pending"
)
record.update!(
payment: payment,
status: "completed"
)
payment
end
end
private
def fingerprint
Digest::SHA256.hexdigest(
{ amount: @amount.to_d.to_s("F") }.to_json
)
end
end
This example illustrates the core idea, but it is not a complete payment-processing implementation. A real system must handle concurrent requests, transaction isolation, failure recovery, key ownership, currency and other relevant request fields, and payment-provider interactions.
For a local database operation, storing the business record and the idempotency result in the same transaction can provide atomicity: either both changes commit, or neither does.
For an external payment provider, a local database transaction cannot roll back a charge that has already succeeded at the provider. That requires additional safeguards.
Never call an external payment provider inside a database transaction and assume the transaction makes the entire workflow atomic.
7. What About External Payment Providers?
Imagine the payment provider successfully charges the card, but your Rails server crashes before saving the successful result.
Rails application Payment provider Database
| | |
|---- Charge $100 ------>| |
| | |
|<--- Payment success ---| |
| | |
X Server crashes | |
| |
Retry after recovery
If the application creates a new payment operation blindly, it could charge the customer twice.
A robust design uses the same logical operation identifier when communicating with a provider that supports idempotency.
Checkout request
|
v
Persist payment intent + operation key
|
v
Send charge request with stable provider key
|
v
Provider processes the operation
|
v
Persist/reconcile provider result
|
v
Return the known result to the client
If the request times out, the application retries with the same provider key or retrieves the existing operation's status.
A practical architecture often includes:
- A durable payment intent.
- A stable idempotency key.
- A database uniqueness constraint.
- A payment state machine, such as
pending,succeeded, andfailed. - Reconciliation for uncertain outcomes.
- Webhook deduplication.
- Monitoring and alerts for operations stuck in
pending.
Important: Provider idempotency guarantees and retention periods vary. Follow the provider's documented behavior rather than assuming that keys remain valid forever.
8. What New Problems Does Idempotency Create?
Every reliability technique introduces design decisions of its own.
Problem 1: Concurrent requests
Two requests with the same key may arrive at almost exactly the same time.
Request A ----\
>---- Server ----> Database
Request B ----/
If both requests check for the key before either inserts it, both might attempt the operation.
Solution: Use a unique constraint, transactional processing, and a clear conflict-handling strategy. A database constraint protects against races that application-level checks cannot prevent.
Problem 2: The same key, different payload
A client submits a $100 payment using key abc123, then accidentally submits $500 using that same key.
The server must not silently treat the second request as a valid retry of the first.
Solution: Store a request fingerprint and reject mismatched payloads.
Problem 3: In-progress requests
What happens when a duplicate arrives while the original operation is still processing?
Possible approaches include:
- Wait for the first operation to finish.
- Return a documented conflict or in-progress response.
- Return a status resource the client can poll.
Choose behavior that clients can understand and implement consistently.
Problem 4: Key expiration
If the server deletes an idempotency record too soon, a delayed retry might create a duplicate operation.
Solution: Choose retention periods based on the operation's risk and the retry window. For high-value operations, retain a durable business-level uniqueness record even if temporary response caches expire.
Problem 5: External side effects
A database transaction cannot atomically commit both a local database change and an unrelated external payment, email, or shipping-provider action.
Solution: Use durable workflows, the transactional outbox pattern where appropriate, provider idempotency, retries, and reconciliation.
9. Idempotency in Stateless vs Stateful Systems
This is where the concept becomes particularly interesting in system design.
A stateless application server does not depend on its own local memory to remember previous requests.
But idempotency requires the system to remember which logical operations have already been processed.
Does that make the application stateful?
Not necessarily.
Load Balancer
/ \
v v
Rails Server A Rails Server B
\ /
\ /
v v
Shared Database
----------------
Idempotency keys
Operation status
Stored results
Both application servers can be stateless with respect to the request history because the shared database stores that history.
This gives us an important distinction:
- Stateless application server: Does not rely on its own local memory to retain request history between calls.
- Stateful system: Maintains durable information about operations, users, transactions, or other business facts.
A stateless application tier can be part of a stateful overall system.
This architecture supports horizontal scaling: adding more application servers does not require each server to maintain its own copy of idempotency history.
Why not store the keys in Redis only?
Redis can be useful for fast lookups, coordination, or temporary records. However, its suitability depends on durability settings, eviction policies, failover behavior, and the consequences of losing a key.
For critical financial operations, durable database constraints and the payment provider's own safeguards are usually more important than a cache-only deduplication strategy.
Use Redis as part of the design when appropriate, not as a substitute for a reliable source of truth without understanding its failure modes.
10. Idempotency and the Transactional Outbox
Suppose a Rails application creates an order and then publishes an event to a background worker.
Two things must happen:
- Save the order in the database.
- Ensure the system can eventually publish an
OrderCreatedevent.
If the order commits but event publication fails, the system may never notify the downstream service.
The transactional outbox pattern addresses this dual-write problem.
Within one database transaction, save both the order and an outbox event.
Database transaction
|
+---- Create order
|
+---- Create outbox event
|
+---- Commit both
|
v
Background publisher
|
v
Message broker
|
v
Consumer processes event
The publisher may send an event more than once, so the consumer should be idempotent too.
For example, a consumer can store the processed event ID under a unique database constraint before applying the business effect in the same transaction.
Remember: The outbox pattern helps prevent lost events during the database-to-message-broker handoff. It does not, by itself, guarantee exactly-once processing across the entire system.
11. Common Misconceptions and Beginner Mistakes
Misconception 1: Idempotency means a request executes only once.
Incorrect. The request may execute repeatedly. The goal is to prevent unintended repeated effects.
Misconception 2: Every PUT request is automatically safe.
Not necessarily. An implementation can violate the intended HTTP semantics by introducing additional side effects.
Misconception 3: PATCH is always non-idempotent.
Incorrect. A PATCH operation that sets a field to a fixed value can be idempotent.
Misconception 4: A successful HTTP response means a retry is unnecessary.
A client may never receive that response. The server could have completed the operation before the connection failed.
Misconception 5: A frontend button disabled after clicking solves the problem.
It helps prevent accidental double-clicks, but it cannot reliably handle network retries, multiple devices, background workers, or concurrent requests.
Misconception 6: A unique idempotency key alone solves everything.
Not without durable storage, appropriate uniqueness constraints, concurrency handling, payload validation, and a strategy for uncertain external outcomes.
Misconception 7: Idempotency guarantees exactly-once execution.
It does not. A request or message can be delivered multiple times. The system must make repeated processing safe or deduplicate the business effect.
12. How Is Idempotency Asked in Interviews and Exams?
Here are common interview questions and concise answers.
Q1. Define idempotency.
An operation is idempotent if repeating it has the same intended effect on system state as performing it once.
Q2. Which HTTP methods are idempotent?
GET, HEAD, PUT, DELETE, OPTIONS, and TRACE are defined as idempotent. POST is not inherently idempotent, and PATCH is not guaranteed to be idempotent.
Q3. Is DELETE idempotent if the second request returns 404?
Yes. The response can differ while the intended server state remains unchanged.
Q4. How would you prevent duplicate payment processing?
Use a stable idempotency key, durable storage, unique constraints, atomic state transitions where possible, the provider's idempotency mechanism, and reconciliation for uncertain outcomes.
Q5. How can a stateless application server support idempotency?
Store idempotency records and operation results in shared durable storage rather than relying on server-local memory.
Q6. Does idempotency guarantee exactly-once execution?
No. It makes repeated attempts safe with respect to the protected effect; it does not guarantee that the underlying code executes only once.
Q7. How do you detect an idempotency failure in production?
Monitor duplicate business records, repeated charges, key conflicts, concurrent-request errors, unexpected state transitions, and operations stuck in processing. Add alerts and reconciliation for high-risk workflows.
13. Test Your Understanding
Try answering these five questions without looking at the explanations above.
1 — Easy: You send the same GET /api/orders/101 request five times. What should remain unchanged, and does idempotency require identical responses?
2 — Easy to Medium: Why is DELETE /api/orders/101 considered idempotent even if the second request returns 404 Not Found?
3 — Medium: A client sends POST /api/payments and times out. Explain how an idempotency key prevents an unintended second charge.
4 — Hard: Two Rails servers receive the same idempotency key simultaneously. Why is checking the database first insufficient, and what database-level protection would you add?
5 — Interview Challenge: A payment provider charges a customer successfully, but your application crashes before saving the result. Design a recovery strategy that avoids an unintended duplicate charge. Explain the role of durable state, provider idempotency, and reconciliation.
Final Revision Summary
- Idempotency: Repeating an operation produces the same intended effect as performing it once.
- Why it matters: Network failures and retries can cause duplicate business effects.
- HTTP methods: GET, HEAD, PUT, DELETE, OPTIONS, and TRACE are idempotent by definition; POST is not inherently idempotent; PATCH may or may not be.
- Safe retries: Use stable idempotency keys, durable records, database constraints, payload validation, and concurrency handling.
- Payments: Combine local persistence, provider idempotency, explicit operation states, and reconciliation.
- Stateless architecture: Application servers can remain stateless while shared storage retains idempotency records.
- Distributed systems: The outbox pattern helps avoid lost events, while idempotent consumers handle repeated delivery.
- Key distinction: Idempotency does not mean exactly-once execution.
The mental model to remember:
Networks can fail. Responses can disappear. Requests can be retried. Design your system so that repeating the same logical operation does not repeat its unintended effects.
That is the essence of idempotency—and a core skill for building reliable distributed systems.
Top comments (0)