Retries Are Not Reliability: Understanding Idempotency in Distributed Systems

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻

Modern software systems expect failures to happen. Networks can timeout, servers can restart, and requests can fail halfway through execution.

To recover from these failures, systems commonly use retries.

But retries introduce another problem: the same operation may execute more than once.

So the important question becomes:

If an operation runs twice, can our system safely handle it?

This is where idempotency becomes an important software-engineering concept.

---

𝗧𝗵𝗲 𝗥𝗲𝗮𝗹 𝗣𝗿𝗼𝗯𝗹𝗲𝗺: 𝗔 𝗧𝗶𝗺𝗲𝗼𝘂𝘁 𝗗𝗼𝗲𝘀𝗻'𝘁 𝗠𝗲𝗮𝗻 𝗙𝗮𝗶𝗹𝘂𝗿𝗲

Imagine a payment request reaches the server and the payment succeeds—but the response is lost because of a network problem.

Client
↓
Request
↓
Server
↓
Payment succeeds
↓
Response lost
↓
Client timeout

The client doesn't know whether the operation failed or succeeded.

So it retries—and the same operation may happen again.

The real challenge isn't retrying. It's making retries safe.

---

𝗪𝗵𝗮𝘁 𝗜𝘀 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆?

Idempotency means that repeating the same operation produces the same intended final state as performing it once.

For example:

Set account status = ACTIVE

Whether we run it once or five times, the final state is still:

ACTIVE

But consider:

Add ₹100 to account

Running it five times adds ₹500.

So the important idea is:

If a request is retried, the system should not accidentally repeat the business effect.

This is why idempotency is important when designing reliable software.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝘁 𝘃𝘀 𝗡𝗼𝗻-𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝘁 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀

Not every operation is automatically idempotent.

A simple example of an idempotent operation is:

Set user status = ACTIVE

Running it multiple times still produces:

ACTIVE

A non-idempotent operation changes the result every time it runs:

Add ₹100 to balance

Run once → ₹100 added
Run twice → ₹200 added
Run three times → ₹300 added

So when designing a system, we need to ask:

"What happens if this operation runs again?"

𝗦𝗶𝗺𝗽𝗹𝗲 𝗱𝗶𝗮𝗴𝗿𝗮𝗺:

             Operation
                │
         ┌──────┴──────┐
         ↓             ↓
    Idempotent     Non-idempotent
         │             │
   Same result     Result changes
   when repeated   when repeated

---

𝗪𝗵𝘆 𝗥𝗲𝘁𝗿𝗶𝗲𝘀 𝗖𝗮𝗻 𝗖𝗿𝗲𝗮𝘁𝗲 𝗗𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀

Retries are useful because temporary failures are common in real systems.

Request
↓
Temporary failure
↓
Retry
↓
Success

The problem starts when the first request actually succeeded, but the client never received the response.

First request → Operation succeeds → Response lost
↓
Client retries
↓
Operation runs again

The same problem can happen at different layers: clients, APIs, load balancers, message queues, and background workers can all retry work.

Retries 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 𝗿𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆, but without idempotency, they can also create duplicate work.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗞𝗲𝘆𝘀: 𝗚𝗶𝘃𝗶𝗻𝗴 𝗮 𝗥𝗲𝗾𝘂𝗲𝘀𝘁 𝗮𝗻 𝗜𝗱𝗲𝗻𝘁𝗶𝘁𝘆

One common way to make non-idempotent operations safer is to give each request a unique idempotency key.

For example:

Idempotency-Key: payment-abc123

The server stores this key when it successfully processes the request.

If the same request arrives again:

Same key
↓
Already processed?
↓
YES → Return the previous result

Instead of creating another payment or order, the system recognizes that it has already handled that request.

The key idea is simple:

The request can be repeated, but the business operation should happen only once.

---

𝗛𝗼𝘄 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗞𝗲𝘆𝘀 𝗔𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗪𝗼𝗿𝗸

The basic flow is:

Client
↓
Request + Idempotency Key
↓
Server
↓
Check stored key

If the key doesn't exist, the server processes the operation and stores the result.

If the same key appears again for the same operation, the server can recognize it as a repeated request and return the stored result.

New key → Process → Store result

Same key → Find result → Return result

This approach is useful for operations such as payments, order creation, job submission, and other requests where duplicate execution could cause a problem.

Idempotency keys turn an 𝗼𝘁𝗵𝗲𝗿𝘄𝗶𝘀𝗲 𝗮𝗺𝗯𝗶𝗴𝘂𝗼𝘂𝘀 𝗿𝗲𝘁𝗿𝘆 𝗶𝗻𝘁𝗼 𝗮 𝗿𝗲𝗾𝘂𝗲𝘀𝘁 𝘁𝗵𝗲 𝘀𝘆𝘀𝘁𝗲𝗺 𝗰𝗮𝗻 𝗿𝗲𝗰𝗼𝗴𝗻𝗶𝘇𝗲.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗗𝗲𝘀𝗶𝗴𝗻

An idempotency key is useful only if the system can reliably remember it.

A simple approach is to store the key in the database:

idempotency_key

---

payment-abc123

When another request arrives with the same key, the system can detect that the operation was already processed.

But checking the key in application code alone is not enough. Two identical requests can arrive at almost the same time.

That's why the database should also enforce uniqueness, for example with a 𝘂𝗻𝗶𝗾𝘂𝗲 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁.

Application logic can detect duplicates, but the database should provide the final protection against duplicate records.

---

𝗥𝗮𝗰𝗲 𝗖𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝘀: 𝗪𝗵𝗲𝗻 𝗧𝘄𝗼 𝗥𝗲𝗾𝘂𝗲𝘀𝘁𝘀 𝗔𝗿𝗿𝗶𝘃𝗲 𝗧𝗼𝗴𝗲𝘁𝗵𝗲𝗿

Now consider two requests using the same idempotency key at almost exactly the same time.

Request A ──→ Check key → Not found

Request B ──→ Check key → Not found

Both requests may think they are allowed to continue.

This is called a race condition: the result depends on the timing of concurrent operations.

A unique database constraint can prevent both requests from creating the same record.

Request A ──→ Database ──→ INSERT ✓

Request B ──→ Database ──→ DUPLICATE ✗

The important lesson is:

𝗖𝗵𝗲𝗰𝗸𝗶𝗻𝗴 𝗳𝗶𝗿𝘀𝘁 𝗮𝗻𝗱 𝗶𝗻𝘀𝗲𝗿𝘁𝗶𝗻𝗴 𝗹𝗮𝘁𝗲𝗿 𝗶𝘀 𝗻𝗼𝘁 𝗮𝗹𝘄𝗮𝘆𝘀 𝘀𝗮𝗳𝗲 𝘄𝗵𝗲𝗻 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗿𝗲𝗾𝘂𝗲𝘀𝘁𝘀 𝗰𝗮𝗻 𝗿𝘂𝗻 𝗮𝘁 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝘁𝗶𝗺𝗲.

Concurrency needs to be handled deliberately.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗶𝗻 𝗤𝘂𝗲𝘂𝗲-𝗕𝗮𝘀𝗲𝗱 𝗦𝘆𝘀𝘁𝗲𝗺𝘀

Idempotency is not limited to HTTP APIs. It is equally important in background job processing.

Consider a worker processing a job:

Queue
↓
Worker
↓
Process job
↓
Worker fails
↓
Job is retried

The retry may cause the worker to process the same job again.

For example, a worker might successfully update the database and then crash before confirming that the job finished.

The queue may then deliver the job again.

This means a reliable worker should be designed so that 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗷𝗼𝗯 𝗺𝗼𝗿𝗲 𝘁𝗵𝗮𝗻 𝗼𝗻𝗰𝗲 𝗱𝗼𝗲𝘀 𝗻𝗼𝘁 𝗰𝗿𝗲𝗮𝘁𝗲 𝗮𝗻 𝗶𝗻𝗰𝗼𝗿𝗿𝗲𝗰𝘁 𝗳𝗶𝗻𝗮𝗹 𝘀𝘁𝗮𝘁𝗲.

This is one reason idempotency is important in systems built with 𝗾𝘂𝗲𝘂𝗲𝘀, 𝘄𝗼𝗿𝗸𝗲𝗿𝘀, 𝗿𝗲𝘁𝗿𝗶𝗲𝘀, 𝗮𝗻𝗱 𝗮𝘀𝘆𝗻𝗰𝗵𝗿𝗼𝗻𝗼𝘂𝘀 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗜𝘀 𝗡𝗼𝘁 𝘁𝗵𝗲 𝗦𝗮𝗺𝗲 𝗮𝘀 𝗘𝘅𝗮𝗰𝘁𝗹𝘆-𝗢𝗻𝗰𝗲 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻

Idempotency does not mean the system physically executes an operation only once.

A request or job may actually be processed more than once.

The goal is different:

𝗠𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻𝘀 𝘀𝗵𝗼𝘂𝗹𝗱 𝗻𝗼𝘁 𝗰𝗿𝗲𝗮𝘁𝗲 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗲𝗳𝗳𝗲𝗰𝘁𝘀.

Execute → Execute → Execute
↓
Same intended final state

This distinction is important in distributed systems because guaranteeing that something happens exactly once across multiple systems can be difficult.

Instead, engineers often design operations to be 𝘀𝗮𝗳𝗲 𝘄𝗵𝗲𝗻 𝗿𝗲𝗽𝗲𝗮𝘁𝗲𝗱.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗶𝗻 𝗥𝗲𝗮𝗹 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗦𝘆𝘀𝘁𝗲𝗺𝘀

Idempotency is useful anywhere a repeated request could create an unwanted result.

Common examples include:

Creating an order
Processing a payment
Submitting a background job
Sending notifications
Updating user information
Handling webhook events

The implementation can differ between systems, but the principle remains the same:

Same operation
↓
Repeated request
↓
Safe final result

This makes the system more reliable when requests are retried or messages are delivered more than once.

---

𝗧𝗵𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗠𝗶𝗻𝗱𝘀𝗲𝘁: 𝗗𝗲𝘀𝗶𝗴𝗻 𝗳𝗼𝗿 𝗥𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻

The biggest lesson is not simply "use an idempotency key."

The deeper lesson is to design software with the assumption that requests may be repeated.

Before implementing an operation, ask:

What if this request runs twice?
What if it runs three times?
What if the response is lost?
What if two requests arrive together?

These questions help us discover duplicate operations, race conditions, and data-consistency problems before they become production issues.

𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗶𝘀 𝗻𝗼𝘁 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝘁𝗵𝗮𝘁 𝗻𝗲𝘃𝗲𝗿 𝗳𝗮𝗶𝗹𝘀. 𝗜𝘁 𝗶𝘀 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝘁𝗵𝗮𝘁 𝗯𝗲𝗵𝗮𝘃𝗲𝘀 𝘀𝗮𝗳𝗲𝗹𝘆 𝘄𝗵𝗲𝗻 𝗳𝗮𝗶𝗹𝘂𝗿𝗲𝘀 𝗮𝗻𝗱 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻𝘀 𝗵𝗮𝗽𝗽𝗲𝗻.

---

𝗪𝗵𝗮𝘁 𝗦𝗵𝗼𝘂𝗹𝗱 𝗮𝗻 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗥𝗲𝗰𝗼𝗿𝗱 𝗦𝘁𝗼𝗿𝗲?

When a system receives an idempotent request, it needs to remember what happened.

Depending on the system, an idempotency record may contain information such as:

Idempotency Key
User ID
Request Status
Response
Created At

For example:

payment-abc123
↓
COMPLETED
↓
Previous response

When the same key arrives again, the system can return the stored result instead of processing the operation again.

𝗧𝗵𝗲 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗽𝗼𝗶𝗻𝘁 𝗶𝘀 𝘁𝗵𝗮𝘁 𝗶𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗶𝘀 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝗮𝗯𝗼𝘂𝘁 𝘀𝘁𝗼𝗿𝗶𝗻𝗴 𝗮 𝗸𝗲𝘆—𝗶𝘁 𝗶𝘀 𝗮𝗯𝗼𝘂𝘁 𝗿𝗲𝗺𝗲𝗺𝗯𝗲𝗿𝗶𝗻𝗴 𝘁𝗵𝗲 𝗼𝘂𝘁𝗰𝗼𝗺𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻.

---

𝗪𝗵𝗮𝘁 𝗜𝗳 𝘁𝗵𝗲 𝗙𝗶𝗿𝘀𝘁 𝗥𝗲𝗾𝘂𝗲𝘀𝘁 𝗙𝗮𝗶𝗹𝘀?

We also need to think about failures.

Suppose a request creates an idempotency record but the actual operation fails.

Request
↓
Store key
↓
Operation fails

Should the same key be allowed to retry?

The answer depends on the system's business rules and how the idempotency record is designed.

Some systems store the final failure and return the same result. Others allow the operation to be retried when the failure is considered safe to retry.

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝘁𝗵𝗲𝗿𝗲𝗳𝗼𝗿𝗲 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝘀 𝗮 𝗰𝗹𝗲𝗮𝗿 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗮𝗯𝗼𝘂𝘁 𝗯𝗼𝘁𝗵 𝘀𝘂𝗰𝗰𝗲𝘀𝘀𝗳𝘂𝗹 𝗮𝗻𝗱 𝗳𝗮𝗶𝗹𝗲𝗱 𝗮𝘁𝘁𝗲𝗺𝗽𝘁𝘀.

---

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 𝗜𝘀 𝗮 𝗕𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻, 𝗡𝗼𝘁 𝗝𝘂𝘀𝘁 𝗮 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗙𝗲𝗮𝘁𝘂𝗿𝗲

There is no single idempotency strategy that works for every operation.

Consider:

 * Create Payment
 * Create Order
 * Send Email
 * Submit Job
 * Update Profile

Each operation has 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗿𝘂𝗹𝗲𝘀 and different consequences when repeated.

Idempotency is not simply a technology that we add to every API. 𝗜𝘁 𝗶𝘀 𝗮 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗮𝗻𝗱 𝘀𝘆𝘀𝘁𝗲𝗺-𝗱𝗲𝘀𝗶𝗴𝗻 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻.

Before implementing idempotency, an engineer should understand:

𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻'𝘀 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗲𝗳𝗳𝗲𝗰𝘁, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝘀𝗵𝗼𝘂𝗹𝗱 𝗵𝗮𝗽𝗽𝗲𝗻 𝗶𝗳 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗿𝗲𝗾𝘂𝗲𝘀𝘁 𝗮𝗿𝗿𝗶𝘃𝗲𝘀 𝗮𝗴𝗮𝗶𝗻?

Good system design starts with that question—not with choosing a database column or library.

---

𝗧𝗵𝗲 𝗕𝗶𝗴𝗴𝗲𝗿 𝗟𝗲𝘀𝘀𝗼𝗻

Idempotency looks like a small API feature, but it teaches a much bigger software-engineering lesson.

Distributed systems are full of uncertainty:

Requests can repeat.
Responses can disappear.
Workers can retry.
Messages can arrive again.
Multiple requests can run together.

We cannot always prevent these situations.

What we can do is design our systems so that these situations do not create incorrect results.

𝗧𝗵𝗮𝘁'𝘀 𝘁𝗵𝗲 𝗿𝗲𝗮𝗹 𝗽𝘂𝗿𝗽𝗼𝘀𝗲 𝗼𝗳 𝗶𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆: 𝗻𝗼𝘁 𝗽𝗿𝗲𝘃𝗲𝗻𝘁𝗶𝗻𝗴 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻, 𝗯𝘂𝘁 𝗺𝗮𝗸𝗶𝗻𝗴 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻 𝘀𝗮𝗳𝗲.

---

𝗙𝗶𝗻𝗮𝗹 𝗧𝗵𝗼𝘂𝗴𝗵𝘁𝘀

Idempotency taught me an important lesson about building reliable software:

𝗪𝗲 𝗰𝗮𝗻𝗻𝗼𝘁 𝗮𝗹𝘄𝗮𝘆𝘀 𝗽𝗿𝗲𝘃𝗲𝗻𝘁 𝗳𝗮𝗶𝗹𝘂𝗿𝗲𝘀, 𝗿𝗲𝘁𝗿𝗶𝗲𝘀, 𝗼𝗿 𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗿𝗲𝗾𝘂𝗲𝘀𝘁𝘀—𝗯𝘂𝘁 𝘄𝗲 𝗰𝗮𝗻 𝗱𝗲𝘀𝗶𝗴𝗻 𝗼𝘂𝗿 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 𝘁𝗼 𝗵𝗮𝗻𝗱𝗹𝗲 𝘁𝗵𝗲𝗺 𝘀𝗮𝗳𝗲𝗹𝘆.

The next time you design an API, background job, payment flow, or distributed service, ask one simple question:

"𝗪𝗵𝗮𝘁 𝗵𝗮𝗽𝗽𝗲𝗻𝘀 𝗶𝗳 𝘁𝗵𝗶𝘀 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗿𝘂𝗻𝘀 𝗮𝗴𝗮𝗶𝗻?"

That question can reveal problems that are easy to miss during normal development.

Thanks for reading! 🙌

I hope this article helped make idempotency easier to understand, especially if you're learning distributed systems or building production-level applications.

If you found it useful, feel free to share it with another developer who might benefit from it.

Keep building. Keep learning. Keep questioning how your systems behave when things go wrong. 🚀

𝗔𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗔𝘂𝘁𝗵𝗼𝗿

I'm a software engineer interested in building reliable software systems and continuously improving my understanding of system design, distributed systems, and backend engineering.

This article was written while exploring these concepts through hands-on project development and practical engineering problems.