Why Retry Is One Of The Most Dangerous Keywords In Software

Few lines of code look more innocent than this:

retry(3)

It feels responsible.

Professional.

Resilient.

After all, networks fail.

Servers become unavailable.

Databases occasionally time out.

Retrying seems like the obvious solution.

And sometimes it is.

But after enough years building production systems, I've become convinced of something:

Retry is one of the most dangerous keywords in software.

Not because retries are bad.

Because retries amplify everything.

Good systems become more reliable.

Bad systems become disasters.

The problem is that many developers treat retries as a reliability feature when they're actually a distributed systems feature.

And distributed systems are where simple ideas go to become complicated.

---

Why Retries Exist

Imagine:

await fetch("/api/users");

The request fails.

Maybe:

• Network hiccup
• Temporary database issue
• Load balancer restart
• Service deployment

The operation might succeed if attempted again.

So we write:

retry(3)

Seems reasonable.

And in many cases:

It Works

Which is why retries become popular.

---

The Dangerous Assumption

Most developers unconsciously assume:

Failure
=
Operation Did Not Execute

Unfortunately that's not always true.

A request can:

Execute Successfully
↓
Response Never Arrives

From the client's perspective:

Failure

From the server's perspective:

Success

Now a retry becomes dangerous.

---

The Double Payment Problem

Imagine a payment service.

await chargeCard(order);

The card processor successfully charges:

$100

The response is lost due to a network issue.

Client sees:

Request Failed

and retries.

await chargeCard(order);

again.

Now:

Charge #1 = Success
Charge #2 = Success

The customer paid twice.

Nobody wrote bad logic.

The retry created the bug.

---

The Email Storm Problem

Consider:

await sendWelcomeEmail(user);

Email provider accepts the message.

Response times out.

Application retries.

await sendWelcomeEmail(user);

again.

Customer receives:

Welcome!
Welcome!
Welcome!
Welcome!

Support ticket created.

Marketing team confused.

The retry succeeded.

Too well.

---

Retries Amplify Side Effects

This is the core issue.

Pure operations:

2 + 2

can run forever.

Nothing changes.

Side effects are different.

Examples:

Charge Card
Create Order
Send Email
Book Seat
Reserve Inventory
Send SMS

Each execution changes reality.

Retries repeat reality.

And reality doesn't always appreciate repetition.

---

The Thundering Herd Problem

One failed request isn't scary.

Ten thousand retries are.

Imagine:

Service A

becomes slow.

Clients start retrying.

Traffic doubles.

Service becomes slower.

More retries occur.

Traffic doubles again.

Eventually:

Small Failure
↓
Massive Outage

This is known as:

The Thundering Herd Problem

And retries are often the cause.

---

When Retries Attack Databases

Suppose:

Database

is under heavy load.

Queries start timing out.

Application retries automatically.

Now:

More Queries
↓
More Load
↓
More Timeouts
↓
More Retries

You have accidentally built a denial-of-service attack against your own database.

---

Why Idempotency Matters

In the previous article we discussed:

Idempotency

This is where it becomes critical.

Without idempotency:

Retry
=
Repeat Side Effects

With idempotency:

Retry
=
Same Result

A retry becomes safe.

That's why reliable systems almost always combine:

Retries
+
Idempotency

rather than using retries alone.

---

Not Every Failure Should Be Retried

A common mistake:

retry(3)

for every error.

Consider:

400 Bad Request

Retrying won't help.

The request is invalid.

Or:

401 Unauthorized

Retrying won't magically authenticate the user.

Good retry policies distinguish between:

Transient Failures

and

Permanent Failures

---

Exponential Backoff Exists For A Reason

Bad:

Retry Immediately
Retry Immediately
Retry Immediately

Better:

1 Second
↓
2 Seconds
↓
4 Seconds
↓
8 Seconds

This is:

Exponential Backoff

and it prevents systems from overwhelming already struggling services.

---

Real World Example: Flight Booking

Imagine:

Reserve Seat

times out.

Client retries.

Without protection:

Seat Reserved Twice

or:

Two Different Seats Reserved

Now inventory becomes inconsistent.

Airlines spend enormous effort preventing these scenarios.

Because retries happen constantly.

---

Real World Example: Webhooks

Webhook providers often retry automatically.

For example:

Payment Completed

may arrive:

1 Time
2 Times
5 Times

depending on delivery conditions.

Systems that assume:

Exactly Once

processing usually fail eventually.

Systems that expect retries survive.

---

Real World Example: Message Queues

Kafka.

RabbitMQ.

SQS.

Azure Service Bus.

All assume:

Messages May Be Delivered Again

because reliability is more important than uniqueness.

Consumers must be designed accordingly.

---

Common Retry Mistakes

Retrying Everything

Not every failure is recoverable.

---

Retrying Immediately

Often makes outages worse.

---

Ignoring Idempotency

Creates duplicate side effects.

---

Infinite Retries

Eventually becomes infinite damage.

---

Hiding Failures

Retries should not become a substitute for monitoring.

---

Pros Of Retries

1. Better Reliability

Transient failures disappear.

2. Better User Experience

Temporary outages become invisible.

3. Improved Resilience

Systems tolerate instability.

4. Reduced Manual Intervention

Many failures self-heal.

5. Better Distributed Systems

Network failures become manageable.

---

Cons Of Retries

1. Duplicate Operations

Without idempotency.

2. Traffic Amplification

Can worsen outages.

3. Cascading Failures

One issue spreads across systems.

4. Increased Complexity

Backoff strategies become necessary.

5. Hidden Production Problems

Retries can mask deeper issues.

---

The Real Lesson

Most developers think retries exist to make software more reliable.

That's only partially true.

Retries don't eliminate failures.

They change failures.

Sometimes they transform:

Temporary Network Problem

into:

Duplicate Payment

Sometimes they transform:

Slow Database

into:

Full System Outage

That's why experienced engineers don't ask:

Should We Retry?

They ask:

What Happens If This Operation Executes Twice?

Because once retries enter the picture, duplicate execution is no longer an edge case.

It's a certainty.

And reliable systems are designed with that reality in mind.

---

What's Next?

In the next article we'll discuss:

The Myth Of Stateless Systems

Because many systems described as "stateless" are actually storing state somewhere else.

And that distinction turns out to be extremely important.

---

About The Author

Hi, I'm Amrish Khan.

I enjoy building developer tools, exploring software architecture, and writing about the deeper ideas behind everyday programming concepts.

I'm also building Aruvix — a growing ecosystem of local-first developer tools designed to process data directly in the browser without unnecessary uploads.

Here's a detailed blog on Aruvix:

https://dev.to/amrishkhan05/aruvix-the-ultimate-offline-first-developer-toolkit-e0i

You can follow my work and thoughts here:

Portfolio:
https://www.amrishkhan.dev

LinkedIn:
https://www.linkedin.com/in/amrishkhan

GitHub:
https://www.github.com/amrishkhan05

If you enjoyed this article, consider following for more deep dives into JavaScript, architecture, local-first software, and performance engineering.