Circuit Breaker Pattern Explained: How to Prevent Cascading Failures in Distributed Systems

Circuit Breaker Pattern

What happens when one service fails, but your entire system keeps calling it anyway ?

Imagine your payment service is down. Normally, every purchase request follows a path like this:

User
 ↓
Order Service
 ↓
Payment Service
 ↓
Bank

Now the Payment Service is unavailable — or perhaps it’s simply extremely slow. A request that normally takes 200 ms now takes 10 seconds. If your system continues sending requests to it, what happens ?

• Request queues grow.
• Threads and connections remain occupied.
• Timeouts increase.
• Latency goes up.

And eventually, services that were perfectly healthy can start failing too.

In other words, one failing dependency can potentially bring down an entire system.

This is where the Circuit Breaker pattern comes in.

Instead of continuously calling a dependency that is already failing, the system temporarily stops sending requests to it and fails fast.

The goal isn’t to fix the broken service. The goal is to prevent its failure from spreading.

What Is the Circuit Breaker Pattern ?

The Circuit Breaker is a resilience pattern commonly used in distributed systems to protect services from repeatedly calling an unhealthy dependency.

The idea comes from electrical circuit breakers. When electrical current becomes dangerously high, a circuit breaker interrupts the circuit to prevent further damage.

Software can apply a similar idea:

Healthy dependency
↓
Requests flow
↓
Dependency starts failing
↓
Failure threshold reached
↓
Circuit opens
↓
Requests fail fast

A Circuit Breaker doesn’t repair the dependency.

Instead, it can:

• Stop unnecessary requests to a failing service
• Protect your own application’s resources
• Reduce additional load on the unhealthy dependency
• Help prevent cascading failures
• Give the dependency time to recover
• Allow the system to degrade gracefully when possible

A useful way to think about it is:

Circuit Breaker doesn’t eliminate failure. It prevents failure from becoming a larger system-wide failure.

Why Do Distributed Systems Need Circuit Breakers ?

In a simple monolithic application, you might call a function like:

calculatePrice()

If the function fails, handling the failure is usually relatively straightforward. But distributed systems are different.

A call to another service crosses a network boundary:

Service A
↓
Service B
↓
Service C
↓
External APIService A
↓
Service B
↓
Service C
↓
External API

Unlike a local function call, a remote call can:

• Time out
• Lose its connection
• Become extremely slow
• Return a 5xx error
• Be rate-limited
• Become completely unavailable

And the more important problem is that failure can propagate between services.

For example:

Payment Service
↓
starts failing
↓
Order Service waits
↓
Threads become occupied
↓
Requests start queuing
↓
Latency increases
↓
Error rate increases
↓
The entire service comes under pressure

This is a cascading failure.

The original problem may have started in one service, but the resulting resource exhaustion can spread to otherwise healthy parts of the system.

Circuit Breaker is one of the patterns that can help interrupt this chain.

Reliability, High Availability, and Circuit Breakers

Two concepts often appear in discussions about distributed systems:

Reliability: is about a system behaving dependably in the presence of failures and recovering appropriately when failures occur.

High Availability: is about keeping a system accessible to users even when parts of its infrastructure or dependencies experience problems.

A Circuit Breaker does not make an external service permanently available.

Instead, it can help isolate its failure:

Dependency Failure
↓
Circuit Breaker
↓
Failure Isolation
↓
Graceful Degradation
↓
System remains responsive

This is where the idea of failure isolation becomes important.

Instead of allowing one dependency’s failure to spread across the entire system, we try to limit its blast radius.

How Does a Circuit Breaker Work ?

A Circuit Breaker is commonly modeled as a state machine with three primary states:

• Closed
• Open
• Half-Open

CLOSED - Everything Is Normal

When the Circuit Breaker is CLOSED, requests are allowed to reach the dependency.

Client
↓
Circuit Breaker
↓
Payment Service

The Circuit Breaker monitors the results of those requests. For example:

100 requests
95 successes
5 failures

If the failure rate exceeds the configured threshold, the circuit transitions to OPEN.

But an important detail is that a failure threshold doesn’t necessarily mean:

Five requests failed consecutively.

The threshold can be based on different measurements, such as:

• Number of failures
• Failure percentage
• Number of timeouts
• Failures within a time window
• A combination of multiple metrics

For example:

50 failures in the last 100 requests

or:

Failure Rate > 30% during the last 30 seconds

The right threshold depends on the actual behavior of the dependency and the reliability objectives of your system. There is no universal magic number.

OPEN - Stop Calling the Failing Service

When the Circuit Breaker transitions to OPEN, its behavior changes completely.

Before:

Request
↓
Circuit Breaker
↓
Payment Service

After:

Request
↓
Circuit Breaker
↓
FAIL FAST

The request never reaches the Payment Service. This is the central idea behind the Circuit Breaker pattern. If we already have strong evidence that a dependency is unhealthy, why should every new request wait for another network timeout ?

Suppose the Payment Service has a 10-second timeout. Now imagine 1,000 concurrent requests all waiting for that timeout:

1,000 requests × 10 seconds

A significant amount of your application’s resources can remain occupied simply waiting for a dependency that is already failing.

With an open circuit:

Request
↓
Circuit = OPEN
↓
Immediate failure

The system can reject the request much earlier.

That can help:

• Free threads
• Preserve connections
• Reduce resource consumption
• Keep latency more predictable
• Prevent additional traffic from reaching the unhealthy dependency

Sometimes failing quickly is better than failing slowly.

But the circuit should not remain open forever. If it did, the system would never discover that the dependency has recovered.

After a configured period, the Circuit Breaker can transition to HALF-OPEN.

HALF-OPEN - Has the Service Recovered ?

This is one of the most subtle parts of the pattern.

Suppose the Payment Service was unavailable, but now appears to be healthy again. Should we immediately send all traffic back to it ?

Not necessarily. The service may have recovered only partially. For example, it might have just restarted and currently be capable of handling only a small amount of traffic.

Instead of immediately allowing thousands of requests through, the Circuit Breaker can allow a limited number of test requests.
For example:

HALF-OPEN
↓
5 test requests
↓
Are they successful ?

If the requests succeed:

HALF-OPEN
↓
Successful requests
↓
CLOSED

The circuit returns to normal operation.

If the test requests fail:

HALF-OPEN
↓
Failure
↓
OPEN

The circuit opens again.

Why Does HALF-OPEN Allow Only a Few Requests ?

Because service recovery doesn’t necessarily mean full recovery. Imagine a Payment Service that was down and has just come back online. It might currently be capable of processing only:

20 requests / second

If your system suddenly sends:

10,000 requests

you could create another overload:

Recovery
↓
Traffic spike
↓
Overload
↓
Failure again

In other words, your system can accidentally overwhelm the dependency immediately after it recovers.

Half-Open provides a controlled way to test recovery:

Instead of:

0 requests
↓
10,000 requests

we can do:

0 requests
↓
A few test requests
↓
Evaluate results
↓
Gradually increase traffic

The exact number of test requests is implementation- and system-dependent. There is no universal value that works for every architecture.

A Real-World Scenario: Payment Service Failure

Let’s put everything together. Imagine an online store:

User
↓
Order Service
↓
Payment Service
↓
Bank Gateway

Under normal conditions:

Order Service
↓
Circuit Breaker
↓
Payment Service
↓
Bank

Everything is working normally.

Step 1: The Failure Starts

The Bank Gateway begins experiencing problems. The Payment Service starts timing out:

Payment Request
↓
Timeout

The Circuit Breaker records these failures.

Step 2: The Failure Threshold Is Reached

Suppose our policy is:

100 requests
40 failures

and our configured rule is:

Failure Rate > 30% → OPEN

The threshold has been exceeded. The Circuit Breaker opens.

Step 3: New Requests Fail Fast

Now a new request arrives:

User
↓
Order Service
↓
Circuit = OPEN
↓
Immediate Failure

The Order Service no longer calls the Payment Service.

This prevents the application from repeatedly waiting for a dependency that is already known to be unhealthy.

Step 4: The System Degrades Gracefully

Once the circuit is open, the system still has to decide what the user should experience. For example:

Payment is temporarily unavailable. Please try again in a few minutes.

Or the application might have another valid strategy:

• Keep the order in a PENDING state
• Put the operation into a queue
• Switch to another payment provider
• Return non-critical information from a cache

This is where fallback becomes important.

What Is a Fallback?

A fallback answers a simple question:

If the primary path fails, do we have a meaningful alternative ?

For example:

Primary Payment Provider
↓
FAILED
↓
Secondary Payment Provider

But a fallback is not always possible. For a payment operation, you cannot simply return Payment successful, when the payment was never actually processed.

The fallback must therefore be consistent with the business logic. A fallback is not a fake success. It is an alternative behavior that is safe and meaningful for that particular operation.

Circuit Breaker ≠ Retry

These two concepts are often confused. They solve different problems.

Retry says:

The request failed. Maybe trying again will succeed.

Circuit Breaker says:

This dependency has been failing repeatedly. Stop sending requests to it for now.

Retry can be useful for transient failures, such as:

• Temporary network glitch
• Temporary timeout
• Temporary throttling

But if the dependency is genuinely down, excessive retries can make the situation worse.

How Retries Can Cause a Cascading Failure

Suppose you have 1,000 requests and every failed request is retried three times. You might end up with:

1,000 original requests + 3,000 retries = 4,000 requests

That’s exactly what you don’t want when the dependency is already overloaded.

Now imagine:

A → B → C

Service C fails. Service B retries its requests to C. Service A retries its requests to B.

The result can look like this:

C fails
↓
B retries
↓
A retries
↓
More traffic
↓
C becomes even more overloaded
↓
More failures
↓
Cascading failure

This can lead to a retry storm.

So should we eliminate retries ?

No. The problem isn’t retrying itself. The problem is uncontrolled retries. A more resilient design might combine:

Request
↓
Timeout
↓
Retry
↓
Exponential Backoff + Jitter
↓
Circuit Breaker

The important point is that retries should generally be limited to failures where another attempt has a reasonable chance of succeeding.

For example, retrying a transient 503 Service Unavailable may make sense in some systems.

Retrying a 401 Unauthorized generally does not solve the underlying problem.

Exponential Backoff and Jitter

Imagine 10,000 requests fail at approximately the same time. If every client retries exactly one second later:

1 second
↓
10,000 requests

You’ve created another traffic spike.

With exponential backoff, the delay between retries increases:

1s
2s
4s
8s
…

This helps spread retries over a longer period.

But there’s still a problem. If every client follows exactly the same schedule, they can still retry at roughly the same moments. That’s where jitter helps.
Instead of:

1s
2s
4s
8s

different clients might retry at slightly different times:

1.2s
1.8s
2.4s
3.1s
…

The goal is to spread retry traffic instead of allowing thousands of clients to synchronize their retries.

Idempotency: The Problem You Must Consider When Retrying

Retries introduce another important problem: duplicate side effects.
Imagine:

POST /payments

The request reaches the Payment Service. The payment is successfully processed. But the response is lost because of a network failure. What does the client see?

Timeout

From the client’s perspective, it doesn’t know whether the payment succeeded. So it retries:

POST /payments

Now you potentially have:

Request #1 → Payment successful
↓
Response lost
↓
Retry
↓
Request #2 → Payment successful again

You may have charged the customer twice. This is where idempotency becomes critical. For example, the client could send an idempotency key:

Idempotency-Key: ABC123

The Payment Service can use that key to recognize that the same logical operation has already been processed.

Conceptually:

Retry
↓
Same Idempotency Key
↓
Same Logical Operation
↓
No Duplicate Side Effect

Therefore, whenever you introduce retries, ask:

Is this operation actually safe to retry ?

Important Circuit Breaker Configuration Parameters

Implementing a Circuit Breaker isn’t simply a matter of adding an if statement. Several parameters need to be defined.

1. Failure Threshold

How many failures should cause the circuit to open ?
For example:

Failure Rate > 50%

A threshold that is too low may cause the circuit to open because of temporary failures.

A threshold that is too high may allow the dependency to cause significant damage before the circuit reacts.

2. Recovery Timeout

Once the circuit is open, how long should we wait before testing the dependency again ?

For example:

30 seconds

A timeout that is too short can produce this cycle:

OPEN
↓
HALF-OPEN
↓
Failure
↓
OPEN
↓
HALF-OPEN
↓
…

On the other hand, a timeout that is too long may prevent requests from reaching a dependency even after it has recovered.

3. Number of HALF-OPEN Requests

When the circuit enters HALF-OPEN, how many requests should be allowed through ?

For example:

1 request

or:

5 requests

or a limited percentage of traffic.

Allowing more requests can provide more information about recovery, but it also creates more load on the recovering dependency.

4. Which Failures Should Open the Circuit ?

Not every error should necessarily be treated as evidence that a dependency is unhealthy.

For example:

Timeout → likely relevant
503 → likely relevant
Connection error → likely relevant
401 → probably not
400 → probably not

If every 4xx error contributes to the Circuit Breaker threshold, you could incorrectly conclude that the entire dependency is unavailable.

Circuit Breaker Has a Cost Too

No resilience pattern is free. Circuit Breaker provides important benefits, but it also introduces complexity.

Potential benefits

• Helps prevent cascading failures
• Reduces load on an unhealthy dependency
• Enables fail-fast behavior
• Protects application resources
• Supports graceful degradation
• Improves system resilience
• Enables controlled recovery

Potential costs

• More application complexity
• Additional monitoring requirements
• Threshold tuning
• State management
• Fallback design
• More difficult debugging
• False positives
• False negatives

For example, if your threshold is too sensitive:

Temporary failure
↓
Circuit opens
↓
Requests rejected

even though the dependency wasn’t actually down. If the threshold is too permissive:

Dependency is failing
↓
Circuit remains CLOSED
↓
More failures
↓
More resource consumption

So configuring a Circuit Breaker is a trade-off, not a search for one perfect number.

Where Should the Circuit Breaker Live ?

A common placement is close to the remote call:

Order Service
↓
Circuit Breaker
↓
Payment Service

This keeps the decision close to the dependency being protected.

However, Circuit Breaking can also be implemented at other layers, depending on the architecture:

• Application layer
• API Gateway
• Service Mesh
• Sidecar

A Service Mesh for example, can provide resilience features without requiring every application to implement the same logic. But that introduces another trade-off. Moving more resilience behavior into infrastructure can simplify application code, while potentially making debugging and understanding system behavior more complicated.

Do You Always Need a Circuit Breaker ?

No.

Circuit Breaker is not automatically appropriate for every operation.

For example:

function calculateTax()

If this is a local, fast operation with no meaningful remote dependency, adding a Circuit Breaker may simply introduce unnecessary complexity.

Circuit Breakers become more interesting when dealing with things such as:

• Remote APIs
• External providers
• Database dependencies
• Microservices
• Third-party services

The key question isn’t:

Do we use microservices ?

The better question is:

What happens to our system when this dependency becomes slow, unavailable, or unreliable ?

Circuit Breaker Alone Is Not Resilience

One common misconception is Circuit Breaker = Resilience. It isn’t.

Circuit Breaker is only one tool in a broader resilience strategy.

A more complete design might look like:

     ┌─────────────┐
     │   Timeout   │
     └──────┬──────┘
            ↓
     ┌─────────────┐
     │    Retry    │
     └──────┬──────┘
            ↓
┌───────────────────────┐
│ Exponential Backoff   │
│       + Jitter        │
└───────────┬───────────┘
            ↓
    ┌─────────────┐
    │   Circuit   │
    │   Breaker   │
    └──────┬──────┘
           ↓
    ┌─────────────┐
    │  Fallback   │
    └─────────────┘

Depending on the system, other patterns and mechanisms may also be relevant:

• Bulkheads
• Rate limiting
• Queues
• Caching
• Health checks
• Observability
• Idempotency

Resilience usually comes from combining the right mechanisms, not from adding one pattern everywhere.

How Do You Know Your Circuit Breaker Is Working ?

A Circuit Breaker without observability is difficult to operate. At a minimum, you should be able to monitor metrics such as:

• Circuit state
• Failure rate
• Success rate
• Timeout rate
• Number of circuit openings
• Half-Open attempts
• Recovery time
• Fallback rate
• Request latency

State transitions can also be important monitoring events.
For example:

Circuit: CLOSED → OPEN

This may indicate a significant problem with an external dependency.

Without monitoring, you may know that requests are failing — but not know why the Circuit Breaker opened, how often it opens, or whether it is actually helping.

The Goal Isn’t to Eliminate Errors

This is one of the most important ideas behind the pattern. A Circuit Breaker does not necessarily reduce the number of errors users see. In fact, after a circuit opens, you may see more immediate failures. But those failures can be much more controlled.

Without a Circuit Breaker:

1,000 requests
↓
1,000 timeouts
↓
10 seconds each
↓
Thread exhaustion
↓
Entire service becomes unhealthy

With a Circuit Breaker:

1,000 requests
↓
Circuit OPEN
↓
1,000 fast failures
↓
Dependency protected
↓
Main service remains responsive

In both cases, there are failures. But the second scenario prevents those failures from consuming resources indefinitely. That’s the key distinction.

The goal of a Circuit Breaker isn’t to make failure disappear. It’s to turn an uncontrolled failure into a controlled one.

And that is one of the fundamental ideas behind resilient distributed systems.

The State Machine in One Picture

Let’s summarize the state transitions:

CLOSED: Requests are allowed through.

CLOSED
↓
Requests flow normally

If failures exceed the configured threshold:

CLOSED → OPEN

OPEN: Requests fail fast without calling the dependency.

OPEN
↓
Fail Fast

After the configured recovery timeout:

OPEN → HALF-OPEN

HALF-OPEN: A limited number of test requests are allowed.

If recovery succeeds:

HALF-OPEN → CLOSED

If the dependency fails again:

HALF-OPEN → OPEN

So the complete lifecycle is:

Image by geeksforgeeks

The Bigger Lesson

It’s useful to remember the three states, But understanding why they exist is much more important.

When a dependency is failing, continuing to send more requests isn’t necessarily resilience. Sometimes the most resilient thing your system can do is stop making the problem worse.

Circuit Breaker, together with carefully designed timeouts, limited retries, exponential backoff, jitter, fallback strategies, and idempotency, can help prevent a localized dependency failure from turning into a much larger outage.

The central idea is simple:

When a dependency is unhealthy, protect your system first — and give the dependency a controlled chance to recover.

Final Takeaway

The Circuit Breaker pattern is a resilience mechanism for systems that depend on remote services or external resources where failures, timeouts, and temporary unavailability are possible.

Its three primary states are:

CLOSED
↓
Requests flow normally

OPEN
↓
Requests fail fast

HALF-OPEN
↓
A limited number of requests test recovery

The most important lesson isn’t memorizing these states. It’s understanding the philosophy behind them:

A resilient system doesn’t just know how to handle failure. It knows when to stop making a failure worse.

A Final Thought Experiment

Imagine your Payment Service is completely down. Your system receives 10,000 payment requests per minute. Which behavior makes more sense ?

Option 1: Retry every request three times.
Option 2: Open the Circuit Breaker after the failure rate crosses the configured threshold and fail fast.
Option 3: Use a combination of:

The interesting part isn’t memorizing which pattern to use. The real engineering challenge is understanding how these mechanisms interact under failure. That’s where resilient distributed-system design begins.

Further Reading

• Microsoft Azure — Circuit Breaker Pattern
• Microsoft Azure — Transient Fault Handling
• Microsoft Azure — Retry Storm Antipattern
• AWS — Using the Circuit Breaker Pattern with Step Functions and DynamoDB
• AWS Builders’ Library — Making retries safe with idempotent APIs
• Martin Fowler — Circuit Breaker
• Circuit Breaker Design Pattern
• Resilient Distributed Systems
• Top Strategies to Improve Reliability in Distributed Systems
• How failure cascades in Distributed Systems
• Recovery in Distributed Systems
• Monolithic Architecture
• Finite-State Machine
• Service-Level Objective
• Retries Strategies in Distributed Systems
• Failure Detection and Recovery in Distributed Systems
• Graceful Degradation in Distributed Systems
• Recovering from Transient Errors
• Retry Storm Antipattern
• Understanding Retries, Exponential Backoffs, and Circuit Breakers in Distributed Systems
• Exponential Backoff And Jitter
• What Are False Positives and Negatives in Software Testing ?
• What is a Service Mesh ?
• Sidecar Design Pattern for Microservices
• Distributed Systems and Microservices: Guide to Streamline Software Architectures
• Bulkhead Pattern
• Rate Limiting in Distributed System
• Distributed Task Queue — Distributed Systems
• Health Checks and Graceful Degradation in Distributed Systems
• Observability in Distributed Systems
• Idempotency in Distributed Systems: When and Why It Matters
• Distributed Computing