Your app submits a request to an external API.
Usually, everything works:
Your Application → External API → Response
But what happens when the external API becomes slow?
Or stops responding completely?
Your application may keep waiting, requests may start piling up, and eventually other parts of your system can become affected.
This is why production applications need to be designed not only for successful requests, but also for failures.
In this post, we'll look at three important mechanisms for handling unreliable dependencies:
- Timeouts
- Retries
- Circuit breakers
We'll also look at exponential backoff and graceful degradation.
Why API Failures Are Different in Production
When developing locally, an API call often looks simple:
Application
↓
API Request
↓
External Service
↓
Response
In production, there are many things that can go wrong.
An external service might:
- Take too long to respond
- Return a 500 error
- Become temporarily unavailable
- Rate-limit your requests
- Experience network problems
- Return an unexpected response
For example, imagine an e-commerce application that calls a recommendation service:
User
↓
E-commerce API
↓
Recommendation Service
If the recommendation service becomes unavailable, should the entire application stop working?
Usually, no.
The application should have a strategy for handling that failure.
That's where resilience patterns become important.
Timeouts: Don't Wait Forever
The first rule is simple:
Never allow an external request to wait indefinitely.
Suppose your application sends a request to a payment service:
Application → Payment API
If the payment API doesn't respond, your application might continue waiting.
If enough requests do this simultaneously, you can end up with many requests consuming connections and resources.
A timeout places a limit on how long the application will wait.
Application
↓
Payment API
↓
No response
↓
Timeout
For example:
response = requests.get(
"https://api.example.com/payment",
timeout=5
)
The exact timeout value depends on the operation and system requirements, but the important principle is:
Every external dependency should have an intentional timeout.
A timeout doesn't fix the underlying failure.
It prevents one slow dependency from holding your application hostage indefinitely.
Retries: Temporary Failures Can Recover
Some failures are temporary.
A network request might fail once and succeed a moment later.
In those cases, retrying can be useful.
A basic retry flow looks like this:
Request
↓
Failure
↓
Retry
↓
Success
For example:
Attempt 1 → Failed
Attempt 2 → Failed
Attempt 3 → Success
This can be useful for transient problems such as temporary network failures or certain server errors.
However, retries should not be added blindly.
Imagine 1,000 requests reach a service that is already struggling.
If every request immediately retries three times, the service could suddenly receive thousands of additional requests. Instead of recovering, the situation can become worse. This is sometimes referred to as a retry storm.
Exponential Backoff
One way to make retries safer is to introduce a delay between attempts.
Instead of:
Retry → Retry → Retry
you can use:
Request
↓
Failure
↓
Wait
↓
Retry
↓
Failure
↓
Wait longer
↓
Retry
This is called exponential backoff.
A simplified example might look like:
Attempt 1 → Failure
Wait 1 second
Attempt 2 → Failure
Wait 2 seconds
Attempt 3 → Failure
Wait 4 seconds
The delay increases after each failed attempt.
In distributed systems, jitter is often added to these delays as well. This prevents many clients from retrying at exactly the same time.
A practical retry strategy therefore usually includes:
- A maximum number of retries
- Exponential backoff
- Jitter
- Appropriate timeout values
- Rules for which errors are retryable
Don't Retry Every Request
This is an important distinction. Not every failed request should automatically be retried.
For example, retrying a read operation may be relatively safe in many cases.
But consider a request that creates a payment or places an order.
If the server processed the request successfully but the response was lost, your client may not know whether the operation actually happened.
Automatically sending the same request again could potentially create a duplicate operation.
This is why idempotency matters.
For operations that may be retried, systems often use idempotency keys or other mechanisms to ensure that repeating the request doesn't accidentally repeat the business operation.
The key question isn't:
“Can we retry this request?”
It's:
“What happens if we retry this request?”
Circuit Breakers: Stop Calling a Failing Service
Now consider a different situation.
An external service isn't failing once or twice.
It's been failing continuously for several minutes.
Should your application keep sending requests?
Probably not.
This is where a circuit breaker can help.
A circuit breaker generally has three states:
Success
┌───────────┐
│ ↓
CLOSED → OPEN → HALF-OPEN
↑ │
└─────────────────┘
Recovery
Closed
Everything is operating normally.
Requests are allowed through.
Application → External Service
Open
The dependency is consistently failing.
The circuit opens and stops sending requests to that dependency.
Application → Circuit Breaker → Fallback
This prevents your application from repeatedly calling a service that is already unavailable.
Half-Open
After some time, the circuit breaker allows a limited request through to check whether the dependency has recovered.
If the request succeeds:
Half-Open → Closed
If it fails again:
Half-Open → Open
This gives the failing service time to recover while protecting your application from repeated failures.
Graceful Degradation
Sometimes the best solution isn't retrying at all.
Instead, your application can provide a reduced version of the functionality.
Consider an online shopping application:
Product Page
↓
Recommendation Service
The recommendation service goes down.
You could make the entire product page fail.
Or you could simply hide the recommendations:
Product Page
↓
Recommendation Service ❌
↓
Show Product Without Recommendations
The user can still view the product and continue shopping.
This is called graceful degradation.
The idea is simple:
When one feature fails, don't necessarily let the entire system fail with it.
Putting the Patterns Together
These mechanisms work particularly well when combined.
A simplified architecture could look like:
┌───────────────┐
│ Your Service │
└───────┬───────┘
↓
┌─────────────┐
│ Timeout │
└──────┬──────┘
↓
┌─────────────┐
│ Retry │
└──────┬──────┘
↓
┌──────────────────┐
│ Circuit Breaker │
└─────────-────────┘
↓
External Service
↓
┌─────────┐
│Fallback │
└─────────┘
The exact implementation depends on your architecture, but the concepts work together:
Timeouts prevent requests from waiting indefinitely.
Retries handle certain temporary failures.
Backoff prevents retries from overwhelming a failing service.
Circuit breakers interrupt repeated calls to an unhealthy Service.
Fallbacks allow your application to continue operating when possible.
What Should You Monitor?
Adding resilience mechanisms isn't enough.
You also need to know when they're being triggered.
Useful metrics include:
- Request failure rate
- Timeout rate
- Retry count
- Retry success rate
- Circuit breaker state
- External API latency
- Fallback frequency
- Dependency availability
For example, if your retry success rate suddenly drops from 90% to 10%, that could indicate that the dependency has a larger problem.
Without monitoring, your system may appear to be working while silently relying on retries and fallbacks.
A Practical Checklist
When your application depends on external services, consider the following:
Timeout
Set a reasonable timeout for every external request.
Retry
Retry only failures that are safe and potentially temporary.
Backoff
Avoid immediate repeated retries. Use exponential backoff and, where appropriate, jitter.
Idempotency
Make sure retrying an operation doesn't accidentally duplicate a business action.
Circuit Breaker
Stop repeatedly calling a dependency that is consistently failing.
Fallback
Decide whether your application can continue with reduced functionality.
Observability
Track failures, retries, latency, and dependency health.
Conclusion
External services will eventually become slow, unavailable, or unpredictable.
The objective is not to create a system that will never fail.
That's unrealistic.
The goal is to build a system that fails predictably, limits the impact of failures, and recovers when dependencies become healthy again.
Timeouts, retries, exponential backoff, circuit breakers, and graceful degradation are some of the building blocks that make that possible.
The next time you add an external API to your application, don't ask only:
“What happens when the API works?”
Also ask:
“What happens when it doesn't?”
Top comments (0)