One of the biggest misconceptions about modern distributed systems is that they fail because servers are too slow or because there are not enough machines handling requests. While insufficient infrastructure can certainly become a bottleneck, it is rarely the root cause behind large-scale outages. In reality, some of the most significant production incidents have occurred in systems that had thousands of servers, multiple data centres, redundant databases, and sophisticated load balancers. The problem was not a lack of hardware. The problem was that the system was not prepared for failure.
Imagine an e-commerce company launching its biggest sale of the year. Months of preparation have gone into the event. Engineers have increased server capacity, databases have been replicated across multiple regions, caches have been warmed, and additional instances of critical services have been deployed. Everything appears ready. At exactly 10:00 AM, the sale begins. Within seconds, millions of users start refreshing the homepage, searching for products, adding items to their carts, and attempting to complete purchases.
Initially, everything works as expected. Requests are distributed across multiple application servers, response times remain low, and the infrastructure appears healthy. Then, one small problem begins to emerge. The inventory database, which keeps track of available stock, starts responding slightly slower than usual. Instead of taking 50 milliseconds to answer a query, it now takes 500 milliseconds. Half a second may not sound like much, but in a distributed system processing thousands of requests every second, that additional delay quickly becomes significant.
Every checkout request must wait for the inventory service to confirm that a product is still available. Because the inventory service is now waiting longer for the database, the checkout service also begins waiting. Those waiting requests continue occupying server threads, database connections, and memory. Meanwhile, new customers continue arriving and generating even more requests. Within a few minutes, queues begin forming across multiple services. CPU utilisation rises sharply, memory consumption increases, and response times continue growing. Soon, services that were originally healthy also begin slowing down because they are waiting for other services that are already struggling.
What started as a minor slowdown inside a single database has now spread throughout the entire application.
This phenomenon is one of the defining challenges of distributed systems. Unlike monolithic applications, where most operations happen within a single process, distributed applications consist of dozens or even hundreds of independent services communicating continuously over the network. A single user request may pass through an API Gateway, an authentication service, a product service, an inventory service, a payment service, a notification service, and several databases before the user finally receives a response. Every additional dependency introduces another opportunity for delays, failures, or unexpected behaviour.
At first glance, this architecture appears highly resilient because every service can scale independently. However, this same flexibility also creates a new type of risk. Services become interconnected. If one component slows down, every service depending on it begins slowing down as well. If those services become overloaded, they start affecting other services. Gradually, what began as an isolated problem spreads throughout the entire system like a chain reaction.
Engineers often describe this behaviour as a cascading failure. Much like a line of dominoes falling one after another, a failure in one component triggers failures in neighbouring components until eventually the entire application becomes unstable. Importantly, the original component that caused the issue may not even be the one users notice. Customers simply experience a website that feels slow, unresponsive, or completely unavailable.
Large technology companies spend enormous amounts of time designing systems that prevent exactly this situation. Their objective is not merely to ensure that individual services remain operational. Instead, they aim to ensure that failures remain isolated. A slow recommendation engine should never prevent users from placing orders. A malfunctioning notification service should never stop payments from being processed. An external API outage should not cause every application server to exhaust its available resources while waiting for responses that may never arrive.
Achieving this level of resilience requires a fundamental shift in how engineers think about software design. Beginners often design applications assuming that every dependency will behave correctly. Every service will respond quickly. Every database query will succeed. Every network request will complete. Experienced system designers make exactly the opposite assumption. They expect services to fail. They expect networks to become unreliable. They expect traffic spikes to occur unexpectedly. Rather than hoping these situations never happen, they build systems specifically designed to continue operating when they do.
This philosophy explains why modern distributed systems deliberately make decisions that initially seem counterintuitive. Sometimes they reject incoming requests even though additional users are trying to access the application. Sometimes they temporarily stop communicating with another service, even though that service may recover in a few seconds. Sometimes they return partial information instead of attempting to generate a complete response. These decisions may appear strange until one understands the goal behind them.
The purpose is not to maximise the number of successful requests at any given moment.
The purpose is to maximise the long-term health of the system.
Consider a hospital emergency room during a major disaster. Doctors do not attempt to treat every patient simultaneously because doing so would overwhelm the entire medical staff. Instead, patients are prioritised based on urgency, and new admissions may even be redirected to nearby hospitals if capacity has been reached. Although this means some patients must wait longer, the hospital remains capable of treating those in the greatest need.
Distributed systems behave in much the same way. Rather than allowing unlimited requests to consume every available resource, resilient applications establish protective mechanisms that regulate traffic, isolate failures, and recover gracefully when dependencies become unhealthy. These mechanisms prevent small problems from becoming catastrophic outages.
Three design patterns have become particularly important for achieving this level of resilience.
The first is Rate Limiting, which controls how many requests clients are allowed to send, preventing systems from becoming overwhelmed during sudden traffic spikes or malicious attacks.
The second is the Circuit Breaker Pattern, which prevents applications from repeatedly communicating with services that are already failing, allowing both the calling service and the failing dependency time to recover.
The third is Graceful Failure Handling, where applications continue providing as much functionality as possible even when certain components become unavailable, ensuring that users experience degraded service rather than complete failure.
Individually, each of these patterns addresses a different aspect of reliability. Together, they form one of the most important defensive strategies in distributed systems. Rather than attempting to eliminate failures, a practically impossible goal—they acknowledge that failures are inevitable and focus on containing their impact.
At first, the idea of intentionally rejecting user requests seems almost contradictory. After all, the primary responsibility of a web application is to serve its users. If someone sends a request, shouldn't the system always try its best to process it?
While this sounds reasonable in theory, it often leads to the exact opposite outcome in practice.
Imagine a restaurant that has enough chefs to prepare one hundred meals every hour. Under normal conditions, around seventy customers arrive during that time, leaving enough capacity for the kitchen to operate efficiently. Orders are prepared quickly, waiters deliver food on time, and customers leave satisfied.
Now imagine that a famous food blogger unexpectedly recommends the restaurant to millions of followers. Within minutes, five thousand customers arrive at the entrance. The restaurant has only two choices. It can either allow everyone inside, creating chaos in the kitchen, overwhelming the staff, exhausting ingredients, and ultimately disappointing every customer. Or it can temporarily stop accepting new customers until it regains control of the situation.
Most well-managed restaurants choose the second option.
Although some customers may have to wait or return later, the restaurant continues functioning. More importantly, the customers who are already inside still receive the quality of service they expect.
Distributed systems face the same challenge.
Every application has a finite amount of CPU, memory, network bandwidth, database connections, and processing threads. Regardless of whether the application runs on a single server or thousands of cloud instances, those resources are never unlimited. If incoming requests exceed the system's ability to process them, queues begin growing, response times increase, memory usage rises, and eventually the application becomes unstable.
The surprising realisation is that processing every request is not always the best strategy. Sometimes, rejecting a small percentage of requests allows the remaining ninety-nine percent to complete successfully. This principle forms the foundation of Rate Limiting.
Understanding Rate Limiting
Rate limiting is a mechanism that controls how many requests a client is allowed to make within a specified period of time. Instead of allowing unlimited traffic to reach backend services, the system establishes clear boundaries that define acceptable behaviour. Once a client exceeds those boundaries, additional requests are temporarily rejected until the allowed time window resets.
Although this may sound restrictive, rate limiting serves multiple purposes beyond simply reducing traffic. It protects applications from accidental overload, prevents abuse by automated bots, limits the impact of malicious attacks, and ensures that no single client consumes a disproportionate share of system resources.
Consider a public weather API that serves millions of developers around the world. If the API allowed unlimited requests, a single poorly written application caught in an infinite retry loop could generate thousands of requests every second. Even though the bug exists in only one client application, its behaviour could significantly degrade performance for every other developer using the same API.
By enforcing a limit such as one hundred requests per minute for each API key, the service protects itself from misuse while still providing fair access to everyone else.
This idea of fair resource allocation is one of the primary motivations behind rate limiting.
Why Large Internet Companies Depend on Rate Limiting
Almost every major technology company relies on rate limiting, even if users rarely notice it.
When you use the GitHub API, you cannot make unlimited requests indefinitely. The platform enforces request limits to ensure that automated scripts do not monopolise shared infrastructure.
Payment providers such as Stripe apply strict limits to protect financial systems from accidental duplicate requests and malicious abuse.
Cloud providers implement request quotas to ensure that one customer cannot unintentionally consume resources needed by thousands of others.
Even conversational AI platforms implement request limits. Without them, a handful of users—or even automated programs—could generate enormous volumes of requests, degrading response quality for everyone else using the service.
These limits are not signs of weak infrastructure.
Quite the opposite.
They are evidence that the infrastructure has been designed to remain stable under unpredictable conditions.
Where Should Rate Limiting Be Applied?
One of the most common questions engineers ask is where rate limiting should actually occur.
Should it be implemented inside every microservice?
Should databases reject excessive queries?
Should the web server perform the limiting?
While multiple approaches exist, the most common practice is to perform rate limiting as early as possible, before expensive backend operations begin.
For this reason, many distributed systems enforce request limits at the API Gateway or Load Balancer.
This placement provides an important advantage.
Instead of allowing excessive traffic to travel through multiple services before finally being rejected, unwanted requests are filtered immediately at the system's entry point. Backend services never spend CPU cycles, memory, or database connections processing requests that should not have been accepted in the first place.
This approach is remarkably similar to airport security. Rather than allowing passengers to enter restricted areas before checking identification, airports verify passengers at the entrance, preventing unnecessary congestion throughout the terminal.
Different Types of Rate Limits
Not every client should necessarily receive the same limits.
Consider a cloud platform serving both anonymous visitors and paying enterprise customers.
Anonymous users might be allowed only a small number of requests each minute to discourage abuse. Registered users may receive higher limits, while enterprise customers paying for premium services could receive substantially larger quotas.
Similarly, login endpoints often use much stricter limits than product search endpoints.
A user might be allowed to search for products hundreds of times every minute, but only attempt five password submissions before the system temporarily blocks additional login attempts. This simple restriction dramatically reduces the effectiveness of brute-force password attacks.
Rate limiting therefore becomes more than just a performance optimisation.
It also becomes an important security mechanism.
How Does a System Know When a User Has Exceeded the Limit?
At this point, the overall idea of rate limiting is relatively straightforward.
The more interesting question is how software actually keeps track of requests.
Imagine an API serving ten million users.
Every second, requests arrive from different countries, devices, and applications.
How does the system determine whether a particular client has already exceeded one hundred requests during the current minute?
The answer is that every incoming request is associated with some form of identity.
Depending on the application, this identity could be an IP address, an authenticated user account, an API key, a session identifier, or even an organisation ID.
Each request updates a small counter maintained by the rate-limiting system.
Whenever another request arrives, the current count is checked against the configured limit.
If the client remains within the permitted quota, the request proceeds normally.
If the limit has already been exceeded, the request is rejected immediately, often with an HTTP 429 Too Many Requests response.
The implementation sounds simple.
However, designing a rate limiter that remains accurate while processing millions of requests every second across multiple servers is surprisingly difficult.
Questions quickly begin to arise.
What happens when the clock reaches the next minute?
Should every counter immediately reset?
How do multiple servers maintain consistent request counts?
How can the system avoid sudden bursts of traffic exactly when counters reset?
Answering these questions has led engineers to develop several sophisticated rate-limiting algorithms, each designed to solve different types of workload patterns.
Some algorithms prioritise simplicity.
Others prioritise fairness.
Others are specifically designed to accommodate short bursts of traffic without allowing sustained overload.
By now, we understand why modern distributed systems need rate limiting. Every application has finite resources, and allowing unlimited traffic inevitably leads to congestion, increased latency, and eventually service outages. However, knowing that requests should be limited is only half the problem. The more interesting challenge is deciding how those limits should actually be enforced.
At first glance, the solution appears deceptively simple. Suppose an API allows one hundred requests per minute for every user. We could simply maintain a counter for each user, increment it every time a request arrives, and reset the counter when the next minute begins. If the counter reaches one hundred, every subsequent request is rejected until the minute resets.
While this approach certainly works, engineers quickly discovered that it introduces an unexpected problem.
Imagine the current time is 10:00:59. A client has not made a single request during the last minute. Suddenly, they send one hundred requests within a fraction of a second. Since the current minute has not yet ended, every request is accepted. One second later, the clock reaches 10:01:00, all counters are reset, and the client immediately sends another one hundred requests.
Technically, the client has respected the configured limit of one hundred requests per minute.
In reality, however, the server has processed two hundred requests within roughly two seconds.
The system followed the rule perfectly.
The rule simply was not designed well enough.
This seemingly small observation has led to several different rate-limiting algorithms, each attempting to balance fairness, simplicity, performance, and implementation complexity. Rather than viewing these algorithms as competing solutions, it is more helpful to think of them as different tools designed for different workloads.
The Fixed Window Algorithm
The simplest and perhaps most intuitive rate-limiting strategy is the Fixed Window algorithm.
Time is divided into fixed intervals—for example, one minute. Every client receives a counter associated with the current window. Each incoming request increments that counter. Once the counter reaches the configured limit, all remaining requests during that window are rejected. When the next window begins, the counter is reset to zero.
The primary advantage of this approach is its simplicity. It requires very little memory, is straightforward to implement, and performs extremely well even under heavy traffic. This is why many early APIs adopted the Fixed Window strategy.
However, simplicity often comes with trade-offs.
As we saw earlier, requests arriving near the boundary between two windows can effectively double the intended request rate. This phenomenon, commonly known as the boundary problem, makes Fixed Window less suitable for systems where traffic patterns fluctuate rapidly.
For many internal applications, this limitation is acceptable. For large public APIs serving millions of users, however, engineers generally prefer more sophisticated approaches.
Sliding Window: A Fairer Approach
Instead of dividing time into rigid intervals, the Sliding Window algorithm continuously evaluates requests over the most recent time period.
Suppose an API allows one hundred requests every sixty seconds.
Rather than asking, "How many requests occurred during the current minute?" the Sliding Window algorithm asks a different question:
"How many requests has this client made during the last sixty seconds?"
Notice the difference.
The observation window moves forward continuously with time.
Because the window never resets abruptly, sudden bursts around minute boundaries disappear. Traffic becomes much smoother, and clients receive a more consistent experience.
The trade-off is computational complexity.
Instead of maintaining a single counter, the system must remember when recent requests occurred so that older requests can gradually expire from the sliding window. For applications processing millions of requests every second, this additional bookkeeping increases both memory usage and implementation complexity.
Despite these challenges, Sliding Window is widely used because it produces significantly fairer traffic distribution than Fixed Window.
The Leaky Bucket Algorithm
Imagine pouring water into a bucket with a small hole at the bottom.
Water may enter the bucket quickly, but it leaves at a constant rate.
If water arrives faster than it can drain, the bucket eventually overflows.
The Leaky Bucket algorithm applies the same idea to incoming requests.
Instead of processing every request immediately, requests are placed into a queue and released at a steady rate.
This approach produces an important benefit.
Even if traffic arrives in sudden bursts, downstream services receive requests at a predictable and stable rate.
Applications that perform expensive operations—such as generating reports, processing images, or communicating with slower external systems—often benefit from this smoothing effect.
The drawback is that bursty traffic may experience increased waiting times. Requests that arrive together must wait for earlier requests to leave the queue, increasing latency even when the backend remains healthy.
In other words, the Leaky Bucket prioritises consistency over responsiveness.
Token Bucket: Balancing Protection and Flexibility
Perhaps the most widely adopted algorithm today is the Token Bucket algorithm because it combines strong protection with enough flexibility to accommodate normal traffic patterns.
The concept is surprisingly elegant.
Imagine a bucket that gradually fills with tokens at a fixed rate. Every incoming request must consume one token before it can proceed. If tokens are available, the request is immediately accepted. If the bucket becomes empty, additional requests are rejected until new tokens are generated.
Unlike the Leaky Bucket, the Token Bucket allows short bursts of activity.
Suppose a user remains inactive for several seconds. During that time, unused tokens accumulate inside the bucket. When the user suddenly sends multiple requests, those stored tokens allow the requests to proceed immediately without violating the long-term rate limit.
This makes the Token Bucket particularly well suited for interactive applications.
Human users rarely generate perfectly uniform traffic. They may spend several minutes reading content before suddenly clicking several buttons in quick succession. Allowing these short bursts creates a smoother user experience while still protecting backend services from sustained overload.
For this reason, Token Bucket has become the preferred choice for many API gateways, cloud platforms, and networking systems.
Choosing the Right Algorithm
At this point, it should be clear that no single algorithm is universally superior.
Each algorithm optimises for different priorities.
The Fixed Window algorithm is simple, efficient, and easy to implement, but suffers from boundary effects.
Sliding Window produces much fairer request distribution, although it requires more computational resources.
Leaky Bucket excels at smoothing traffic and protecting downstream services from sudden bursts, making it useful for systems that require predictable processing rates.
Token Bucket offers an excellent balance between fairness and flexibility by allowing temporary bursts while still enforcing long-term request limits, which explains its popularity in modern distributed systems.
The most appropriate algorithm therefore depends not only on the application's traffic patterns but also on its business requirements.
An authentication service protecting against brute-force attacks may prioritise strict enforcement. A streaming platform may prefer flexibility to accommodate natural user behaviour. A financial API may choose consistency above all else.
Understanding these trade-offs is significantly more valuable than memorising the algorithms themselves.
Up to this point, we have focused on protecting systems before requests enter the application. Rate limiting controls incoming traffic and prevents services from becoming overwhelmed by excessive demand.
But what happens when the problem does not originate from users at all?
What if traffic levels remain perfectly normal, yet one of your dependencies suddenly becomes slow or completely unavailable?
Should your application continue sending requests to a service that is already failing?
Surprisingly, repeatedly retrying those requests often makes the situation even worse.
To solve this problem, distributed systems employ another powerful resilience pattern known as the Circuit Breaker. Instead of allowing failures to spread across services, a circuit breaker detects unhealthy dependencies, temporarily stops communication with them, and gives both systems time to recover.
Up to this point, we have focused on protecting a system from excessive incoming traffic. Rate limiting acts like a security guard standing at the entrance of a building, ensuring that only a manageable number of people are allowed inside at any given time. By controlling the rate at which requests enter the system, applications prevent themselves from becoming overwhelmed before any real damage occurs.
However, not every failure originates from incoming traffic.
Sometimes the application itself is healthy, the number of users is perfectly normal, and the infrastructure has more than enough resources to handle the workload. Yet requests still begin timing out, response times suddenly increase, and users start experiencing errors.
In situations like these, the problem often lies outside the service itself.
Modern applications rarely operate in isolation. A typical microservice communicates with authentication services, payment providers, recommendation engines, notification systems, databases, search engines, cloud storage services, and numerous third-party APIs. Every one of these dependencies represents another system that can become slow, unavailable, or completely fail.
The challenge is that applications generally assume these dependencies will respond within a reasonable amount of time. When they stop responding, the calling service often continues waiting patiently, hoping that the next request will succeed. Unfortunately, thousands of requests making the same assumption simultaneously can quickly exhaust the application's own resources.
To understand why this happens, imagine an online travel booking platform.
A customer searches for flights, selects one, enters passenger details, and finally clicks the Book Now button. The Booking Service receives the request and immediately contacts the Payment Service to complete the transaction. Under normal conditions, the payment provider responds within a few hundred milliseconds, allowing the booking process to continue smoothly.
Now imagine that the external payment provider experiences an outage.
The Booking Service sends the payment request and waits.
After several seconds, the request eventually times out.
Meanwhile, another customer clicks Book Now.
A second payment request is sent.
It also waits.
Within a few moments, hundreds of booking requests are simultaneously waiting for responses that will never arrive.
Every waiting request occupies memory, worker threads, and network connections. Although the Booking Service itself is functioning perfectly, it gradually runs out of resources because it is spending all of its time waiting for another service.
Ironically, repeatedly calling the failing dependency only makes the situation worse.
Instead of giving the Payment Service time to recover, thousands of additional requests continue arriving every second, increasing its workload even further. Both systems become overloaded, and what originally affected only one dependency now begins affecting the entire application.
This behaviour is another example of cascading failure, but unlike the earlier example involving excessive traffic, this time the chain reaction begins with an unhealthy dependency rather than an overloaded server.
The obvious question is therefore:
Should an application continue communicating with a service that is already failing?
The answer, surprisingly, is no.
Instead of repeatedly attempting operations that are almost certain to fail, resilient systems deliberately stop sending requests for a short period of time. This allows the failing service to recover while simultaneously protecting the calling application from wasting valuable resources.
This idea is known as the Circuit Breaker Pattern.
Understanding the Circuit Breaker Pattern
The name "Circuit Breaker" comes from electrical engineering.
In a home, electrical circuits contain protective switches called circuit breakers. Under normal conditions, electricity flows freely through the circuit. However, if excessive current begins flowing—perhaps because of a short circuit—the breaker immediately disconnects the circuit. Although electricity temporarily stops flowing, the interruption prevents far more serious damage such as overheating, equipment failure, or even fire.
Software systems apply the same principle.
Instead of electricity flowing between electrical components, requests flow between software services.
Instead of excessive electrical current, the danger comes from repeated failures and long response times.
Instead of physically disconnecting a wire, the application temporarily stops sending requests to the failing dependency.
The objective is remarkably similar in both cases:
Prevent a small problem from becoming a much larger one.
How a Circuit Breaker Protects Services
Imagine two microservices communicating with one another.
The Order Service depends on the Payment Service to complete customer purchases.
Under normal conditions, every order request results in a successful call to the Payment Service.
Now suppose the Payment Service becomes unavailable.
Without any protection, every new order continues attempting payment.
Each failed request consumes additional threads, memory, and network resources.
As customer traffic continues increasing, the Order Service gradually becomes overloaded—not because it is malfunctioning, but because it is endlessly waiting for a dependency that cannot respond.
Now imagine placing a circuit breaker between the two services.
Initially, requests continue flowing normally.
However, once the circuit breaker observes that the Payment Service has failed repeatedly, it changes its behaviour.
Instead of forwarding every request, it immediately rejects new calls without contacting the Payment Service at all.
The Order Service no longer wastes time waiting for responses that are unlikely to arrive.
The failing dependency receives an opportunity to recover without being flooded with additional traffic.
More importantly, the rest of the application remains healthy.
The Three States of a Circuit Breaker
Although the concept sounds straightforward, circuit breakers are not simply "enabled" or "disabled."
Instead, they move through three different operational states depending on the health of the downstream service.
Closed State
When everything is functioning normally, the circuit breaker remains Closed.
In this state, every request is forwarded to the downstream service exactly as if the circuit breaker did not exist.
The circuit breaker quietly monitors response times, error rates, and timeouts while allowing traffic to pass.
At this stage, users are completely unaware that a circuit breaker exists.
It simply observes the health of the dependency.
Open State
Suppose the downstream service suddenly begins returning failures.
After the circuit breaker observes that a predefined failure threshold has been exceeded—for example, fifty consecutive failures or an error rate above a configured percentage—it assumes the service is unhealthy.
Rather than continuing to send requests that are almost guaranteed to fail, the circuit breaker opens.
Once open, incoming requests never reach the failing dependency.
Instead, they fail immediately, allowing the application to respond much more quickly than waiting for repeated timeouts.
Although this may appear harsh, it significantly improves the overall stability of the system.
A fast failure is often preferable to a slow one.
Half-Open State
Of course, services eventually recover.
The circuit breaker therefore cannot remain open forever.
After waiting for a configurable period, it enters an intermediate state known as Half-Open.
Rather than forwarding every request, it allows only a small number of carefully selected requests to reach the downstream service.
These requests act as health probes.
If they succeed, the circuit breaker concludes that the dependency has recovered and transitions back to the Closed state.
If they fail again, the breaker immediately returns to the Open state and waits before testing once more.
This gradual recovery process prevents thousands of waiting clients from simultaneously overwhelming a service that has only just restarted.
Instead, traffic is restored carefully and progressively.
Why Fast Failure Is Better Than Slow Failure
Many engineers instinctively believe that applications should keep retrying failed requests because the next attempt might succeed.
Retries certainly have their place, especially when failures are temporary.
However, unlimited retries against an already failing service usually create more problems than they solve.
Every retry generates additional network traffic.
Every retry occupies another application thread.
Every retry increases the workload on a dependency that is already struggling.
Eventually, the retries themselves become part of the outage.
Circuit breakers solve this problem by recognising that continuing to attempt impossible operations serves no useful purpose.
Sometimes the healthiest decision a distributed system can make is to stop trying—for a little while.
This simple principle has made circuit breakers one of the most widely adopted resilience patterns in modern microservice architectures.
Throughout this article, we have explored two of the most important resilience patterns used in distributed systems. We began with Rate Limiting, where applications deliberately control incoming traffic to prevent themselves from becoming overwhelmed. We then examined the Circuit Breaker Pattern, which prevents services from repeatedly communicating with dependencies that are already failing.
Both patterns share a common objective.
They protect the system before failures spread.
However, they still leave us with an important question.
What should the application do after a failure has already occurred?
Suppose a recommendation service is unavailable.
Should the entire website stop loading?
If a notification service fails, should customers be prevented from placing orders?
If an analytics platform becomes unreachable, should payments stop processing?
In many cases, the answer is clearly no.
Although these services are valuable, they are not equally important. Some features are essential to the application's core functionality, while others simply enhance the user experience. A resilient distributed system understands this distinction and is designed to continue providing its most critical functionality even when less important components become unavailable.
This philosophy is known as Graceful Failure Handling, or more commonly, Graceful Degradation.
Rather than allowing a single failure to bring down the entire application, the system accepts that some features may temporarily become unavailable while ensuring that users can still accomplish their primary objectives.
Not Every Failure Should Become an Outage
Imagine opening your favourite e-commerce website.
The homepage loads successfully.
Products appear almost instantly.
You search for a laptop, compare specifications, and decide to purchase one.
During checkout, however, you notice that the "Recommended Products" section is missing.
Would you abandon the purchase because personalised recommendations failed to load?
Probably not.
The recommendation engine improves the shopping experience, but it is not essential to completing an order.
Now imagine a different situation.
The recommendation engine works perfectly, but the payment service fails every time you attempt to check out.
In this case, the website becomes practically useless because the application's most important business function has stopped working.
This simple comparison illustrates one of the most important principles in system design:
Not every service deserves the same level of priority.
Understanding which services are mission-critical and which are optional allows architects to design systems that fail intelligently instead of failing.
Graceful Degradation in Practice
Consider the architecture of a modern streaming platform.
When a user opens the application, several independent services work together to build the experience.
One service authenticates the user.
Another retrieves the list of available movies.
A recommendation engine suggests content based on viewing history.
A separate analytics service records user interactions.
Another service sends notifications about newly released shows.
Now imagine that the recommendation service suddenly becomes unavailable.
Should the entire streaming platform stop functioning?
Of course not.
A much better approach is to temporarily hide the recommendation section while continuing to display the movie catalogue.
The user can still browse content, watch movies, and enjoy the platform.
The application has degraded gracefully rather than failing completely.
This approach prioritises the features that matter most to users while temporarily sacrificing less important functionality.
Fallbacks: Having a Plan B
One of the most common techniques used during graceful degradation is the concept of a fallback.
A fallback is simply an alternative response that the application returns when the preferred operation cannot be completed.
Imagine a weather application that normally retrieves forecasts from a third-party weather provider.
If the provider becomes unavailable, the application has several possible choices.
It could display the most recently cached forecast.
It could return a simplified forecast from another provider.
Or it could simply inform the user that live weather information is temporarily unavailable.
Any of these options is generally preferable to allowing the entire application to crash.
Fallbacks are widely used across distributed systems because they allow applications to continue serving useful responses even when external dependencies experience problems.
Although the returned information may not be perfect, it is often significantly better than returning an error page.
Why Timeouts Matter
One subtle but extremely important aspect of graceful failure handling is knowing when to stop waiting.
Suppose an application sends a request to another service.
If no response arrives after one second, should it continue waiting?
What about ten seconds?
Or one minute?
Without clearly defined limits, applications can spend enormous amounts of time waiting for responses that may never arrive.
Those waiting requests continue consuming memory, threads, and network connections.
Eventually, the application itself begins slowing down despite being perfectly healthy.
This is why modern distributed systems almost always configure timeouts for communication between services.
Instead of waiting indefinitely, requests are automatically abandoned after a predefined period.
The application can then trigger a fallback response, retry the request if appropriate, or simply inform the user that the operation could not be completed.
Timeouts work particularly well when combined with circuit breakers.
The timeout detects that a dependency has become slow.
The circuit breaker notices repeated failures.
Eventually, the breaker opens, preventing further requests from reaching the unhealthy service until recovery begins.
Together, these mechanisms stop small delays from escalating into widespread outages.
Bringing Everything Together
By now, it should be clear that these resilience patterns are not independent techniques.
They complement one another.
Imagine a customer attempting to place an order during an unusually busy shopping event.
The request first reaches the API Gateway, where Rate Limiting ensures that incoming traffic remains within safe operating limits.
The request is then forwarded to the Order Service, which communicates with the Payment Service through a Circuit Breaker.
If the payment provider begins failing, the circuit breaker quickly opens, preventing thousands of additional requests from overwhelming both services.
Finally, if certain non-essential services such as recommendation engines or notification systems become unavailable, Graceful Degradation ensures that the checkout process continues while those optional features are temporarily disabled.
The entire flow looks something like this.
Notice how each resilience pattern solves a different problem.
Rate limiting protects the application from excessive demand.
Circuit breakers protect services from failing dependencies.
Graceful degradation protects the user experience when failures inevitably occur.
Individually, each technique improves reliability.
Together, they create systems capable of surviving situations that would otherwise result in complete outages.
Reliability Is About Trade-offs, Not Perfection
One recurring theme throughout this System Design series is that there are very few absolute answers.
No architecture guarantees perfect scalability.
No consistency model is always correct.
There is no caching strategy that works for every workload.
Reliability follows the same pattern.
Every resilience mechanism introduces trade-offs.
Rate limiting may reject legitimate requests during periods of exceptionally high demand.
Circuit breakers may temporarily deny requests even after a service has recovered if recovery thresholds are configured too conservatively.
Fallback responses may return stale or incomplete information instead of live data.
Yet these trade-offs are almost always preferable to complete system failure.
A slightly degraded application is far more valuable than one that is entirely unavailable.
Modern distributed systems are therefore designed with a simple philosophy:
Accept that failures will happen, isolate them quickly, and recover gracefully without affecting the rest of the system.
That single idea lies at the heart of resilient software architecture.
Final Thoughts
As software systems continue evolving into increasingly distributed architectures, resilience is no longer a luxury—it is a fundamental requirement. Every dependency introduces another opportunity for failure, and every new feature adds another layer of complexity. Engineers cannot eliminate these failures, but they can control how systems respond to them.
Rate limiting prevents applications from accepting more work than they can safely process.
Circuit breakers prevent failures from spreading across service boundaries.
Graceful degradation ensures that users continue receiving value even when parts of the application become unavailable.
Together, these patterns transform software from a collection of independent services into a resilient system capable of operating under real-world conditions, where failures are not exceptional events but expected realities.
Perhaps the most important lesson is that reliability is not measured by how rarely a system fails. It is measured by how well the system continues serving users when failures inevitably occur. That mindset separates software that merely works from software that remains dependable at scale.
Top comments (0)