DEV Community: Sushant Gaurav

Rate Limiting, Circuit Breakers & Handling Failures Gracefully

Sushant Gaurav — Tue, 22 Sep 2026 05:41:00 +0000

One of the biggest misconceptions about modern distributed systems is that they fail because servers are too slow or because there are not enough machines handling requests. While insufficient infrastructure can certainly become a bottleneck, it is rarely the root cause behind large-scale outages. In reality, some of the most significant production incidents have occurred in systems that had thousands of servers, multiple data centres, redundant databases, and sophisticated load balancers. The problem was not a lack of hardware. The problem was that the system was not prepared for failure.

Imagine an e-commerce company launching its biggest sale of the year. Months of preparation have gone into the event. Engineers have increased server capacity, databases have been replicated across multiple regions, caches have been warmed, and additional instances of critical services have been deployed. Everything appears ready. At exactly 10:00 AM, the sale begins. Within seconds, millions of users start refreshing the homepage, searching for products, adding items to their carts, and attempting to complete purchases.

Initially, everything works as expected. Requests are distributed across multiple application servers, response times remain low, and the infrastructure appears healthy. Then, one small problem begins to emerge. The inventory database, which keeps track of available stock, starts responding slightly slower than usual. Instead of taking 50 milliseconds to answer a query, it now takes 500 milliseconds. Half a second may not sound like much, but in a distributed system processing thousands of requests every second, that additional delay quickly becomes significant.

Every checkout request must wait for the inventory service to confirm that a product is still available. Because the inventory service is now waiting longer for the database, the checkout service also begins waiting. Those waiting requests continue occupying server threads, database connections, and memory. Meanwhile, new customers continue arriving and generating even more requests. Within a few minutes, queues begin forming across multiple services. CPU utilisation rises sharply, memory consumption increases, and response times continue growing. Soon, services that were originally healthy also begin slowing down because they are waiting for other services that are already struggling.

What started as a minor slowdown inside a single database has now spread throughout the entire application.

This phenomenon is one of the defining challenges of distributed systems. Unlike monolithic applications, where most operations happen within a single process, distributed applications consist of dozens or even hundreds of independent services communicating continuously over the network. A single user request may pass through an API Gateway, an authentication service, a product service, an inventory service, a payment service, a notification service, and several databases before the user finally receives a response. Every additional dependency introduces another opportunity for delays, failures, or unexpected behaviour.

At first glance, this architecture appears highly resilient because every service can scale independently. However, this same flexibility also creates a new type of risk. Services become interconnected. If one component slows down, every service depending on it begins slowing down as well. If those services become overloaded, they start affecting other services. Gradually, what began as an isolated problem spreads throughout the entire system like a chain reaction.

Engineers often describe this behaviour as a cascading failure. Much like a line of dominoes falling one after another, a failure in one component triggers failures in neighbouring components until eventually the entire application becomes unstable. Importantly, the original component that caused the issue may not even be the one users notice. Customers simply experience a website that feels slow, unresponsive, or completely unavailable.

Large technology companies spend enormous amounts of time designing systems that prevent exactly this situation. Their objective is not merely to ensure that individual services remain operational. Instead, they aim to ensure that failures remain isolated. A slow recommendation engine should never prevent users from placing orders. A malfunctioning notification service should never stop payments from being processed. An external API outage should not cause every application server to exhaust its available resources while waiting for responses that may never arrive.

Achieving this level of resilience requires a fundamental shift in how engineers think about software design. Beginners often design applications assuming that every dependency will behave correctly. Every service will respond quickly. Every database query will succeed. Every network request will complete. Experienced system designers make exactly the opposite assumption. They expect services to fail. They expect networks to become unreliable. They expect traffic spikes to occur unexpectedly. Rather than hoping these situations never happen, they build systems specifically designed to continue operating when they do.

This philosophy explains why modern distributed systems deliberately make decisions that initially seem counterintuitive. Sometimes they reject incoming requests even though additional users are trying to access the application. Sometimes they temporarily stop communicating with another service, even though that service may recover in a few seconds. Sometimes they return partial information instead of attempting to generate a complete response. These decisions may appear strange until one understands the goal behind them.

The purpose is not to maximise the number of successful requests at any given moment.

The purpose is to maximise the long-term health of the system.

Consider a hospital emergency room during a major disaster. Doctors do not attempt to treat every patient simultaneously because doing so would overwhelm the entire medical staff. Instead, patients are prioritised based on urgency, and new admissions may even be redirected to nearby hospitals if capacity has been reached. Although this means some patients must wait longer, the hospital remains capable of treating those in the greatest need.

Distributed systems behave in much the same way. Rather than allowing unlimited requests to consume every available resource, resilient applications establish protective mechanisms that regulate traffic, isolate failures, and recover gracefully when dependencies become unhealthy. These mechanisms prevent small problems from becoming catastrophic outages.

Three design patterns have become particularly important for achieving this level of resilience.

The first is Rate Limiting, which controls how many requests clients are allowed to send, preventing systems from becoming overwhelmed during sudden traffic spikes or malicious attacks.

The second is the Circuit Breaker Pattern, which prevents applications from repeatedly communicating with services that are already failing, allowing both the calling service and the failing dependency time to recover.

The third is Graceful Failure Handling, where applications continue providing as much functionality as possible even when certain components become unavailable, ensuring that users experience degraded service rather than complete failure.

Individually, each of these patterns addresses a different aspect of reliability. Together, they form one of the most important defensive strategies in distributed systems. Rather than attempting to eliminate failures, a practically impossible goal—they acknowledge that failures are inevitable and focus on containing their impact.

At first, the idea of intentionally rejecting user requests seems almost contradictory. After all, the primary responsibility of a web application is to serve its users. If someone sends a request, shouldn't the system always try its best to process it?

While this sounds reasonable in theory, it often leads to the exact opposite outcome in practice.

Imagine a restaurant that has enough chefs to prepare one hundred meals every hour. Under normal conditions, around seventy customers arrive during that time, leaving enough capacity for the kitchen to operate efficiently. Orders are prepared quickly, waiters deliver food on time, and customers leave satisfied.

Now imagine that a famous food blogger unexpectedly recommends the restaurant to millions of followers. Within minutes, five thousand customers arrive at the entrance. The restaurant has only two choices. It can either allow everyone inside, creating chaos in the kitchen, overwhelming the staff, exhausting ingredients, and ultimately disappointing every customer. Or it can temporarily stop accepting new customers until it regains control of the situation.

Most well-managed restaurants choose the second option.

Although some customers may have to wait or return later, the restaurant continues functioning. More importantly, the customers who are already inside still receive the quality of service they expect.

Distributed systems face the same challenge.

Every application has a finite amount of CPU, memory, network bandwidth, database connections, and processing threads. Regardless of whether the application runs on a single server or thousands of cloud instances, those resources are never unlimited. If incoming requests exceed the system's ability to process them, queues begin growing, response times increase, memory usage rises, and eventually the application becomes unstable.

The surprising realisation is that processing every request is not always the best strategy. Sometimes, rejecting a small percentage of requests allows the remaining ninety-nine percent to complete successfully. This principle forms the foundation of Rate Limiting.

Understanding Rate Limiting

Rate limiting is a mechanism that controls how many requests a client is allowed to make within a specified period of time. Instead of allowing unlimited traffic to reach backend services, the system establishes clear boundaries that define acceptable behaviour. Once a client exceeds those boundaries, additional requests are temporarily rejected until the allowed time window resets.

Although this may sound restrictive, rate limiting serves multiple purposes beyond simply reducing traffic. It protects applications from accidental overload, prevents abuse by automated bots, limits the impact of malicious attacks, and ensures that no single client consumes a disproportionate share of system resources.

Consider a public weather API that serves millions of developers around the world. If the API allowed unlimited requests, a single poorly written application caught in an infinite retry loop could generate thousands of requests every second. Even though the bug exists in only one client application, its behaviour could significantly degrade performance for every other developer using the same API.

By enforcing a limit such as one hundred requests per minute for each API key, the service protects itself from misuse while still providing fair access to everyone else.

This idea of fair resource allocation is one of the primary motivations behind rate limiting.

Why Large Internet Companies Depend on Rate Limiting

Almost every major technology company relies on rate limiting, even if users rarely notice it.

When you use the GitHub API, you cannot make unlimited requests indefinitely. The platform enforces request limits to ensure that automated scripts do not monopolise shared infrastructure.

Payment providers such as Stripe apply strict limits to protect financial systems from accidental duplicate requests and malicious abuse.

Cloud providers implement request quotas to ensure that one customer cannot unintentionally consume resources needed by thousands of others.

Even conversational AI platforms implement request limits. Without them, a handful of users—or even automated programs—could generate enormous volumes of requests, degrading response quality for everyone else using the service.

These limits are not signs of weak infrastructure.

Quite the opposite.

They are evidence that the infrastructure has been designed to remain stable under unpredictable conditions.

Where Should Rate Limiting Be Applied?

One of the most common questions engineers ask is where rate limiting should actually occur.

Should it be implemented inside every microservice?

Should databases reject excessive queries?

Should the web server perform the limiting?

While multiple approaches exist, the most common practice is to perform rate limiting as early as possible, before expensive backend operations begin.

For this reason, many distributed systems enforce request limits at the API Gateway or Load Balancer.

This placement provides an important advantage.

Instead of allowing excessive traffic to travel through multiple services before finally being rejected, unwanted requests are filtered immediately at the system's entry point. Backend services never spend CPU cycles, memory, or database connections processing requests that should not have been accepted in the first place.

This approach is remarkably similar to airport security. Rather than allowing passengers to enter restricted areas before checking identification, airports verify passengers at the entrance, preventing unnecessary congestion throughout the terminal.

Different Types of Rate Limits

Not every client should necessarily receive the same limits.

Consider a cloud platform serving both anonymous visitors and paying enterprise customers.

Anonymous users might be allowed only a small number of requests each minute to discourage abuse. Registered users may receive higher limits, while enterprise customers paying for premium services could receive substantially larger quotas.

Similarly, login endpoints often use much stricter limits than product search endpoints.

A user might be allowed to search for products hundreds of times every minute, but only attempt five password submissions before the system temporarily blocks additional login attempts. This simple restriction dramatically reduces the effectiveness of brute-force password attacks.

Rate limiting therefore becomes more than just a performance optimisation.

It also becomes an important security mechanism.

How Does a System Know When a User Has Exceeded the Limit?

At this point, the overall idea of rate limiting is relatively straightforward.

The more interesting question is how software actually keeps track of requests.

Imagine an API serving ten million users.

Every second, requests arrive from different countries, devices, and applications.

How does the system determine whether a particular client has already exceeded one hundred requests during the current minute?

The answer is that every incoming request is associated with some form of identity.

Depending on the application, this identity could be an IP address, an authenticated user account, an API key, a session identifier, or even an organisation ID.

Each request updates a small counter maintained by the rate-limiting system.

Whenever another request arrives, the current count is checked against the configured limit.

If the client remains within the permitted quota, the request proceeds normally.

If the limit has already been exceeded, the request is rejected immediately, often with an HTTP 429 Too Many Requests response.

The implementation sounds simple.

However, designing a rate limiter that remains accurate while processing millions of requests every second across multiple servers is surprisingly difficult.

Questions quickly begin to arise.

What happens when the clock reaches the next minute?

Should every counter immediately reset?

How do multiple servers maintain consistent request counts?

How can the system avoid sudden bursts of traffic exactly when counters reset?

Answering these questions has led engineers to develop several sophisticated rate-limiting algorithms, each designed to solve different types of workload patterns.

Some algorithms prioritise simplicity.

Others prioritise fairness.

Others are specifically designed to accommodate short bursts of traffic without allowing sustained overload.

By now, we understand why modern distributed systems need rate limiting. Every application has finite resources, and allowing unlimited traffic inevitably leads to congestion, increased latency, and eventually service outages. However, knowing that requests should be limited is only half the problem. The more interesting challenge is deciding how those limits should actually be enforced.

At first glance, the solution appears deceptively simple. Suppose an API allows one hundred requests per minute for every user. We could simply maintain a counter for each user, increment it every time a request arrives, and reset the counter when the next minute begins. If the counter reaches one hundred, every subsequent request is rejected until the minute resets.

While this approach certainly works, engineers quickly discovered that it introduces an unexpected problem.

Imagine the current time is 10:00:59. A client has not made a single request during the last minute. Suddenly, they send one hundred requests within a fraction of a second. Since the current minute has not yet ended, every request is accepted. One second later, the clock reaches 10:01:00, all counters are reset, and the client immediately sends another one hundred requests.

Technically, the client has respected the configured limit of one hundred requests per minute.

In reality, however, the server has processed two hundred requests within roughly two seconds.

The system followed the rule perfectly.

The rule simply was not designed well enough.

This seemingly small observation has led to several different rate-limiting algorithms, each attempting to balance fairness, simplicity, performance, and implementation complexity. Rather than viewing these algorithms as competing solutions, it is more helpful to think of them as different tools designed for different workloads.

The Fixed Window Algorithm

The simplest and perhaps most intuitive rate-limiting strategy is the Fixed Window algorithm.

Time is divided into fixed intervals—for example, one minute. Every client receives a counter associated with the current window. Each incoming request increments that counter. Once the counter reaches the configured limit, all remaining requests during that window are rejected. When the next window begins, the counter is reset to zero.

The primary advantage of this approach is its simplicity. It requires very little memory, is straightforward to implement, and performs extremely well even under heavy traffic. This is why many early APIs adopted the Fixed Window strategy.

However, simplicity often comes with trade-offs.

As we saw earlier, requests arriving near the boundary between two windows can effectively double the intended request rate. This phenomenon, commonly known as the boundary problem, makes Fixed Window less suitable for systems where traffic patterns fluctuate rapidly.

For many internal applications, this limitation is acceptable. For large public APIs serving millions of users, however, engineers generally prefer more sophisticated approaches.

Sliding Window: A Fairer Approach

Instead of dividing time into rigid intervals, the Sliding Window algorithm continuously evaluates requests over the most recent time period.

Suppose an API allows one hundred requests every sixty seconds.

Rather than asking, "How many requests occurred during the current minute?" the Sliding Window algorithm asks a different question:

"How many requests has this client made during the last sixty seconds?"

Notice the difference.

The observation window moves forward continuously with time.

Because the window never resets abruptly, sudden bursts around minute boundaries disappear. Traffic becomes much smoother, and clients receive a more consistent experience.

The trade-off is computational complexity.

Instead of maintaining a single counter, the system must remember when recent requests occurred so that older requests can gradually expire from the sliding window. For applications processing millions of requests every second, this additional bookkeeping increases both memory usage and implementation complexity.

Despite these challenges, Sliding Window is widely used because it produces significantly fairer traffic distribution than Fixed Window.

The Leaky Bucket Algorithm

Imagine pouring water into a bucket with a small hole at the bottom.

Water may enter the bucket quickly, but it leaves at a constant rate.

If water arrives faster than it can drain, the bucket eventually overflows.

The Leaky Bucket algorithm applies the same idea to incoming requests.

Instead of processing every request immediately, requests are placed into a queue and released at a steady rate.

This approach produces an important benefit.

Even if traffic arrives in sudden bursts, downstream services receive requests at a predictable and stable rate.

Applications that perform expensive operations—such as generating reports, processing images, or communicating with slower external systems—often benefit from this smoothing effect.

The drawback is that bursty traffic may experience increased waiting times. Requests that arrive together must wait for earlier requests to leave the queue, increasing latency even when the backend remains healthy.

In other words, the Leaky Bucket prioritises consistency over responsiveness.

Token Bucket: Balancing Protection and Flexibility

Perhaps the most widely adopted algorithm today is the Token Bucket algorithm because it combines strong protection with enough flexibility to accommodate normal traffic patterns.

The concept is surprisingly elegant.

Imagine a bucket that gradually fills with tokens at a fixed rate. Every incoming request must consume one token before it can proceed. If tokens are available, the request is immediately accepted. If the bucket becomes empty, additional requests are rejected until new tokens are generated.

Unlike the Leaky Bucket, the Token Bucket allows short bursts of activity.

Suppose a user remains inactive for several seconds. During that time, unused tokens accumulate inside the bucket. When the user suddenly sends multiple requests, those stored tokens allow the requests to proceed immediately without violating the long-term rate limit.

This makes the Token Bucket particularly well suited for interactive applications.

Human users rarely generate perfectly uniform traffic. They may spend several minutes reading content before suddenly clicking several buttons in quick succession. Allowing these short bursts creates a smoother user experience while still protecting backend services from sustained overload.

For this reason, Token Bucket has become the preferred choice for many API gateways, cloud platforms, and networking systems.

Choosing the Right Algorithm

At this point, it should be clear that no single algorithm is universally superior.

Each algorithm optimises for different priorities.

The Fixed Window algorithm is simple, efficient, and easy to implement, but suffers from boundary effects.

Sliding Window produces much fairer request distribution, although it requires more computational resources.

Leaky Bucket excels at smoothing traffic and protecting downstream services from sudden bursts, making it useful for systems that require predictable processing rates.

Token Bucket offers an excellent balance between fairness and flexibility by allowing temporary bursts while still enforcing long-term request limits, which explains its popularity in modern distributed systems.

The most appropriate algorithm therefore depends not only on the application's traffic patterns but also on its business requirements.

An authentication service protecting against brute-force attacks may prioritise strict enforcement. A streaming platform may prefer flexibility to accommodate natural user behaviour. A financial API may choose consistency above all else.

Understanding these trade-offs is significantly more valuable than memorising the algorithms themselves.

Up to this point, we have focused on protecting systems before requests enter the application. Rate limiting controls incoming traffic and prevents services from becoming overwhelmed by excessive demand.

But what happens when the problem does not originate from users at all?

What if traffic levels remain perfectly normal, yet one of your dependencies suddenly becomes slow or completely unavailable?

Should your application continue sending requests to a service that is already failing?

Surprisingly, repeatedly retrying those requests often makes the situation even worse.

To solve this problem, distributed systems employ another powerful resilience pattern known as the Circuit Breaker. Instead of allowing failures to spread across services, a circuit breaker detects unhealthy dependencies, temporarily stops communication with them, and gives both systems time to recover.

Up to this point, we have focused on protecting a system from excessive incoming traffic. Rate limiting acts like a security guard standing at the entrance of a building, ensuring that only a manageable number of people are allowed inside at any given time. By controlling the rate at which requests enter the system, applications prevent themselves from becoming overwhelmed before any real damage occurs.

However, not every failure originates from incoming traffic.

Sometimes the application itself is healthy, the number of users is perfectly normal, and the infrastructure has more than enough resources to handle the workload. Yet requests still begin timing out, response times suddenly increase, and users start experiencing errors.

In situations like these, the problem often lies outside the service itself.

Modern applications rarely operate in isolation. A typical microservice communicates with authentication services, payment providers, recommendation engines, notification systems, databases, search engines, cloud storage services, and numerous third-party APIs. Every one of these dependencies represents another system that can become slow, unavailable, or completely fail.

The challenge is that applications generally assume these dependencies will respond within a reasonable amount of time. When they stop responding, the calling service often continues waiting patiently, hoping that the next request will succeed. Unfortunately, thousands of requests making the same assumption simultaneously can quickly exhaust the application's own resources.

To understand why this happens, imagine an online travel booking platform.

A customer searches for flights, selects one, enters passenger details, and finally clicks the Book Now button. The Booking Service receives the request and immediately contacts the Payment Service to complete the transaction. Under normal conditions, the payment provider responds within a few hundred milliseconds, allowing the booking process to continue smoothly.

Now imagine that the external payment provider experiences an outage.

The Booking Service sends the payment request and waits.

After several seconds, the request eventually times out.

Meanwhile, another customer clicks Book Now.

A second payment request is sent.

It also waits.

Within a few moments, hundreds of booking requests are simultaneously waiting for responses that will never arrive.

Every waiting request occupies memory, worker threads, and network connections. Although the Booking Service itself is functioning perfectly, it gradually runs out of resources because it is spending all of its time waiting for another service.

Ironically, repeatedly calling the failing dependency only makes the situation worse.

Instead of giving the Payment Service time to recover, thousands of additional requests continue arriving every second, increasing its workload even further. Both systems become overloaded, and what originally affected only one dependency now begins affecting the entire application.

This behaviour is another example of cascading failure, but unlike the earlier example involving excessive traffic, this time the chain reaction begins with an unhealthy dependency rather than an overloaded server.

The obvious question is therefore:

Should an application continue communicating with a service that is already failing?

The answer, surprisingly, is no.

Instead of repeatedly attempting operations that are almost certain to fail, resilient systems deliberately stop sending requests for a short period of time. This allows the failing service to recover while simultaneously protecting the calling application from wasting valuable resources.

This idea is known as the Circuit Breaker Pattern.

Understanding the Circuit Breaker Pattern

The name "Circuit Breaker" comes from electrical engineering.

In a home, electrical circuits contain protective switches called circuit breakers. Under normal conditions, electricity flows freely through the circuit. However, if excessive current begins flowing—perhaps because of a short circuit—the breaker immediately disconnects the circuit. Although electricity temporarily stops flowing, the interruption prevents far more serious damage such as overheating, equipment failure, or even fire.

Software systems apply the same principle.

Instead of electricity flowing between electrical components, requests flow between software services.

Instead of excessive electrical current, the danger comes from repeated failures and long response times.

Instead of physically disconnecting a wire, the application temporarily stops sending requests to the failing dependency.

The objective is remarkably similar in both cases:

Prevent a small problem from becoming a much larger one.

How a Circuit Breaker Protects Services

Imagine two microservices communicating with one another.

The Order Service depends on the Payment Service to complete customer purchases.

Under normal conditions, every order request results in a successful call to the Payment Service.

Now suppose the Payment Service becomes unavailable.

Without any protection, every new order continues attempting payment.

Each failed request consumes additional threads, memory, and network resources.

As customer traffic continues increasing, the Order Service gradually becomes overloaded—not because it is malfunctioning, but because it is endlessly waiting for a dependency that cannot respond.

Now imagine placing a circuit breaker between the two services.

Initially, requests continue flowing normally.

However, once the circuit breaker observes that the Payment Service has failed repeatedly, it changes its behaviour.

Instead of forwarding every request, it immediately rejects new calls without contacting the Payment Service at all.

The Order Service no longer wastes time waiting for responses that are unlikely to arrive.

The failing dependency receives an opportunity to recover without being flooded with additional traffic.

More importantly, the rest of the application remains healthy.

The Three States of a Circuit Breaker

Although the concept sounds straightforward, circuit breakers are not simply "enabled" or "disabled."

Instead, they move through three different operational states depending on the health of the downstream service.

Closed State

When everything is functioning normally, the circuit breaker remains Closed.

In this state, every request is forwarded to the downstream service exactly as if the circuit breaker did not exist.

The circuit breaker quietly monitors response times, error rates, and timeouts while allowing traffic to pass.

At this stage, users are completely unaware that a circuit breaker exists.

It simply observes the health of the dependency.

Open State

Suppose the downstream service suddenly begins returning failures.

After the circuit breaker observes that a predefined failure threshold has been exceeded—for example, fifty consecutive failures or an error rate above a configured percentage—it assumes the service is unhealthy.

Rather than continuing to send requests that are almost guaranteed to fail, the circuit breaker opens.

Once open, incoming requests never reach the failing dependency.

Instead, they fail immediately, allowing the application to respond much more quickly than waiting for repeated timeouts.

Although this may appear harsh, it significantly improves the overall stability of the system.

A fast failure is often preferable to a slow one.

Half-Open State

Of course, services eventually recover.

The circuit breaker therefore cannot remain open forever.

After waiting for a configurable period, it enters an intermediate state known as Half-Open.

Rather than forwarding every request, it allows only a small number of carefully selected requests to reach the downstream service.

These requests act as health probes.

If they succeed, the circuit breaker concludes that the dependency has recovered and transitions back to the Closed state.

If they fail again, the breaker immediately returns to the Open state and waits before testing once more.

This gradual recovery process prevents thousands of waiting clients from simultaneously overwhelming a service that has only just restarted.

Instead, traffic is restored carefully and progressively.

Why Fast Failure Is Better Than Slow Failure

Many engineers instinctively believe that applications should keep retrying failed requests because the next attempt might succeed.

Retries certainly have their place, especially when failures are temporary.

However, unlimited retries against an already failing service usually create more problems than they solve.

Every retry generates additional network traffic.

Every retry occupies another application thread.

Every retry increases the workload on a dependency that is already struggling.

Eventually, the retries themselves become part of the outage.

Circuit breakers solve this problem by recognising that continuing to attempt impossible operations serves no useful purpose.

Sometimes the healthiest decision a distributed system can make is to stop trying—for a little while.

This simple principle has made circuit breakers one of the most widely adopted resilience patterns in modern microservice architectures.

Throughout this article, we have explored two of the most important resilience patterns used in distributed systems. We began with Rate Limiting, where applications deliberately control incoming traffic to prevent themselves from becoming overwhelmed. We then examined the Circuit Breaker Pattern, which prevents services from repeatedly communicating with dependencies that are already failing.

Both patterns share a common objective.

They protect the system before failures spread.

However, they still leave us with an important question.

What should the application do after a failure has already occurred?

Suppose a recommendation service is unavailable.

Should the entire website stop loading?

If a notification service fails, should customers be prevented from placing orders?

If an analytics platform becomes unreachable, should payments stop processing?

In many cases, the answer is clearly no.

Although these services are valuable, they are not equally important. Some features are essential to the application's core functionality, while others simply enhance the user experience. A resilient distributed system understands this distinction and is designed to continue providing its most critical functionality even when less important components become unavailable.

This philosophy is known as Graceful Failure Handling, or more commonly, Graceful Degradation.

Rather than allowing a single failure to bring down the entire application, the system accepts that some features may temporarily become unavailable while ensuring that users can still accomplish their primary objectives.

Not Every Failure Should Become an Outage

Imagine opening your favourite e-commerce website.

The homepage loads successfully.

Products appear almost instantly.

You search for a laptop, compare specifications, and decide to purchase one.

During checkout, however, you notice that the "Recommended Products" section is missing.

Would you abandon the purchase because personalised recommendations failed to load?

Probably not.

The recommendation engine improves the shopping experience, but it is not essential to completing an order.

Now imagine a different situation.

The recommendation engine works perfectly, but the payment service fails every time you attempt to check out.

In this case, the website becomes practically useless because the application's most important business function has stopped working.

This simple comparison illustrates one of the most important principles in system design:

Not every service deserves the same level of priority.

Understanding which services are mission-critical and which are optional allows architects to design systems that fail intelligently instead of failing.

Graceful Degradation in Practice

Consider the architecture of a modern streaming platform.

When a user opens the application, several independent services work together to build the experience.

One service authenticates the user.

Another retrieves the list of available movies.

A recommendation engine suggests content based on viewing history.

A separate analytics service records user interactions.

Another service sends notifications about newly released shows.

Now imagine that the recommendation service suddenly becomes unavailable.

Should the entire streaming platform stop functioning?

Of course not.

A much better approach is to temporarily hide the recommendation section while continuing to display the movie catalogue.

The user can still browse content, watch movies, and enjoy the platform.

The application has degraded gracefully rather than failing completely.

This approach prioritises the features that matter most to users while temporarily sacrificing less important functionality.

Fallbacks: Having a Plan B

One of the most common techniques used during graceful degradation is the concept of a fallback.

A fallback is simply an alternative response that the application returns when the preferred operation cannot be completed.

Imagine a weather application that normally retrieves forecasts from a third-party weather provider.

If the provider becomes unavailable, the application has several possible choices.

It could display the most recently cached forecast.

It could return a simplified forecast from another provider.

Or it could simply inform the user that live weather information is temporarily unavailable.

Any of these options is generally preferable to allowing the entire application to crash.

Fallbacks are widely used across distributed systems because they allow applications to continue serving useful responses even when external dependencies experience problems.

Although the returned information may not be perfect, it is often significantly better than returning an error page.

Why Timeouts Matter

One subtle but extremely important aspect of graceful failure handling is knowing when to stop waiting.

Suppose an application sends a request to another service.

If no response arrives after one second, should it continue waiting?

What about ten seconds?

Or one minute?

Without clearly defined limits, applications can spend enormous amounts of time waiting for responses that may never arrive.

Those waiting requests continue consuming memory, threads, and network connections.

Eventually, the application itself begins slowing down despite being perfectly healthy.

This is why modern distributed systems almost always configure timeouts for communication between services.

Instead of waiting indefinitely, requests are automatically abandoned after a predefined period.

The application can then trigger a fallback response, retry the request if appropriate, or simply inform the user that the operation could not be completed.

Timeouts work particularly well when combined with circuit breakers.

The timeout detects that a dependency has become slow.

The circuit breaker notices repeated failures.

Eventually, the breaker opens, preventing further requests from reaching the unhealthy service until recovery begins.

Together, these mechanisms stop small delays from escalating into widespread outages.

Bringing Everything Together

By now, it should be clear that these resilience patterns are not independent techniques.

They complement one another.

Imagine a customer attempting to place an order during an unusually busy shopping event.

The request first reaches the API Gateway, where Rate Limiting ensures that incoming traffic remains within safe operating limits.

The request is then forwarded to the Order Service, which communicates with the Payment Service through a Circuit Breaker.

If the payment provider begins failing, the circuit breaker quickly opens, preventing thousands of additional requests from overwhelming both services.

Finally, if certain non-essential services such as recommendation engines or notification systems become unavailable, Graceful Degradation ensures that the checkout process continues while those optional features are temporarily disabled.

The entire flow looks something like this.

Notice how each resilience pattern solves a different problem.

Rate limiting protects the application from excessive demand.

Circuit breakers protect services from failing dependencies.

Graceful degradation protects the user experience when failures inevitably occur.

Individually, each technique improves reliability.

Together, they create systems capable of surviving situations that would otherwise result in complete outages.

Reliability Is About Trade-offs, Not Perfection

One recurring theme throughout this System Design series is that there are very few absolute answers.

No architecture guarantees perfect scalability.

No consistency model is always correct.

There is no caching strategy that works for every workload.

Reliability follows the same pattern.

Every resilience mechanism introduces trade-offs.

Rate limiting may reject legitimate requests during periods of exceptionally high demand.

Circuit breakers may temporarily deny requests even after a service has recovered if recovery thresholds are configured too conservatively.

Fallback responses may return stale or incomplete information instead of live data.

Yet these trade-offs are almost always preferable to complete system failure.

A slightly degraded application is far more valuable than one that is entirely unavailable.

Modern distributed systems are therefore designed with a simple philosophy:

Accept that failures will happen, isolate them quickly, and recover gracefully without affecting the rest of the system.

That single idea lies at the heart of resilient software architecture.

Final Thoughts

As software systems continue evolving into increasingly distributed architectures, resilience is no longer a luxury—it is a fundamental requirement. Every dependency introduces another opportunity for failure, and every new feature adds another layer of complexity. Engineers cannot eliminate these failures, but they can control how systems respond to them.

Rate limiting prevents applications from accepting more work than they can safely process.

Circuit breakers prevent failures from spreading across service boundaries.

Graceful degradation ensures that users continue receiving value even when parts of the application become unavailable.

Together, these patterns transform software from a collection of independent services into a resilient system capable of operating under real-world conditions, where failures are not exceptional events but expected realities.

Perhaps the most important lesson is that reliability is not measured by how rarely a system fails. It is measured by how well the system continues serving users when failures inevitably occur. That mindset separates software that merely works from software that remains dependable at scale.

Designing for Reliability: Redundancy, Replication, and Fault Tolerance

Sushant Gaurav — Tue, 08 Sep 2026 05:41:00 +0000

If there is one assumption every software engineer eventually learns to make, it is this:

Failures are not exceptional events—they are inevitable.

Servers crash. Hard disks fail. Network cables get unplugged. Data centers lose power. Cloud regions experience outages. Software contains bugs. Human operators make mistakes. Even seemingly reliable hardware eventually reaches the end of its lifespan.

When applications are small, these failures are often inconvenient but manageable. A company may simply restart the server or restore a backup. Users might experience a few minutes of downtime, but the impact remains relatively limited.

As systems grow larger, however, this mindset no longer works.

Imagine an online banking platform serving millions of customers. Or a streaming service with viewers spread across the globe. Or an e-commerce platform processing thousands of orders every second during a major shopping festival.

For these applications, even a few minutes of downtime can translate into millions of dollars in lost revenue, damaged customer trust, and long-term reputational harm.

The question therefore changes.

Instead of asking,

"How do we prevent failures?"

experienced system designers ask,

"How do we keep the system running even when failures occur?"

That single shift in thinking defines modern distributed systems.

Reliability is not achieved by building components that never fail.

Reliability is achieved by designing systems that continue functioning despite failures.

Reliability Is More Than Just Uptime

When people hear the word reliable, they often think of uptime.

If an application remains online, it must be reliable.

While availability is certainly an important aspect of reliability, the two concepts are not identical.

Consider an online shopping website.

Suppose customers can still browse products, but every payment attempt fails because the payment service has crashed.

Technically, the website is still online.

Yet very few people would call it reliable.

Now imagine another scenario.

The payment succeeds, but inventory updates are lost because one database server failed before writing the latest stock count.

Again, the application appears available.

However, it is no longer behaving correctly.

A reliable system is therefore one that continues delivering its intended functionality despite unexpected failures.

It is not simply about remaining accessible.

It is about continuing to perform the operations users depend upon.

Why Failures Become More Common as Systems Grow

A single-server application contains relatively few moving parts.

One application.

One database.

One machine.

Naturally, there are fewer opportunities for something to go wrong.

Modern distributed systems look very different.

A single user request might involve an API Gateway, an authentication service, an order service, a payment service, an inventory service, a cache, multiple databases, a message broker, and several third-party APIs.

Every additional component improves scalability, flexibility, or maintainability.

However, every additional component also introduces another potential point of failure.

If even one critical dependency becomes unavailable, the entire request may fail.

Ironically, the very architectures that allow systems to scale also make them more vulnerable to failures.

This is why reliability becomes one of the central concerns in distributed system design.

The Philosophy of Designing for Failure

One of the biggest differences between beginner and experienced engineers is how they think about failures.

Beginners often assume that failures are rare exceptions.

Experienced engineers assume failures are guaranteed.

This difference influences every architectural decision.

Instead of asking,

"What happens if the server crashes?"

they ask,

"When the server crashes, what happens next?"

Instead of asking,

"What if the database becomes unavailable?"

they ask,

"How quickly can another database take over?"

This philosophy is commonly known as designing for failure.

Rather than attempting to eliminate failures—which is practically impossible—the goal is to ensure that failures remain isolated and recoverable.

A well-designed distributed system accepts that individual components will eventually fail.

The overall system should not.

Three Pillars of Reliability

Although reliability encompasses many different engineering practices, most distributed systems achieve it through three fundamental ideas.

The first is redundancy.

Critical components should never exist as a single copy. If one server fails, another should already be available to continue serving requests.

The second is replication.

Important data should be stored on multiple machines rather than on a single storage device. If hardware fails, information remains available elsewhere.

The third is fault tolerance.

Even when failures occur, the application should continue operating with little or no disruption to users.

Although these concepts are closely related, they solve different problems.

Understanding the distinction between them is essential because they are often confused with one another.

The remainder of this article will explore each concept individually before showing how they work together to build highly reliable systems.

Redundancy: Never Depend on a Single Component

Imagine a suspension bridge supported by only one steel cable.

If that cable snaps, the bridge immediately collapses.

Now imagine the same bridge supported by dozens of independent cables.

Losing one cable no longer causes disaster because the remaining cables continue carrying the load.

Distributed systems apply the same principle.

Whenever a component is critical to the application's operation, relying on a single instance becomes extremely risky.

Suppose an application is deployed on just one server.

Everything works perfectly until that server experiences a hardware failure.

At that moment, the application becomes completely unavailable.

Now consider a different architecture.

Instead of depending on one server, requests are distributed across several identical instances.

If one server crashes, the load balancer simply redirects traffic to the remaining healthy servers.

Most users may never even notice that a failure occurred.

This is redundancy in its simplest form.

The purpose of redundancy is not to make systems faster.

Its primary purpose is to eliminate single points of failure.

What Is a Single Point of Failure?

A Single Point of Failure (SPOF) is any component whose failure causes the entire system—or a critical part of it—to stop functioning.

Single points of failure are dangerous because they concentrate risk.

If everything depends on one component, the reliability of the entire application becomes limited by that component.

Consider a system with three application servers but only one database.

Although the application layer is redundant, the database remains a single point of failure.

If that database crashes, every application server loses access to the data it needs.

The application effectively stops working.

Identifying and eliminating single points of failure is therefore one of the first responsibilities of system architects.

Every critical dependency should eventually have a backup plan.

By running several application servers instead of one, the system can continue serving users even if an individual server crashes.

At first glance, it may seem that redundancy alone is enough to build a reliable system.

After all, if every component has a backup, what else could go wrong?

The answer lies in one important observation.

Application servers can usually be recreated. Data cannot.

If an application server crashes, we can simply start another one from the same deployment package or container image.

If the only copy of a customer's payment history, medical records, or financial transactions disappears, recreating that data is impossible.

Infrastructure can be rebuilt.

Lost data often cannot.

This is why distributed systems treat data very differently from application code.

Rather than simply deploying multiple servers, they create multiple copies of the data itself.

This brings us to one of the most fundamental concepts in distributed systems:

Replication.

What Is Replication?

Replication is the process of maintaining multiple copies of the same data on different machines.

The goal is straightforward.

If one database server becomes unavailable, another server should already contain the same information and be capable of continuing operations.

Notice the subtle difference between redundancy and replication.

Redundancy creates additional infrastructure.

Replication creates additional copies of data.

Although these concepts work together, they solve different problems.

Consider a database deployed on a single machine.

Every customer record, transaction, product, and order exists in exactly one location.

As long as that machine remains healthy, everything works normally.

But if its storage device fails, the application immediately loses access to its data.

Now imagine introducing replicas.

The application still has one primary database responsible for accepting writes.

However, every change made to the primary is copied to multiple replicas.

Now, if one machine fails, another already contains the same information.

The system has become significantly more resilient.

Why Replication Matters

Imagine running an online banking platform.

Every deposit, withdrawal, and transfer updates customer balances stored in a database.

Suppose that database exists on only one physical server.

One afternoon, the server's storage device experiences a catastrophic failure.

Without replication, every transaction that has not been backed up is gone.

Even if backups exist, restoring them may take hours.

Customers lose access to their accounts.

Businesses cannot process payments.

Trust disappears almost instantly.

Now consider the same application with replicated databases.

The primary server unexpectedly crashes.

Although one database becomes unavailable, another replica already contains nearly identical information.

Instead of restoring backups from scratch, the application promotes one of the replicas to become the new primary.

Users experience only a brief interruption—or perhaps none at all.

The application survives because the data survived.

Replication protects one of the most valuable assets any software system possesses:

Its information.

Primary-Replica Architecture

The most common replication strategy is known as Primary-Replica Replication, sometimes called Leader-Follower Replication.

In this architecture, one database acts as the primary.

Every write operation flows through this server.

The remaining replicas continuously receive updates from the primary.

Whenever a customer places an order, the write is first stored on the primary database.

The primary then propagates the change to its replicas.

These replicas remain synchronised so that they can take over if the primary becomes unavailable.

Many relational databases, including PostgreSQL and MySQL, support this replication model because it balances simplicity with reliability.

Reads and Writes Are Treated Differently

One interesting property of primary-replica architectures is that not every request must go to the primary database.

Write operations almost always target the primary because allowing multiple independent servers to modify the same data simultaneously introduces significant coordination challenges.

Read operations, however, are often distributed across replicas.

This architecture provides two important benefits.

First, the primary handles fewer read requests, allowing it to focus on processing writes efficiently.

Second, multiple replicas can serve read traffic simultaneously, significantly increasing the system's overall throughput.

This approach is extremely common in applications where reads greatly outnumber writes.

Social media platforms, e-commerce websites, streaming services, and news portals all spend far more time serving content than modifying it.

Replication therefore improves not only reliability but also scalability.

What Happens When the Primary Fails?

Replication becomes truly valuable during failures.

Imagine that the primary database suddenly becomes unavailable.

Without replication, every application attempting to write data immediately fails.

With replication, however, the system can perform a process known as failover.

Failover simply means transferring responsibility from a failed component to a healthy one.

Instead of waiting for the failed server to recover, one of the replicas becomes the new primary.

Applications reconnect to the newly promoted server and continue operating.

From the user's perspective, the interruption may last only a few seconds.

This automatic transition is one of the defining characteristics of highly available systems.

Replication Is Not Free

At this point, replication sounds like the perfect solution.

More copies of the data.

Better availability.

Improved read scalability.

Automatic recovery from failures.

So why doesn't every distributed system simply replicate everything?

Because replication introduces its own challenges.

Every time data changes, every replica must eventually receive the update.

Until synchronisation completes, different replicas may temporarily contain different versions of the same information.

If a customer updates their shipping address and immediately performs another request, one replica may already contain the new address while another still returns the previous one.

Suddenly, we encounter a familiar concept.

Consistency.

In our previous article, we explored the trade-offs between strong consistency and eventual consistency.

Replication is one of the primary reasons those consistency models exist.

The more copies of data we maintain, the harder it becomes to ensure that every copy remains synchronised at every moment.

In other words, replication improves reliability while simultaneously introducing coordination challenges.

This is one of the recurring themes throughout distributed systems.

Every solution creates new engineering problems that must be solved thoughtfully.

Redundancy vs Replication

Because these two terms are frequently confused, it is useful to summarise the distinction before moving forward.

Redundancy focuses on components.

If one application server fails, another should already be available.

Replication focuses on data.

If one database fails, another should already contain the same information.

A useful way to remember the difference is this:

Redundancy protects services. Replication protects information.

Modern distributed systems almost always employ both.

Running multiple application servers without replicated databases still leaves the data vulnerable.

Replicating databases without redundant application servers still leaves users unable to access the application when servers fail.

Reliable systems require both infrastructure redundancy and data replication working together.

What Is Fault Tolerance?

Fault tolerance is the ability of a system to continue operating correctly even when one or more of its components fail.

Notice the wording carefully.

The objective is not to prevent failures.

Failures are inevitable.

Instead, the objective is to prevent those failures from becoming service outages.

Imagine driving a modern car.

If one tyre suddenly bursts, the vehicle does not instantly stop functioning.

Safety systems help the driver maintain control until they can safely stop.

The failure still occurs.

The system simply handles it gracefully.

Distributed systems follow the same philosophy.

When an application server crashes, another server should automatically begin handling requests.

When a database becomes unavailable, a healthy replica should take over.

When one availability zone experiences an outage, traffic should be redirected to another.

Users should experience little or no disruption.

Fault tolerance is therefore about maintaining service despite failures, not eliminating failures.

Building Systems That Expect Failure

One of the biggest mindset shifts in distributed systems is moving from failure prevention to failure management.

Traditional software often assumes that servers will remain available.

Distributed systems assume exactly the opposite.

Every component is treated as if it could fail at any moment.

This assumption influences every architectural decision.

Instead of asking,

"Will this service ever fail?"

engineers ask,

"What happens when this service fails?"

Suppose an application depends on a recommendation engine to suggest products to customers.

If the recommendation service crashes, should the checkout process stop working?

Probably not.

A better design allows customers to continue purchasing products while temporarily hiding personalised recommendations.

The system loses a non-essential feature.

The core business functionality continues operating.

This approach is known as graceful degradation.

Graceful Degradation

Not every component inside an application has the same level of importance.

Some services are essential.

Others simply improve the user experience.

A well-designed distributed system recognises this difference.

Imagine watching a movie on a streaming platform.

If the recommendation service becomes unavailable, you might stop seeing personalised suggestions.

However, the movie itself should continue playing without interruption.

Similarly, an online shopping platform should continue accepting orders even if its analytics service temporarily stops processing events.

Customers care far more about completing purchases than updating business dashboards.

This idea is illustrated below.

Instead of allowing one failing service to bring down the entire application, the system isolates the failure and continues providing its most important functionality.

This philosophy is one of the defining characteristics of resilient software.

Detecting Failures Automatically

Fault tolerance depends on quickly identifying unhealthy components.

A system cannot recover from failures it does not detect.

For this reason, distributed systems continuously monitor the health of their services.

Load balancers periodically send lightweight health check requests to application servers.

If a server responds successfully, it continues receiving traffic.

If it stops responding, the load balancer automatically removes it from the pool.

Notice that users do not need to know a server has failed.

The infrastructure quietly redirects requests to healthy instances.

Once the failed server recovers, it can safely rejoin the system.

Health checks form the foundation of automatic recovery in modern cloud platforms and container orchestration systems such as Kubernetes.

Automatic Recovery and Self-Healing Systems

Detecting failures is only the first step.

The next challenge is recovering automatically.

Modern distributed systems increasingly rely on self-healing infrastructure.

Suppose an application is configured to always run four instances.

If one instance crashes unexpectedly, the orchestration platform immediately notices that only three instances remain.

Instead of waiting for a human operator, it automatically launches a replacement.

This idea may seem simple, but it fundamentally changes system operations.

Applications no longer depend on engineers manually restarting failed servers.

The infrastructure continuously restores the desired state on its own.

Cloud-native platforms such as Kubernetes have made this style of self-healing architecture a standard practice.

Reliability Requires Layers of Protection

One of the most important lessons in system design is that no single technique makes a system reliable.

Instead, reliability emerges from multiple layers working together.

Consider an online banking platform.

Multiple application servers provide redundancy.

Database replication protects customer information.

Health checks identify failed services.

Load balancers redirect traffic.

Automatic failover promotes replicas when necessary.

Backups protect against catastrophic data loss.

Monitoring systems alert engineers when unusual behaviour occurs.

Each mechanism addresses a different type of failure.

Together, they create a system capable of surviving situations that would completely disable a simpler architecture.

This layered approach is often referred to as defence in depth.

Rather than relying on one perfect solution, distributed systems combine multiple protective mechanisms so that if one layer fails, another continues providing protection.

Reliability in the Real World

Companies operating internet-scale platforms rarely ask whether failures will occur.

Instead, they invest significant effort into ensuring that failures remain localised.

Streaming platforms continue serving videos even when recommendation services experience issues.

Cloud providers replicate customer data across multiple availability zones so that hardware failures do not result in permanent data loss.

Large e-commerce companies deploy services across multiple data centres, allowing customer traffic to continue flowing even if an entire region experiences an outage.

In each case, the objective is not perfection.

The objective is resilience.

Reliable systems acknowledge that failures are unavoidable and design architectures capable of recovering from them quickly and automatically.

Bringing Everything Together

By now, you may have noticed that the three concepts discussed in this article build upon one another.

Redundancy ensures that critical infrastructure never depends on a single component.

Replication ensures that valuable data exists in multiple locations instead of one.

Fault tolerance combines these ideas with automatic detection, failover, recovery, and graceful degradation to ensure that the overall system continues functioning despite failures.

The relationship between these concepts can be visualized as follows.

Although each concept provides value individually, modern distributed systems almost always combine all three.

Removing any one of them weakens the overall architecture.

Final Thoughts

As applications grow from small monoliths into globally distributed platforms, reliability becomes one of the defining characteristics of good system design.

Users rarely notice a system that works flawlessly.

They immediately notice one that fails.

Designing reliable software therefore means assuming that servers will crash, networks will become unreliable, databases will fail, and unexpected situations will occur.

Instead of treating these events as disasters, modern distributed systems treat them as routine operating conditions.

Redundancy ensures that critical services always have backups.

Replication protects the information those services manage.

Fault tolerance allows the entire application to continue functioning while failures are detected, isolated, and automatically recovered.

Perhaps the most important lesson from this article is that reliability is not a feature that can be added later.

It is an architectural principle that influences every design decision from the very beginning.

The most reliable systems are not the ones that never fail.

They are the ones that are designed to recover so quickly and gracefully that most users never realise a failure occurred.

Consistency Models in Distributed Systems: Strong vs Eventual Consistency

Sushant Gaurav — Tue, 25 Aug 2026 05:41:00 +0000

One of the biggest promises of distributed systems is that they allow applications to scale beyond the limits of a single machine. By distributing data across multiple servers, applications become capable of handling millions of users, surviving hardware failures, and serving requests from different parts of the world.

At first glance, this sounds like the perfect solution.

If one database server is no longer enough, simply add another.

If one data centre becomes overloaded, deploy another in a different region.

If a server fails, allow another replica to take over.

Everything appears straightforward.

However, the moment multiple copies of the same data begin existing on different machines, distributed systems encounter one of their most fundamental challenges.

How do we ensure that every copy of the data remains correct?

This question sounds deceptively simple.

In reality, it is one of the hardest problems in computer science.

A Single Database Never Faces This Problem

Imagine an application that stores all of its information inside a single database server.

A customer updates their delivery address.

The application writes the new address into the database.

Every future request reads that exact value.

There is no confusion.

There is only one copy of the data, so every user sees the same information.

Life is simple.

This simplicity is one of the reasons monolithic applications are relatively easy to reason about.

There is only one source of truth.

As long as that database is healthy, every request returns identical information.

Now imagine the application becomes enormously successful.

Millions of users begin accessing it every day.

The single database starts struggling under the load.

To improve availability and scalability, engineers decide to introduce database replication.

Instead of maintaining one database server, they now maintain several identical copies.

At first glance, nothing appears to have changed.

The application still stores the same data.

Users still perform the same operations.

But internally, the architecture has become dramatically more complicated.

There is no longer one copy of the data.

There are several.

And those copies must somehow remain synchronised.

When Copies Disagree

Suppose a customer changes their password.

The application successfully writes the new password to the primary database.

Immediately afterwards, the updated information begins replicating to the remaining database replicas.

But replication is not instantaneous.

It takes time.

Perhaps only a few milliseconds.

Sometimes hundreds of milliseconds.

Occasionally even longer.

Now imagine another request arrives before every replica has received the update.

One server contains the new password.

Another still contains the old one.

Which version should the application return?

Consider another example.

A customer transfers money between two bank accounts.

The transaction completes successfully.

One database replica reflects the new account balance.

Another replica has not yet received the update.

If the customer immediately refreshes their banking application, should they see the old balance or the new one?

Neither answer feels satisfactory.

Returning outdated information can confuse users.

Waiting indefinitely until every server agrees can make the system painfully slow.

This is the central dilemma of consistency in distributed systems.

Understanding Consistency

When engineers talk about consistency, they are not referring to data correctness in the traditional sense.

Instead, they are asking a very specific question:

If one client writes new data, when should every other client be able to observe that change?

Notice what this definition focuses on.

It is not asking whether the data is valid.

It is asking when different parts of the system should agree on the latest value.

That single word—when—is incredibly important.

Some systems require agreement immediately.

Others can tolerate small delays.

The acceptable answer depends entirely on the application.

Why Perfect Consistency Is Surprisingly Difficult

At first, the solution seems obvious.

Whenever data changes, simply update every server before responding to the user.

Problem solved.

Unfortunately, distributed systems operate over networks.

Networks introduce latency.

Messages can be delayed.

Servers can temporarily become unavailable.

Entire data centres can lose connectivity.

Imagine an application with database replicas in New York, London, Singapore, and Sydney.

Every update must now travel thousands of miles across the internet.

Even travelling at nearly the speed of light, information still requires time to move between continents.

Network congestion, routing changes, and temporary failures increase that delay even further.

Now consider a customer in Singapore updating their profile picture.

Should a user in London immediately see the new image?

What if the network cable connecting Europe and Asia is temporarily unavailable?

Should the application wait?

Should it reject all requests?

Or should it temporarily allow users to see slightly outdated information?

These are not theoretical questions.

Large internet companies make decisions like these every day.

The Relationship Between Replication and Consistency

In our previous article about Data Partitioning and Sharding, we discussed how systems distribute data across multiple machines to improve scalability.

In another article, we explored Replication, where multiple copies of the same data improve availability and fault tolerance.

Replication solves one problem while creating another.

The moment multiple copies of the same information exist, the system must decide how quickly those copies should become identical.

The faster replicas synchronise, the more consistent the system becomes.

The longer synchronisation takes, the greater the possibility that different users temporarily observe different versions of the same information.

Replication and consistency are therefore deeply connected.

One cannot exist without influencing the other.

A Real-World Example Everyone Has Experienced

Most people have experienced eventual consistency without realising it.

Suppose you change your profile picture on a social media platform.

Immediately afterwards, you refresh your own profile and see the new image.

A friend opens your profile from another country a few seconds later.

Surprisingly, they still see the old picture.

A short time afterwards, they refresh again.

Now the new picture appears.

Was the system broken?

Not at all.

The update simply required time to propagate across multiple servers distributed around the world.

During those few seconds, different users observed different versions of the same data.

This behaviour is completely normal in many distributed systems.

The important question is whether that temporary inconsistency is acceptable.

For a profile picture, probably yes.

For a bank balance, almost certainly not.

And that distinction leads us to the two primary consistency models used in distributed systems:

Strong Consistency and Eventual Consistency.

Although both models attempt to solve the same problem, they make very different trade-offs between correctness, availability, latency, and user experience.

Understanding Strong Consistency

Imagine you withdraw money from an ATM.

Your account balance is $2,500.

You withdraw $500.

The transaction completes successfully.

A few seconds later, you check your balance using your banking application's mobile app.

How much money should it display?

There is really only one acceptable answer.

It should display $2,000.

Seeing the old balance—even for a few seconds—would immediately reduce your trust in the banking system.

You would naturally wonder:

"Did the withdrawal actually happen?"

This is precisely the guarantee that strong consistency provides.

Once a write operation has been acknowledged as successful, every subsequent read must return that latest value, regardless of which server processes the request.

There is only one version of the truth.

Every client sees it immediately.

From the user's perspective, it feels almost as though there is only a single database, even though the data may actually be replicated across multiple machines.

How Strong Consistency Is Achieved

Achieving strong consistency sounds simple in theory.

Whenever data changes, ensure that every replica receives the update before confirming success to the client.

However, implementing this guarantee in a distributed environment is considerably more challenging.

Consider a distributed database with one primary node and three replicas.

Notice the order of events.

The client does not immediately receive a success response after writing to the primary database.

Instead, the primary waits until the replicas acknowledge the update.

Only after sufficient replicas confirm the change does the system inform the client that the operation has completed successfully.

This waiting period is what allows every future read to return the same value.

The consistency guarantee comes from coordination.

And coordination always has a cost.

Why Strong Consistency Increases Latency

One of the recurring themes throughout this System Design series has been that every architectural decision involves trade-offs.

Strong consistency is no exception.

Imagine a database replicated across three continents.

A customer in India updates their shipping address.

The primary database accepts the request.

Before confirming success, however, it must replicate that update to servers located in Europe and North America.

Although data travels incredibly fast across fibre-optic networks, it is still constrained by the laws of physics.

Information cannot travel instantaneously.

Every additional replica introduces communication delays.

Every acknowledgement requires another network round trip.

The client remains waiting until the coordination process finishes.

As a result, the response time increases.

In other words, stronger consistency often comes at the cost of higher latency.

This is one of the most important trade-offs in distributed systems.

Applications demanding immediate agreement frequently sacrifice speed to achieve correctness.

The Cost of Waiting

Now imagine that one of the replicas becomes temporarily unreachable.

Perhaps a network cable has been damaged.

Perhaps a data centre is experiencing maintenance.

Perhaps temporary congestion has increased network delays.

The primary database now faces a difficult decision.

Should it continue waiting?

Should it reject the write request?

Or should it proceed without updating every replica?

If the system chooses to wait, users experience delays.

If it chooses to reject requests, availability decreases.

If it proceeds anyway, consistency is no longer guaranteed.

Notice how this dilemma resembles the trade-offs we discussed in the article on CAP Theorem.

Maintaining immediate consistency during failures often requires sacrificing availability.

This is one reason why strong consistency is relatively expensive to maintain in globally distributed systems.

Strong Consistency in the Real World

Despite these challenges, many applications simply cannot tolerate inconsistent data.

Imagine purchasing the last available ticket for an international flight.

If two customers simultaneously purchase what appears to be the final seat because different replicas disagree, the airline now has a serious operational problem.

The same issue appears in hotel reservations.

Suppose only one room remains available.

Two booking requests arrive at different replicas before synchronisation completes.

Without strong consistency, both customers may successfully reserve the same room.

Financial systems provide another obvious example.

If your account balance differs depending on which database replica processes your request, users quickly lose confidence in the system.

Healthcare systems also depend heavily on strong consistency.

Imagine two doctors viewing different versions of a patient's allergy information because replicas have not yet synchronised.

The consequences could be severe.

Inventory management systems face similar challenges.

When only one product remains in stock, every customer must observe the same inventory count.

Otherwise, multiple customers may purchase an item that no longer exists.

In each of these scenarios, correctness is significantly more important than minimising response time.

Users are generally willing to wait an additional few hundred milliseconds if it guarantees that the information they receive is accurate.

Consensus: Getting Multiple Servers to Agree

Strong consistency introduces another fascinating challenge.

How do multiple independent servers actually agree on the latest value?

Suppose four replicas receive slightly different information because of temporary network delays.

Which version becomes the correct one?

Distributed databases solve this using consensus algorithms.

Although the underlying algorithms are mathematically sophisticated, the central idea is surprisingly intuitive.

Before accepting certain operations, multiple servers participate in a coordinated decision-making process.

Rather than allowing individual replicas to decide independently, the cluster reaches agreement collectively.

This process ensures that every server eventually shares the same understanding of the system's state.

Several well-known distributed databases rely on consensus algorithms such as Raft or Paxos to provide strong consistency guarantees.

While these algorithms are beyond the scope of this article, it is important to recognise that strong consistency is not achieved through replication alone.

It requires coordination, agreement, and carefully designed protocols capable of handling failures.

Strong Consistency Is About Trust

Perhaps the easiest way to understand strong consistency is to think about trust.

Whenever users interact with a banking application, stock trading platform, payment gateway, airline reservation system, or healthcare database, they expect the system to provide one definitive answer.

There should never be uncertainty regarding the latest account balance, available seat, completed payment, or medical record.

Strong consistency provides that confidence.

Every successful write becomes immediately visible to every subsequent read.

The system behaves as though there were only a single, perfectly synchronised source of truth.

Achieving this guarantee is technically demanding.

It increases latency.

It requires coordination.

It becomes more difficult during network failures.

Yet for many applications, those costs are entirely justified because the business value of correctness far outweighs the additional complexity.

However, not every application requires this level of precision.

Many systems prioritise responsiveness, scalability, and availability over immediate agreement.

Instead of forcing every server to synchronise instantly, they allow updates to propagate gradually throughout the system.

This alternative approach is known as Eventual Consistency, and it has become one of the defining characteristics of many modern internet-scale applications.

Understanding Eventual Consistency

Imagine updating your profile picture on a social media platform.

You upload a new image.

Within a second, you refresh your own profile and see the updated picture.

A friend living on another continent opens your profile almost immediately afterwards.

Surprisingly, they still see the old picture.

A few moments later, they refresh again.

Now the new image appears.

Nothing was broken.

The system simply required time to distribute the update across its globally distributed infrastructure.

This is the essence of eventual consistency.

Unlike strong consistency, eventual consistency does not require every replica to agree immediately.

Instead, it guarantees something slightly different.

If no new updates occur, all replicas will eventually converge to the same value.

The word eventually is doing all the work here.

The system accepts that different users may temporarily observe different versions of the same data.

However, given enough time, every replica will synchronise and reach the same final state.

Why Eventual Consistency Exists

At first, allowing inconsistent data sounds like a terrible idea.

After all, haven't we spent the entire article discussing why consistency matters?

The answer depends entirely on the type of data involved.

Consider the homepage of a video streaming platform.

Suppose a movie's thumbnail changes.

If some users continue seeing the previous thumbnail for five seconds while others see the updated version immediately, has the application failed?

Probably not.

Now imagine an online news website.

An article receives a new featured image.

Readers in Europe see the change immediately.

Readers in Asia see the previous image for another ten seconds.

Again, the user experience remains almost unaffected.

The cost of delaying every request until every server agrees would likely outweigh the temporary inconsistency.

In situations like these, eventual consistency becomes an extremely practical trade-off.

Rather than waiting for worldwide synchronisation, applications prioritise speed and availability.

Users receive responses almost instantly, while the system quietly propagates updates in the background.

How Eventual Consistency Works

Let's revisit the replicated database architecture from the previous section.

This time, however, the write process follows a different sequence.

Instead of waiting for every replica to acknowledge the update, the primary database accepts the write and immediately responds to the client.

Replication happens afterwards.

Notice the difference from strong consistency.

The user receives confirmation almost immediately.

The application does not pause while waiting for every replica to synchronise.

Instead, replication continues in the background.

This significantly improves response time.

However, it also creates a short period during which different replicas may contain different versions of the data.

That temporary disagreement is an intentional design decision.

Living With Temporary Inconsistency

To understand eventual consistency, it helps to stop thinking of inconsistency as an error.

Instead, think of it as a transition period.

Suppose an e-commerce company updates the description of a product.

Some users immediately receive the updated description.

Others continue seeing the previous version for a short time.

Eventually, every replica contains the new information.

The inconsistency existed only while the update was propagating.

The same idea appears in many cloud services.

When a DNS record changes, internet providers around the world do not update simultaneously.

Some DNS servers refresh their records immediately.

Others continue serving cached information until their cache expires.

Eventually, every DNS server reflects the new record.

The internet itself relies heavily on eventual consistency.

Eventual Consistency in Everyday Applications

One of the reasons eventual consistency is so widely adopted is that users rarely notice it.

Consider a messaging application.

You send a message to a group chat.

Some participants receive it almost instantly.

Others experience a slight delay because of network conditions.

Despite these temporary differences, everyone eventually receives the same message.

Similarly, when you like a social media post, the like count may not increase simultaneously for every user viewing that page.

One person may see 1,024 likes while another briefly sees 1,023 likes.

A few seconds later, both users observe the same value.

Streaming platforms exhibit similar behaviour.

Suppose a creator uploads a new video.

The video may appear immediately in one region while taking a little longer to become available elsewhere.

This delay is often caused by content propagation across geographically distributed servers.

Again, eventual consistency allows the platform to scale globally without forcing every server to synchronise before serving users.

These examples illustrate an important principle.

Not every piece of information requires perfect synchronisation.

For many user experiences, slight delays are practically invisible.

Why Eventual Consistency Improves Availability

Another significant advantage of eventual consistency appears during failures.

Imagine that one replica temporarily loses network connectivity.

In a strongly consistent system, writes may need to pause until coordination is restored.

An eventually consistent system often behaves differently.

The primary database continues accepting updates.

Healthy replicas continue receiving changes.

The unavailable replica simply catches up later after connectivity returns.

This ability to continue operating despite temporary failures makes eventual consistency particularly attractive for globally distributed applications.

Rather than sacrificing availability whenever communication becomes difficult, the system tolerates temporary divergence and resolves it later.

This design aligns closely with the trade-offs discussed in our article on the CAP Theorem, where distributed systems frequently prioritise availability over immediate consistency during network partitions.

Eventual Does Not Mean Random

One misconception beginners often have is that eventual consistency means replicas simply update whenever they feel like it.

That is not how distributed systems work.

Modern databases use carefully designed replication mechanisms, version tracking, timestamps, conflict resolution strategies, and synchronisation protocols to ensure replicas eventually converge toward the same state.

The delay may vary depending on network conditions, infrastructure, or workload, but the synchronisation process is deliberate and carefully managed.

Without these mechanisms, replicas could permanently diverge, creating conflicting versions of the same data.

Eventual consistency accepts temporary disagreement.

It does not accept permanent disagreement.

That distinction is crucial.

Eventual Consistency Is About Practicality

Perhaps the easiest way to understand eventual consistency is to think about priorities.

Instead of asking,

"Can every server agree immediately?"

the system asks,

"Is immediate agreement actually necessary?"

For profile pictures, product descriptions, recommendation lists, analytics dashboards, news feeds, and countless other types of information, the answer is often no.

Users generally prefer receiving information immediately—even if it is a few seconds behind—rather than waiting longer for perfect synchronisation.

This practical mindset is one of the reasons eventual consistency powers so many of today's largest distributed platforms.

It acknowledges an important reality of globally distributed systems:

Sometimes delivering information quickly is more valuable than delivering perfectly synchronised information.

A Side-by-Side Comparison

One of the easiest ways to understand these consistency models is to compare how they behave when the same event occurs.

Imagine a user updates their profile information.

Under Strong Consistency

The system waits until the update has been synchronised according to its consistency requirements before acknowledging success.

Every subsequent request immediately observes the updated information.

There is never any ambiguity regarding which version is correct.

Users receive one consistent answer regardless of which server processes the request.

Under Eventual Consistency

The system acknowledges the update almost immediately.

Replication continues in the background.

For a short period, different users may observe different versions of the data depending on which replica serves their request.

Eventually, every replica converges to the same state.

The difference is not whether synchronisation happens.

The difference is when synchronisation becomes visible to users.

Visualising the Difference

The following diagram illustrates the behaviour of both consistency models after a write operation.

Notice that both systems ultimately reach the same final state.

The distinction lies in whether the client waits for synchronisation or whether synchronisation continues after the response has already been returned.

Choosing the Right Model Depends on the Business Problem

A common mistake among beginners is assuming that every application should strive for strong consistency because it sounds safer.

In reality, insisting on strong consistency everywhere often leads to unnecessary complexity and reduced performance.

Imagine an online retail platform.

When a customer updates their shipping address, seeing the previous address for two or three seconds is unlikely to create significant problems.

However, when that same customer submits a payment, displaying an outdated account balance would be completely unacceptable.

The application therefore treats different pieces of information differently.

Profile information may tolerate eventual consistency.

Financial transactions generally require strong consistency.

The same application may therefore use multiple consistency models simultaneously.

This idea surprises many engineers when they first encounter distributed systems.

Consistency is rarely an application-wide decision.

It is usually made at the level of individual business operations.

Real-World Examples

Large technology companies rarely commit themselves exclusively to one consistency model.

Instead, they carefully choose the appropriate model for each type of data.

Consider a typical social media platform.

When you upload a new profile picture, the update propagates across content delivery networks and replicated databases around the world.

Some users may briefly continue seeing the previous image.

This temporary inconsistency has almost no business impact.

The platform therefore favours eventual consistency because it delivers excellent scalability and responsiveness.

Now consider the payment system used by that same company.

Suppose a user purchases a premium subscription.

The payment must only be processed once.

The account balance must remain accurate.

Subscription status must not fluctuate between different servers.

In this scenario, strong consistency becomes essential.

The same organisation therefore employs both models—simply for different purposes.

Can a Single Application Use Both?

Absolutely.

In fact, this is exactly how many modern distributed systems are designed.

Consider an e-commerce platform.

Notice how different business capabilities have different requirements.

The payment service cannot afford inconsistent account balances.

The inventory service must accurately determine whether products remain available.

On the other hand, recommendation systems, analytics dashboards, search indexes, and notification services can safely process information asynchronously.

Attempting to force every component to use strong consistency would increase latency and reduce scalability without providing meaningful business value.

Conversely, making payment processing eventually consistent could result in duplicate transactions or oversold inventory.

Good system design therefore involves identifying which operations genuinely require immediate correctness and which can tolerate temporary inconsistency.

Strong Consistency vs Eventual Consistency

The following table summarises the primary differences between the two models.

Characteristic	Strong Consistency	Eventual Consistency
Data visibility	Latest data is always returned	Temporary stale reads are possible
Read behavior	Every client observes the same value	Different replicas may briefly return different values
Latency	Typically higher because of coordination	Usually lower because responses are returned immediately
Availability during failures	May decrease while replicas coordinate	Generally higher because updates continue despite temporary failures
Scalability	More difficult at global scale	Easier to scale across regions
Typical applications	Banking, payments, reservations, inventory	Social media, analytics, recommendations, DNS, caching

Rather than viewing these differences as advantages or disadvantages, think of them as design trade-offs.

Every distributed system decides where it wants to position itself along this spectrum.

Consistency Is a Business Decision

One of the most important lessons for system designers is that consistency is not merely a technical concern.

It is ultimately a business decision.

Technology provides the available options.

Business requirements determine which option is appropriate.

A stock trading platform cannot allow customers to observe outdated portfolio values.

An airline cannot sell the same seat to multiple passengers.

A hospital cannot display different versions of a patient's medical history.

These applications willingly accept additional complexity to guarantee correctness.

Conversely, a social media platform benefits far more from responsiveness and availability than from forcing every profile update to synchronise globally before users can continue browsing.

The acceptable level of inconsistency depends entirely on the consequences of stale data.

Understanding those consequences is often more important than understanding the underlying algorithms.

Final Thoughts

Throughout this System Design series, one idea has consistently appeared across every topic we have explored.

No architecture is universally optimal.

Monolithic systems trade flexibility for simplicity.

Distributed systems trade simplicity for scalability.

Horizontal scaling trades hardware limitations for operational complexity.

Microservices trade organisational independence for increased communication overhead.

Consistency models follow the same pattern.

Strong consistency prioritises correctness, even if that means accepting additional latency and coordination.

Eventual consistency prioritises scalability, responsiveness, and availability, while accepting that replicas may temporarily disagree.

Neither philosophy is inherently better.

They simply optimise for different priorities.

As you continue studying distributed systems, you will notice that nearly every architectural decision revolves around balancing competing trade-offs rather than maximising a single metric.

The best engineers are not those who always choose the most advanced technology.

They are the ones who understand the strengths and limitations of every approach and select the one that best serves the problem they are trying to solve.

That, ultimately, is the essence of system design.

Communication Between Services: REST, gRPC, and Message Queues

Sushant Gaurav — Tue, 11 Aug 2026 05:41:00 +0000

In the previous article, we explored how large applications gradually evolve from monoliths into collections of smaller, independently deployable services. Instead of one massive application responsible for authentication, payments, inventory, notifications, and analytics, we now have multiple services, each responsible for a single business capability.

At first glance, this seems like a clean and elegant solution. Every service owns its own logic, its own database, and can be deployed independently without affecting the rest of the system.

But splitting an application into multiple services immediately creates a new challenge.

The components that once lived inside the same process now live on different machines.

Inside a monolithic application, communication is almost effortless. If the payment module needs information about a user, it simply calls a function from the user module. The call happens inside the same process, uses the same memory space, and usually completes in a fraction of a millisecond. Developers rarely stop to think about this communication because, from their perspective, it is almost free.

Microservices change this assumption entirely.

The payment service no longer has direct access to the user service. They may be running on different servers, in different containers, or even in different regions of the world. Every interaction between them must now travel through a network.

What was once a simple function call has become a network request.

This seemingly small architectural change has enormous consequences.

Network requests introduce latency. They can fail unexpectedly. They may experience congestion, timeouts, packet loss, or temporary service outages. Unlike local function calls, network communication is inherently unreliable.

This is why experienced engineers often say that moving to microservices means moving into the world of distributed systems.

The application is no longer just executing code.

It is coordinating communication between independent systems.

Why Communication Becomes a System Design Problem

Imagine a customer placing an order on an e-commerce platform.

From the customer's perspective, the process appears straightforward. They click the "Place Order" button, wait a few seconds, and receive a confirmation message.

Behind the scenes, however, the request may travel through several independent services before the order is successfully processed.

The order service must verify the customer.

The inventory service must confirm that the requested items are available.

The payment service must process the transaction.

The notification service must send a confirmation email or SMS.

Each of these services performs its own responsibility, but together they must behave as if they were a single application.

Notice something interesting about this interaction.

The Order Service is not solving business problems alone.

It spends a significant amount of time communicating with other services.

As systems grow larger, communication itself becomes one of the biggest consumers of time and resources.

This is one of the reasons distributed systems are fundamentally different from traditional applications.

The challenge is no longer writing business logic.

The challenge is coordinating independent systems efficiently and reliably.

Not Every Conversation Between Services Is the Same

Suppose the Order Service needs to verify whether a product exists before allowing a purchase.

The answer is needed immediately.

The customer is waiting.

The Order Service cannot continue until it receives a response.

Now consider a different situation.

A payment has been completed successfully, and the customer should receive a confirmation email.

Does the customer really need to wait until the email has been delivered?

Probably not.

The order should complete successfully even if the email arrives a few seconds later.

Now imagine another scenario.

A recommendation engine wants to learn which products users purchase so that it can improve future recommendations.

Should the payment process slow down while waiting for the recommendation engine to process analytics?

Again, the answer is no.

These examples illustrate an important principle in distributed systems:

Not every interaction between services has the same urgency.

Some requests require an immediate response.

Others can happen later.

Some require a direct conversation.

Others simply need to notify another service that an event has occurred.

Understanding these differences is the key to choosing the right communication mechanism.

Synchronous vs Asynchronous Communication

Before discussing REST, gRPC, or messaging systems, we need to understand two fundamental communication models that appear throughout distributed systems.

The first is synchronous communication.

In synchronous communication, one service sends a request and waits for the response before continuing.

It is very similar to a phone call.

When you call someone, the conversation happens in real time. You ask a question and wait for the answer before continuing the discussion.

Most web applications work this way.

When your browser requests a webpage, it waits until the server responds.

Likewise, when one microservice calls another using REST or gRPC, it often waits for the response before moving forward.

This approach is simple to understand because it closely resembles traditional programming.

However, it also creates dependencies.

If Service B becomes slow, Service A also becomes slow.

If Service B becomes unavailable, Service A may fail entirely.

In other words, synchronous communication couples the availability and performance of multiple services.

Asynchronous communication takes a very different approach.

Instead of waiting for an immediate response, one service simply publishes a message and continues its own work.

Receiving services process that message whenever they are ready.

A useful analogy is email.

When you send an email, you do not wait with your computer open until the recipient replies.

You send the message and continue with your day.

The recipient responds later.

Distributed systems often behave in exactly the same way.

This model dramatically reduces coupling between services.

The Order Service no longer needs to know whether the Notification Service is currently running.

Its responsibility ends once the message has been successfully placed into the queue.

The notification can be processed seconds—or even minutes—later without affecting the customer experience.

As systems become larger, this asynchronous style of communication becomes increasingly common because it improves resilience and allows services to operate independently.

Three Ways Services Usually Communicate

Almost every distributed system relies on one of three primary communication mechanisms.

The first is REST, which has become the standard way of building web APIs over the last two decades. It is simple, human-readable, and supported by virtually every programming language and framework.

The second is gRPC, a high-performance communication framework designed for fast and efficient communication between services. Instead of exchanging human-readable JSON, gRPC uses a compact binary protocol, making it significantly faster in many internal service-to-service scenarios.

The third approach uses message queues, where services communicate indirectly by publishing events instead of sending direct requests. Rather than waiting for immediate responses, services exchange information through brokers such as queues or event streams, enabling asynchronous communication and improving system resilience.

Although all three approaches allow services to communicate, they solve different problems.

Choosing between them is rarely about which technology is better.

It is about understanding the nature of the conversation taking place between services.

Should the sender wait for a response?

Does the receiver need to respond immediately?

Can the work happen later?

Should failures affect the user?

The answers to these questions determine the most appropriate communication model.

And this is where our discussion truly begins.

REST: The Language of the Web

Long before microservices became popular, applications already needed a way to communicate over networks.

Web browsers needed to request web pages.

Mobile applications needed to fetch user profiles.

Third-party developers needed access to payment systems, maps, weather data, and social media platforms.

The internet itself needed a common language that every application, regardless of the programming language it was written in, could understand.

This is where REST entered the picture.

REST, which stands for Representational State Transfer, is not a protocol or a programming language. It is an architectural style proposed by Roy Fielding in his doctoral dissertation in the year 2000. Rather than introducing a completely new communication protocol, REST embraced technologies that were already powering the web—primarily HTTP.

This decision played a significant role in REST's widespread adoption.

Instead of asking developers to learn an entirely new ecosystem, REST leveraged concepts they were already familiar with. URLs identified resources, HTTP methods described operations, and responses were returned using formats such as JSON or XML.

Over time, REST became the de facto standard for building APIs, and today, it powers a significant portion of the modern internet.

Whether you're checking your bank balance, booking a flight, ordering food, or scrolling through social media, there's a good chance your device is communicating with backend services through REST APIs.

Thinking in Resources Instead of Functions

One of the biggest mindset shifts when learning REST is understanding that it is resource-oriented, not function-oriented.

In traditional programming, we tend to think in terms of actions.

We write functions like:

createUser()
getUser()
deleteUser()
updateUser()

REST encourages us to think differently.

Instead of focusing on actions, we focus on resources.

A user is a resource.

An order is a resource.

A product is a resource.

A payment is a resource.

Once a resource exists, HTTP methods describe what we want to do with it.

For example:

GET    /users/125
POST   /users
PUT    /users/125
DELETE /users/125

Notice how the URL represents what we are working with, while the HTTP method represents what operation we want to perform.

This separation makes APIs easier to understand because they resemble the structure of the real-world entities they represent.

How a REST Request Travels Through the System

Let's consider a simple example.

A customer opens an e-commerce application and wants to view the details of a product.

The frontend application sends an HTTP request to the Product Service.

The Product Service processes the request, retrieves the necessary information from its database, converts the result into JSON, and sends it back to the client.

The interaction looks something like this:

From the user's perspective, this entire interaction feels almost instantaneous.

Behind the scenes, however, several things have happened.

The request travelled through the internet.

The server authenticated the client.

Business logic was executed.

A database query was performed.

The response was serialised into JSON.

Finally, the data was transmitted back over the network.

Every REST request follows this general request-response lifecycle.

Why REST Became So Popular

One of REST's greatest strengths is its simplicity.

Almost every programming language today can send an HTTP request.

Browsers understand HTTP natively.

Firewalls are designed to work with HTTP traffic.

Cloud platforms, API gateways, reverse proxies, load balancers, and CDNs all understand HTTP exceptionally well.

Because of this universal support, REST APIs can be consumed by virtually anything.

A web browser.

A mobile application.

A desktop application.

Another backend service.

Even command-line tools like curl.

This universality is one of the reasons REST became the default communication mechanism for modern applications.

It reduced friction.

Developers no longer needed specialised libraries or proprietary communication protocols.

If two systems could speak HTTP, they could usually communicate with one another.

REST Is Stateless - And That Is a Good Thing

One of the defining characteristics of REST is that it is stateless.

At first, the word sounds technical, but the idea is remarkably simple.

Every request should contain all the information required to process it.

The server should not rely on memory from previous requests.

Imagine asking someone for directions.

If every time you ask, you provide your current location and destination, they can answer immediately.

They don't need to remember your previous conversations.

REST works in the same way.

Suppose a client sends the following request:

GET /orders/1254
Authorization: Bearer <token>

The request already contains everything the server needs.

It specifies:

Which resource is being requested.
Who is making the request.
Authentication credentials.
Any additional headers or parameters.

The server processes the request, sends the response, and then forgets everything about that interaction.

The next request starts from scratch.

This stateless design provides significant scalability benefits.

Since servers do not need to remember client sessions, incoming requests can be distributed across multiple application instances using a load balancer.

Any server can process any request because every request is self-contained.

This aligns perfectly with the horizontal scaling principles we discussed earlier in this series.

JSON: The Universal Language of REST

Although REST itself does not require JSON, the two have become almost inseparable.

JSON is lightweight, human-readable, and supported by virtually every modern programming language.

A typical REST response might look like this:

{
  "id": 105,
  "name": "Mechanical Keyboard",
  "price": 89.99,
  "stock": 42
}

One reason developers appreciate JSON is that it is easy to inspect.

If something goes wrong, developers can open browser developer tools, examine the response, and immediately understand what the server returned.

This readability makes debugging significantly easier than many binary communication formats.

However, this convenience comes with a trade-off.

Text-based data is generally larger than binary data.

A JSON document contains field names, punctuation, quotation marks, and whitespace, all of which increase the size of the response.

For applications serving millions of requests every minute, these additional bytes become significant.

This limitation eventually motivated the development of faster communication mechanisms such as gRPC, which we'll explore in the next part.

Where REST Truly Shines

REST is exceptionally well suited for communication between clients and servers.

When a web browser requests product information, when a mobile application fetches a user's profile, or when an external partner integrates with your platform, REST is often an excellent choice.

Its simplicity, interoperability, and widespread tooling make it ideal for public-facing APIs.

It is also highly cache-friendly.

Since REST is built on HTTP, it naturally benefits from HTTP caching, proxy servers, reverse proxies, and Content Delivery Networks (CDNs). Responses that do not change frequently can often be cached closer to users, reducing latency and improving performance.

This is one of the reasons many public APIs continue to rely on REST despite the emergence of newer communication technologies.

Its ecosystem is mature, battle-tested, and universally understood.

REST Is Not Perfect

Despite its popularity, REST is not the answer to every communication problem.

Because REST relies on HTTP and typically exchanges JSON, each request carries a certain amount of overhead.

Headers must be transmitted.

JSON must be serialised by the sender.

JSON must be parsed by the receiver.

Every interaction involves opening, processing, and completing an HTTP request-response cycle.

For occasional communication, this overhead is almost negligible.

But imagine hundreds of microservices communicating thousands of times every second.

The cumulative cost becomes noticeable.

Applications requiring extremely low latency or high throughput often begin looking for more efficient alternatives.

Another limitation is that REST is fundamentally request-driven.

One service asks another for information and waits until the response arrives.

As we discussed in the previous part, this synchronous style of communication creates dependencies between services.

If one service becomes slow, the calling service also slows down.

If one service becomes unavailable, requests may begin failing throughout the system.

These limitations do not make REST a poor choice.

They simply highlight that every communication model involves trade-offs.

And understanding those trade-offs is precisely what system design is about.

gRPC: Built for High-Performance Service-to-Service Communication

REST revolutionised the way applications communicate over the web, and even today it remains one of the most widely adopted architectural styles for building APIs. It is simple, readable, and supported by virtually every programming language and framework.

However, as organisations embraced microservices, engineers began noticing a different kind of challenge.

The majority of service-to-service communication was no longer happening between browsers and backend servers.

Instead, it was happening between backend services themselves.

A single user request could trigger communication between dozens of internal services.

An Order Service might call the Inventory Service, which in turn communicates with the Pricing Service, the Recommendation Service, the Shipping Service, and the Notification Service before the original request is completed.

Each REST call may take only a few milliseconds, but when hundreds of these calls occur during the processing of a single request—and millions more occur every day—the overhead begins to accumulate.

The issue wasn't that REST was slow.

The issue was that REST was designed primarily for interoperability and simplicity rather than maximum performance.

This is where gRPC enters the picture.

Developed by Google, gRPC is a high-performance Remote Procedure Call (RPC) framework specifically designed for efficient communication between services.

Unlike REST, which encourages thinking in terms of resources and HTTP operations, gRPC focuses on calling methods on remote services almost as if they were local functions.

This makes service-to-service communication feel much closer to traditional programming while retaining the advantages of distributed systems.

Understanding Remote Procedure Calls

Before understanding gRPC, it helps to understand what a Remote Procedure Call (RPC) actually means.

Imagine writing the following function in your application:

calculateShipping(orderId)

Inside a monolithic application, calling this function is straightforward.

The code executes within the same process, accesses local memory, and immediately returns a result.

Now imagine that the shipping logic has been extracted into its own microservice.

The function still appears to exist, but in reality it now executes on an entirely different machine.

Instead of calling local code, the application must:

Serialise the request.
Send it across the network.
Wait for the remote server to execute the operation.
Receive the response.
Deserialise the result.

From the developer's perspective, however, the interaction still resembles a normal function call.

This is the philosophy behind RPC systems.

Rather than thinking in terms of URLs and HTTP resources, developers think in terms of methods and services.

How gRPC Communicates

One of the biggest differences between REST and gRPC lies in how data is transmitted.

REST typically exchanges data using JSON.

JSON is human-readable, easy to debug, and universally supported.

However, it is also relatively verbose.

Consider a simple JSON response:

{
  "productId": 101,
  "name": "Mechanical Keyboard",
  "price": 89.99
}

Every field name is transmitted over the network.

Quotation marks, commas, braces, and whitespace all contribute to the size of the payload.

For applications serving millions of requests every second, these extra bytes matter.

gRPC takes a different approach.

Instead of JSON, it uses Protocol Buffers (Protobuf), a compact binary serialisation format.

Rather than transmitting descriptive text, Protobuf represents data in a highly efficient binary format.

The resulting payloads are significantly smaller.

Smaller payloads require less bandwidth.

Less bandwidth means faster transmission.

Faster transmission contributes to lower latency and higher throughput.

Although these improvements may appear minor for a single request, they become substantial when multiplied across billions of service calls every day.

Defining APIs Before Writing Code

One particularly elegant aspect of gRPC is that communication contracts are defined before implementation begins.

Instead of writing API endpoints directly, developers first create a Protocol Buffer definition file, commonly called a .proto file.

This file describes:

The services available.
The methods each service exposes.
The structure of every request.
The structure of every response.

Conceptually, it looks like this:

service PaymentService {
    rpc ProcessPayment(PaymentRequest)
        returns (PaymentResponse);
}

From this single definition, tools automatically generate client libraries and server code in multiple programming languages.

This process eliminates much of the repetitive work developers traditionally perform when building APIs.

More importantly, both the client and the server now share the same contract.

This significantly reduces integration errors because both sides are generated from the same specification.

HTTP/2 Makes a Difference

Another reason gRPC performs exceptionally well is that it is built on top of HTTP/2 rather than traditional HTTP/1.1.

Without diving too deeply into networking internals, HTTP/2 introduces several improvements that make communication more efficient.

Instead of opening multiple independent connections, HTTP/2 allows many requests and responses to travel simultaneously over a single connection.

This reduces connection overhead and allows multiple conversations to happen concurrently.

HTTP/2 also compresses headers and improves how data is transmitted across the network.

The result is faster communication with lower latency, especially in environments where services exchange thousands of requests every second.

For large distributed systems, these efficiencies accumulate into significant performance improvements.

Streaming: A Capability REST Doesn't Naturally Provide

Perhaps one of gRPC's most impressive features is its support for streaming.

Traditional REST communication generally follows a simple pattern:

One request.

One response.

The conversation ends.

Sometimes, however, applications need continuous communication rather than isolated requests.

Imagine a live stock market dashboard.

A multiplayer online game.

A GPS navigation system.

A live sports score application.

A monitoring dashboard displaying server metrics in real time.

In these scenarios, repeatedly sending HTTP requests every second becomes inefficient.

gRPC supports multiple communication models.

A client can send one request and receive a continuous stream of responses.

A server can continuously receive streamed requests.

Or both sides can exchange messages simultaneously.

This capability makes gRPC particularly attractive for applications involving live updates and continuous data exchange.

Where gRPC Excels

gRPC is particularly well suited for communication inside distributed systems.

When hundreds of microservices continuously exchange information, performance becomes increasingly important.

The smaller payload sizes, efficient serialisation, persistent HTTP/2 connections, and automatic code generation all contribute to a communication model optimised for speed.

This is why many organisations use gRPC internally while continuing to expose REST APIs to external clients.

External developers appreciate REST because it is simple, human-readable, and easy to integrate.

Internal services benefit from gRPC because efficiency matters more than human readability.

Many modern architectures therefore combine both approaches.

This hybrid architecture allows systems to take advantage of the strengths of each communication model.

gRPC Is Not Always the Right Choice

Despite its impressive capabilities, gRPC is not intended to replace REST entirely.

Because it uses binary Protocol Buffers, developers cannot simply open a browser and inspect responses as easily as they can with JSON.

Debugging often requires specialised tooling.

Public APIs also tend to favour REST because nearly every programming language, browser, and third-party integration platform already understands HTTP and JSON.

REST therefore remains an excellent choice for communication between external clients and backend systems.

gRPC shines when communication happens primarily between trusted internal services where performance, efficiency, and strong API contracts become more important than human readability.

As with every architectural decision we have discussed throughout this series, neither approach is universally better.

REST optimises for simplicity and interoperability.

gRPC optimises for efficiency and performance.

Understanding the requirements of the system is what determines the better choice.

Up to this point, every communication model we have discussed has shared one common characteristic.

Whether we were using REST or gRPC, one service directly contacted another service and waited for it to perform some work.

The communication was immediate.

The sender knew exactly who the receiver was.

The receiver processed the request and returned a response.

While this model works exceptionally well for many scenarios, it also creates an important dependency.

If the receiving service is unavailable, the sender cannot continue.

If the receiving service becomes slow, the sender also becomes slow.

In other words, the health of one service directly affects another.

As distributed systems become larger, these dependencies begin to accumulate.

Imagine a modern e-commerce platform during a festival sale.

A customer clicks the "Place Order" button.

At first glance, the operation seems simple.

The order is created, payment is processed, inventory is updated, a confirmation email is sent, loyalty points are added, analytics are recorded, invoices are generated, warehouse systems are notified, and recommendation engines learn from the purchase.

Although these activities are all triggered by the same user action, they do not all have the same urgency.

The customer certainly expects payment to be processed immediately.

However, the customer does not care whether analytics are updated within five milliseconds or five seconds.

Similarly, recommendation engines can learn about the purchase later without affecting the shopping experience.

This observation leads to one of the most important ideas in distributed systems:

Not every task needs to happen immediately.

Once engineers recognise this, a completely different communication model becomes possible.

Instead of directly asking another service to perform work, a service can simply announce that something has happened.

Any interested service can process that information whenever it is ready.

This is the philosophy behind message queues.

Thinking in Events Instead of Requests

Traditional APIs are request-driven.

One service asks another service to act.

Message queues are event-driven.

Instead of saying,

"Please send an email."

a service simply announces,

"An order has been created."

The service publishing the event does not know who will consume it.

It does not need to know.

Its responsibility ends after successfully publishing the message.

Other services independently decide whether that event is relevant to them.

This subtle difference fundamentally changes the architecture.

Instead of tightly coupling services together, communication becomes loosely coupled.

Imagine dropping a letter into a mailbox.

Your responsibility ends once the letter has been posted.

You do not stand beside the mailbox waiting for the postal worker.

You trust that the postal system will eventually deliver the message.

Message queues operate in much the same way.

Introducing the Message Broker

A message queue introduces an intermediary known as a message broker.

Rather than communicating directly with one another, services communicate through this broker.

The architecture now looks something like this:

Notice what has changed.

The Order Service no longer needs to know anything about the Notification Service.

It does not know whether the Notification Service is running.

It does not know whether analytics are temporarily unavailable.

It simply publishes an event.

The broker takes responsibility for delivering that message to interested consumers.

This greatly reduces dependencies between services.

A Real-World Example

Let's revisit our e-commerce platform.

A customer successfully places an order.

Inside the Order Service, the business transaction completes successfully.

Immediately afterwards, an event named OrderCreated is published to the message broker.

Notice something remarkable.

The Order Service only performs one communication.

The broker handles everything else.

As additional services are introduced in the future—perhaps fraud detection, inventory forecasting, recommendation systems, or customer rewards—the Order Service remains unchanged.

The architecture naturally supports growth.

Why Message Queues Improve Reliability

Suppose the Notification Service suddenly crashes.

What happens?

In a synchronous REST-based system, the Order Service attempts to contact the Notification Service and receives an error.

Depending on how the application is designed, this failure may delay or even fail the entire user request.

With a message queue, the situation is very different.

The Order Service publishes the event successfully.

The broker safely stores the message.

The Notification Service can process it later after recovering.

The customer still receives a successful order confirmation because the critical business operation—the purchase itself—has already completed.

This ability to temporarily decouple producers and consumers is one of the biggest reasons message queues are so widely adopted.

Services no longer have to be available at exactly the same moment.

The broker absorbs temporary failures and smooths communication between systems.

Handling Traffic Spikes

Another major advantage of message queues appears during periods of unusually high traffic.

Imagine an online retailer during Black Friday.

Millions of customers begin placing orders simultaneously.

The Order Service processes purchases as quickly as possible.

However, sending emails, generating invoices, updating analytics, and notifying warehouse systems all require additional processing time.

If every service attempted to perform all this work immediately, the entire platform could become overwhelmed.

Message queues naturally absorb these spikes.

Incoming events accumulate inside the queue.

Consumers process them at a sustainable rate.

This buffering effect prevents downstream services from becoming overloaded.

Instead of rejecting requests, the system gracefully handles temporary bursts in demand.

This is one reason message queues are often compared to waiting lines at supermarkets.

Customers continue arriving.

Some wait briefly.

Cashiers process them one at a time.

The line absorbs fluctuations in arrival rates.

Scaling Consumers Independently

Suppose the Notification Service cannot keep up with incoming messages.

Unlike synchronous communication, we do not necessarily need a faster server.

Instead, we can simply start more consumer instances.

Each consumer retrieves messages independently.

The workload becomes distributed across multiple workers.

As demand increases, additional consumers can be added.

As traffic decreases, unnecessary consumers can be removed.

This makes message queues naturally compatible with horizontal scaling, one of the concepts we explored earlier in this series.

Rather than scaling the producer, we scale the workers responsible for processing queued tasks.

Popular Message Queue Technologies

Over the years, several technologies have become industry standards for asynchronous communication.

RabbitMQ is one of the most widely used traditional message brokers. It provides reliable message delivery, flexible routing, acknowledgements, retries, and mature tooling, making it an excellent choice for many business applications.

Apache Kafka approaches the problem from a slightly different perspective. Rather than acting solely as a message queue, Kafka functions as a distributed event streaming platform capable of handling enormous volumes of events every second. Organisations use Kafka extensively for log aggregation, analytics pipelines, financial systems, IoT platforms, and real-time data processing.

Cloud providers also offer managed messaging services such as Amazon SQS, Google Cloud Pub/Sub, and Azure Service Bus, allowing teams to adopt asynchronous communication without managing broker infrastructure themselves.

Although these technologies differ in implementation, they all pursue the same goal:

Allow independent systems to communicate without requiring them to be available at exactly the same time.

The Cost of Asynchronous Communication

Message queues provide remarkable flexibility, but they also introduce new challenges.

One obvious trade-off is that communication is no longer immediate.

If an email service processes messages five seconds later, that delay is usually acceptable.

If a payment confirmation arrives five seconds later, the customer may become concerned.

Choosing asynchronous communication therefore requires understanding which operations are time-sensitive and which are not.

Another challenge is debugging.

In synchronous systems, following a request is relatively straightforward.

With asynchronous systems, events may pass through brokers, retries, dead-letter queues, and multiple consumers before finally completing.

Understanding the lifecycle of a single business operation becomes considerably more difficult.

This is why mature event-driven systems rely heavily on distributed tracing, centralised logging, and observability tools.

As systems become more asynchronous, visibility becomes just as important as functionality.

Asynchronous Communication Is About Independence

Perhaps the most important lesson about message queues is that they are not simply another communication technology.

They represent a different philosophy.

REST asks another service to perform work immediately.

gRPC performs the same idea more efficiently.

Message queues remove the requirement for immediate cooperation altogether.

Instead of saying,

"Do this now."

they simply say,

"This happened."

That small change dramatically reduces coupling, improves resilience, smooths traffic spikes, and allows systems to evolve independently.

Throughout this article, we have explored three fundamentally different ways in which services communicate inside modern distributed systems.

We began with REST, the communication style that powers much of today's internet. We then looked at gRPC, a high-performance framework designed specifically for efficient service-to-service communication. Finally, we explored message queues, which abandon direct conversations altogether in favor of asynchronous event-driven communication.

At first, these technologies may appear to compete with one another.

After all, they all allow one system to communicate with another.

So a natural question arises:

Which one should we use?

Interestingly, experienced system designers rarely ask that question.

Instead, they ask a different one.

What kind of conversation are these services trying to have?

That subtle shift in thinking often leads to the correct architectural decision.

There Is No Universal Winner

One of the biggest mistakes engineers make when learning system design is searching for the "best" technology.

Should we always use gRPC because it is faster?

Should we replace every REST API with Kafka?

Should every microservice communicate asynchronously?

The answer to all of these questions is no.

Each communication model was designed to solve a different problem.

Using the wrong one is similar to using a screwdriver to hammer a nail.

The tool is not bad.

It is simply solving a different problem.

REST optimises for simplicity.

gRPC optimises for efficiency.

Message queues optimise for independence.

Understanding those goals is far more valuable than memorising implementation details.

When REST Is the Right Choice

REST remains the most appropriate choice whenever communication involves external clients.

Browsers, mobile applications, desktop applications, and third-party developers all understand HTTP exceptionally well.

If your application exposes a public API, REST is often the safest and most practical option.

Imagine a food delivery application.

A customer's mobile app needs to:

View nearby restaurants.
Browse menus.
Place an order.
Track delivery status.
View previous orders.

Each of these actions follows a straightforward request-response pattern.

The client asks for information.

The server responds immediately.

REST fits naturally into this style of interaction.

It is also an excellent choice when readability and interoperability are more important than absolute performance.

Because REST commonly uses JSON, developers can inspect requests with a browser, Postman, or curl, making development and debugging remarkably convenient.

When gRPC Becomes a Better Choice

Now consider the communication happening inside the backend.

The Order Service may need to communicate with the Inventory Service dozens of times every second.

The Recommendation Service may continuously exchange information with machine learning systems.

A Search Service may call several ranking engines before returning results.

These services are all controlled by the same organisation.

There is no need for human-readable JSON.

There is no requirement for browser compatibility.

Performance becomes the primary concern.

This is precisely where gRPC shines.

Its binary serialisation, HTTP/2 transport, persistent connections, and strongly defined contracts make it ideal for internal communication between trusted services.

The user never interacts with gRPC directly.

Instead, it quietly enables fast communication behind the scenes.

Many organisations therefore expose REST APIs publicly while using gRPC internally.

This hybrid architecture combines the accessibility of REST with the efficiency of gRPC.

When Message Queues Are the Better Solution

Some operations simply do not require immediate responses.

Suppose a customer successfully purchases a product.

Several additional activities may need to occur afterwards.

A confirmation email must be sent.

Analytics should record the purchase.

Inventory forecasting should update demand predictions.

Warehouse systems should prepare packaging.

Recommendation engines should learn from customer behaviour.

Notice that none of these tasks should delay the checkout experience.

The customer only cares that the order has been placed successfully.

Everything else can happen afterwards.

This makes asynchronous communication a far better choice than synchronous requests.

Instead of contacting every downstream service directly, the Order Service simply publishes an OrderCreated event.

Every interested service processes that event independently.

The checkout experience remains fast.

The architecture becomes more resilient.

New services can subscribe to the event without modifying existing code.

This ability to evolve naturally is one of the greatest strengths of event-driven architectures.

Modern Systems Rarely Choose Just One

One of the most interesting observations about large technology companies is that they rarely commit to a single communication mechanism.

Instead, they combine several approaches, allowing each to solve the problems it handles best.

A typical architecture might resemble the following:

Notice how each communication model has a clearly defined responsibility.

The mobile application communicates with backend services using REST because HTTP and JSON are universally supported.

Internal services communicate with one another using gRPC because efficiency matters more than readability.

Business events are distributed through message queues because many downstream services do not require immediate responses.

Rather than competing, these technologies complement one another.

Choosing the Right Communication Style

Whenever designing communication between services, it is useful to begin by asking a series of simple questions.

The first question is whether the caller requires an immediate response.

If the answer is yes, synchronous communication such as REST or gRPC is usually appropriate.

The second question concerns performance.

If communication happens frequently between internal services and latency is critical, gRPC often provides significant advantages.

The third question is whether the work must happen immediately.

If the answer is no, asynchronous messaging usually results in a simpler and more resilient architecture.

Finally, consider ownership.

If the API will be consumed by external customers, partners, or public developers, REST remains the most approachable option because of its universal adoption and extensive tooling.

Thinking about the nature of the conversation rather than the technology itself almost always leads to better architectural decisions.

Communication Is About Trade-offs

Throughout this System Design series, one theme has appeared repeatedly.

There is rarely a perfect solution.

Monoliths trade flexibility for simplicity.

Distributed systems trade simplicity for scalability.

Horizontal scaling trades hardware upgrades for coordination.

Microservices trade organisational independence for operational complexity.

Communication mechanisms follow the same pattern.

REST is easy to understand but introduces additional overhead.

gRPC improves efficiency but sacrifices some simplicity and requires specialised tooling.

Message queues increase resilience and decoupling but introduce eventual consistency and make systems harder to observe and debug.

Good system design is therefore not about selecting the newest technology.

It is about selecting the technology whose trade-offs best match the problem you are trying to solve.

Final Thoughts

As applications evolve from monoliths into distributed systems, communication becomes one of the most important architectural concerns.

The quality of a distributed system is determined not only by how well individual services are implemented but also by how effectively those services cooperate.

REST, gRPC, and message queues each represent a different philosophy of communication.

REST emphasises simplicity and broad compatibility.

gRPC emphasises speed, efficiency, and strongly defined contracts.

Message queues emphasise independence, resilience, and asynchronous processing.

The most successful systems are rarely built around just one of these approaches.

Instead, they combine all three, allowing each communication model to solve the problems for which it was designed.

Perhaps that is the most important lesson from this article.

In system design, the goal is not to find a single technology that solves every problem.

The goal is to understand the strengths and limitations of each tool well enough to know when to use it.

And that ability to choose the right tool for the right problem is what ultimately distinguishes a good software engineer from a great system designer.

Microservices Architecture: Benefits, Challenges, and When to Use It

Sushant Gaurav — Tue, 28 Jul 2026 05:41:00 +0000

For many engineers, microservices are one of the first concepts that come to mind when discussing modern system design.

Job descriptions mention them constantly. Conference talks celebrate them. Architecture diagrams often contain dozens or even hundreds of small services communicating with one another. It is easy to come away with the impression that microservices represent the final destination of software architecture - that every successful application eventually becomes a collection of independently deployed services.

The reality is considerably more nuanced.

Microservices are not the next version of monoliths in the same way that smartphones replaced feature phones. They are not universally better, nor are they the inevitable future of every application. Instead, microservices are an architectural response to a very specific set of problems that begin to appear as systems and organisations grow.

To understand why microservices exist, we first need to understand the world before them.

For a very long time, most applications were built as monoliths. User authentication, product management, payments, inventory, notifications, analytics, reporting, and administration all lived inside a single codebase and were deployed together as a single application.

For small and medium-sized systems, this approach worked remarkably well.

A developer could clone the repository, run the application locally, and understand the request flow from beginning to end. Debugging was relatively straightforward because everything lived in one place. Deployments involved building and releasing a single artefact. Transactions across modules were simple because all the code ran within the same process and often accessed the same database.

This simplicity is one of the reasons monoliths continue to power a large percentage of software systems around the world.

However, successful systems have a tendency to grow in ways that architecture diagrams rarely anticipate.

Teams become larger.
Features become more numerous.
Release cycles become more frequent.
Different parts of the application begin evolving at different speeds.
What started as a clean and elegant codebase slowly begins to feel heavier.
A small change to one module unexpectedly affects another.
Deployment times increase.
Testing becomes slower.
The number of developers working in the same repository continues growing until coordination itself becomes a challenge.

Eventually organizations discover that the problem is no longer computational complexity.

It is organisational complexity.

A payment team wants to deploy a new feature, but must wait for the inventory team to finish testing their changes.

The recommendation engine requires additional compute resources during peak traffic periods, but scaling means scaling the entire application, including components that are barely being used.

Different teams want to adopt different technologies, databases, and deployment strategies, yet the architecture forces everyone into the same decisions.

The application begins behaving less like a product and more like a city that has grown without urban planning.

This is the environment from which microservices emerged.

Instead of treating the application as a single deployable unit, the system is divided into smaller services that each own a specific business capability.

The payment service owns payments.
The inventory service owns inventory.
The notification service owns notifications.
The recommendation service owns recommendations.

Each service becomes responsible for its own logic, data, deployment, and scaling requirements.

Conceptually, the architecture starts looking something like this:

The most important aspect of this diagram is not the number of services.

It is ownership.

Each service becomes a small system with clear responsibilities and well-defined boundaries. Teams can work independently because they no longer need to understand the entire application to make progress in their own domain.

This shift introduces one of the biggest philosophical differences between monoliths and microservices.

In a monolith, components communicate through function calls.
In microservices, communication happens over a network.

At first, this sounds insignificant.

After all, a service call is just another request.

In reality, this changes almost everything.

A function call inside a monolith takes microseconds and rarely fails.

A network request may take milliseconds, may experience congestion, may time out, may be retried, or may fail.

What used to be a simple method invocation suddenly becomes a distributed systems problem involving latency, retries, availability, and fault tolerance.

This is one of the reasons many engineers say that adopting microservices means adopting distributed systems.

The complexity does not disappear.

It simply moves.

Instead of managing complexity inside a codebase, we begin managing complexity between services.

This is neither good nor bad.

It is a trade-off.

And like every major architectural decision in system design, understanding those trade-offs is far more important than understanding the technology itself.

Perhaps the greatest misconception surrounding microservices is the belief that they are primarily a scaling strategy.

They are not.

Companies rarely move to microservices because their servers cannot handle traffic.

More often, they move because their teams cannot handle coordination.

When hundreds or thousands of engineers are working on the same product, organisational scalability becomes just as important as technical scalability.

Microservices allow teams to move independently, deploy independently, and evolve independently.

This alignment between software architecture and team structure is so common that it is often summarised by Conway's Law:

Organisations tend to design systems that mirror their communication structures.

A company with ten independent teams often ends up building ten independent services.

A company with a single tightly integrated team often builds a monolith.

Architecture and organisational design are frequently reflections of one another.

This is why the decision to adopt microservices is rarely just an engineering decision.

It is often a business decision, a team decision, and an operational decision all at the same time.

Understanding this context is essential because, without it, microservices can appear to be an obvious improvement over monoliths.

They are not.

They solve certain problems extremely well.

They also create entirely new categories of problems that monoliths never had to worry about.

Independent Deployment Changes the Development Process

One of the biggest advantages of microservices is that they allow different parts of the system to evolve independently.

Consider an e-commerce platform built as a monolith.

The payment team develops a new payment integration. At the same time, the recommendation team is experimenting with a new machine learning model, while the notification team is redesigning the email delivery pipeline.

In a monolithic architecture, all these changes eventually converge into the same application deployment.

Even though these teams are working on completely unrelated features, they still share release cycles, testing pipelines, deployment windows, and rollback procedures.

This creates coordination overhead.

A delay in one team may delay everyone else.

A bug in one component may prevent unrelated features from reaching production.

As organisations grow, these dependencies become increasingly painful.

Microservices attempt to solve this by allowing services to be deployed independently.

The payment team can deploy payment changes without waiting for the recommendation team.

The notification service can release a new version without affecting inventory management.

The recommendation service can experiment rapidly without introducing risk into unrelated parts of the application.

The result is not necessarily faster code execution.

The result is faster organisational movement.

Independent Scaling Allows Resources to Follow Demand

Another significant advantage appears when different parts of the system experience different workloads.

In most applications, traffic is rarely distributed evenly.

An online marketplace may process millions of product searches every hour while receiving far fewer payment requests.

A video streaming platform may generate enormous traffic for video delivery while user profile services remain relatively idle.

A social network may receive vastly more feed requests than account creation requests.

In a monolithic architecture, scaling one part of the application often means scaling everything.

Even if only the search functionality is under pressure, additional instances of the entire application may need to be deployed.

This increases infrastructure costs and wastes resources.

Microservices allow organisations to scale individual services according to their own requirements.

The search service may run on twenty instances while the payment service operates comfortably on three.

This granularity allows infrastructure decisions to reflect actual usage patterns rather than architectural limitations.

Technology Diversity Becomes Possible

One of the less discussed advantages of microservices is technological flexibility.

Different problems are often best solved using different tools.

A recommendation engine performing machine learning inference may benefit from Python.

A high-performance messaging system may be better suited to Go.

A payment system requiring mature transactional guarantees may prefer Java and relational databases.

An analytics pipeline may rely heavily on distributed data processing frameworks.

In a monolith, these decisions are constrained by the technology choices of the entire application.

Microservices make this separation possible because services communicate through APIs rather than internal language constructs.

The recommendation service does not need to know how the payment service is implemented.

It only needs to understand the contract exposed by its API.

This flexibility is powerful, but as we will see shortly, it can also become dangerous if left uncontrolled.

Fault Isolation Improves Resilience

Failures are inevitable in software systems.

Servers crash.

Databases become unavailable.

Network partitions occur.

Dependency time out.

The question is rarely whether failure will happen.

The question is how much of the system fails when it does.

In a tightly coupled monolith, failures can propagate rapidly.

A memory leak in one component may exhaust resources for the entire application.

A slow database query may impact unrelated functionality.

An overloaded subsystem may cause widespread degradation.

Microservices improve isolation by creating boundaries between components.

If the recommendation service experiences problems, users may temporarily lose personalised recommendations while the rest of the platform continues operating normally.

The application degrades gracefully rather than collapsing completely.

This idea is often referred to as the blast radius of failure.

Microservices aim to reduce the blast radius.

Reducing the impact of failures is one of the strongest arguments in favour of service decomposition.

The Complexity That Replaces the Monolith

At this point, microservices can sound almost ideal.

Independent deployments.

Independent scaling.

Fault isolation.

Technology flexibility.

So why doesn't every company immediately move to microservices?

Because microservices solve one type of complexity by introducing another.

Remember the observation from the previous part:

Complexity does not disappear. It moves.

In a monolith, communication happens through local function calls.

In microservices, communication happens through networks.

And networks are fundamentally unreliable.

A function call rarely fails.

A network request can fail for dozens of reasons:

Packet loss
Timeouts
Congestion
DNS failures
Load balancer failures
Service crashes
Partial outages

Suddenly, developers must think about retries, circuit breakers, backoff strategies, and timeout management.

The application is no longer just software.

It is a distributed system.

Data Becomes One of the Hardest Problems

Monoliths often enjoy a luxury that microservices lose:

A shared database.

A single transaction can update multiple tables atomically.

Consistency is relatively easy to achieve.

Microservices intentionally avoid this model because shared databases create coupling between services.

Instead, each service typically owns its own data.

This improves service independence but introduces difficult questions.

What happens when a payment succeeds but order creation fails?

How do we maintain consistency across multiple databases?

How do we roll back distributed operations?

How do services coordinate business workflows?

These problems eventually lead engineers toward concepts such as event-driven architectures, sagas, outbox patterns, and distributed transactions.

Problems that never existed inside a simple monolith suddenly become central architectural concerns.

Observability Stops Being Optional

Debugging a monolith can often be as simple as following logs from a single application instance.

Microservices remove that convenience.

A single user request may travel through:

API Gateway
Authentication Service
User Service
Payment Service
Inventory Service
Notification Service

When something goes wrong, finding the source of the problem becomes significantly harder.

This is why mature microservice architectures invest heavily in:

Distributed tracing
Centralised logging
Metrics collection
Monitoring systems

Observability evolves from a useful feature into a critical requirement.

By this point, microservices can feel both exciting and intimidating at the same time.

On one hand, they offer independent deployments, independent scaling, fault isolation, and organisational flexibility. On the other hand, they introduce distributed systems complexity, operational overhead, and entirely new categories of failure.

This naturally leads to the most important question in the entire discussion:

When should you actually use microservices?

Interestingly, the answer is not "as early as possible."

In fact, for many systems, that answer would be actively harmful.

One of the biggest misconceptions in modern software engineering is the belief that successful companies started with microservices and scaled effortlessly from day one.

That is not what happened.

Many of today's largest technology companies began with monolithic architectures.

The early versions of companies like Amazon, Netflix, and Facebook were significantly more monolithic than many engineers realise.

This was not because the engineers lacked knowledge.

It was because monoliths optimise for something that startups desperately need:

speed.

When a company is trying to validate an idea, acquire users, and find product-market fit, the biggest risk is rarely scalability.

The biggest risk is building the wrong product.

During this stage, simplicity is a competitive advantage.

A monolith allows teams to move quickly. Features can be built without worrying about service boundaries, network contracts, distributed tracing, or inter-service communication protocols.

Developers can change database schemas quickly.

Refactoring is easier.

Testing is simpler.

Deployment pipelines are smaller.

Operational costs are lower.

Most importantly, the engineering team can spend its energy solving business problems instead of infrastructure problems.

This is why one of the most common pieces of advice in software architecture is:

Start with a monolith and earn your microservices.

That statement is not a criticism of microservices.

It is recognition that complexity should arrive only when it solves a real problem.

Microservices are not free.

Every service introduces additional deployment pipelines, monitoring requirements, infrastructure costs, API contracts, and operational responsibilities.

A system with fifty microservices is not managing one application.

It is managing fifty applications.

That distinction becomes incredibly important as organisations grow.

Signs That a Team May Be Ready for Microservices

While there is no universal threshold, certain patterns appear repeatedly in organisations that successfully adopt microservices.

One common signal is organisational growth.

When dozens or hundreds of engineers are contributing to the same codebase, coordination begins to slow development. Teams start stepping on each other's changes. Releases become increasingly risky because every deployment contains modifications from many unrelated teams.

At some point, the bottleneck is no longer technical architecture.

It is communication.

Microservices can reduce this friction by allowing teams to own individual services and deploy independently.

Another signal appears when different parts of the application have drastically different scaling requirements.

Consider a streaming platform.

The video processing pipeline may require enormous compute resources while account management requires comparatively little infrastructure.

Scaling both components together becomes inefficient.

Microservices allow resources to follow actual demand.

The same pattern appears in machine learning systems, recommendation engines, search platforms, and analytics pipelines.

Some components naturally become much larger than others.

Independent scaling starts becoming valuable.

A third signal appears when release velocity becomes a problem.

If a minor update to the notification system requires coordination across multiple teams and weeks of testing for unrelated components, the deployment process itself becomes an obstacle to business growth.

Independent deployments can dramatically improve development speed in these environments.

Signs That a Team Probably Should Not Use Microservices

Interestingly, identifying situations where microservices are unnecessary is often easier.

A small engineering team working on a relatively straightforward application rarely benefits from distributed complexity.

If five developers are building an internal business application with moderate traffic, microservices often create more problems than they solve.

The team now needs:

Service discovery
Monitoring infrastructure
Centralised logging
API versioning
Deployment orchestration
Network security policies
Inter-service authentication
Distributed tracing

None of these problems existed before.

The architecture becomes more sophisticated while the business problem remains the same.

This is one of the reasons many organisations accidentally build what engineers jokingly call:

a distributed monolith.

A distributed monolith has all the operational complexity of microservices but none of their advantages.

Services are tightly coupled.

Deployments still require coordination.

Failures cascade between systems.

Scaling remains difficult.

The architecture becomes harder to understand without becoming more flexible.

This is often considered one of the most painful outcomes of premature microservice adoption.

The Hybrid Reality of Modern Systems

The discussion is often framed as a choice between monoliths and microservices.

Real systems are usually far more nuanced.

Many organisations operate somewhere in the middle.

A company may begin with a monolith and gradually extract services as scaling requirements emerge.

Certain domains may remain inside the monolith for years while others become independent services.

A recommendation engine may become a standalone service because it has unique computational requirements.

Payment processing may become isolated because it has strict security and compliance requirements.

Analytics pipelines may move into separate services because they require entirely different storage and processing technologies.

Meanwhile, user management and administration may continue living inside the monolith.

The result is neither a pure monolith nor a pure microservice architecture.

It is a pragmatic architecture shaped by business needs rather than ideology.

This gradual evolution is far more common than complete rewrites.

In fact, many experienced architects prefer incremental extraction because it allows systems to evolve naturally rather than forcing large architectural migrations.

The Most Important Lesson About Microservices

Perhaps the most valuable lesson in this entire discussion is that architecture is ultimately a tool for managing complexity.

Monoliths manage complexity by keeping everything together.

Microservices manage complexity by separating things apart.

Neither approach eliminates complexity.

They simply decide where it lives.

A small system with microservices may end up more complicated than necessary.

A massive global platform with a monolith may eventually become impossible to maintain.

Good architecture is rarely about following trends.

It is about understanding trade-offs.

The best architects are not the ones who always choose microservices.

They are the ones who know when not to.

Final Thoughts

Microservices changed the software industry because they allowed systems and organisations to grow beyond the limits of a single application and a single team.

They enabled independent deployment, independent scaling, and independent ownership.

But they achieved these benefits by embracing the realities of distributed systems: unreliable networks, partial failures, eventual consistency, and operational complexity.

That trade-off is the essence of microservices.

They are not an upgrade from monoliths.

They are a different answer to a different problem.

And understanding the problem is far more important than understanding the technology.

Data Partitioning and Sharding: How Systems Scale Data

Sushant Gaurav — Tue, 14 Jul 2026 05:41:00 +0000

When engineers first learn about scalability, the conversation usually revolves around application servers.

If traffic increases, add more servers. If requests increase, add a load balancer. If latency increases, introduce caching. For a while, this works remarkably well.

Application servers are relatively easy to scale because they are typically stateless. If one server becomes overloaded, another can be added behind the load balancer, and traffic can simply be distributed between them.

Databases are different.

Unlike application servers, databases hold state. They contain user information, transactions, product catalogues, messages, orders, and every other piece of information that gives an application meaning. They are the memory of the system.

And because databases hold state, scaling them becomes significantly harder.

Eventually, almost every growing system encounters the same problem:

The application can still handle more traffic, but the database cannot.

This is where data partitioning enters the picture.

Why Bigger Databases Are Not Always the Answer

The first response to a struggling database is usually straightforward.

Increase the machine size.
Add more CPU cores.
Add more memory.
Upgrade to faster SSDs.
Move to a more powerful server.

This approach is known as vertical scaling, and just like application servers, databases can benefit from it for a surprisingly long time.

For many startups and medium-sized applications, vertical scaling is often enough.

However, every machine eventually reaches physical and financial limits.

There is always a larger server available—until there isn't.

At some point, upgrading hardware stops providing meaningful improvements. Even worse, the entire system becomes dependent on a single machine.

No matter how powerful that machine becomes, it is still only one machine.

If it fails, the database fails. If the database fails, the application fails.

This creates both a scalability problem and an availability problem.

The question therefore, changes from:

"How do we make this database bigger?"

to:

"How do we distribute the database itself?"

Understanding Data Partitioning

Data partitioning is the process of dividing a large dataset into smaller and more manageable pieces.

Instead of storing all users, all products, and all orders on a single database server, the data is split into multiple partitions and distributed across several machines.

The idea itself is surprisingly intuitive.

Imagine a library containing one hundred million books.

Storing every book inside a single building would eventually create problems. The building would become too large, too expensive, and too difficult to maintain.

A more practical solution would be to divide the books among multiple buildings.

Each building stores only a subset of the collection. Together, the buildings form a complete library system. Individually, each building only manages a portion of the data.

Databases use the same principle.

From the perspective of the application, the database may still appear to be a single logical system.

Behind the scenes, however, the data is physically distributed across multiple machines.

This separation between logical simplicity and physical complexity is one of the defining characteristics of modern distributed systems.

What Is Sharding?

The terms partitioning and sharding are often used interchangeably, and in many practical discussions they effectively mean the same thing.

Strictly speaking, partitioning refers to the broader concept of splitting data into smaller pieces.

Sharding is a specific type of partitioning in which those pieces are distributed across different physical machines.

You can think of sharding as horizontal partitioning of data.

Instead of making a single database server larger, we make the database wider by adding more machines.

Each shard becomes responsible for storing a subset of the overall dataset.

The application no longer talks to a single database server.

Instead, it communicates with a routing layer that determines which shard owns the requested data.

This may sound like a small architectural change.

In reality, it changes everything.

Because the moment data exists in multiple locations, entirely new problems begin to emerge.

Why Replication Alone Cannot Solve This Problem

When discussing database scaling, many engineers initially assume replication can solve everything.

After all, if one database becomes overloaded, why not simply create additional copies of it?

Replication certainly helps, but it solves a very different problem.

Replication creates multiple copies of the same data.

Sharding divides the data into different pieces.

Consider an application with one hundred million users.

With replication, every replica contains all one hundred million users.

This approach improves read scalability because read requests can be distributed across multiple replicas.

It also improves availability because the system can continue operating if one replica fails.

However, the primary database still receives every write request.

Every new user registration, every order creation, and every transaction still goes to the same machine.

Eventually, that machine becomes the bottleneck.

Sharding addresses a completely different dimension of scalability.

Instead of asking:

"How do we serve more readers?"

It asks:

"How do we store and write more data?"

Why Large Internet Systems Depend on Sharding

Modern internet companies operate at scales that simply cannot be supported by a single database server.

A social media platform may store billions of posts.

An e-commerce platform may process millions of transactions every day.

A ride-sharing platform may continuously ingest location updates from millions of devices.

The volume of data alone makes single-node databases impractical.

Sharding allows these systems to continue growing incrementally.

When storage requirements increase, additional shards can be introduced.

When write throughput increases, new shards can absorb part of the workload.

Instead of upgrading to increasingly expensive hardware, capacity can be expanded horizontally.

This is one of the reasons why sharding is often described as bringing horizontal scaling to databases.

The philosophy is the same as horizontal scaling for application servers.

Rather than building larger machines, we build larger systems.

The Hidden Complexity of Sharding

At this point, sharding may sound almost too good to be true.

More storage.
More throughput.
Better scalability.

So why doesn't every application use it from day one?

Because sharding introduces complexity that simply does not exist in single-database systems.

The system must now answer difficult questions.

How does the application know which shard contains a particular user's data?
What happens when one shard becomes significantly larger than the others?
How are transactions handled when data spans multiple shards?
What happens if a shard fails?
How are backups managed?
How are joins performed across shards?
These problems are not implementation details.
They are some of the hardest challenges in distributed systems.

And solving them requires entirely new techniques and architectural patterns.

The Fundamental Question of Sharding

The moment a system decides to split its data across multiple database servers, a surprisingly simple question becomes one of the most important architectural decisions in the entire system:

How do we decide which shard stores which data?

A single database never has to think about this problem. Every query goes to the same machine. Every user record lives in the same place. Every order, product, message, and transaction can be found by asking a single database server.

Sharding changes this completely.

Now the system must determine where data belongs before it can even execute a query.

If a user with ID 125847 logs into the application, the system first needs to answer:

Which shard owns user 125847?

Only then can the actual database query begin.

This process is known as a sharding strategy, and the choice of strategy has enormous implications for scalability, performance, and operational complexity.

Range-Based Sharding

One of the earliest and most intuitive approaches is range-based sharding.

The idea is simple.

Each database server becomes responsible for a specific range of values.

For example:

Shard 1 stores users with IDs from 1 to 1 million.
Shard 2 stores users with IDs from 1 million to 2 million.
Shard 3 stores users with IDs from 2 million to 3 million.

Visually, the architecture looks something like this:

At first glance, this approach seems almost perfect.

It is easy to understand. It is easy to implement. It is easy to debug. If someone asks where user 2,345,678 lives, the answer is immediately obvious.

The problem is that real-world data is rarely distributed evenly.

Imagine a social media platform where new users are constantly signing up.

Most new users will have the latest IDs.

This means almost all write traffic ends up hitting the newest shard while older shards remain relatively idle.

One database server becomes overloaded while others sit underutilised.

This phenomenon is known as a hotspot, and it is one of the biggest challenges in distributed databases.

Understanding Hotspots

Hotspots occur when certain shards receive significantly more traffic than others.

The issue may arise from user growth patterns, geographical concentration, or uneven business activity.

Consider an e-commerce platform during a flash sale.

If products are partitioned based on category and electronics happens to be the most popular category, the shard responsible for electronics suddenly receives an overwhelming amount of traffic.

Meanwhile, other shards continue operating normally.

The irony is that the system may have dozens of database servers available, yet performance still suffers because the load distribution itself is uneven.

This is one of the reasons engineers started looking for more balanced approaches.

Hash-Based Sharding

Hash-based sharding attempts to solve this problem by distributing data more evenly across servers.

Instead of storing data according to ranges, the system applies a mathematical hash function to the shard key.

The result of the hash determines where the data will live.

Conceptually, it works like this:

hash(user_id) % number_of_shards

Suppose the system currently has four shards.

If the hash value of a user ID results in 2, that user is stored on shard two.

If the result is 3, the user belongs to shard three.

The actual values are less important than the outcome:

The distribution becomes much more uniform.

Unlike range-based sharding, newly created users are unlikely to end up on the same shard.

Writes become naturally distributed across the cluster.

This significantly reduces hotspots and improves scalability.

However, hash-based sharding introduces a new problem.

Humans lose predictability.

With range sharding, engineers immediately know where a user lives.

With hash sharding, finding the location of a record requires executing the hashing algorithm.

The routing layer becomes mandatory.

The Problem of Growth

Eventually, every successful system encounters another difficult question:

What happens when four shards are no longer enough?

The obvious answer seems simple:

Add a fifth shard.

Unfortunately, hash-based systems make this surprisingly painful.

Consider the following example:

hash(user_id) % 4

Now suppose we add a fifth shard:

hash(user_id) % 5

Suddenly, almost every user maps to a different shard than before.

Data that previously belonged to shard one may now belong to shard three.

Data from shard three may move to shard five.

A huge percentage of the database needs to be migrated.

For systems containing billions of records, this migration can become extraordinarily expensive and risky.

This challenge led to one of the most elegant ideas in distributed systems.

Consistent Hashing

Consistent hashing was developed specifically to minimise data movement when infrastructure changes.

Instead of mapping data directly to servers, both servers and data are placed on a logical ring.

Each piece of data is assigned to the next available server in the ring.

Now imagine adding a new server.

Instead of redistributing the entire dataset, only a small portion of the data moves to the new machine.

Most records remain exactly where they were.

This dramatically reduces migration costs and makes scaling much safer.

Consistent hashing is widely used in distributed systems because infrastructure growth becomes a normal operational activity rather than a major migration project.

Systems such as distributed caches, distributed databases, and object storage platforms frequently rely on this technique.

Choosing the Right Shard Key

Interestingly, one of the most important decisions in sharding is often overlooked:

choosing the shard key.

The shard key is the attribute used to determine where data lives.

Examples include:

User ID
Customer ID
Geographic region
Product category
Organisation ID

Choosing the wrong shard key can create a severe imbalance in the system.

Suppose a ride-sharing platform partitions data by city.

This may work well initially.

But if one city suddenly becomes ten times larger than every other city, that shard becomes a permanent hotspot.

Similarly, partitioning social media users by country may appear reasonable until a handful of countries dominate global traffic.

A good shard key distributes data evenly while still allowing efficient queries.

Finding that balance is often one of the hardest design decisions in large systems.

Why Cross-Shard Queries Become Expensive

In a traditional database, a query can freely join tables because all the data lives on the same machine.

Sharding changes this assumption.

Imagine a query asking for information that spans multiple shards.

The database can no longer answer the question locally.

Instead, the system must:

Send requests to multiple shards.
Wait for responses from each server.
Merge the results.
Return the final response.

A query that once required a single machine now becomes a distributed operation involving network communication and coordination.

This is why sharding often forces teams to rethink their data models and access patterns.

The Reality of Modern Systems

The truth is that most large internet companies use a combination of techniques rather than relying on a single strategy.

Sharding is combined with:

Replication for availability.
Caching for performance.
Load balancing for traffic distribution.
CDNs for global content delivery.

Modern distributed systems are rarely built from one idea alone.

They are ecosystems of complementary techniques working together.

A shard may contain replicas.

Those replicas may sit behind load balancers.

Frequently accessed data may never reach the database at all because it is served from a cache.

This layered architecture is what allows modern systems to operate at a global scale.

Final Thoughts

Sharding is often introduced as a database optimisation technique.

In reality, it is much more than that.

It represents the moment when data itself becomes distributed.

And once data becomes distributed, the system enters an entirely new world of trade-offs involving routing, coordination, consistency, and fault tolerance.

Scaling application servers is relatively straightforward.

Scaling data is where distributed systems become truly interesting.

Because at the internet scale, the challenge is no longer simply storing information.

The challenge is knowing where that information lives, how to find it quickly, and how to keep the entire system functioning while millions of users are trying to access it simultaneously.

Databases in System Design - SQL vs NoSQL (When and Why)

Sushant Gaurav — Tue, 30 Jun 2026 05:41:00 +0000

Every system, no matter how simple or complex, eventually converges to a single fundamental need: it must remember things.

Users sign up, data gets generated, transactions happen, content is created, relationships form, and all of this needs to be stored, retrieved, and updated reliably. At a small scale, this seems trivial. You pick a database, store your data, and move on.

But as systems grow, data stops being passive.

It becomes the centre of gravity around which everything else revolves.

Performance depends on it. Scalability depends on it. Consistency, availability, and even user experience are shaped by how data is stored and accessed.

And this is where one of the most important decisions in system design emerges:

Should you use a relational database (SQL), or a non-relational database (NoSQL)?

At first glance, this appears to be a technology choice. But in reality, it is a design philosophy decision; one that reflects how your system models the world.

The Structured World of SQL

Relational databases, often referred to as SQL databases, are built on a simple but powerful idea: data should be organised into structured tables with clearly defined relationships.

This model has been around for decades and is grounded in mathematical principles. Each piece of data fits into a predefined schema. Tables are connected through relationships, and queries allow you to retrieve and combine data in flexible ways.

This structure brings clarity.

When you design a relational schema, you are effectively defining the shape of your data upfront. You decide:

What entities exist
What attributes do they have
How they relate to each other

Once defined, this structure is enforced strictly. Every piece of data must conform to it.

This may feel restrictive at first, but it provides a powerful guarantee:

The data in the system is consistent, predictable, and reliable.

This is why relational databases are widely used in systems where correctness is critical. Financial systems, transactional platforms, and enterprise applications rely on the guarantees provided by SQL databases to maintain integrity.

For example, systems within companies like Oracle and Microsoft have long relied on relational databases to manage structured, high-integrity data.

The Power of Relationships

One of the defining strengths of SQL databases is their ability to model relationships between data.

Consider a simple scenario:

A user places an order
An order contains multiple items
Each item belongs to a product

In a relational database, these relationships are explicitly defined. You can join tables together and retrieve complex, interconnected data with a single query.

This ability to perform joins allows systems to answer rich questions:

What products has a user purchased?
Which items are most frequently bought together?
What is the total value of all orders in a given time period?

This expressiveness is one of the reasons SQL remains dominant in many domains.

Consistency

Relational databases are designed with strong consistency in mind.

They follow properties often referred to as ACID:

Atomicity
Consistency
Isolation
Durability

While these terms may sound theoretical, their impact is very real.

They ensure that:

Transactions either complete fully or not at all
The database remains in a valid state
Concurrent operations do not interfere in harmful ways
Data is not lost, even in the event of failure

This makes SQL databases particularly well-suited for systems where correctness cannot be compromised.

Think of banking systems. If money is deducted from one account but not credited to another due to a failure, the system becomes unreliable. SQL databases are designed to prevent such scenarios.

The Limits Begin to Appear

Despite their strengths, relational databases begin to show limitations as systems scale.

The first challenge is scalability.

SQL databases are traditionally designed to scale vertically. You increase the power of a single machine to handle more load. But as we explored earlier, vertical scaling has limits, both in terms of hardware and cost.

The second challenge is flexibility.

Because the schema is predefined, making changes to the structure of data can be complex. Adding new fields, modifying relationships, or evolving the data model often requires careful migrations.

In rapidly evolving systems, this rigidity can slow down development.

The third challenge is distribution.

Distributing relational databases across multiple nodes while maintaining strong consistency is difficult. It introduces coordination overhead, which can impact performance and availability.

These challenges do not make SQL obsolete, but they create space for a different approach.

Enter NoSQL - A Different Philosophy

NoSQL databases emerged not as a replacement for SQL, but as a response to the challenges of scale, flexibility, and distribution.

Instead of enforcing a rigid schema, NoSQL systems embrace a more flexible approach to data modelling.

Data is often stored in formats that closely resemble how it is used in applications:

Documents
Key-value pairs
Wide-column structures
Graphs

This flexibility allows developers to evolve data models quickly without being constrained by predefined schemas.

But more importantly, NoSQL databases are designed with distribution as a first principle.

From the ground up, they assume:

Data will be spread across multiple machines
Systems will operate across regions
Failures will occur

This makes them naturally suited for horizontally scaled systems.

Companies like Amazon and Netflix have leveraged NoSQL databases to handle massive amounts of distributed data with high availability.

A Shift in Trade-offs

With NoSQL, the trade-offs begin to shift.

Instead of strict consistency, many NoSQL systems embrace eventual consistency.

Instead of rigid schemas, they offer flexibility.

Instead of centralised control, they enable distributed scalability.

But these benefits come at a cost:

More responsibility on the application to manage data integrity
Reduced ability to perform complex joins
Increased complexity in ensuring consistency when needed

This is not a drawback; it is a design choice.

And understanding this shift is key to choosing the right database.

Where This Is Heading

At this point, we have two distinct worlds:

SQL: structured, consistent, relationship-driven
NoSQL: flexible, scalable, distribution-friendly

But the real question is not which one is better.

It is:

When should you use each?

This is where database selection moves from theory to system design strategy.

The Many Faces of NoSQL

When people say NoSQL, they often imagine a single alternative to relational databases. In reality, NoSQL databases vary significantly in how they store and retrieve data.

Each type exists because different applications have different needs.

Document Databases - Modelling Data as It Is Used

Document databases store data in a format similar to JSON. Instead of splitting data across multiple tables, related information is often stored together in a single document.

This structure mirrors how applications actually consume data.

Instead of performing multiple joins to assemble related information, everything is already grouped together. This reduces query complexity and improves performance for read-heavy workloads.

This model is widely used in systems where:

Data structures evolve frequently
Relationships are hierarchical
Fast reads are more important than complex joins

Key-Value Stores

Key-value databases are the simplest form of NoSQL.

Data is stored as a collection of key-value pairs. You provide a key, and the system returns the associated value.

This simplicity allows for extremely fast lookups and high scalability.

However, it comes with limitations:

No support for complex queries
No inherent relationships between data

These databases are often used for:

Caching layers
Session storage
Simple lookup services

Wide-Column Databases

Wide-column databases organise data into rows and columns, but unlike SQL, each row can have a different set of columns.

This flexibility allows systems to handle large-scale, sparse datasets efficiently.

These databases are designed for:

High write throughput
Distributed storage across many nodes
Analytical workloads at scale

Graph Databases

While SQL can model relationships, graph databases treat them as the core of the system.

Data is stored as nodes and edges, making it ideal for highly connected data.

This model excels in scenarios where relationships are complex and deeply interconnected, such as:

Social networks
Recommendation systems
Fraud detection

Where SQL Still Shines

Despite the rise of NoSQL, relational databases remain dominant in many critical systems.

This is not by accident.

SQL databases are still the best choice when:

Data relationships are complex and require joins
Transactions must be strongly consistent
Data integrity is critical
The schema is stable and well-defined

For example, financial systems and transactional platforms within companies like Microsoft continue to rely heavily on relational databases.

Because in these systems, correctness is not negotiable.

Where NoSQL Excels

NoSQL systems shine in environments where scale and flexibility are more important than strict consistency.

They are particularly effective when:

The system needs to scale horizontally
Data models change frequently
High availability is required
Workloads are distributed globally

Platforms like Netflix use NoSQL databases to handle massive volumes of user activity, content metadata, and streaming data.

In such systems, slight inconsistencies are acceptable if they enable better performance and availability.

The Hybrid Reality

At this point, it might seem like you must choose one approach over the other.

But in modern system design, that is rarely the case.

Most large-scale systems use a combination of SQL and NoSQL databases, depending on the requirements of each component.

For example, a system might:

Use SQL for payments and transactions
Use NoSQL for user activity and analytics
Use caching for performance

This approach allows systems to utilise the strengths of each model while minimising their weaknesses.

Companies like Amazon follow this strategy extensively, using different databases for different parts of their architecture.

The Real Decision Framework

Choosing between SQL and NoSQL is not about trends or popularity.

It comes down to understanding your system’s priorities.

Ask yourself:

Do I need strict consistency or can I tolerate eventual consistency?
Is my data highly structured or flexible?
Will my system scale vertically or horizontally?
Are relationships central to my queries?

These questions guide the decision more than any feature comparison ever could.

Final Thought

Databases are not just storage systems.

They define how your system thinks about data, how it evolves, and how it scales.

Choosing the right database is not about picking the most powerful tool:

It is about choosing the tool that aligns with your system’s reality.

Because in system design, the best solutions are not the most sophisticated ones.

They are the ones that fit the problem.

Caching in System Design - The Secret to High Performance

Sushant Gaurav — Tue, 16 Jun 2026 05:41:00 +0000

There is a point in every system’s growth where adding more servers stops being enough.

You scale horizontally. You introduce load balancers. You distribute traffic efficiently. And yet, something still feels off.

Requests are slower than expected. Databases are under constant pressure. Systems that should scale effortlessly begin to struggle under repeated work.

And then you realise something fundamental:

The system is doing the same work again and again.

The same queries.
The same computations.
The same responses.

Over and over.

This is not a scaling problem.

It is a redundancy problem.

And the solution to this problem is one of the most powerful ideas in system design:

Caching.

What Is Caching, Really?

At a surface level, caching is often defined as storing frequently accessed data in a faster storage layer so it can be retrieved quickly.

But this definition, while correct, does not capture its true significance.

Caching is not just a performance optimisation.

It is a shift in how systems think about work.

Instead of asking:

Can we compute this quickly?

Caching asks:

Do we need to compute this at all?

That shift from computation to reuse is what makes caching so powerful.

The Cost of Repeated Work

To understand why caching matters, we need to look at what happens without it.

Imagine a system where every user request requires:

Fetching data from a database
Performing business logic
Formatting a response

This process may take only a few milliseconds per request. But at scale, those milliseconds add up.

When thousands or millions of users request the same data, the system is forced to:

Execute identical database queries repeatedly
Perform the same computations
Generate the same responses

This creates unnecessary load on the system, especially on components like databases, which are often the most expensive and limited resources.

Over time, this repeated work becomes the bottleneck.

Caching addresses this by eliminating redundant effort.

The Core Idea - Store Once, Serve Many

At its heart, caching is simple.

When a request is processed, instead of discarding the result, the system stores it in a cache. The next time the same request arrives, the system can return the cached result instead of recomputing it.

This introduces two fundamental concepts:

Cache Hit - The data is found in the cache and returned immediately
Cache Miss - The data is not in the cache, so it must be computed and then stored

The effectiveness of a caching system is often measured by its cache hit rate.

Higher hit rate → fewer expensive operations → better performance.

Why Caching Changes Everything

Caching has a profound impact on system behaviour.

Reduced Latency

Fetching data from memory is significantly faster than querying a database or calling an external service.

Increased Throughput

By reducing the load on core systems, caching allows more requests to be handled simultaneously.

Lower System Load

Databases, APIs, and backend services experience less pressure, improving overall system stability.

Better Scalability

Systems can handle larger traffic without proportionally increasing infrastructure.

This is why caching is used extensively in large-scale systems.

Platforms like Netflix and Google rely heavily on caching at multiple layers to serve massive amounts of data efficiently.

Where Does the Cache Live?

One of the most important design decisions in caching is where to place the cache.

Because caching is not a single layer, it can exist at multiple points in the system.

Application-Level Cache

The simplest form of caching happens within the application itself.

Data is stored in memory inside the server process.

This is:

Extremely fast
Easy to implement

But it has limitations:

Not shared across servers
Lost when the server restarts

This works well for small-scale systems or single-node setups.

Distributed Cache

As systems scale horizontally, caching must also scale.

Instead of storing cache locally, systems use distributed caching systems that are shared across multiple servers.

This allows:

Consistent access to cached data
Better cache utilisation
Scalability across nodes

However, it introduces:

Network overhead
Cache synchronisation challenges

Edge Cache (CDN)

At the highest level, caching can move closer to the user.

Content Delivery Networks (CDNs) store cached data in geographically distributed locations.

When a user requests content, it is served from the nearest location rather than the origin server.

This drastically reduces latency and server load.

This is how platforms like Amazon and Netflix deliver content globally with high performance.

The Trade-off Begins

At this point, caching may seem like a perfect solution.

Faster responses.
Lower load.
Better scalability.

So why not cache everything?

Because caching introduces a new and unavoidable challenge:

Data can become stale.

When the underlying data changes, cached data may no longer be accurate.

And this leads us to one of the hardest problems in system design:

Cache invalidation.

But as systems grow, a deeper and more uncomfortable truth begins to emerge:

Caching is easy to add… but very hard to get right.

Because the moment you introduce a cache, you are no longer just optimising performance, you are managing two versions of reality:

The source of truth (database)
The cached copy (fast, but potentially outdated)

And keeping these two in sync is where the real challenge begins.

Cache Invalidation - The Hardest Problem

There’s a well-known saying in system design:

There are only two hard things in Computer Science: cache invalidation and naming things.

It sounds like a joke, but it isn’t.

Cache invalidation is the process of ensuring that cached data remains accurate when the underlying data changes.

Let’s say a product’s price changes in the database. If the old price is still stored in the cache, users may see outdated information.

So the system must decide:

When should the cache be updated?
Should it be updated immediately or later?
Should it be removed entirely?

Each choice comes with trade-offs between consistency, performance, and complexity.

Common Approaches to Cache Invalidation

There is no single correct way to handle cache invalidation. Instead, systems use different strategies depending on their requirements.

Time-Based Expiration (TTL)

One of the simplest approaches is to assign a time-to-live (TTL) to cached data.

After a fixed duration, the cache entry expires and is removed.

This approach is:

Easy to implement
Predictable
Widely used

But it has limitations.

If the TTL is too long:

Data may remain stale for too long

If the TTL is too short:

Cache effectiveness decreases (more misses)

So choosing the right TTL becomes a balancing act.

Write-Based Invalidation

Another approach is to update or invalidate the cache whenever data changes.

For example:

When a product is updated → update or delete its cache entry

This ensures better consistency, but introduces complexity:

Every write operation must handle cache updates
Failures in cache updates can lead to inconsistencies

This approach works well when accuracy is critical.

Explicit Invalidation

Sometimes, systems explicitly remove cache entries when they know data has changed.

Instead of updating the cache, they simply delete it, forcing the next request to fetch fresh data.

This is simple and safe, but may temporarily increase load due to cache misses.

Caching Strategies - When to Read and Write

Beyond invalidation, another important question is:

When should the system interact with the cache?

This leads to different caching strategies.

Cache-Aside (Lazy Loading)

This is the most commonly used strategy.

When a request arrives:

Check the cache
If data exists → return it (cache hit)
If not → fetch from database, store in cache, then return

This approach is:

Simple
Flexible
Widely adopted

But it can lead to stale data if not invalidated properly.

Write-Through Cache

In this strategy, data is written to both:

Cache
Database

simultaneously.

This ensures that the cache is always up-to-date.

However:

Writing becomes slower
More coordination is required

Write-Back (Write-Behind)

Here, data is first written to the cache, and the database is updated later.

This improves write performance but introduces risk:

If the cache fails before writing to the database, data may be lost

This strategy is used when performance is prioritised over immediate consistency.

Cache Eviction - Making Space for New Data

Caches are not infinite.

At some point, they run out of space.

So the system must decide:

Which data should be removed to make room for new data?

This is handled through eviction policies.

LRU (Least Recently Used)

Removes data that has not been accessed recently.

This works well because frequently accessed data tends to remain in the cache.

LFU (Least Frequently Used)

Removes data that is accessed the least often.

This is useful when certain data is consistently popular.

TTL-Based Eviction

Data is removed after a fixed time, regardless of usage.

Each policy reflects a different assumption about how users interact with data.

Choosing the right one depends on your access patterns.

When Caching Goes Wrong

Caching is powerful, but when misused, it can create serious problems.

Stale Data Issues

Users see outdated information, leading to inconsistencies.

Cache Stampede

When a popular cache entry expires, many requests hit the database simultaneously, overwhelming it.

Increased Complexity

Managing cache logic, invalidation, and consistency adds significant engineering overhead.

Hidden Bugs

Caching can mask underlying issues, making debugging harder.

The Deeper Insight

At this point, caching should no longer feel like a simple optimisation.

It is a trade-off system.

You trade:

Freshness for speed
Simplicity for performance
Consistency for scalability

And like everything in system design, there is no perfect choice.

Only the choice that best fits your requirements.

Final Thought

The most important thing to understand about caching is this:

Caching does not make your system faster; it makes your system do less work.

And at scale, doing less work is the only way to survive.

Because the systems that scale are not the ones that compute faster:

They are the ones that avoid unnecessary computation altogether.

Load Balancing Explained - How Systems Handle Millions of Requests

Sushant Gaurav — Tue, 02 Jun 2026 05:41:00 +0000

There is a moment in the life of every growing system when a single server quietly becomes a bottleneck.

At first, the system works exactly as expected. Users send requests, the server processes them, and responses are returned almost instantly. Everything feels smooth, predictable, and under control.

But as usage grows, something subtle begins to change.

Requests start arriving faster than they can be processed. The server becomes overloaded. Response times increase. Eventually, some requests begin to fail, not because the logic is incorrect, but because the system simply cannot keep up.

This is not a bug.

It is a limitation.

And this is the moment when adding more servers becomes necessary.

But adding more servers introduces a new challenge:

How do you decide which server should handle which request?

Because without coordination, adding more machines does not solve the problem; it just spreads the chaos.

This is where load balancing enters the picture.

What Is Load Balancing, Really?

At its core, load balancing is the process of distributing incoming requests across multiple servers so that no single machine becomes overwhelmed.

But that definition, while correct, does not capture the full picture.

Load balancing is not just about distribution.

It is about:

Maximising resource utilisation
Minimising response time
Ensuring high availability
Preventing system overload

It acts as the invisible layer that allows systems to scale horizontally without collapsing under pressure.

The Role of a Load Balancer

A load balancer sits between users and your servers.

Instead of users directly interacting with a specific machine, they send requests to the load balancer, which then decides where those requests should go.

From the user’s perspective, the system still appears as a single entity.

But behind the scenes, requests are being intelligently distributed across multiple machines.

This abstraction is powerful.

It allows you to:

Add or remove servers without affecting users
Handle traffic spikes dynamically
Improve fault tolerance

But the real complexity lies in how the load balancer makes decisions.

The Simplest Approach - Equal Distribution

The most intuitive way to distribute requests is to spread them evenly across all servers.

Each incoming request is sent to the next server in line.

This is known as round-robin distribution.

At first glance, this seems perfectly fair.

Every server gets an equal share of requests.

But real systems are rarely that simple.

Because not all requests are equal.

The Problem with Equal Distribution

In practice, different requests require different amounts of work.

Some requests might:

Query large datasets
Perform complex computations
Call multiple downstream services

While others may be simple and lightweight.

If all requests are distributed equally without considering their complexity, some servers may become overloaded while others remain underutilised.

This is where more intelligent load-balancing strategies come into play.

Smarter Distribution - Load-Aware Routing

Instead of blindly distributing requests, load balancers can make decisions based on the current state of each server.

For example:

Send requests to the server with the least number of active connections
Route traffic to the server with the lowest response time
Avoid servers that are showing signs of stress

This allows the system to adapt dynamically to changing conditions.

But this introduces a new requirement:

The load balancer must constantly monitor the health and performance of servers.

Health Checks - Knowing When a Server Is Failing

One of the most critical responsibilities of a load balancer is detecting when a server is no longer able to handle requests.

This is done through health checks.

The load balancer periodically sends requests to each server to verify:

Is the server responding?
Is it responding within an acceptable time?
Is it returning correct responses?

If a server fails these checks, it is temporarily removed from the pool.

This ensures that user requests are only sent to healthy servers, improving reliability and user experience.

Load Balancing and Availability

Load balancing plays a crucial role in achieving high availability.

Without it, even a horizontally scaled system can fail.

Imagine having multiple servers, but users are directly connected to just one of them. If that server goes down, the system becomes unavailable, even though other servers are perfectly functional.

A load balancer prevents this by acting as a single entry point, ensuring that traffic is always routed to available resources.

This is one of the reasons why large-scale systems, including those built by Netflix, rely heavily on load balancing at multiple layers.

Where This Is Heading

So far, we’ve explored:

Why load balancing is necessary
How it distributes traffic
Why simple strategies are not enough
How health checks ensure reliability

But as systems grow more complex, the role of a load balancer expands beyond simple request routing.

Because not all requests are the same.
Not all servers behave the same.
And not all failures look the same.

To handle real-world traffic at scale, load balancing itself evolves into a multi-layered system with different levels of intelligence.

Layer 4 vs Layer 7 - Two Ways to Think About Traffic

To understand modern load balancing, we need to look at the network stack; specifically, how requests move through it.

At a high level, load balancers operate at two common layers:

Layer 4 (Transport Layer)
Layer 7 (Application Layer)

These are not just technical distinctions; they define how much the load balancer understands about the request.

Layer 4 Load Balancing - Fast and Blind

A Layer 4 load balancer operates at the transport level (TCP/UDP). It does not inspect the content of the request. Instead, it makes decisions based on basic network information like:

IP address
Port number

From its perspective, a request is just a stream of packets.

This makes Layer 4 load balancing:

Extremely fast
Low overhead
Highly efficient for raw traffic distribution

But it also means it lacks context.

It cannot differentiate between:

/login vs /checkout
API calls vs static content
High-cost vs low-cost operations

It simply forwards traffic.

Layer 7 Load Balancing - Intelligent Routing

Layer 7 load balancers operate at the application level (HTTP/HTTPS). They can inspect the actual content of the request:

URL paths
Headers
Cookies
Request types

This allows for much smarter routing decisions.

For example:

Send /images requests to a caching server
Route /api/payments to a specific service
Direct mobile users to a different backend

This level of control is powerful, but it comes with trade-offs:

Higher latency than Layer 4
More computational overhead
Increased complexity

This leads to a common pattern in real systems:

Use Layer 4 for speed, Layer 7 for intelligence, often together.

Sticky Sessions - When State Gets in the Way

Earlier, we discussed how scalable systems aim to be stateless. But in practice, not all systems achieve this immediately.

Some applications rely on session state stored on individual servers.

This creates a problem.

If a user’s first request goes to Server 1, and their session data is stored there, sending their next request to Server 3 may break the experience.

To handle this, systems sometimes use sticky sessions.

Sticky sessions ensure that requests from the same user are always routed to the same server.

While this solves the immediate problem, it introduces new challenges:

Uneven load distribution
Reduced fault tolerance (if the server fails, the session is lost)
Difficulty scaling dynamically

This is why modern systems try to avoid sticky sessions and instead externalise state into shared systems like distributed caches.

Global Load Balancing - Scaling Across Regions

So far, we’ve discussed load balancing within a single region. But large-scale systems operate globally.

Users from different parts of the world expect fast response times. Sending every request to a single data centre is not practical.

This is where global load balancing comes into play.

Instead of routing traffic between servers, global load balancing routes traffic between regions.

The system decides:

Which region is closest to the user
Which region is currently healthy
Which region has capacity

This reduces latency and improves availability.

If one region goes down, traffic can be redirected to another.

This is how companies like Google and Amazon maintain global-scale systems that remain responsive even under failures.

Load Balancing Meets Caching and CDNs

As systems scale further, load balancing does not operate in isolation.

It works alongside:

Caching layers
CDNs (Content Delivery Networks)

A CDN can serve requests directly from edge locations, reducing the load on origin servers.

This introduces a powerful optimisation:

The best request is the one that never reaches your server.

By serving content closer to users, CDNs reduce latency, decrease server load, and improve scalability.

Load balancers then handle the remaining dynamic traffic efficiently.

The Deeper Insight

At this point, load balancing should no longer feel like a simple routing mechanism.

It is a control system that:

Distributes load
Detects failures
Optimises performance
Enables scalability

And more importantly, it connects multiple system design concepts:

Horizontal scaling
Availability
Latency optimization
Fault tolerance

This is why load balancing is often one of the first components introduced when systems begin to scale.

Final Thought

When systems are small, you think in terms of servers.

When systems grow, you think in terms of clusters.

But when systems reach scale, you think in terms of traffic flow.

Where requests come from, how they move, and where they are handled become the defining factors of system performance and reliability.

And at the centre of that flow is the load balancer.

Not just distributing requests, but shaping how the system behaves under pressure.

Redis Essentials: Architecture, Caching, and Setup

Sushant Gaurav — Tue, 26 May 2026 05:41:00 +0000

Redis is often a misunderstood tool in the backend developer's arsenal. While many view it simply as a "topic" to be covered in an hour, its role in modern system design is pivotal for building high-performance, scalable applications. This article explores what Redis is, why it is used, and how to set it up locally for development.

Understanding Redis: The In-Memory Powerhouse

At its core, Redis is an in-memory data store, often referred to as a "lightning-fast" hash map or key-value store. Unlike traditional databases like MongoDB or PostgreSQL that primarily store data on a hard disk (SSD or HDD), Redis keeps its state in the RAM (Random Access Memory).

Core Concept: The In-Memory Advantage

The fundamental difference between Redis and traditional databases (like MongoDB or PostgreSQL) is where they store data. While standard databases primarily use disk storage (SSDs/HDDs), Redis keeps its state in RAM (Random Access Memory).

Because RAM access is significantly faster than mechanical or electronic disk reads, Redis is often described as "lightning fast".

Architecture: The Caching Layer

In a typical application, Redis acts as an intermediary between the backend application and the primary database. This setup creates two primary scenarios:

Cache Hit: The backend finds the required data in Redis and returns it immediately to the user, bypassing the slower database.
Cache Miss: If the data isn't in Redis, the backend queries the primary database. It then stores a copy of this "hot record" in Redis for future requests before responding to the user.

This architecture dramatically reduces "read pressure" on the primary database, which should remain the "Source of Truth" for permanent records.

Key Features and Data Management

Persistence: Contrary to the myth that in-memory data is always lost on restart, Redis offers persistence features. It can load data from saved files back into memory upon a server reboot.
Key-Value Pairs: Redis stores data in simple pairs. Developers are encouraged to use human-readable, colon-separated keys (for example, user:session:123 or product:all) to avoid collisions and simplify debugging.
TTL (Time to Live): This is one of Redis's most powerful features. You can set an expiration time on a key (for example, 90 seconds). Once the time expires, Redis automatically deletes the record, ensuring the memory remains uncluttered.

Advanced Use Cases

Beyond simple data caching, Redis is used for:

Session Management: Storing user login states (Active/Inactive) across multiple distributed servers.
OTP Management: Holding temporary One-Time Passwords for a few minutes, they are valid.
Rate Limiting: Tracking IP addresses or user IDs to prevent abuse (for example, blocking a user for 10 minutes after too many failed login attempts).
Job Queues: Maintaining lists of background tasks. "Workers" (secondary backend applications) pull jobs from Redis to process time-consuming tasks like sending emails in batches.
Shared Counters: Tracking live metrics like page views or "likes" across various application instances.

Strategic Local Setup

For development, the sources recommend using Docker and Docker Compose to spin up a local environment. A standard configuration involves:

Redis Image: Using redis:7-alpine for a lightweight footprint.
Port Mapping: Binding the default Redis port 6379 to the host machine.
Persistence Command: Running the server with --appendonly yes to ensure data is written to a log.

In a Node.js environment, the ioredis library is the industry-standard package for communication. A basic connection is established by creating a new Redis client using the local URL: redis://localhost:6379. Developers can test the connection using the PING command, which should return a PONG response from the server.

When to Use Redis

Redis is not a solution for every problem. Use it if your application needs to:

Remove read pressure from the primary DB.
Manage rapidly expiring temporary data.
Handle background job queues or shared counters.

However, it is not a replacement for a primary database.

When NOT to use Redis

Redis is not a "magic bullet". It should not be used if you don't have a clear bottleneck or if your data doesn't fit the patterns described above. If you have a write-heavy application where data doesn't need to be read frequently, or if you are trying to use it as a primary database for complex relational data, Redis may not be the right solution.

Scalability in System Design - Vertical vs Horizontal Scaling

Sushant Gaurav — Tue, 19 May 2026 05:41:00 +0000

There comes a point in every system’s life where things start to break; not because the system is poorly designed, but because it is being used more than it was ever intended to handle.

At first, everything feels smooth. Requests are processed quickly, users are satisfied, and the system behaves predictably. But as usage grows, subtle changes begin to appear. Pages take longer to load. APIs respond more slowly. Databases struggle to keep up. Eventually, what once worked effortlessly starts becoming unreliable.

This is not a failure of design.

It is a signal.

The system has reached the limits of its current capacity.

And this is where scalability enters the conversation.

What Does Scalability Really Mean?

Scalability is often misunderstood as simply handling more users. But that definition is incomplete.

A system is considered scalable if it can handle increasing load without a proportional drop in performance or reliability.

Notice the nuance here.

It is not just about handling more requests - it is about doing so efficiently. A system that doubles its resources every time the load increases is not truly scalable; it is simply brute-forcing the problem.

True scalability is about growing intelligently.

And to achieve that, systems typically rely on two fundamental approaches:

Scaling up (vertical scaling)
Scaling out (horizontal scaling)

At a high level, both aim to solve the same problem: increasing capacity. But the way they approach it, and the consequences of those choices, are fundamentally different.

Vertical Scaling - Growing Taller

Vertical scaling, often referred to as scaling up, is the simpler and more intuitive approach.

Instead of changing the structure of the system, you make the existing machine more powerful.

You increase:

CPU
RAM
Disk capacity

In essence, you are upgrading the machine so it can handle more work.

From an engineering perspective, vertical scaling feels natural.

There is no need to redesign the system. The application continues to run as it always has, just on better hardware. Databases remain centralised. Communication patterns remain unchanged. There is no need to think about distribution, coordination, or synchronisation.

This simplicity is incredibly valuable - especially in the early stages of a system.

It allows teams to focus on building features rather than solving infrastructure complexity.

Why Vertical Scaling Works So Well (Initially)

In the early lifecycle of a product, vertical scaling often provides the fastest path to growth.

If your database is slowing down, you can upgrade it to a machine with more memory. If your application server is under load, you can increase its CPU capacity.

The system continues to function exactly as before, just with more headroom.

This is why many systems, including those built by companies like Instagram in their early days, start with vertically scaled architectures.

The benefits are clear:

Minimal architectural changes
Lower operational complexity
Faster implementation

For a small team trying to move quickly, this is often the most practical choice.

The Limits of Vertical Scaling

But like everything in system design, vertical scaling has limits.

The first limitation is physical.

A machine can only be upgraded to a certain extent. There is a maximum amount of CPU, memory, and storage you can add. Beyond that point, scaling up is no longer possible.

The second limitation is cost.

As machines become more powerful, their cost increases disproportionately. A machine that is twice as powerful is often significantly more than twice as expensive.

This leads to diminishing returns.

The third and perhaps most critical limitation is risk.

When your entire system depends on a single machine, that machine becomes a single point of failure.

If it goes down:

The entire system goes down

No matter how powerful the machine is, it cannot protect you from hardware failures, network issues, or unexpected crashes.

This is where the need for a different approach begins to emerge.

Horizontal Scaling - Growing Wider

Horizontal scaling, or scaling out, takes a fundamentally different approach.

Instead of making a single machine more powerful, you add more machines and distribute the workload among them.

Now, instead of relying on one powerful server, the system relies on multiple smaller servers working together.

This introduces a new concept: distribution of work.

Requests are no longer handled by a single machine. They are spread across multiple nodes, often using a load balancer that decides where each request should go.

At first, this might seem like a straightforward extension of vertical scaling. But in reality, it changes the nature of the system entirely.

Because the moment you introduce multiple machines, you introduce:

Network communication
Data synchronisation
Failure handling across nodes

In other words, you are stepping into the world of distributed systems.

A Shift in Complexity

Vertical scaling keeps complexity low but limits growth.

Horizontal scaling removes those limits but introduces a new kind of complexity.

This is not just an implementation detail; it is a fundamental shift in how systems are designed and reasoned about.

In a vertically scaled system:

There is one source of truth
Communication is local
Failures are simpler to reason about

In a horizontally scaled system:

Data may exist in multiple places
Communication happens over networks
Failures become partial and unpredictable

This is the same shift we saw earlier when moving from monolithic to distributed systems.

Because in many ways, horizontal scaling is what forces systems to become distributed.

Where This Is Heading

At this point, we’ve built the intuition:

Vertical scaling is simple, powerful, and limited
Horizontal scaling is flexible, scalable, and complex

But this is only the surface. To truly understand horizontal scaling, we need to answer a deeper question:

What actually happens to data and traffic when a system scales horizontally?

Because adding more machines is easy.

Making them work together correctly and efficiently is the real challenge.

Distributing Traffic - The Role of Load Balancing

The moment you introduce multiple servers, you need a way to decide:

Which request goes to which machine?

This is the job of a load balancer.

A load balancer sits between users and your servers, acting as a traffic controller. Instead of users directly hitting a specific server, their requests are routed through the load balancer, which distributes them across available machines.

At a surface level, this seems simple—just spread requests evenly. But in practice, it involves subtle decisions:

Should requests be distributed round-robin?
Should they go to the least loaded server?
Should user sessions stick to the same server?

These choices affect both performance and correctness.

And more importantly, they introduce a critical requirement:

Each server should be able to handle requests independently.

This leads to a key design principle in scalable systems:

Statelessness.

Stateless vs Stateful Systems

In a vertically scaled system, state is easy to manage. Since everything runs on a single machine, user sessions, data, and temporary state can be stored locally.

But in a horizontally scaled system, this approach breaks down.

If a user’s request goes to Server 1, and their next request goes to Server 3, that second server must still understand the user’s context.

This is why scalable systems aim to make application servers stateless.

Instead of storing session data locally, they store it in shared systems such as:

Databases
Distributed caches
External storage systems

This allows any server to handle any request, making load balancing effective.

But this shift pushes complexity elsewhere, into how data is stored and accessed.

Scaling Data - The Real Challenge

Handling more requests is only part of the problem.

The bigger challenge is handling more data.

In a vertically scaled system, data typically lives in a single database. As the load increases, you upgrade the database server. But just like application servers, databases have limits.

This is where horizontal scaling forces a fundamental shift in data strategy.

Two major approaches emerge:

Replication - Copying Data Across Nodes

Replication involves creating multiple copies of the same data across different machines.

This allows:

Multiple servers to read data simultaneously
Improved availability if one node fails

For example, one database node may handle writes, while multiple replicas handle read requests.

This improves throughput, but introduces consistency challenges—something we explored earlier in CAP.

Sharding - Splitting Data Across Nodes

Sharding takes a different approach.

Instead of copying data, it divides it.

Each server is responsible for a subset of the data:

User A–M on one server
User N–Z on another

This allows the system to scale almost indefinitely by adding more shards.

But it introduces new complexity:

How do you decide which shard stores which data?
What happens when data needs to move between shards?
How do you handle queries that span multiple shards?

Sharding improves scalability dramatically, but at the cost of operational and architectural complexity.

Availability - Why Horizontal Scaling Wins

One of the most powerful advantages of horizontal scaling is fault tolerance.

In a vertically scaled system, everything depends on a single machine. If it fails, the system goes down.

In a horizontally scaled system, failure becomes partial.

If one server crashes:

Other servers continue handling requests
The system degrades, but does not collapse

This is the foundation of high-availability systems.

It is also why companies like Netflix design their systems to run across multiple machines, zones, and even regions.

They assume failure will happen—and design systems that survive it.

Cost, Complexity, and Trade-offs

At this point, horizontal scaling may seem like the obvious choice.

But it comes with trade-offs that cannot be ignored.

Cost Dynamics

While horizontal scaling can start with cheaper machines, the total cost can grow quickly as you add more infrastructure, networking, and operational overhead.

Engineering Complexity

You now need to handle:

Distributed communication
Data consistency
Failures across nodes
Monitoring and observability

Debugging Challenges

A single request may pass through multiple machines. Debugging issues becomes significantly harder compared to a single-node system.

This leads to a critical insight:

Horizontal scaling solves scalability problems by introducing distributed systems complexity.

The Hybrid Reality

In practice, most systems do not rely purely on vertical or horizontal scaling.

They combine both.

A common approach is:

Scale vertically first (quick wins, low complexity)
Introduce horizontal scaling as limits are reached

For example:

Start with a powerful database server
Add read replicas as traffic grows
Eventually introduce sharding when needed

This gradual evolution allows systems to grow without unnecessary complexity early on.

The Deeper Insight

At its core, scalability is not about choosing between vertical and horizontal scaling.

It is about understanding when each approach makes sense.

Vertical scaling is about simplicity and speed.
Horizontal scaling is about resilience and long-term growth.

And the transition between them is one of the most important decisions in system design.

Because once you move toward horizontal scaling, you are no longer just scaling a system:

You are designing a distributed system.

Final Thought

Scalability is often seen as a technical challenge.

But in reality, it is a reflection of success.

Systems only need to scale when they are being used, when they are growing, when they matter.

And the way you scale them defines:

Their performance
Their reliability
Their future evolution

Because in the end, scalability is not just about handling more users:

It is about building systems that can grow without breaking.

CAP Theorem Explained Simply (And Why It Matters in Real Systems)

Sushant Gaurav — Tue, 05 May 2026 05:41:00 +0000

There is a moment in every system design journey where things stop feeling simple.

Until that point, systems seem manageable. You think in terms of databases, APIs, scaling strategies, maybe even caching layers. But then you encounter distributed systems in their true form—data spread across machines, services communicating over unreliable networks, failures happening in unpredictable ways.

And suddenly, a question emerges that is far more difficult than it first appears:

How do we ensure that all parts of a system behave correctly when they are no longer in the same place?

This is the question that gave rise to the CAP Theorem.

At first glance, CAP is often presented as a rule - something to memorize:

A distributed system can only guarantee two out of three: Consistency, Availability, and Partition Tolerance.

But this simplified statement, while technically correct, hides the deeper truth.

CAP is not just a rule.

It is a constraint imposed by reality.

To truly understand it, we need to go beyond definitions and step into the world where distributed systems actually operate.

The Reality of Distribution

In a monolithic system, everything runs within a single environment. Data is stored in one place, and operations happen in a predictable sequence. If you update a value, every part of the system immediately sees that update.

But in a distributed system, things are fundamentally different.

Data is no longer centralized. It is spread across multiple nodes—possibly across regions, continents, or even different cloud providers. These nodes communicate over a network, and that network is not perfect.

Messages can be delayed.
Packets can be lost.
Connections can break.

And when that happens, parts of the system can no longer talk to each other.

This situation is known as a network partition.

Partition Tolerance — The Unavoidable Reality

Partition tolerance refers to a system’s ability to continue functioning even when communication between nodes is disrupted.

And here’s the critical insight:

In distributed systems, partitions are not optional—they are inevitable.

You cannot design a real-world distributed system and assume that the network will always be reliable. Sooner or later, something will fail.

This means that partition tolerance is not a choice you make.

It is a condition you must accept.

Once you accept this, the CAP theorem becomes much clearer.

Because now the real question is:

When a partition happens, what do you prioritize - consistency or availability?

Consistency — One Truth Across the System

Consistency, in the context of CAP, means that all nodes see the same data at the same time.

If a user updates a piece of data, any subsequent read—no matter which node it comes from—should return that updated value.

There is a single, unified truth.

This is straightforward in a centralized system. But in a distributed system, maintaining this guarantee requires coordination between nodes.

When a write happens, all replicas must agree on the updated value before it is considered complete.

This coordination takes time. And during a network partition, it may not be possible at all.

So if you insist on strong consistency, the system must sometimes refuse to respond rather than risk returning incorrect data.

Availability — Always Responding

Availability means that every request to the system receives a response.

It does not necessarily mean the response is correct or up-to-date—only that the system does not fail to respond.

In highly available systems, the priority is to keep the system operational, even under failure conditions.

This often means allowing different nodes to respond independently, even if they do not have the latest data.

The system continues to function, but it may temporarily serve inconsistent data.

The Core Trade-off

Now we arrive at the heart of CAP.

When a network partition occurs, you are forced to make a decision:

If you choose consistency, you may have to reject requests to ensure correctness.
If you choose availability, you may return outdated or inconsistent data.

You cannot guarantee both at the same time.

This is the essence of the CAP theorem.

It is not about picking any two out of three in general conditions. It is about what happens during a partition.

And since partitions are inevitable, this trade-off is unavoidable.

Why This Matters More Than You Think

At this point, CAP might seem like an abstract concept. But in reality, it influences almost every large-scale system you interact with.

When you see slightly outdated data on a social media feed, that is a system choosing availability over strict consistency.

When a payment system refuses to process a transaction until it confirms the latest state, that is a system prioritizing consistency over availability.

These are not accidental behaviours. They are deliberate design choices shaped by CAP.

Companies like Amazon often design different parts of their systems with different priorities. For example, product catalogues may favour availability, while payment systems enforce strict consistency.

This highlights an important idea:

CAP is not applied to an entire system uniformly; it is applied at the level of individual components.

Now the natural question becomes:

What kinds of systems make which choices? And how do real-world architectures actually deal with this trade-off?

To answer that, we need to look at how CAP is commonly categorized in practice.

The Three CAP Categories (And the Truth Behind Them)

CAP is often explained using three system types:

CP (Consistency + Partition Tolerance)
AP (Availability + Partition Tolerance)
CA (Consistency + Availability)

At first glance, this looks like a clean classification. But there is a subtle—and very important—truth hidden here.

In real distributed systems, CA is not actually achievable.

Why?

Because partition tolerance is not optional.

If your system is distributed, you cannot ignore the possibility of network failures. And the moment you accept partitions as inevitable, you are always operating in the world of P (Partition Tolerance).

So in practice, the real trade-off is always:

CP vs AP

CP Systems — Choosing Consistency Over Availability

A CP system prioritizes correctness above all else.

When a partition occurs, and nodes cannot communicate reliably, the system chooses to reject or delay requests rather than risk returning inconsistent data.

This often means:

Some parts of the system become temporarily unavailable
Users may experience errors or delays
But the data remains correct and trustworthy

This approach is critical in systems where correctness is non-negotiable.

Think about financial transactions. If your bank shows two different balances depending on which server you hit, the system is fundamentally broken.

This is why systems dealing with payments, inventory management, or critical state often lean toward CP.

For example, services within Google that require strict coordination (like distributed databases with strong guarantees) are designed to favor consistency, even if it means temporarily sacrificing availability.

AP Systems — Choosing Availability Over Consistency

AP systems take the opposite approach.

When a partition occurs, they continue to serve requests no matter what, even if that means returning stale or inconsistent data.

This results in:

High availability
Faster response times during failures
Temporary inconsistencies across nodes

But here’s the key: these inconsistencies are not permanent.

AP systems rely on a concept called eventual consistency, where all nodes will converge to the same state once the network stabilizes.

This model works well for systems where perfect accuracy at every moment is not required.

For example, platforms like Facebook prioritize keeping the platform responsive. If your feed shows a slightly outdated like count for a few seconds, it does not break the user experience.

The system favours availability and responsiveness over strict consistency.

Why CA Systems Don’t Really Exist

It is tempting to think that some systems can achieve both consistency and availability.

And technically, in systems that are not distributed, this is true.

A single-node database can provide both:

Immediate consistency
Always-available responses (as long as the node is up)

But the moment you distribute the system across multiple nodes, the network becomes a factor.

And once the network becomes a factor, partitions become inevitable.

So any system that claims to be CA is either:

Not truly distributed
Or quietly sacrificing partition tolerance (which is unrealistic in real-world systems)

Real Systems Don’t Pick One Side Completely

Here’s where things get even more interesting.

Real-world systems rarely choose to be purely CP or purely AP.

Instead, they mix and match based on the needs of different components.

For example, in a large system like Amazon:

The shopping cart might be AP
(you can still add items even if some nodes are out of sync)
The payment system is CP
(transactions must be accurate and consistent)
The product catalog might lean toward AP
(slight delays in updates are acceptable)

This layered approach allows systems to optimize for different trade-offs depending on the context.

This is a critical mindset shift:

CAP is not a system-wide decision. It is a per-component design choice.

How Modern Systems Soften the Trade-off

While CAP defines a hard constraint, modern systems use clever techniques to reduce the pain of the trade-off.

They cannot eliminate it—but they can make it less noticeable.

Eventual Consistency with Conflict Resolution

AP systems often allow temporary inconsistencies but resolve them later using strategies like:

Last write wins
Version vectors
Conflict-free data structures

Retries and Idempotency

Systems retry failed requests intelligently, ensuring that operations can be safely repeated without corrupting data.

Graceful Degradation

Instead of failing completely, systems reduce functionality under stress:

Showing cached data
Disabling non-critical features

Geo-Partitioning

Data is partitioned geographically so that most operations happen locally, reducing the impact of global partitions.

Common Misconceptions About CAP

Even experienced engineers sometimes misunderstand CAP. Let’s clear up a few common myths.

You can choose any two at any time

No — the trade-off only matters during a partition.

AP systems don’t care about consistency

They do — they just relax when consistency is achieved.

CP systems are always better

Not necessarily — they can lead to poor user experience during failures.

CAP is outdated

Not at all — it is still one of the most fundamental constraints in distributed systems.

The Real Lesson of CAP

CAP is not about memorizing three letters.

It is about understanding this deeper truth:

In distributed systems, failure forces you to make trade-offs.

And those trade-offs are not bugs.

They are design decisions.

The best system designers are not the ones who avoid trade-offs—they are the ones who choose them wisely, based on the needs of the system.

Final Thought

If you truly understand CAP, you start seeing systems differently.

You begin to ask better questions:

What happens when this service cannot reach another service?
Is it better to fail or return stale data?
Where can we tolerate inconsistency, and where can we not?

And those questions lead to better designs.

Because at scale, systems are not defined by how they behave when everything works—

They are defined by how they behave when things break.