Load Balancing: How Do Large Systems Distribute Traffic?

---

Imagine millions of users accessing your application at the same time.

If every request goes to a single server:

Users
  ↓
One Server 🔥

Eventually, that server becomes overloaded.

Requests become slower. Some may fail. And if the server goes down, the entire application can become unavailable.

So how do large systems handle increasing traffic?

One of the fundamental building blocks is load balancing.

A load balancer distributes incoming traffic across multiple servers so that no single server has to handle all the workload.

But load balancing is more than simply sending requests to different servers.

Let's understand how it works and why it matters in scalable system design.

The Basic Architecture

A simple load-balanced architecture looks like this:

                 Users
                   ↓
            Load Balancer
                   ↓
        ┌──────────┼──────────┐
        ↓          ↓          ↓
     Server 1   Server 2   Server 3
        │          │          │
        └──────────┼──────────┘
                   ↓
                Database

Instead of depending on one application server, we run multiple instances and distribute incoming requests between them.

This provides several benefits:

• ⚡ Higher request-handling capacity
• 📈 Horizontal scalability
• 🛡️ Better availability
• 🔄 Fault tolerance

But how does the load balancer decide which server should receive a request?

How Does a Load Balancer Choose a Server?

A load balancer can use different algorithms depending on the workload.

1. Round Robin

Requests are distributed sequentially:

Request 1 → Server 1
Request 2 → Server 2
Request 3 → Server 3
Request 4 → Server 1
Request 5 → Server 2

This works well when servers have similar capacity and requests have relatively similar processing costs.

It is simple, predictable, and easy to implement.

2. Weighted Round Robin

What if the servers don't have the same capacity?

For example:

Server 1 → 20%
Server 2 → 30%
Server 3 → 50%

A more powerful server can receive more traffic.

This is useful when your infrastructure contains machines with different resource capacities.

3. Least Connections

Instead of distributing requests in a fixed sequence, the load balancer sends new connections to the server currently handling the fewest active connections.

For example:

Server 1 → 120 connections
Server 2 → 80 connections
Server 3 → 45 connections

The next connection would likely go to Server 3.

This can be useful when requests or connections have significantly different lifetimes.

4. IP Hash

The client's IP address can be used to determine which server receives the request.

Conceptually:

Client IP
   ↓
  Hash
   ↓
Server Selection

This can provide a form of session affinity, where requests from the same client tend to reach the same server.

However, relying heavily on session affinity can make scaling and failover more complicated, so stateless designs are often preferred when practical.

What Happens When a Server Fails?

This is one of the most important features of load balancing.

A load balancer can perform health checks on backend servers.

For example:

Server 1 → Healthy ✅
Server 2 → Failed ❌
Server 3 → Healthy ✅

If Server 2 stops responding correctly, the load balancer can remove it from the active pool.

Traffic is then routed to the healthy servers:

                 Users
                   ↓
            Load Balancer
              ↙         ↘
         Server 1      Server 3
             ✅            ✅

         Server 2 ❌

Users may continue using the application without knowing that one server has failed.

This is one reason load balancing improves availability and fault tolerance.

Load Balancing + Auto Scaling

Load balancing becomes even more powerful when combined with auto scaling.

Imagine traffic gradually increasing:

Traffic ↑
   ↓
Load Balancer
   ↓
More Servers Added
   ↓
Traffic Distributed

During low traffic:

             Load Balancer
                  ↓
               Server 1

During high traffic:

             Load Balancer
                  ↓
           ┌──────┼──────┐
           ↓      ↓      ↓
          S1     S2     S3

When demand increases, additional application instances can be added.

When demand decreases, unnecessary instances can be removed.

The load balancer continues distributing traffic across the available instances.

This is a key pattern behind elastic and horizontally scalable systems.

Layer 4 vs Layer 7 Load Balancing

Load balancing can happen at different layers of the network stack.

Two common approaches are Layer 4 and Layer 7.

Layer 4

Layer 4 load balancing operates using transport-level information such as:

• IP address
• TCP/UDP
• Port

It does not need to understand the details of the HTTP request.

Because of this, it can be relatively lightweight and fast.

Layer 7

Layer 7 load balancing operates at the application layer and can understand information such as:

• URL path
• Hostname
• HTTP headers
• Cookies

For example:

/api/*       → API Servers

/images/*    → Image Servers

/admin/*     → Admin Servers

This allows the load balancer to make routing decisions based on the actual application request.

The choice between Layer 4 and Layer 7 depends on the requirements and routing needs of the system.

A Load Balancer Doesn't Solve Every Bottleneck

Here's an important system-design lesson.

Suppose we build:

Users
  ↓
Load Balancer
  ↓
Servers
  ↓
Database 🔥

We now have 20 application servers.

But the database can process only 2,000 operations per second.

Adding more application servers won't magically make the database faster.

The database is still the bottleneck.

You may need a different solution, such as:

• Better database queries
• Indexing
• Caching
• Read replicas
• Partitioning
• Reducing unnecessary database operations

This leads to a fundamental principle:

Before scaling a component, identify what is actually limiting the system.

Adding infrastructure everywhere is not the same as designing a scalable system.

What About Sessions?

Load balancing becomes much easier when application servers are stateless.

Consider this architecture:

Server 1
   ↓
Local Session

If a user's next request goes to Server 2, Server 2 may not have access to that session.

This creates a problem.

Instead, shared state can be stored in an external system when appropriate:

Server 1 ─┐
Server 2 ─┼──→ Shared Session Store
Server 3 ─┘

Now any application server can handle the user's request.

This makes horizontal scaling easier because requests don't need to be tied to one particular server.

Putting It All Together

A larger application might eventually look something like this:

                         Users
                           ↓
                          CDN
                           ↓
                    Load Balancer
                           ↓
              ┌────────────┼────────────┐
              ↓            ↓            ↓
           API 1         API 2         API 3
              └────────────┼────────────┘
                           ↓
                         Cache
                           ↓
                        Database

Each component solves a different problem:

• CDN → serves content closer to users
• Load Balancer → distributes traffic
• Application Servers → process requests
• Cache → reduces repeated expensive operations
• Database → stores persistent data

The load balancer's job is relatively simple:

Distribute traffic efficiently and keep unhealthy servers out of the request path.

But that simple responsibility becomes extremely important as the system grows.

Common Load Balancing Mistakes

1. Adding a Load Balancer Too Early

A small application running on a single server may not need one.

Start with the simplest architecture that satisfies your requirements.

2. Assuming It Fixes Every Performance Problem

A load balancer cannot fix:

• Slow database queries
• Inefficient application code
• Memory leaks
• Slow external APIs

It distributes traffic. It doesn't automatically make every component faster.

3. Ignoring Health Checks

If unhealthy servers remain in the routing pool, users can continue receiving failed requests.

4. Creating a Single Point of Failure

If your entire architecture depends on one load balancer instance, that load balancer itself can become a failure point.

Large systems typically consider redundancy at this layer as well.

5. Ignoring Application State

Stateful application servers can make horizontal scaling more complicated.

Understand where sessions and other shared state live before adding more servers.

The Bigger System Design Lesson

Load balancing is not just about distributing requests.

It represents a broader system-design idea:

When one component becomes a limitation, distribute the workload across multiple components.

We move from:

Users
  ↓
One Server

to:

Users
  ↓
Load Balancer
  ↓
Multiple Servers

But don't stop there.

Ask:

• What happens when traffic increases?
• What happens when a server fails?
• What happens when the database becomes the bottleneck?
• What happens when one region goes down?
• Can we add more servers easily?
• How do we know when the system is overloaded?

These questions lead to the larger world of scalable system design.

Conclusion

Load balancing is one of the fundamental building blocks of scalable applications.

It allows us to distribute incoming traffic across multiple servers, remove unhealthy instances from the request path, and increase application capacity by adding more servers.

But the most important lesson is not:

"Use a load balancer."

It is:

"Understand why you need one."

When traffic grows, distribute the workload.

When servers fail, route around them.

When demand increases, add capacity.

And when performance problems appear, find the actual bottleneck before adding more infrastructure.

That's the mindset behind good system design.