Imagine opening a popular website during a major event.
Thousands—or even millions—of users may be requesting pages, loading images, submitting forms, calling APIs, and accessing data at the same time.
If all those requests were sent to a single server, the server would eventually become overloaded.
So how do large-scale applications handle this traffic?
One of the key pieces of the architecture is the Load Balancer.
The Problem: One Server Isn't Enough
Consider a simple application:
Users
↓
Application Server
Every request reaches the same server.
As traffic increases, that server has to handle more:
- CPU usage
- Memory consumption
- Network traffic
- Database connections
- Concurrent requests
Eventually, the server becomes a bottleneck.
And if that server goes down, the application goes down with it.
This creates two major problems:
Scalability and availability.
The Solution: Multiple Servers
Instead of relying on one server, we can run multiple instances of the application.
┌── Server 1
Users → Load ├── Server 2
├── Server 3
└── Server 4
Now the workload can be distributed across multiple servers.
But there's a problem.
How does the system decide which server should receive each request?
That's where the load balancer comes in.
What Is a Load Balancer?
A load balancer is a component that sits between clients and application servers.
Instead of users directly communicating with individual servers, they communicate with the load balancer.
Client
↓
Load Balancer
↓
Application Servers
The load balancer receives incoming requests and decides where each request should go.
Its job isn't simply to "split traffic."
It can also monitor server health, route requests, handle failures, and help applications scale horizontally.
How Does It Distribute Requests?
There are different load-balancing algorithms.
1. Round Robin
Requests are distributed sequentially.
Request 1 → Server 1
Request 2 → Server 2
Request 3 → Server 3
Request 4 → Server 1
Request 5 → Server 2
It's simple and works well when servers have similar capacity.
2. Least Connections
The load balancer sends the request to the server currently handling the fewest active connections.
Server 1 → 100 connections
Server 2 → 45 connections
Server 3 → 70 connections
The next request would likely go to Server 2.
This can be useful when requests have different processing times.
3. Weighted Routing
Not every server necessarily has the same capacity.
For example:
Server 1 → 50%
Server 2 → 30%
Server 3 → 20%
A more powerful server can therefore receive a larger share of the traffic.
But What If a Server Crashes?
This is where load balancers become particularly valuable.
Suppose we have:
Server 1 → Healthy
Server 2 → Healthy
Server 3 → Failed
The load balancer can perform health checks on the servers.
If Server 3 stops responding correctly, the load balancer can stop sending new requests to it.
Traffic can continue through the healthy servers.
┌── Server 1 ✓
Users → Load ────┼── Server 2 ✓
└── Server 3 ✗
The user may never know that one of the servers failed.
This improves availability and fault tolerance.
Horizontal Scaling
Load balancing becomes especially powerful when combined with horizontal scaling.
Instead of making one server increasingly powerful:
Small Server
↓
Bigger Server
↓
Even Bigger Server
we can add more servers:
Server 1
Server 2
Server 3
Server 4
Server 5
This is called horizontal scaling or scaling out.
When traffic increases, additional application instances can be added behind the load balancer.
When traffic decreases, unnecessary instances can be removed.
This approach is fundamental to many cloud-native architectures.
What Happens During a Traffic Spike?
Imagine an application normally receives:
10,000 requests per minute
Then suddenly a major event causes:
500,000 requests per minute
A single server may struggle to handle that sudden increase.
With a scalable architecture, additional application instances can be created and placed behind the load balancer.
┌── Server 1
├── Server 2
Users → Load Balancer ───┼── Server 3
├── Server 4
├── Server 5
└── Server 6
The load balancer distributes incoming traffic across the available instances.
This is one of the fundamental ideas behind highly scalable applications.
Is the Load Balancer the Only Thing That Makes an Application Scalable?
No.
This is an important distinction.
A load balancer can distribute traffic, but it doesn't magically make the entire application scalable.
Other components may also become bottlenecks:
- Database
- Cache
- Message queues
- External APIs
- Network
- Storage
- Application code
For example, you could have 100 application servers but still have a database that can handle only a fraction of the traffic.
The architecture therefore needs to be designed as a complete system.
Where Does a Load Balancer Fit?
A simplified modern architecture might look like this:
Users
↓
DNS
↓
Load Balancer
↓
Application Servers
↓
Cache / Message Queue
↓
Database
In larger cloud-native systems, this architecture can become much more sophisticated, with components such as:
- API Gateways
- CDNs
- Kubernetes
- Service Meshes
- Auto Scaling
- Distributed Caches
- Message Brokers
- Multiple Database Instances
The load balancer is one important piece of the larger architecture.
Load Balancing Isn't Just About Performance
One common misconception is that load balancers exist only to make applications faster.
Their role is much broader.
They can help provide:
Scalability
Distribute traffic across multiple application instances.
High Availability
Continue serving requests when individual servers fail.
Fault Tolerance
Detect unhealthy instances and route traffic elsewhere.
Traffic Management
Control how requests are distributed across infrastructure.
Horizontal Scaling
Allow applications to grow by adding more instances.
The Bigger Picture
When you visit a large website, you usually don't know which physical or virtual server handled your request.
And that's exactly the point.
The infrastructure is designed so that the application can continue operating even while traffic changes, servers fail, and new instances are added.
A simple request:
User → Website
can actually travel through a much larger system:
User
↓
DNS
↓
Load Balancer
↓
Application Instance
↓
Cache / Services
↓
Database
↓
Response
The user sees a website.
Behind the scenes, an entire distributed system may be working to deliver that single response.
Final Takeaway
A load balancer isn't simply a traffic distributor.
It's an important building block for designing applications that can scale, survive failures, and handle unpredictable traffic.
The real power comes when load balancing is combined with horizontal scaling, health checks, caching, databases, auto-scaling, and resilient application design.
That's how modern systems move from:
"One server handles everything."
to:
"The system can adapt as traffic changes."
And that shift is one of the foundations of scalable software architecture.
What other system-design concept would you like to see explained this way?
Top comments (0)