This is Part 4 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
What You'll Learn
By the end of this article, you'll understand:
- Why adding another server creates a new challenge.
- What a Load Balancer is and why it exists.
- How a Load Balancer distributes incoming requests.
- Why users never know multiple servers are serving their requests.
- The difference between Vertical Scaling and Horizontal Scaling.
In the previous article, we learned why applications eventually become slow as more users start using them. Every request consumes CPU time and RAM, and although upgrading to a more powerful server can temporarily solve the problem, every machine has physical and financial limits.
This naturally leads to an important question.
If one server is no longer enough, what should we do next?
At first, the answer seems obvious.
Instead of running your application on one server, why not run it on two?
Imagine your application currently looks like this.
Users
│
▼
+-------------+
| Server |
+-------------+
If this server can no longer handle the traffic, you could simply add another one.
+-------------+
| Server A |
+-------------+
+-------------+
| Server B |
+-------------+
This immediately gives your application more CPU power, more RAM, and more processing capacity. In theory, your application should now be able to serve far more users than before.
It sounds like the perfect solution.
However, adding another server introduces a completely new problem.
Imagine you open your browser and visit your application.
Your browser sends an HTTP request exactly as it always has.
But this time, there isn't just one server waiting to receive it.
There are two.
So which one should handle your request?
Should it go to Server A?
Or should it go to Server B?
Your browser has no idea.
It only knows one thing: the address of your website.
It doesn't know how many servers are running behind the scenes, where they are located, or which one is currently free to process the next request.
In other words, adding more servers solves one problem but creates another.
Someone now needs to decide where every incoming request should go.
That "someone" is called a Load Balancer.
--
Meet the Load Balancer
A Load Balancer sits between your users and your servers.
It doesn't store your application's data.
It doesn't execute your business logic.
It doesn't communicate with the database.
Instead, it has one very specific responsibility.
It decides which server should handle each incoming request.
To understand this, imagine a busy hospital.
Patients keep arriving throughout the day.
Some need to visit a general physician, while others need to see a specialist. If every patient walked directly into a doctor's room without any coordination, some doctors would become overwhelmed while others might remain idle.
Instead, hospitals usually have a reception desk.
When a patient arrives, the receptionist asks a few questions and then directs them to the most appropriate doctor.
The receptionist doesn't treat the patient.
The receptionist simply decides where the patient should go.
A Load Balancer performs a very similar role.
Whenever a user opens your application, the browser doesn't send the request directly to one of your servers anymore. Instead, the request first reaches the Load Balancer.
The Load Balancer quickly examines the incoming request and decides which server should process it.
The overall flow now looks something like this.
User
│
▼
+------------------+
| Load Balancer |
+------------------+
│ │
▼ ▼
+----------+ +----------+
| Server A | | Server B |
+----------+ +----------+
Suppose three users visit your website at almost the same time.
Instead of sending all three requests to Server A, the Load Balancer can distribute them across multiple servers.
User 1 ─────────► Server A
User 2 ─────────► Server B
User 3 ─────────► Server A
As a result, neither server has to handle all of the traffic by itself.
The workload is shared.
This is where the name Load Balancer comes from.
It attempts to balance the workload across multiple servers so that no single machine becomes overwhelmed while others remain underutilized.
One interesting thing happens here.
From the user's perspective, absolutely nothing has changed.
The user still visits the same website.
They still type the same URL into their browser.
They still receive the same response.
The user has no idea whether the application is running on one server, two servers, or even hundreds of servers spread across different parts of the world.
All of that complexity is hidden behind the Load Balancer.
This is one of the reasons why Load Balancers are such an important part of modern web architecture.
They allow engineers to increase the capacity of an application without changing the way users interact with it.
As traffic grows, new servers can be added behind the Load Balancer, and users continue accessing the application exactly as they always have.
Of course, this raises another interesting question.
If there are multiple servers available, how does the Load Balancer decide which one should receive the next request?
Does it simply alternate between them?
Does it choose a server randomly?
Or is there a smarter strategy?
Let's find out.
--
How Does the Load Balancer Decide?
At this point, we know that a Load Balancer receives every incoming request before it reaches one of our servers.
But another question naturally follows.
How does it decide which server should receive the next request?
At first, you might think the decision is random.
After all, if two identical servers are running the same application, does it really matter which one handles the request?
Sometimes it doesn't.
However, as applications become larger and traffic becomes more unpredictable, choosing the right server can make a noticeable difference in performance.
Over the years, engineers have developed several strategies for distributing requests. Let's look at some of the most common ones.
1. Round Robin
The simplest strategy is called Round Robin.
The idea is straightforward.
Instead of sending every request to the same server, the Load Balancer simply takes turns.
Request 1 ─────► Server A
Request 2 ─────► Server B
Request 3 ─────► Server A
Request 4 ─────► Server B
Every new request goes to the next server in the list.
If there are three servers, the pattern becomes:
Request 1 ─────► Server A
Request 2 ─────► Server B
Request 3 ─────► Server C
Request 4 ─────► Server A
This approach is easy to implement and works surprisingly well when all of the servers have similar hardware and each request requires roughly the same amount of work.
However, real-world applications are rarely that predictable.
Some requests might take only a few milliseconds, while others could require several seconds to complete.
If one server is still busy handling long-running requests, blindly sending it another request may not be the best decision.
This leads us to another strategy.
2. Least Connections
Instead of taking turns, the Load Balancer can look at how busy each server currently is.
Suppose the situation looks like this.
Server A → 120 Active Requests
Server B → 38 Active Requests
If a new request arrives, sending it to Server B is usually the better choice because it has fewer active connections.
This strategy is called Least Connections.
Rather than following a fixed pattern, the Load Balancer continuously tries to distribute traffic based on the current workload of each server.
As traffic becomes more dynamic, this often leads to better resource utilization.
3. Random Selection
Believe it or not, some systems simply choose a server at random.
Although this sounds inefficient, randomness often produces a surprisingly balanced distribution when the number of requests is very large.
For smaller applications, this simple approach can perform remarkably well without requiring the Load Balancer to constantly monitor every server.
The important thing to remember isn't the names of these algorithms.
It's the idea behind them.
Every Load Balancer is trying to answer the same question.
"Which server is in the best position to handle this request right now?"
Different applications answer that question in different ways, but the objective is always the same.
Keep the workload balanced so that no single server becomes overwhelmed while others remain underutilised.
--
Horizontal Scaling: Growing by Adding More Servers
Earlier in this series, we explored a technique called Vertical Scaling. When a server began running out of CPU power or memory, we upgraded it to a more powerful machine.
That approach works well for many applications, especially during the early stages of development.
However, as we learned in the previous article, every server eventually reaches its limits. At some point, buying a larger machine becomes too expensive, technically impossible, or simply no longer provides enough performance to justify the cost.
This is where engineers start thinking differently.
Instead of asking,
"How can we make this server bigger?"
they begin asking,
"Why should one server do all the work?"
Imagine your application is receiving ten thousand requests every second.
Instead of expecting one powerful server to handle every request, you deploy four identical servers, each running the same application.
Your architecture now looks like this.
Users
│
▼
+------------------+
| Load Balancer |
+------------------+
│ │ │ │
▼ ▼ ▼ ▼
+--------+ +--------+ +--------+ +--------+
|Server A| |Server B| |Server C| |Server D|
+--------+ +--------+ +--------+ +--------+
Rather than processing every request on a single machine, the workload is now shared across four servers.
Each server handles only a portion of the incoming traffic.
As a result, CPU usage and memory consumption are spread across multiple machines instead of being concentrated on one.
This approach is called Horizontal Scaling.
Unlike Vertical Scaling, where we improve a single server, Horizontal Scaling increases the overall capacity of an application by adding more servers.
This idea offers an important advantage.
Suppose your application suddenly becomes twice as popular.
With Vertical Scaling, your first instinct might be to replace your server with an even larger one.
With Horizontal Scaling, you often don't need to replace anything.
Instead, you simply add another server behind the Load Balancer.
The Load Balancer immediately begins sending some of the incoming requests to the new machine, reducing the workload on the existing servers.
From the user's perspective, nothing changes.
The website still has the same URL.
The application behaves exactly as before.
Most users never realize that their requests are being handled by different servers behind the scenes.
This ability to increase capacity by adding more machines is one of the reasons Horizontal Scaling has become the preferred approach for modern cloud applications.
Companies like Netflix, Amazon, Spotify, and Instagram don't rely on a single enormous server.
Instead, they rely on many servers working together as a single system.
Of course, introducing multiple servers creates another challenge.
What happens if one of those servers suddenly crashes?
Should the Load Balancer continue sending requests to it?
Or should it detect the failure and automatically redirect traffic somewhere else?
As it turns out, modern Load Balancers are capable of much more than simply distributing requests.
They also help ensure that your application remains available even when individual servers fail.
--
Wrapping Up
At the beginning of this article, we asked a simple question.
What happens when one server is no longer enough?
At first, adding another server sounded like the perfect solution. More servers mean more CPU power, more memory, and the ability to handle more users.
However, adding multiple servers introduced a new problem.
If several servers are running the same application, who decides where each request should go?
That's exactly the problem a Load Balancer solves.
Rather than allowing users to connect directly to your servers, every request first passes through the Load Balancer. It quietly decides which server should process the request, ensuring that the workload is shared instead of overwhelming a single machine.
We also learned that not every request has to be routed in the same way. Some Load Balancers simply alternate between servers, while others make decisions based on which machine is currently handling the fewest active requests. Regardless of the strategy, the objective remains the same—keep the workload balanced and make the best possible use of the available servers.
Finally, we introduced one of the most important ideas in modern System Design.
Instead of continuously upgrading a single server, we can increase an application's capacity by adding more servers.
This approach is known as Horizontal Scaling, and it forms the foundation of almost every large-scale application running on the internet today.
At this point, our architecture has evolved considerably.
We started with a single server.
Users
│
▼
+-------------+
| Server |
+-------------+
Today, our application looks more like this.
Users
│
▼
+------------------+
| Load Balancer |
+------------------+
│ │ │
▼ ▼ ▼
+--------+ +--------+ +--------+
|Server A| |Server B| |Server C|
+--------+ +--------+ +--------+
Our application can now serve far more users than before.
But introducing multiple servers creates another challenge.
Imagine Server B suddenly crashes.
Should the Load Balancer continue sending users to a server that is no longer responding?
How does it know that a server has failed?
And if one server goes offline, how can the application continue working without users even noticing?
Those questions introduce another important concept in System Design: High Availability.
We'll explore exactly how modern systems stay online—even when individual servers fail—in the next part of this series.
See you in Part 5.
Top comments (0)