Imagine you have built a backend API that works perfectly.
You deploy it to a server, connect your database, and everything is running smoothly.
Then your application becomes popular.
Instead of 100 requests per minute, you're suddenly handling 10,000. Then 100,000.
Your single server now has too much work to handle.
It becomes slow.
Eventually, it crashes.
So what do you do?
One solution is to make the server more powerful. But there is a limit to how much you can scale a single machine.
A more practical approach is to run multiple backend servers and distribute incoming traffic between them.
This is where load balancing comes in.
In this article, we'll explore what load balancing is, why it matters, how different algorithms work, the difference between Layer 4 and Layer 7 load balancing, health checks, sticky sessions, and how load balancing fits into a real-world backend architecture.
What Is Load Balancing?
Load balancing is the process of distributing incoming network traffic across multiple servers.
The system responsible for doing this is called a load balancer.
Instead of clients communicating directly with one backend server, they communicate with the load balancer.
The load balancer then decides which backend server should handle each request.
A simplified architecture looks like this:
Clients
│
▼
┌──────────────┐
│ Load Balancer│
└──────┬───────┘
│
┌────────┼────────┐
│ │ │
▼ ▼ ▼
┌──────┐ ┌──────┐ ┌──────┐
│ API 1│ │ API 2│ │ API 3│
└──────┘ └──────┘ └──────┘
The clients don't necessarily need to know that three backend servers exist.
From their perspective, they're communicating with one application.
The load balancer handles the distribution behind the scenes.
Why Do We Need Load Balancing?
The biggest reason is scalability.
Suppose your application initially has one server:
Users
│
▼
┌─────────┐
│ Server 1│
└─────────┘
This might work perfectly when you have a small number of users.
But as traffic increases:
Thousands of Users
│
▼
┌─────────┐
│ Server 1│
└─────────┘
│
💥
The server eventually becomes a bottleneck.
You could upgrade the server:
2 CPU → 8 CPU
8 GB RAM → 32 GB RAM
This is known as vertical scaling.
But vertical scaling has physical and financial limits.
Eventually, you may need a different approach:
Users
│
▼
Load Balancer
│
┌──────────┼──────────┐
▼ ▼ ▼
Server 1 Server 2 Server 3
This is horizontal scaling.
Instead of making one machine significantly more powerful, you add more machines.
Vertical Scaling vs Horizontal Scaling
There are two common ways to scale a system.
Vertical Scaling
Vertical scaling means making an existing server more powerful.
For example:
Before:
4 CPU
8 GB RAM
↓
After:
16 CPU
64 GB RAM
The advantage is simplicity.
You don't necessarily need to change your application architecture.
However, the machine has a physical limit.
There is also another problem.
If that server goes down, your entire application goes down.
Horizontal Scaling
Horizontal scaling means adding more servers.
For example:
Before:
Server
│
▼
API App
After:
Load Balancer
│
┌─────────┼─────────┐
▼ ▼ ▼
API 1 API 2 API 3
Now traffic can be distributed across multiple machines.
If one server fails, the others can continue serving requests.
This gives us both scalability and availability.
How Does a Load Balancer Work?
At a high level, the process looks like this:
1. Client sends request
│
▼
2. Load balancer receives request
│
▼
3. Load balancer selects backend
│
▼
4. Backend processes request
│
▼
5. Response goes back to client
Suppose a user requests:
GET /api/users
Instead of going directly to:
api-server-1
the request first reaches the load balancer.
The load balancer might decide:
Server 1 already has many active connections. I'll send this request to Server 2.
So the request becomes:
Client
│
│ GET /api/users
▼
Load Balancer
│
│ forwards request
▼
API Server 2
The load balancer is essentially acting as a traffic manager.
Load Balancing Algorithms
The load balancer needs a way to decide:
Which server should receive this request?
There are several strategies for answering that question.
1. Round Robin
Round Robin is one of the simplest approaches.
Requests are distributed sequentially.
Suppose we have three servers:
Server A
Server B
Server C
Requests might be distributed like this:
Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A
Request 5 → Server B
Request 6 → Server C
It basically goes around in a circle.
Advantages
- Simple
- Easy to implement
- Works well when servers have similar capacity
Disadvantages
It doesn't consider how busy a server currently is.
For example:
Server A → 10 active requests
Server B → 2 active requests
Server C → 1 active request
Round Robin might still send the next request to Server A.
2. Weighted Round Robin
Sometimes servers don't have equal capacity.
For example:
Server A → 16 CPU
Server B → 8 CPU
Server C → 4 CPU
Giving each server the same amount of traffic wouldn't necessarily be ideal.
Weighted Round Robin allows us to assign different weights.
For example:
Server A → weight 3
Server B → weight 2
Server C → weight 1
Traffic could approximately look like:
A → A → A
B → B
C
The more powerful server receives more traffic.
This is useful when backend instances have different capacities.
3. Least Connections
Instead of simply counting requests sequentially, the load balancer looks at the number of active connections.
Suppose:
Server A → 20 connections
Server B → 8 connections
Server C → 3 connections
The next request would probably go to Server C.
New Request
│
▼
Least Connections
│
▼
Server C
This can be useful when requests take different amounts of time to complete.
4. IP Hash
With IP Hash, the load balancer uses the client's IP address to determine which backend server receives the request.
Conceptually:
Client IP
│
▼
Hash Function
│
▼
Backend Server
For example:
192.168.1.10 → Server A
192.168.1.20 → Server B
192.168.1.30 → Server C
The goal is often to make requests from the same client consistently reach the same server.
This can be useful in some session-based architectures.
However, it also has limitations.
If the distribution of client IPs is uneven, traffic may become unevenly distributed.
5. Random
Another simple strategy is to randomly select a backend server.
For example:
Request 1 → Server B
Request 2 → Server A
Request 3 → Server A
Request 4 → Server C
Random selection can work surprisingly well with a large number of requests, although more sophisticated algorithms are often preferable when the system needs better control.
Layer 4 vs Layer 7 Load Balancing
This is one of the most important concepts when learning load balancing.
Load balancers can operate at different layers of the network stack.
Two common approaches are:
- Layer 4
- Layer 7
Layer 4 Load Balancing
Layer 4 operates at the transport layer.
It primarily works with protocols such as:
- TCP
- UDP
The load balancer doesn't necessarily need to understand the contents of an HTTP request.
It can make decisions based on information such as:
Source IP
Destination IP
Source Port
Destination Port
Protocol
For example:
Client
│
│ TCP connection
▼
Layer 4 Load Balancer
│
├──────► Server A
│
└──────► Server B
Because it operates at a lower level, Layer 4 load balancing can be fast and efficient.
Layer 7 Load Balancing
Layer 7 operates at the application layer.
For web applications, this usually means understanding HTTP or HTTPS.
Now the load balancer can inspect things such as:
HTTP Method
URL
Headers
Cookies
Host
For example:
GET /api/users
could be routed to one group of servers:
/api/* → API Servers
while:
GET /images/logo.png
could be routed somewhere else:
/images/* → Static Content Servers
This gives Layer 7 load balancers much more control over routing.
Layer 4 vs Layer 7: Simple Comparison
| Feature | Layer 4 | Layer 7 |
|---|---|---|
| Network layer | Transport | Application |
| Common protocols | TCP, UDP | HTTP, HTTPS |
| Understands HTTP | No | Yes |
| Can inspect URL | No | Yes |
| Can inspect headers | No | Yes |
| Routing flexibility | Lower | Higher |
| Typical use | Network-level traffic | Application-aware routing |
Neither is universally better.
The right choice depends on the architecture and requirements of the system.
Health Checks
Imagine you have three servers:
Server A → Healthy
Server B → Healthy
Server C → Crashed
What happens if the load balancer continues sending requests to Server C?
Users will receive errors.
This is why load balancers commonly use health checks.
The load balancer periodically checks whether backend servers are healthy.
For example:
GET /health
The server might respond:
HTTP/1.1 200 OK
The load balancer interprets that as:
This server is healthy.
But if the server repeatedly fails:
Connection refused
Timeout
HTTP 500
the load balancer can temporarily remove it from the pool.
Now traffic becomes:
Load Balancer
│
┌───────┴───────┐
▼ ▼
Server A Server B
Server C
❌ Unhealthy
This is one of the mechanisms that makes load balancing useful for fault tolerance.
Active vs Passive Health Checks
There are different approaches to detecting failures.
Active Health Checks
The load balancer actively sends requests to the server.
For example:
GET /health
If the server responds correctly, it remains available.
If it repeatedly fails, the load balancer removes it.
Passive Health Checks
The load balancer observes actual traffic.
If a backend repeatedly produces failures or connection errors, the load balancer can mark it as unhealthy.
In practice, systems can use a combination of health-check mechanisms.
Sticky Sessions
Here's an interesting problem.
Suppose your application stores user session information directly in server memory.
For example:
Server A
└── Session for User 123
The user's next request might go to Server B.
Request 1 → Server A
Request 2 → Server B
Server B doesn't know about the session stored inside Server A.
The user might suddenly appear logged out.
One solution is sticky sessions.
The load balancer attempts to keep the user connected to the same backend server.
User 123
│
├── Request 1 → Server A
├── Request 2 → Server A
├── Request 3 → Server A
└── Request 4 → Server A
This can solve some problems, but it introduces another dependency.
If Server A fails, the user's session may disappear.
Stateless Applications
A more scalable approach is often to make backend servers stateless.
Instead of storing important session state inside one server:
Server A
└── User Session
we can store shared state somewhere accessible to all servers.
For example:
Load Balancer
│
┌─────────┼─────────┐
▼ ▼ ▼
Server A Server B Server C
│ │ │
└─────────┼─────────┘
▼
Shared Storage
This could be a database, cache, or another shared state-management system.
Now any backend server can handle the request.
This makes horizontal scaling much easier.
Load Balancer vs Reverse Proxy
These concepts are closely related but aren't exactly the same.
A reverse proxy sits between clients and backend servers.
For example:
Client
│
▼
Reverse Proxy
│
▼
Backend
A reverse proxy can perform tasks such as:
- TLS termination
- Request routing
- Compression
- Caching
- Security filtering
- Header manipulation
A load balancer focuses specifically on distributing traffic across multiple backend instances.
However, one piece of software can perform both roles.
For example, Nginx can act as a reverse proxy and load balancer.
A Real-World Backend Architecture
Let's put everything together.
A production application might look something like this:
Internet
│
▼
┌─────────────┐
│ DNS │
└──────┬──────┘
│
▼
┌─────────────┐
│Load Balancer│
└──────┬──────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ API 1 │ │ API 2 │ │ API 3 │
└───┬────┘ └───┬────┘ └───┬────┘
│ │ │
└──────────────┼──────────────┘
│
┌─────────┴─────────┐
▼ ▼
┌─────────┐ ┌─────────┐
│ Database│ │ Redis │
└─────────┘ └─────────┘
A request might travel through the system like this:
User
│
│ HTTPS request
▼
DNS
│
▼
Load Balancer
│
▼
API Server
│
├──► Redis
│
└──► Database
│
▼
Response
│
▼
User
Each component has a specific responsibility.
DNS helps the client find the service.
The load balancer distributes traffic.
The backend handles business logic.
Redis can provide fast access to cached or shared data.
The database provides persistent storage.
This separation allows each part of the system to scale independently.
What Happens When a Server Fails?
Let's say we have:
Server A → Healthy
Server B → Healthy
Server C → Healthy
Then Server B crashes.
Without load balancing:
Users
│
▼
Server B
❌
Requests fail.
With a load balancer:
Load Balancer
/ \
▼ ▼
Server A Server C
Healthy Healthy
Server B
❌
The load balancer detects that Server B is unhealthy and stops sending new traffic to it.
The application can continue operating using the remaining servers.
This is the difference between:
"One server is down."
and:
"The entire application is down."
That distinction is extremely important in highly available systems.
Does a Load Balancer Eliminate Downtime?
No.
A load balancer improves availability, but it doesn't magically eliminate every failure.
For example, the load balancer itself can become a single point of failure.
If your architecture looks like:
Users
│
▼
One Load Balancer
│
├── Server A
├── Server B
└── Server C
what happens if the load balancer crashes?
Everything behind it becomes unreachable.
This is why production systems often use redundant load balancers or managed load-balancing services.
The architecture might look more like:
Users
│
┌────────┴────────┐
▼ ▼
Load Balancer 1 Load Balancer 2
│ │
└────────┬────────┘
│
┌────────┼────────┐
▼ ▼ ▼
Server A Server B Server C
Now the load-balancing layer itself has redundancy.
Common Load Balancing Mistakes
Understanding load balancing also means understanding what it doesn't solve.
1. Adding More Servers Doesn't Fix Everything
If your database is the bottleneck:
Load Balancer
│
┌───┼───┐
▼ ▼ ▼
API API API
\ | /
\ | /
Database
💥
Adding more API servers won't necessarily solve the problem.
The database might still be overloaded.
2. Ignoring Health Checks
If unhealthy servers continue receiving traffic, the load balancer becomes part of the problem rather than the solution.
3. Poor Session Management
Sticky sessions can hide architectural problems.
If your application depends heavily on one specific backend server, scaling becomes more difficult.
4. Ignoring Connection Limits
Every backend server has limits.
CPU, memory, network connections, database connections, file descriptors, and other resources can become bottlenecks.
A load balancer doesn't remove those limits.
It simply distributes traffic.
Load Balancing and Backend Engineering
As a backend developer, you don't necessarily need to build a load balancer from scratch.
But you should understand what happens around your API.
For example, suppose you build a Go API:
GET /api/users
POST /api/orders
GET /api/products
Initially, you might deploy one instance:
Client
│
▼
Go API
Later, your application grows:
Client
│
▼
Load Balancer
│
├── Go API #1
├── Go API #2
└── Go API #3
Now your Go application needs to behave correctly when multiple instances are running.
This means thinking about:
- Shared state
- Database connections
- Authentication
- Sessions
- Caching
- Idempotency
- Concurrency
- Timeouts
- Retries
- Graceful shutdown
- Health endpoints
This is where load balancing becomes more than just a networking concept.
It starts influencing how you design backend applications.
A Simple Mental Model
When learning load balancing, remember this:
MANY CLIENTS
│
▼
LOAD BALANCER
│
┌──────────┼──────────┐
▼ ▼ ▼
SERVER A SERVER B SERVER C
│ │ │
└──────────┼──────────┘
│
▼
SHARED DATA
The load balancer answers one fundamental question:
"Which backend should handle this request?"
The answer can depend on:
- Round Robin
- Server weight
- Active connections
- Client identity
- Request information
- Server health
Once you understand that, the rest of the topic becomes much easier.
Final Thoughts
Load balancing is one of the fundamental building blocks of scalable backend systems.
When an application is small, one server may be enough.
But as traffic grows, relying on a single server creates bottlenecks and increases the impact of failures.
Load balancing allows us to distribute traffic across multiple backend instances while improving scalability, availability, and fault tolerance.
The important thing isn't just memorizing algorithms like Round Robin or Least Connections.
It's understanding why load balancing exists in the first place.
A scalable backend is rarely just:
Client → Server
Instead, it increasingly becomes:
Client
│
▼
DNS
│
▼
Load Balancer
│
├──────► Backend 1
├──────► Backend 2
└──────► Backend 3
│
▼
Database / Cache
And this is an important shift in thinking for backend engineers:
You're no longer just writing code that works on one machine. You're designing systems that continue working when traffic, users, and failures increase.
That is where concepts like load balancing start to matter.
Top comments (0)