Day 1: the big picture. Day 2: databases. Day 3: caching, and how it's really a consistency problem in disguise. Today: load balancing — how traffic actually gets spread across servers, and the ways that can quietly go wrong.
Why Load Balancers Exist
Once you've horizontally scaled (Day 1 flashback) and have multiple servers instead of one, something needs to decide which server handles which request. That's the load balancer's job: sit between the client and your fleet of servers, and distribute incoming traffic so no single server gets overwhelmed while others sit idle.
It sounds simple — "just spread the requests out" — but how you spread them out turns out to matter a lot.
Load Balancing Algorithms
I went through the main ones:
- Round robin — requests go to servers in sequence, one after another. Simple, but doesn't account for server load — if one request is expensive and another is cheap, round robin doesn't know or care.
- Least connections — send the next request to whichever server currently has the fewest active connections. More adaptive than round robin, especially when requests vary a lot in how long they take.
- Weighted round robin / weighted least connections — same ideas as above, but servers with more capacity get proportionally more traffic. Useful when your fleet isn't made of identical machines.
- IP hash — the client's IP determines which server it's routed to, so the same client tends to land on the same server. This matters for session persistence (also called "sticky sessions") — some applications need a user's requests to keep hitting the same server, e.g., if session data is stored locally on that server instead of in a shared store.
The thing that stood out: there's no universally "best" algorithm. Round robin is fine when requests are roughly uniform. Least connections is better when they're not. IP hash matters when statefulness matters. It's another trade-off decision, not a default setting.
Layer 4 vs. Layer 7 Load Balancing
This distinction took me a minute to really get:
- Layer 4 (transport layer) load balancers route based on IP address and port, without looking at the actual content of the request. Faster, simpler, but less flexible.
-
Layer 7 (application layer) load balancers can inspect the actual request — HTTP headers, URL paths, cookies — and make smarter routing decisions, like sending
/api/videotraffic to one set of servers and/api/searchto another. More flexible, but more overhead per request.
This maps neatly onto the classic latency vs. flexibility trade-off from Day 1: Layer 4 is faster because it knows less; Layer 7 is smarter because it does more work per request.
Health Checks: How the Load Balancer Knows What's Alive
A load balancer is useless if it keeps sending traffic to a server that's already down. Health checks solve this — the load balancer periodically pings each server (or checks a dedicated health endpoint), and stops routing traffic to any server that fails to respond. This was a good reminder that load balancing isn't just about distribution, it's also about failure detection — the load balancer is quietly doing double duty as a monitoring system.
The Load Balancer Becomes a Single Point of Failure
Here's the part that really made me stop and think: I spent three days learning how to avoid single points of failure by scaling horizontally — only to introduce one giant new single point of failure sitting in front of all those servers. If the load balancer itself goes down, it doesn't matter how many healthy backend servers you have.
The common fix is to not have just one load balancer — instead, you run multiple load balancer instances behind something like DNS round robin or a floating/virtual IP, so there's no single machine the entire system depends on. It's a very "turtles all the way down" realization: every layer you add to solve a scaling problem can itself become the next bottleneck or failure point if you don't also scale and replicate that layer.
What Clicked Today
Load balancing isn't just "spread traffic evenly." It's a small system of its own, with its own trade-offs (routing algorithm, OSI layer, statefulness) and its own failure modes (the load balancer itself needs redundancy). I'm noticing a theme across all four days now: every solution to a scaling problem introduces a new, smaller version of the same problem one layer up.
Tomorrow
Next up: message queues and asynchronous processing — why not everything needs to happen in the request-response cycle, and how queues help decouple services from each other.
If you've dealt with a load balancer misconfiguration that caused uneven traffic or a nasty outage, I'd love to hear about it — feels like exactly the kind of thing that's obvious in hindsight and invisible beforehand.
This is part of a 30-day series on learning system design from scratch. Catch up on Day 1, Day 2, and Day 3 if you missed them.
Top comments (0)