Put three application servers behind a load balancer and the architecture looks almost too simple to write an article about:
┌── Server A
Client → LB ─────┼── Server B
└── Server C
A request arrives, the load balancer picks a server, done. That's genuinely the least interesting part of what a production load balancer does, though. The real questions start right after: is Server B actually healthy, not just running? What happens to the 300 requests already in flight on Server C when it gets pulled out for a deploy? Should /payments and /images even go to the same pool of servers? What happens when one request takes 20 milliseconds and the next takes 20 seconds? Where does TLS actually terminate? What happens to a user's session when the server holding it in memory isn't the one that answers their next request?
Once those questions show up, "distribute requests across servers" stops being an adequate description of what's happening. I put together a visual walkthrough of the request flow, the algorithms, and the production failure modes on SeeItFlow, if you'd like to see it rather than read through it. Here's the written version.
Why this exists in the first place
Start with an application running on a single server. Eventually it hits a ceiling, CPU, memory, open connections, network bandwidth, doesn't matter which, you can only buy a bigger machine for so long before vertical scaling stops being an option. So you add more servers. And the moment you have more than one, a new question appears that didn't exist before: who decides which server gets each request? That's the first job a load balancer does.
But horizontal scaling isn't the only reason it exists. Say Server B crashes outright. Without anything watching for that, some clients keep sending traffic to a server that's already dead. With health-aware load balancing, B simply stops receiving requests the moment it's detected as unhealthy, and traffic keeps flowing to A and C without anyone noticing B was ever a problem. So a load balancer is really solving two separate problems at once: throughput, spreading load across more capacity than one machine has, and availability, making sure a dead or struggling server doesn't keep receiving traffic just because nobody told it to stop.
How much should the load balancer actually understand?
One of the more consequential decisions in load balancer design is where it sits in the network stack, and how much of the traffic it's actually allowed to see.
A Layer 4 load balancer works purely with connections, source IP, destination IP, source port, destination port, TCP or UDP. It has no idea what HTTP is, and it doesn't need to, which is exactly what makes it fast and protocol-agnostic. A Layer 7 load balancer, by contrast, actually understands the application protocol riding on top, so for HTTP traffic it can look inside the request itself, the path, the host header, custom headers, even cookies, and route based on what it finds there:
/images/* → image servers
/api/* → API servers
/admin/* → admin service
X-Version:v2 → new backend
That's the real trade-off underneath L4 versus L7: L4 sees less and costs less to process, L7 sees far more and can make far richer routing decisions because of it. Most real architectures don't pick one and stick with it exclusively, they layer both, an L4 balancer handling raw connection distribution with an L7 layer making smarter routing decisions on top.
Round robin works right up until it doesn't
The simplest algorithm is round robin, request one goes to A, request two to B, request three to C, and back around. That works fine when every server has roughly the same capacity and every request costs roughly the same amount of work. It falls apart the moment that stops being true. Picture Server A stuck processing a 20-second request while B and C sit idle, and the next request arrives. Pure round robin doesn't know or care that A is busy, it might send the new request straight to A anyway, because "who's next in rotation" was never the same question as "who actually has capacity right now."
A few other algorithms exist specifically to fix this. Least connections sends each new request to whichever server currently has the fewest active connections, which matters a lot once request durations start varying widely. Weighted round robin accounts for servers that genuinely aren't equal, if A has twice the CPU capacity of B, giving A a weight of 2 against B's weight of 1 means A gets proportionally more traffic instead of being treated as identical. IP hash routes based on a hash of the client's IP, so the same client tends to land on the same server, useful for affinity, though it can produce uneven distribution when a lot of users happen to be sitting behind the same corporate NAT and therefore hash to the same bucket. There's no universally correct choice among these, the right one depends entirely on what's actually varying in your traffic, request duration, server capacity, or the need for a client to consistently land in the same place.
Health checks are easy to get subtly wrong
The basic idea is straightforward: the load balancer periodically checks something like GET /healthz on each server, and after enough consecutive failures, it pulls that server out of rotation, then lets it back in once it starts passing checks again. Simple in concept, and surprisingly easy to implement badly in either direction.
Make the check too shallow, "is the process running", and a server can report healthy while being completely unable to actually serve a request, maybe its database connection pool is exhausted, maybe a critical dependency is down, the process itself never noticed. Make it too deep instead, checking the database, checking Redis, checking two downstream APIs, and now one unrelated dependency having a brief blip can cause every single instance in the fleet to simultaneously report itself unhealthy. At that point the load balancer, doing exactly what it was told, pulls the entire fleet out of rotation at once, and the health check itself just caused the outage it was supposed to prevent.
This is why modern deployments generally separate three distinct questions that sound similar but aren't: startup, has the application finished initializing, liveness, should this specific process be killed and restarted, and readiness, should traffic be sent here right this moment. For the load balancer's actual routing decision, readiness is almost always the one question it should actually be asking, since a process can be alive and still not be in any condition to usefully handle a request.
Sticky sessions fix one problem by creating another
Say a user's login state lives in Server A's memory. Their first request lands on A and gets recorded there. Their next request happens to land on Server B instead, which has never heard of that session, and from the user's perspective they just got randomly logged out for no reason they can see.
Sticky sessions solve this directly, by pinning a given client to the same server every time. It works, but it costs the load balancer some of its freedom to make good decisions elsewhere. If the server a user is pinned to gets overloaded, that user keeps going there anyway, because pinning doesn't know or care about current load. If that server needs to come out of rotation for a deploy, every user pinned to it has to move somewhere else all at once. And if it crashes outright, whatever session state was only living in its memory disappears with it.
The cleaner fix, and the one most systems eventually converge on, is to stop keeping session state on individual servers at all:
┌── Server A ──┐
User → LB ───┼── Server B ──┼── Redis / DB
└── Server C ──┘
Move the session into something shared, Redis, a database, wherever, and every application server becomes genuinely stateless. Any server can now handle any request from any user, which is the same statelessness payoff that shows up everywhere else in distributed system design, once state lives externally instead of in a particular process's memory, the load balancer gets its full freedom back.
The deployment detail people forget: connection draining
Say Server B is being taken out of rotation for a deploy, and it currently has 300 requests actively in progress. Kill it outright and all 300 of those requests fail immediately, from the user's perspective for no visible reason, even though the rest of the infrastructure is completely healthy.
The fix is to stop sending B new traffic first, while letting the requests already running on it finish naturally, and only shut the instance down once those have actually completed. That's connection draining, and it's a small enough detail that it's easy to skip when you're first setting up a load balancer, right up until the first deploy that takes down 300 in-flight requests teaches everyone why it matters. It's genuinely one of the things separating "we have a load balancer" from "we can safely operate a load-balanced system" in production.
Deployments the load balancer makes possible
Once traffic routing is centralized in one place, that same control point turns into a deployment tool, not just a distribution mechanism.
Blue/green deployment keeps two full versions running side by side, the current version, Blue, taking 100% of traffic while the new version, Green, sits at 0% and gets verified. Once it's confirmed healthy, traffic switches, Blue to 0%, Green to 100%, and if something's wrong, switching back is just as immediate.
Canary deployment is more gradual: instead of an all-at-once switch, the new version gets a small slice, say 5%, while the stable version keeps the rest. Latency, error rate, and other metrics get watched closely on that 5%, and if it looks healthy, the percentage climbs, 5% to 25% to 50% to 100%. If anything looks wrong at any step, traffic routes back to the stable version before the bad version ever sees full load. At this point the load balancer has stopped being a traffic distributor and become an actual part of how deployments happen safely.
Who load-balances the load balancer?
There's an obvious uncomfortable question once every request is passing through this one component: doesn't that make the load balancer itself a single point of failure? Yes, if you only deploy one of them. Production systems make the load-balancing layer redundant too, and in most cloud environments this is handled for you, hidden behind a managed service that's already spread across multiple failure zones.
At larger scale, another layer shows up entirely before the regional load balancer:
┌── US Region → Regional LB → Servers
Users → Global LB ───┤
└── EU Region → Regional LB → Servers
GeoDNS or anycast routes a user toward whichever region is closest and healthy, and load balancing becomes genuinely hierarchical, a global layer deciding which region, a regional layer deciding which server within that region.
What this actually looks like assembled together
Put all of the above in one diagram and the trivial three-box picture from the start has grown considerably:
Internet
│
▼
Global Routing / DNS
│
▼
Regional Load Balancer
TLS termination
health checking
request routing
│
┌────────────┼────────────┐
▼ ▼ ▼
Server A Server B Server C
│ │ │
└────────────┼────────────┘
▼
Shared Redis / DB
Servers get added and removed freely. Unhealthy instances stop receiving traffic without anyone having to notice and intervene manually. Deployments drain old instances instead of killing in-flight requests. Canaries take a small, controlled slice of traffic before a rollout goes wide. TLS gets managed in one central place instead of on every individual server. And because the application servers themselves hold no state, any healthy instance really can serve any request from any user.
The mental model worth keeping
A load balancer isn't well described as "something that sends requests to different servers," that framing undersells almost everything it actually ends up doing. A better way to think about it: it's the control point sitting between clients and a pool of backend capacity that's constantly changing, servers coming up, going down, being deployed, being drained. Once something sits in that position, health checking, TLS termination, protocol-aware routing, connection draining, deployment control, regional failover, and observability all naturally accumulate around it, not because anyone planned for the load balancer to do all of that from day one, but because it's the one place in the architecture that already sees every request before it goes anywhere.
References
This post covers the core mechanics and production behavior of load balancing. The full visual breakdown on SeeItFlow covers the fundamentals in more depth, there's a dedicated visual learning walkthrough that shows requests actually moving through the load balancer, health checks failing and recovering, and deployments draining in real time, and an engineering insights guide focused on the trade-offs behind algorithm choice, sticky sessions, and canary rollouts.
Top comments (0)