DEV Community

Cover image for Design a load balancer: the system design interview walkthrough
Gowtham Potureddi
Gowtham Potureddi

Posted on

Design a load balancer: the system design interview walkthrough

Design a load balancer: the system design interview walkthrough

Reaching for a load balancer is table stakes in a system design interview — traffic spikes, you say "I'll put a load balancer in front," the interviewer nods. Then they ask you to design one, and that is a different problem entirely. The brief sounds trivial — hand each request to one of several servers — but the interview asks you to spread a million requests a second across a changing pool of backends, instantly stop sending traffic to a server that just died, route some requests by their URL or cookie, and do all of it without the load balancer itself becoming the single thing whose failure takes the whole site down. That last twist — who balances the balancer — is why "design a load balancer" is a real HLD problem, not a box you drop on the diagram.

The walkthrough turns on three decisions. First, which layer you work at — a fast, connection-level Layer 4 balancer versus a content-aware Layer 7 one, and why large systems run both in sequence. Second, how you pick a backend and know it is alive — the distribution algorithm and the health-check logic that quietly removes a dead server. Third, high availability of the load balancer itself — how a pair of balancers share one address so a failover is invisible to clients. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, the forwarding path and state, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.

Title card for a system design interview walkthrough on designing a load balancer, showing clients talking to a virtual IP in front of a load balancer cluster that distributes requests to a pool of backend servers with health probes, on a dark citrus-and-lavender background.

How the load balancer round is actually run

A load balancer shows up in nearly every high-level design round — you drop one in front of your servers almost reflexively — so an interviewer testing depth will stop you and ask you to design the box you just drew. It runs about 45 minutes on a shared canvas: you draw clients, a virtual IP, the balancer cluster and the backend pool while you talk, and the interviewer interrupts. It is a favourite because "using a load balancer" is something everyone can say, but designing one exposes real tradeoffs — layers, algorithms, health checks, and the balancer's own availability. Get those clean and you have shown you understand the piece of infrastructure every other design leans on.

The interviewer rarely lets "put a load balancer in front and round-robin the requests" stand. "Fine — that one load balancer now handles all your traffic. What happens to the entire site when it dies?" That question — the balancer as the single point of failure you just introduced — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

  1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
  2. Estimate — requests and connections per second, and the uptime target, to justify every later choice.
  3. The forwarding path and state — what the balancer keeps and does per connection.
  4. Deep dive — the two or three decisions that actually decide this system.
  5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you make the balancer itself highly available instead of leaving a single point of failure), communication (did you explain L4 vs L7, and why both in sequence, in a sentence), technical execution (a distribution algorithm, active and passive health checks, a VIP with VRRP failover), and time management (did you spend your 45 minutes on layers, health checks and HA, not on TLS cipher suites). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a load balancer must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

Type Requirement
Functional Accept client connections and forward each to a healthy backend; support both Layer 4 (TCP/UDP) and Layer 7 (HTTP) balancing; distribute by a configurable algorithm; health-check backends and stop routing to unhealthy ones; drain connections from a backend being removed.
Functional (optional) SSL/TLS termination, session persistence (sticky sessions), content-based routing by path/header, autoscaling integration — call these out, then scope most of them for later.
Non-functional Very high throughput (up to ~1M requests/sec); low added latency; the balancer is on the critical path for all traffic, so it must be highly available (99.99%+); backends come and go; horizontally scalable.

The sentence that earns points: "The balancer sits on the critical path for every request, so the design is really two things — how it picks a live backend fast, and how it never becomes the single point of failure it was meant to remove." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the hard axes are raw throughput and availability, because the balancer is in front of everything.

Quantity Assumption Result
Requests up to ~1M requests/sec at peak the data plane must forward at line rate → favours L4 at the edge
Connections long-lived + short-lived mixed per-connection state (tracking, maybe session) adds memory
Added latency must be negligible routing decisions happen at connect/first request, not per packet
Uptime 99.99%+ on the critical path the balancer itself needs redundancy and fast failover

Two conclusions fall out, and you should say both out loud. First, throughput forces the layering — an edge that must survive a million connections a second and DDoS volume wants to be cheap and connection-level (L4), while the smart, content-aware routing (L7) is more expensive per request, which is exactly why large systems put L4 at the edge and L7 behind it. Second, being on the critical path forces redundancy — a single balancer is a single point of failure for the whole site, so you need at least a pair with sub-second failover. You have now pointed the interview at layering and HA — the real deep dives — before anyone asked.

Step 3 — The forwarding path and what state the balancer keeps

A load balancer has no CRUD API; its "interface" is the forwarding path (the data plane) plus the configuration and health state it maintains (the control plane).

Element Role
Data plane on each new connection (L4) or request (L7), pick a backend by the algorithm, proxy the traffic, and track the connection.
Backend pool the configured set of backends, each with a weight/capacity and a current health status; the list changes as servers are added, removed or marked down.
Health state per-backend up/down derived from active probes and passive observation; drives which backends are eligible.
Session table (optional) for sticky sessions, a map from a client (cookie or source IP) to a backend, so a session stays pinned.

The point to make: the data plane is hot and simple (pick, proxy, track), while the control plane (pool membership, health, config) changes it underneath — and keeping the two separated is what lets the balancer stay fast while backends churn.

Step 4 — High-level design: client, VIP, balancer cluster, backend pool

Now the boxes. A load balancer design is a short path with one crucial indirection — clients never talk to a balancer directly, they talk to a virtual IP:

  • Client → virtual IP (VIP): clients resolve to and connect to a single stable address, the VIP, not to any one balancer machine — this indirection is what makes failover invisible.
  • Balancer cluster: the VIP is held by a balancer (or a pair); the active balancer accepts the connection, picks a backend by the configured algorithm, and proxies the traffic.
  • Backend pool + health checks: the balancer continuously health-checks the pool, routes only to healthy backends, and drains connections from any being removed.
  • Two-tier layering (at scale): an L4 edge absorbs raw connection volume cheaply and fans out to a tier of L7 proxies that do content-aware routing and TLS termination before reaching the servers.
  • Control plane: a separate path manages pool membership, weights, algorithm and health thresholds without touching the hot forwarding path.

Say explicitly that clients always talk to the VIP, never a specific balancer — that one indirection is what lets a standby take over in about a second with clients noticing nothing. And name the two-tier L4→L7 pattern, because it is the signal that you know how large systems actually balance. Naming the VIP and the layering unprompted is what the interviewer is listening for.

A load balancer architecture diagram showing clients connecting to a single virtual IP held by an active-passive load balancer cluster, the active balancer picking a backend by a distribution algorithm and proxying traffic to a pool of backend servers, active health probes and passive failure observation marking backends up or down, an L4 edge tier fanning out to an L7 proxy tier for content-based routing and TLS termination, and a separate control plane managing pool membership and health thresholds.

Deep dive: Layer 4 vs Layer 7 load balancing

This is the first decision that actually decides the round, because the layer you work at changes what the balancer can even see.

Layer 4 (transport). An L4 balancer routes on IP and port, making its decision when the connection opens, then shovels bytes without inspecting them. It cannot read a URL or a cookie, but it is fast and cheap — a single box can push on the order of a million connections a second — and its health check is simply "did the TCP connection succeed." This is what you want at the edge, where raw volume and DDoS traffic land.

Layer 7 (application). An L7 balancer terminates the connection and reads the HTTP request, so it can route by path, header or cookie, do TLS termination, strip or inject headers, and run richer health checks (does /healthz return 200). The cost is CPU: a box that handles a million packets a second at L4 might handle only a hundred thousand requests a second at L7.

Both, in sequence. The production answer — and the immediate signal upgrade — is that large systems use both: a cheap L4 tier at the edge absorbs the connection flood and fans out to a tier of L7 proxies that do the smart, content-aware routing. You do not choose one; you layer them.

"Can your load balancer route /api to one pool and /images to another — and if so, what does that cost you?" Only an L7 balancer can, because it reads the HTTP request; L4 only sees IP/port at connection open. L7 adds CPU per request, so big systems front an L4 edge (cheap, high throughput, DDoS) with an L7 tier (content routing, TLS). Name the layer each sees, and the two-tier pattern.

A Layer 4 versus Layer 7 load balancing comparison diagram, two cards. Left card Layer 4: routes on IP and port, decides at connection open, cannot read the payload, very high throughput and low latency, TCP-level health checks, used at the edge for raw volume. Right card Layer 7: terminates and reads the HTTP request, routes by path, header or cookie, does TLS termination and rich health checks, higher CPU per request, used behind the edge; a note that large systems use an L4 edge fanning out to an L7 tier.

Deep dive: distribution algorithms and health checks

This is the decision most candidates rush, and it is where even weighting lives. Picking a backend and knowing it is alive are the balancer's two core jobs.

The algorithm. Round robin hands each new connection to the next backend in turn — O(1), no state — but it is blind to how busy each backend is, so under uneven request weights one server gets hammered. Least connections sends the next request to the backend with the fewest open connections, adapting to heterogeneous load at the cost of a counter per backend; weighted variants fold in each backend's capacity. Many production balancers default to weighted least connections for HTTP, because it respects both configured capacity and real-time load. When a client must keep hitting the same backend — a session cached in memory, or a WebSocket that is pinned for its lifetime — you use session persistence (a cookie or a source-IP/consistent hash), accepting slightly less even distribution.

Health checks. The balancer must stop routing to a dead backend in seconds. Active health checks probe each backend on an interval (a TCP connect at L4, an HTTP /healthz at L7) and mark it down after a failure threshold; passive checks observe real traffic and eject a backend that starts erroring or timing out. Together they keep the pool honest. And when you remove a backend on purpose (a deploy, a scale-down), you drain it — stop sending new connections but let in-flight ones finish — rather than cutting live requests.

"A backend starts timing out on one in three requests but still accepts TCP connections — does your health check catch it, and how fast do you stop sending it traffic?" An L4 TCP check would miss it (the socket still opens); you need an L7 active probe on a real endpoint plus passive observation of error/timeout rates, with a failure threshold so you eject it within a few intervals — then drain, don't cut. Name active and passive checks, and the drain.

System Design
HLD · Infrastructure · 45 min
Design a load balancer out loud, with follow-ups

Practise on Zynter →

Deep dive: making the load balancer highly available

This is the decision that the whole round is secretly about. You added a load balancer to remove a single point of failure and, if you stop there, you created one — every request now flows through one box.

VIP + VRRP (active-passive). The standard fix is two balancers sharing one Virtual IP, coordinated by VRRP (via a tool like Keepalived). The primary owns the VIP and handles all traffic; the backup sits idle, exchanging heartbeats. If the primary dies, the backup claims the VIP in about a second and traffic shifts — clients, who only ever knew the VIP, notice nothing. The VIP is the indirection that makes failover invisible. The tradeoff: active-passive wastes a machine that sits idle.

Active-active and global. When you need to scale the balancer tier itself (or you asked "and if one pair is not enough?"), you go active-active — multiple balancers advertise the same VIP via BGP/anycast, so routers spread clients across all of them and the network itself load-balances the load balancers; this is how hyperscalers run it. Above that, DNS and anycast spread traffic across regions before any single balancer is involved.

"You added the load balancer to kill a single point of failure — but now it is one. Fix it." Two balancers share a Virtual IP via VRRP; the primary owns it, the standby claims it in ~1 second on failure, and clients only ever see the VIP, so failover is invisible. To scale the tier itself, go active-active with the same VIP advertised via anycast/BGP. Name the VIP, the ~1s VRRP failover, and active-active anycast.

A load balancer high-availability diagram showing two load balancers sharing one Virtual IP coordinated by VRRP: in normal operation the primary owns the VIP and serves traffic while the standby exchanges heartbeats, and on failover the standby claims the VIP within about one second so clients that only ever connected to the VIP see no change, with a side note that active-active scaling advertises the same VIP from multiple balancers via anycast and BGP.

System Design
HLD · Infrastructure · 45 min
Defend layering, health checks and HA out loud

Practise on Zynter →

The follow-ups that decide the load balancer round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

  • "The load balancer dies — what happens to the site?" → Nothing, if you built a pair sharing a VIP via VRRP: the standby claims the VIP in ~1s and clients, who only know the VIP, see no change. (problem solving)
  • "Route /api and /images to different pools — which layer?" → L7, because it reads the HTTP request; L4 only sees IP/port. Large systems front an L4 edge with an L7 tier. (technical execution)
  • "Round robin or least connections?" → Least (often weighted least) connections when request duration varies, because round robin is blind to how busy each backend is. (communication)
  • "A backend half-fails but still accepts TCP — do you catch it?" → Only with an L7 active probe plus passive error observation and a failure threshold; an L4 TCP check would miss it. Then drain, don't cut. (problem solving)
  • "How do you scale the balancer tier itself?" → Active-active: multiple balancers advertise the same VIP via anycast/BGP, and the network spreads clients across them. (technical execution)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

  • Days 1–2: Learn the five-step framework until it is reflex. Do the load balancer end to end on paper — requirements, estimate, the forwarding path and state, the client → VIP → cluster → backend pool diagram.
  • Days 3–4: Drill L4 vs L7: what each layer sees, TLS termination, throughput cost, and the two-tier edge pattern. Say each tradeoff in one sentence.
  • Days 5–6: Own algorithms, health checks and HA — round robin vs weighted least connections, session persistence, active vs passive checks and draining, and the VIP/VRRP failover plus active-active anycast. Answer each of the five follow-ups out loud.
  • Day 7: A full timed 45-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — and because a load balancer sits in front of nearly every other design, getting it automatic pays off across the whole list.

Frequently asked questions

How do I design a load balancer step by step?

Follow the five-step HLD framework. Clarify requirements (forward each connection to a healthy backend, support L4 and L7, distribute by a configurable algorithm, health-check and drain — up to ~1M requests/sec, 99.99% uptime because it is on the critical path); estimate to show the hard axes are throughput and availability; describe the forwarding path and the state it keeps (backend pool, health, optional session table); draw the client → virtual IP → balancer cluster → backend pool path with health checks and an L4-edge-to-L7-tier layering; then deep-dive the three decisions that matter — L4 vs L7, distribution algorithms and health checks, and making the balancer itself highly available with a VIP and VRRP failover.

What is the difference between Layer 4 and Layer 7 load balancing?

A Layer 4 balancer works at the transport layer: it routes on IP and port, makes its decision when the connection opens, and forwards bytes without reading them, which makes it very fast and cheap (a box can push around a million connections a second) but blind to the content — it cannot route by URL or cookie, and its health check is just whether the TCP connection succeeds. A Layer 7 balancer terminates the connection and reads the HTTP request, so it can route by path, header or cookie, terminate TLS, and run rich health checks against a real endpoint, at a much higher CPU cost per request. Large systems use both in sequence: a cheap L4 tier at the edge absorbs the connection flood and fans out to an L7 tier that does the smart, content-aware routing.

How does a load balancer know a backend is healthy?

Through a combination of active and passive health checks. Active checks probe each backend on a fixed interval — a TCP connect for an L4 balancer, an HTTP request to a health endpoint like /healthz for an L7 one — and mark a backend down after it fails a configured number of consecutive checks, then back up when it recovers. Passive checks watch the real traffic flowing through and eject a backend that starts returning errors or timing out even though it still accepts connections, which a bare TCP check would miss. When you deliberately remove a backend (a deploy or a scale-down), you drain it: stop sending it new connections but let the in-flight ones finish, rather than cutting live requests.

Which load balancing algorithm should I use?

Round robin hands each new connection to the next backend in turn — it is O(1) and stateless but blind to how busy each backend is, so under uneven request weights one server can get overloaded. Least connections routes to the backend with the fewest open connections, adapting to heterogeneous load at the cost of tracking a counter per backend, and weighted variants fold in each backend's configured capacity; many production balancers default to weighted least connections for HTTP because it respects both capacity and real-time load. When a client must stay on the same backend — an in-memory session or a pinned WebSocket — you add session persistence via a cookie or a source-IP/consistent hash, trading a little evenness for stickiness.

How do you make the load balancer itself highly available?

By never letting a single balancer be the only path. The standard approach is two balancers sharing a Virtual IP (VIP) coordinated by VRRP (for example with Keepalived): the primary owns the VIP and serves all traffic while the backup exchanges heartbeats, and if the primary fails the backup claims the VIP in about a second, so clients — who only ever connected to the VIP — see no change. That is active-passive, which is simple but leaves a machine idle. To scale the balancer tier itself you go active-active, advertising the same VIP from multiple balancers via BGP/anycast so the network spreads clients across all of them, and above that you use DNS and anycast to spread traffic across regions before any single balancer is involved.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can drop a load balancer on the diagram. The round is won on L4-vs-L7 and who balances the balancer.

Draw the client → VIP → cluster → backend-pool path on the canvas and defend it out loud to a voice interviewer that keeps asking "which layer routes by cookie?" and "what happens when the load balancer itself dies?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise on the HLD track →
Browse all interview tracks

Top comments (0)