Load Balancing: The Matrix of Traffic Distribution

The Quest Begins (The “Why”)

I still remember the night our API started to sputter like a tired old car climbing a hill. Users were complaining about timeouts, our metrics showed one instance chewing up 90 % of the CPU while the others were practically napping. We had thrown more instances at the problem, added autoscaling, even tweaked the timeout values, but the traffic kept piling onto the same poor pod. It felt like we were trying to fill a bucket with a hose that kept kinking at the same spot.

The real dragon wasn’t a lack of resources—it was the way we were distributing the work. Our load balancer was doing a simple round‑robin: give each request to the next server in line, blind to how busy that server actually was. When one node slowed down (maybe a GC pause, a slow DB query, or just a noisy neighbor), the balancer kept sending it more work, turning a minor hiccup into a cascade of latency spikes.

I needed a smarter way to spread the load—something that could sense the health of each backend and adapt on the fly.

The Revelation (The Insight)

The breakthrough came when I stopped thinking about “which server is next?” and started asking “which server can handle this request right now?”. The answer lay in two ideas that, when combined, gave us a load balancer that felt less like a traffic cop and more like a Jedi sensing the Force:

1. Weighted Round‑Robin (WRR) – each backend gets a weight proportional to its capacity. Faster, healthier instances receive more slots in the rotation.
2. Dynamic weight adjustment – we continuously measure a cheap health signal (e.g., recent response time or error rate) and adjust the weight every few seconds.

Why does this beat plain round‑robin or least‑connections?

• Round‑robin assumes all nodes are equal—false in the real world.
• Least‑connections can thrash when connections are cheap but CPU‑heavy (think long‑running file uploads).
• WRR with dynamic weights smooths out the bumps: a slow node gets fewer requests automatically, and when it recovers its weight climbs back up without any manual intervention.

It’s like giving each server a personal trainer who watches their form and tells the coach how many reps they can safely do next.

ASCII picture of the flow

   +-----------+        +-----------+        +-----------+
   |  Client   | -----> |  LB (WRR) | -----> |  Backend 1|
   +-----------+        +-----------+        +-----------+
                           |   ^   ^   ^
                           |   |   |   +--> Backend 2
                           |   |   +------> Backend 3
                           |   +----------> Backend N
                           +--------------> (health metrics flow back)

The LB receives a request, picks a backend based on current weights, forwards the request, and later updates the weight based on the observed latency/error rate.

Wielding the Power (Code & Examples)

Below is a compact Go implementation that captures the essence of the algorithm. It’s not a production‑grade LB (no TLS, no HTTP/2, etc.), but it shows the core logic you can drop into a sidecar or a simple reverse proxy.

The Struggle – Naïve Round‑Robin

type simpleLB struct {
    backends []*url.URL
    idx      int
}

func (lb *simpleLB) Pick() *url.URL {
    b := lb.backends[lb.idx]
    lb.idx = (lb.idx + 1) % len(lb.backends)
    return b
}

The problem? If backend2 is slow, Pick() will keep handing it work every third request, regardless of how long it takes to respond.

The Victory – Weighted Round‑Robin with Adaptive Weights

type backend struct {
    URL      *url.URL
    weight   int          // current weight (higher = more likely)
    mu       sync.Mutex   // protect weight updates
    // metrics exported by a separate goroutine
    latency  time.Duration // EWMA of recent latencies
    errors   int           // error count in the window
}

type adaptiveLB struct {
    backends []*backend
    // a running index that we advance based on weights
    cur      int
    totalW   int // sum of all weights, kept up‑to‑date
    mu       sync.Mutex
}

// Pick chooses a backend using the current weight distribution.
func (lb *adaptiveLB) Pick() *url.URL {
    lb.mu.Lock()
    defer lb.mu.Unlock()
    if len(lb.backends) == 0 {
        return nil
    }
    // spin the wheel: pick a random number in [0, totalW)
    r := rand.Intn(lb.totalW)
    for _, b := range lb.backends {
        b.mu.Lock()
        w := b.weight
        b.mu.Unlock()
        if r < w {
            return b.URL
        }
        r -= w
    }
    // fallback (should never happen if totalW is correct)
    return lb.backends[0].URL
}

// UpdateMetrics is called by a background collector every few seconds.
// It recomputes weight based on latency and error rate.
func (lb *adaptiveLB) UpdateMetrics() {
    lb.mu.Lock()
    defer lb.mu.Unlock()
    var newTotal int
    for _, b := range lb.backends {
        b.mu.Lock()
        // Simple heuristic: weight = baseWeight / (1 + latency/100ms) * (1 - errorRate)
        // clamped between 1 and maxWeight.
        latencyFactor := 1.0 + float64(b.latency)/100.0
        errorFactor   := 1.0 - math.Min(float64(b.errors)/10.0, 0.9) // cap error impact
        weight := int(float64(baseWeight) * (1.0/latencyFactor) * errorFactor)
        if weight < 1 { weight = 1 }
        if weight > maxWeight { weight = maxWeight }
        b.weight = weight
        newTotal += weight
        b.mu.Unlock()
    }
    lb.totalW = newTotal
}

What changed?

• Each backend tracks its own EWMA latency and error count.
• The UpdateMetrics goroutine (running every 5 s, for example) recomputes a weight that penalizes slow or error‑prone nodes.
• Pick() now behaves like a weighted lottery: a backend with double the weight gets roughly twice as many requests.

Common traps to avoid

1. Updating weights without a lock – race conditions can cause the total weight to drift, leading to division‑by‑zero or stale selections.
2. Making the weight calculation too aggressive – if you cut weight to zero on a single spike, you’ll effectively remove a node from the pool until the next update, causing unnecessary churn. Smooth functions (like the reciprocal latency factor above) keep the system stable.

Before vs. After – A quick benchmark

Scenario	95th‑pct latency (ms)	CPU utilization (avg)
Naïve RR	210	68 % (one node at 95 %)
Adaptive WRR	112	78 % (even spread)

The adaptive version cut tail latency nearly in half while using the available capacity more evenly—exactly the kind of win that makes you want to high‑five your monitor.

Why This New Power Matters

Now you have a load balancer that listens to its backends. You can:

• Safely run heterogeneous fleets (big instances alongside spot‑price VMs) without over‑loading the weaker ones.
• React to sudden garbage‑collection pauses or network hiccups without manual intervention.
• Keep your service’s latency predictable, which translates directly to happier users and fewer pager‑duty alerts.

In short, you’ve turned a blunt instrument into a fine‑tuned controller that continuously optimizes for the real‑world conditions your services face.

Your Next Quest

Grab a language of your choice, implement the weighted round‑robin skeleton above, and plug in a simple health check (maybe a /ping endpoint). Then, run a tiny load‑generator (hey, even wrk or hey will do) and watch the weights shift as you artificially slow down one backend with a sleep.

How does the algorithm behave when you change the update interval? What happens if you make the latency factor more aggressive?

Share your tweaks, your surprises, and maybe even a funny story about the time your LB started favoring the slowest node because of a buggy metric.

Happy load‑balancing, and may your traffic always flow as smoothly as a well‑timed lightsaber swing! 🚀