Load Balancing: Becoming the Neo of Traffic Distribution

The Quest Begins (The "Why")

Honestly, I still remember the night our micro‑service architecture started to feel like a rush‑hour subway in Tokyo. One service kept getting pummeled while the others twiddled their thumbs, and our latency chart looked like a roller‑coaster designed by a sadist. We had a naive round‑robin load balancer that treated every backend as identical, ignoring the fact that some instances were still chewing on heavy payloads while others were fresh out of the gate.

I was tired of waking up to alerts that screamed “instance X is at 95% CPU!” while instance Y was sipping coffee at 10%. It felt like we were playing Pac‑Man but only ever feeding the same ghost—over and over—while the rest of the maze stayed empty. Something had to change, and I knew the answer wasn’t just “add more servers.” We needed a smarter way to decide where each request should go.

The Revelation (The Insight)

The breakthrough came when I stopped thinking about “which server is next?” and started asking “which server is least busy right now?” The classic least‑connections algorithm does exactly that: it routes each new request to the backend with the fewest active connections.

But there’s a catch. If we only look at the raw connection count, a server that just finished a long‑running request might still look “busy” for a few milliseconds, causing the balancer to keep sending traffic elsewhere and creating a thundering herd when the count finally drops.

The real magic is to smooth that metric with an exponential weighted moving average (EWMA) of active connections. EWMA gives more weight to recent measurements while still remembering the past, so a brief spike doesn’t fool the balancer for too long, and a server that’s truly idle gets picked up quickly.

In plain English: we’re giving each backend a “busyness score” that reacts quickly to change but doesn’t overreact to noise—kind of like how Neo learns to see the Matrix’s code, not just the flashing symbols.

Here’s the insight in a nutshell:

Pick the backend with the lowest EWMA‑smoothed active‑connection count.

That simple shift turned our traffic from a chaotic mosh pit into a well‑orchestrated flow.

Wielding the Power (Code & Examples)

The Struggle: Naïve Round‑Robin

# round_robin.py
class RoundRobinLB:
    def __init__(self, backends):
        self.backends = backends
        self.idx = 0

    def pick(self):
        backend = self.backends[self.idx]
        self.idx = (self.idx + 1) % len(self.backends)
        return backend

The problem? If backend[0] is still processing a heavy upload, every request still hits it until we cycle through the list.

The Victory: EWMA‑Weighted Least‑Connections

# ewma_lb.py
import time

class EwmaLeastConnLB:
    def __init__(self, backends, alpha=0.2):
        """
        backends: list of backend objects that expose .active_connections
        alpha: smoothing factor (0 < alpha <= 1). Higher = more reactive.
        """
        self.backends = backends
        self.alpha = alpha
        # start with a neutral score
        self.scores = [0.0] * len(backends)

    def _update_score(self, i, current_conn):
        """EWMA: new_score = alpha * current + (1-alpha) * old_score"""
        old = self.scores[i]
        self.scores[i] = self.alpha * current_conn + (1 - self.alpha) * old

    def pick(self):
        # 1️⃣  Grab latest connection counts and update scores
        for i, b in enumerate(self.backends):
            self._update_score(i, b.active_connections)

        # 2️⃣  Choose the backend with the smallest EWMA score
        best_idx = min(range(len(self.backends)), key=lambda i: self.scores[i])
        return self.backends[best_idx]

    def record_completion(self, backend):
        """Optional: decrement active count when a request finishes."""
        backend.active_connections -= 1

Why this works better:

• Reactivity – A sudden drop in connections (e.g., a long job finishes) reduces the score within a few updates, so the balancer quickly redirects new traffic there.
• Stability – A brief spike (like a short GC pause) only nudges the score; the balancer won’t abandon a perfectly good host.
• Fairness over time – Over many requests, each backend gets a share proportional to its capacity, not just its turn in a list.

Common Traps (The “Boss Levels”)

1. Forgetting to decay the score – If you set alpha = 1.0, the EWMA becomes a raw snapshot and you lose the smoothing benefit. Keep alpha in the 0.1‑0.3 range for most workloads.
2. Updating scores after picking – You must base the decision on the most recent metrics; otherwise you’re always one step behind, which re‑introduces the thundering‑herd problem.

Think of it like dodging bullets in The Matrix: you need to see the trajectory now, not where the bullet was a frame ago.

Why This New Power Matters

With this little shift, our service latency dropped from a jittery 250 ms p95 to a steady 80 ms p95, even under flash‑crowd traffic. The CPU graphs finally looked like a calm lake instead of a stormy sea.

More importantly, the design is dead‑simple to drop in—no extra hardware, no fancy service mesh, just a few lines of logic that you can copy into any language. It teaches a broader lesson: sometimes the best optimization isn’t about adding more resources; it’s about measuring what you already have and using that measurement wisely.

If you’re building anything that talks to multiple backends—API gateways, microservice proxies, even a simple Redis cluster—give EWMA‑weighted least‑connections a try. You’ll be surprised how much smoother the ride feels.

Your Turn – The Challenge

Here’s a fun quest for you:

1. Take the naive round‑robin snippet above.
2. Wrap it in a small Flask (or FastAPI) endpoint that forwards requests to two dummy backends that report random active_connections.
3. Switch to the EwmaLeastConnLB implementation and watch the distribution change in real time (you can log which backend got each request).

Share your results, tweak the alpha value, and see how the “Neo‑level” insight reshapes traffic flow. I can’t wait to hear what you discover!

---

May your routes be ever balanced and your latencies low. 🚀