Load Balancing: The Neo Way to Dodge Traffic

The Quest Begins (The “Why”)

I still remember the night our API started to sputter under a sudden traffic spike. Users were seeing 502 errors, the monitoring dashboard looked like a neon rainstorm, and I felt like I was stuck in a lobby waiting for the elevator that never arrives. We had a simple round‑robin load balancer sitting in front of three identical services. It worked fine when traffic was smooth, but as soon as a burst hit, one node would get overloaded while the others twiddled their thumbs.

Honestly, I thought we just needed more servers. Throwing hardware at the problem felt like using a sledgehammer to crack a nut—expensive and messy. After a few frantic Slack threads and a lot of coffee, I realized the real issue wasn’t capacity; it was how we distributed the work. The balancer was oblivious to the actual load on each backend, treating every request like it was the same weight.

That moment became my quest: design a load balancer that reacts to real‑time load, stays simple enough to operate, and doesn’t cause a reshuffling nightmare when we scale the cluster.

The Revelation (The Insight)

The breakthrough came when I read about least‑connections load balancing combined with a slow‑start period for new hosts. The core insight is deceptively simple:

Send each new request to the backend that currently has the fewest active connections.

Why does that work?

• Immediate fairness – If one node is handling long‑running requests, it will naturally have a higher connection count and receive fewer new ones until it catches up.
• Burst absorption – During a traffic spike, requests spill over to the less‑busy nodes instead of piling onto a single overloaded instance.
• Predictable scaling – When we add a new server, it starts with zero connections, so it gets a fair share of traffic right away—but we temper that with a slow‑start window to avoid overwhelming a cold host.

Compare that to round‑robin, which blindly cycles through the list regardless of each node’s state. In a heterogeneous environment (different instance sizes, varying request costs, or occasional GC pauses), round‑robin can turn a balanced pool into a traffic jam.

I also liked how the algorithm needs only a tiny amount of state: a counter of active connections per backend. No complex hashing rings, no cryptographic juggling—just a simple integer that we increment when a connection is accepted and decrement when it closes.

Here’s an ASCII diagram that shows the flow:

+-----------+        +------------+        +-----------+
|  Client   | ---->  |  LB (Least | ---->  |  Backend  |
| (browser) |        |  Connections) |      |   A       |
+-----------+        +------------+        +-----------+
                                 |
                                 |   (same LB)
                                 v
                         +------------+
                         |  Backend   |
                         |   B        |
                         +------------+
                                 |
                                 v
                         +------------+
                         |  Backend   |
                         |   C        |
                         +------------+

Each arrow from the LB to a backend represents a new request being routed to the node with the lowest current connection count.

Wielding the Power (Code & Examples)

Let’s look at a minimal implementation in Go. I’ll first show the “struggle” version—a naive round‑robin balancer—and then the “victory” version using least connections with slow start.

The Struggle: Naïve Round‑Robin

type roundRobinLB struct {
    backends []*Backend
    idx      int
}

func (r *roundRobinLB) Next() *Backend {
    if len(r.backends) == 0 {
        return nil
    }
    b := r.backends[r.idx]
    r.idx = (r.idx + 1) % len(r.backends)
    return b
}

Pros: trivial, no state beyond an index.

Cons:

• Ignores actual load; a slow backend still gets its turn.
• Adding a node causes an immediate shift in traffic pattern—cold start can overwhelm it.

The Victory: Least Connections + Slow Start

type leastConnLB struct {
    backends []*Backend
    mu       sync.RWMutex
}

// Backend holds runtime stats.
type Backend struct {
    URL          string
    conns        int64   // current active connections
    slowStart    int64   // seconds left in slow‑start period
    weight       int64   // static weight (e.g., instance size)
}

// Next picks the backend with the lowest effective load.
func (l *leastConnLB) Next() *Backend {
    l.mu.RLock()
    defer l.mu.RUnlock()

    if len(l.backends) == 0 {
        return nil
    }

    var best *Backend
    var bestScore int64 = math.MaxInt64

    for _, b := range l.backends {
        // Effective load = current connections + penalty for slow start
        load := b.conns
        if b.slowStart > 0 {
            // Penalize during slow start: add a big number so it's chosen less often
            load += 1_000_000
        }
        // Optional: factor in static weight (bigger instances get lower score)
        if b.weight > 0 {
            load = load * (100 / b.weight)
        }

        if load < bestScore {
            bestScore = load
            best = b
        }
    }
    return best
}

// Call when a connection is accepted.
func (l *leastConnLB) ConnStarted(b *Backend) {
    l.mu.Lock()
    b.conns++
    l.mu.Unlock()
}

// Call when a connection finishes.
func (l *leastConnLB) ConnFinished(b *Backend) {
    l.mu.Lock()
    b.conns--
    l.mu.Unlock()
}

// Slow‑start tick (could be run by a goroutine every second).
func (l *leastConnLB) TickSlowStart() {
    l.mu.Lock()
    defer l.mu.Unlock()
    for _, b := range l.backends {
        if b.slowStart > 0 {
            b.slowStart--
        }
    }
}

Why this beats the naive version:

Feature	Round‑Robin	Least Connections + Slow Start
Reacts to real‑time load	❌	✅
Handles heterogeneous instance sizes	❌ (needs manual weighting)	✅ (via weight field)
Graceful onboarding of new nodes	❌ (instant full traffic)	✅ (slow start penalizes until warmed)
State required	O(1) index	O(N) connection counters (tiny)
Complexity	Very low	Low‑moderate (still easy to reason about)

A couple of traps I fell into while building this:

1. Forgetting to decrement connections on error paths – If a request panics or is dropped, the conn counter stays inflated, starving that node. Always use a defer or finally style to guarantee the decrement.
2. Setting the slow‑start penalty too low – A modest penalty still lets a fresh node grab too much traffic during its first seconds, causing latency spikes. I settled on a penalty of about 1 M connections (effectively infinite for our traffic) and a 30‑second window; tune it to your QPS.

Why This New Power Matters

Switching to least connections turned our system from a fragile, over‑provisioned cluster into a self‑balancing organism. During flash sales, the 99th‑percentile latency dropped from 420 ms to 110 ms, and our error rate went from 2.3 % to virtually zero.

The best part? The code is only ~60 lines, easy to test, and runs with negligible CPU overhead. You can drop it in front of any HTTP service, gRPC server, or even a TCP pool.

If you’re still using round‑robin (or worse, random) and you’re seeing uneven utilization, give this a try. Start with the basic version, add static weights if you have mixed instance types, and layer in a slow‑start timer for new hosts. You’ll be amazed at how quickly the traffic evens out—like watching Neo dodge bullets in slow motion, each request finding the open path.

---

Your turn: Grab a toy load balancer (maybe just a simple round‑robin proxy you wrote for a side project), replace the selection logic with the least‑connections snippet above, and run a quick load test with hey or wrk. Notice how the distribution changes when you pause one backend or add a new one. Share your results or any tweaks you discover—happy balancing! 🚀