Rate Limiting: The Matrix of Traffic Control

The Quest Begins (The "Why")

I still remember the night our API started returning 503s like confetti at a parade. Users were complaining, the monitoring dashboard was flashing red, and I felt like I was stuck in a boss fight with no health packs. After a frantic dive into the logs, the culprit was obvious: a burst of traffic from a mis‑configured cron job hammering our endpoint with thousands of requests per second. We had no guardrails, and the system was buckling under the load.

That moment sparked a simple question: how do we let good traffic flow while gently throttling the bad, without turning our service into a walled fortress? The answer lay in a classic system design problem — rate limiting. But not just any limiter; I wanted something that felt fair, was easy to reason about, and wouldn’t add latency like a sticky note on a race car.

The Revelation (The Insight)

After reading a handful of blog posts and experimenting with a few implementations, the insight that clicked for me was this: a token bucket algorithm gives you both smooth bursting capability and a hard ceiling, all with O(1) per‑request cost.

Think of the bucket as a container that holds a fixed number of tokens. Tokens drip in at a steady rate (the refill rate). Each incoming request tries to consume a token. If a token is available, the request proceeds; if not, it’s rejected or delayed. The beauty is that the bucket can store up to its capacity, allowing short bursts, while the steady refill prevents the long‑term average from exceeding the limit.

Compare that to the naive fixed‑window counter: you reset a counter every minute and allow up to N requests per window. It’s easy to code, but it lets a client slam N requests at the very start of a window, then another N right after the window rolls over — effectively 2N in a tiny slice of time. That’s the “spike” problem that took down our service that night.

The token bucket smooths those spikes out. It’s like Neo learning to dodge bullets in The Matrix: instead of being hit by a barrage all at once, you learn to flow with the rhythm, moving only when there’s space.

ASCII diagram of the token bucket

+-------------------+
|   Token Bucket    |
|  (capacity = C)   |
+--------+----------+
         ^
         |  tokens arrive at rate R (tokens/sec)
         |
+--------v----------+
|   Refill Timer    |
|  (adds tokens)    |
+--------+----------+
         |
         |  request arrives → try to take 1 token
         v
+--------v----------+
|   Consume?        |
|  if token > 0:    |
|   - decrement token
|   - allow request |
|  else:            |
|   - reject/delay  |
+-------------------+

C = maximum burst size, R = sustainable rate.

Wielding the Power (Code & Examples)

Below is a minimal, production‑ready token bucket in Go. I kept it dependency‑free so you can drop it into any service.

type TokenBucket struct {
    rate      float64 // tokens per second
    capacity  float64 // max tokens
    tokens    float64
    lastSeen  time.Time
    mu        sync.Mutex
}

func NewTokenBucket(rate float64, burst float64) *TokenBucket {
    return &TokenBucket{
        rate:     rate,
        capacity: burst,
        tokens:   burst, // start full so we can burst immediately
        lastSeen: time.Now(),
    }
}

// Allow checks if n tokens can be consumed.
// Returns true if the request is allowed, false otherwise.
func (b *TokenBucket) Allow(n int) bool {
    b.mu.Lock()
    defer b.mu.Unlock()
    now := time.Now()
    // refill based on elapsed time
    elapsed := now.Sub(b.lastSeen).Seconds()
    b.tokens += elapsed * b.rate
    if b.tokens > b.capacity {
        b.tokens = b.capacity
    }
    b.lastSeen = now

    if b.tokens >= float64(n) {
        b.tokens -= float64(n)
        return true
    }
    return false
}

Common “traps” to avoid

1. Forgetting to refill – If you only top off the bucket on request arrival, a long idle period will let the bucket overflow incorrectly, letting a sudden burst exceed the intended rate. The fix is to always compute elapsed time and add tokens before checking.

2. Using integers for rates – With low request volumes (e.g., 0.5 req/sec) integer math truncates the refill, starving legitimate traffic. Keep the token count as a floating‑point number (or use fixed‑point math) to preserve fractional accumulation.

3. Ignoring concurrency – In a high‑throughput service, multiple goroutines will call Allow simultaneously. Without a mutex (or atomic operations), you’ll double‑consume tokens and exceed the limit. The simple sync.Mutex works fine for most cases; if you need extreme scalability, look at lock‑free counters or sharded buckets.

Before & After

Before (fixed‑window counter)

var (
    mu      sync.Mutex
    count   int
    window  time.Time
    limit   = 100
    duration = time.Minute
)

func allow() bool {
    mu.Lock()
    defer mu.Unlock()
    now := time.Now()
    if now.Sub(window) > duration {
        window = now
        count = 0
    }
    if count >= limit {
        return false
    }
    count++
    return true
}

A burst of 150 requests in the first second would blow past the limit, then the counter would reset after a minute, allowing another 150 in the next second — exactly the pattern that knocked us offline.

After (token bucket)

bucket := NewTokenBucket(rate: 100.0, burst: 200.0) // 100 req/s sustained, up to 200 burst

func handler(w http.ResponseWriter, r *http.Request) {
    if !bucket.Allow(1) {
        http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
        return
    }
    // process request...
}

Now the same 150‑request spike is allowed (we have 200 tokens in the bucket), but after that the bucket drains at 100 tokens/sec, so the next second can only sustain ~100 requests — smoothing the traffic and protecting downstream services.

Why This New Power Matters

Adopting a token bucket changed how I think about traffic control. It’s not just a “limit”; it’s a budget that refills over time, giving you the flexibility to handle realistic spikes while still guaranteeing long‑term stability. With this pattern you can:

• Shield databases, third‑party APIs, or downstream micro‑services from thundering herd problems.
• Offer tiered plans (free vs paid) by simply adjusting the rate and burst parameters per API key.
• Implement graceful degradation — instead of hard‑rejecting, you can make clients wait (sleep) until a token becomes available, turning a 429 into a brief pause rather than a hard error.

The best part? The algorithm is tiny, easy to test, and works at any scale — from a side‑project Raspberry Pi service to a globally distributed edge fleet.

Your Turn

Grab the snippet above, plug it into your next service, and play with the rate and burst values. Try simulating a burst with a tool like hey or wrk and watch how the bucket absorbs the shock.

Challenge: Build a middleware that logs the current token count on each request and alerts when the bucket drops below 10 % capacity. Share what you learn — did the insight help you sleep better at night?

Now go forth, limit wisely, and may your APIs stay as smooth as Neo’s bullet‑dodge. 🚀