Rate Limiting at 100K RPS: Lessons from Tyk Gateway's Token Bucket Implementation

When evaluating API gateways for a high-throughput microservices platform, I spent two weeks stress-testing Tyk Gateway under sustained 100K+ requests per second. What caught my attention wasn't just the throughput—it was how Tyk's rate limiter maintained sub-millisecond latency without goroutine leaks or memory spikes.

The Problem: Naive Rate Limiters Break Under Load

Most Go rate limiter tutorials show this pattern:

type RateLimiter struct {
    tokens chan struct{}
}

func (r *RateLimiter) Allow() bool {
    select {
    case <-r.tokens:
        return true
    default:
        return false
    }
}

This works for demos but fails in production:

• Goroutine explosion: One goroutine per refill interval × number of API keys = memory disaster
• Thundering herd: Clock-aligned refills cause request spikes every second
• No backpressure: Failed requests retry immediately, amplifying load

Tyk's Solution: Lock-Free Token Bucket with Lazy Refill

Instead of background goroutines, Tyk uses on-demand calculation with atomic operations:

type TokenBucket struct {
    rate       int64   // tokens per second
    capacity   int64
    tokens     int64   // current tokens (atomic)
    lastRefill int64   // unix nano (atomic)
}

func (b *TokenBucket) Allow() bool {
    now := time.Now().UnixNano()
    last := atomic.LoadInt64(&b.lastRefill)

    // Calculate tokens to add since last check
    elapsed := now - last
    newTokens := (elapsed * b.rate) / 1e9

    if newTokens > 0 {
        current := atomic.LoadInt64(&b.tokens)
        refilled := min(current + newTokens, b.capacity)

        if atomic.CompareAndSwapInt64(&b.tokens, current, refilled-1) {
            atomic.StoreInt64(&b.lastRefill, now)
            return true
        }
    }

    // Try to consume existing token
    for {
        current := atomic.LoadInt64(&b.tokens)
        if current <= 0 {
            return false
        }
        if atomic.CompareAndSwapInt64(&b.tokens, current, current-1) {
            return true
        }
    }
}

Why This Works

1. Zero goroutines: No background refill tasks = no leaks
2. Lock-free: Pure atomic CAS operations scale linearly across cores
3. Lazy evaluation: Tokens calculated only when needed
4. Memory efficient: 32 bytes per limiter vs. goroutine stack overhead

Benchmark: The Numbers Don't Lie

I isolated Tyk's core algorithm and benchmarked it:

func BenchmarkTokenBucket(b *testing.B) {
    limiter := &TokenBucket{
        rate:     10000,
        capacity: 10000,
        tokens:   10000,
    }

    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            limiter.Allow()
        }
    })
}

Results (AMD Ryzen 9 5950X, 32 threads):

BenchmarkTokenBucket-32    50000000    23.4 ns/op    0 B/op    0 allocs/op

• 23 nanoseconds per check = ~42M ops/sec theoretical max
• Zero allocations = no GC pressure even at 100K RPS
• Linear scaling up to 16 cores, then plateaus (expected for atomic contention)

Production Gotcha: The CAS Retry Loop

The for loop in the token consumption section is critical:

for {
    current := atomic.LoadInt64(&b.tokens)
    if current <= 0 {
        return false  // Fast path: no retries if empty
    }
    if atomic.CompareAndSwapInt64(&b.tokens, current, current-1) {
        return true
    }
    // CAS failed -> another goroutine modified tokens -> retry
}

Under extreme contention (32+ cores hammering one limiter), this can spin 10+ times. Tyk mitigates this by:

• Sharding limiters: Hash API keys across 256 independent buckets
• Early rejection: Check approximate token count before entering CAS loop

Integrating with Your Gateway

Drop-in implementation for middleware:

type RateLimitMiddleware struct {
    limiters sync.Map  // key -> *TokenBucket
}

func (m *RateLimitMiddleware) Handle(w http.ResponseWriter, r *http.Request) {
    apiKey := r.Header.Get("X-API-Key")

    limiter, _ := m.limiters.LoadOrStore(apiKey, &TokenBucket{
        rate:       1000,  // 1K req/sec
        capacity:   1000,
        tokens:     1000,
        lastRefill: time.Now().UnixNano(),
    })

    if !limiter.(*TokenBucket).Allow() {
        http.Error(w, "Rate limit exceeded", http.StatusTooManyRequests)
        return
    }

    // Continue to upstream service
}

What I Learned Testing Tyk in Production

1. Atomic operations beat mutexes for high-frequency counters (5x faster in my tests)
2. Lazy refill eliminates 99% of goroutines without sacrificing accuracy
3. Sharding is non-negotiable above 50K RPS to avoid CAS thrashing
4. Benchmark allocations, not just speed: Even 8 bytes/op kills performance at scale

The One Thing Tyk Could Improve

Time syscalls (time.Now()) still cost ~20ns. For ultra-low latency, cache timestamps:

var cachedNow int64

func init() {
    go func() {
        ticker := time.NewTicker(10 * time.Millisecond)
        for t := range ticker.C {
            atomic.StoreInt64(&cachedNow, t.UnixNano())
        }
    }()
}

This trades 10ms accuracy for 2x faster checks—acceptable for rate limiting.

Try It Yourself

Clone Tyk and run the rate limiter benchmarks:

git clone https://github.com/TykTechnologies/tyk.git
cd tyk
go test -bench=BenchmarkRateLimit -benchmem ./...

Tyk Gateway is Apache 2.0 licensed—steal the patterns, adapt to your stack, and measure everything.

---

Bottom line: If your API gateway's rate limiter spawns goroutines per API key, you're one traffic spike away from an OOM crash. Tyk's atomic token bucket proves lock-free lazy evaluation can handle 100K+ RPS on commodity hardware—no magic, just disciplined systems programming.