Building a Rate Limiter Like Neo

The Quest Begins (The "Why")

I was knee‑deep in a side‑project that needed to talk to a third‑party API. The docs said “no more than 10 requests per second per IP”, and I shrugged, slapped a simple setTimeout around each call, and called it a day. Spoiler: that didn’t scale. When the traffic spiked during a launch, my naive throttler turned into a bottleneck, queuing up requests like a line at a coffee shop during rush hour. Latency blew up, users saw errors, and I felt like I’d just watched the hero get knocked down in the first act of a movie.

I dug into the existing rate‑limiter libraries, but they were either over‑engineered for my tiny service or hid the knobs I actually wanted to tweak. I needed something I could understand, tweak on the fly, and reason about under load. So I decided to build my own – not because I thought I could out‑smart the experts, but because I wanted to feel the click when the solution finally made sense.

The Revelation (The Insight)

The breakthrough came when I stopped thinking about “counting requests per second” and started thinking about tokens. Imagine a bucket that leaks at a steady rate. Every incoming request tries to take a token; if there’s one, the request goes through; if not, it gets delayed or rejected. This is the classic token bucket algorithm, and it gave me a mental model that was both simple and tunable.

Here’s why it clicked for me:

• Burst tolerance – The bucket can hold a maximum number of tokens, allowing short bursts without penalizing legitimate traffic.
• Smooth decay – Tokens replenish continuously, so the limiter adapts to changing traffic patterns without needing a reset window.
• Easy to reason about – You only need two numbers: the refill rate (tokens per second) and the bucket capacity (max tokens).

Contrast that with a fixed‑window counter (reset every minute). It’s easy to implement but lets a client blast through the limit right before the window resets, then sit idle for the rest of the minute – a classic “bursty” problem that can overwhelm downstream services. The token bucket smooths that out.

Let me sketch it out:

+-------------------+      request arrives
|   Token Bucket    |<-------------------+
|  (capacity = C)   |                    |
|  tokens = T       |                    |
+-------------------+                    |
        ^                               |
        |  refill at rate R tokens/sec  |
        +-------------------------------+

When a request shows up:

1. Add tokens based on elapsed time since last check (T = min(C, T + R * delta)).
2. If T >= 1, consume a token (T--) and let the request through.
3. Otherwise, either wait or reject.

That’s it. No sliding windows, no complex data structures – just a couple of floats and a timestamp.

Wielding the Power (Code & Examples)

The Struggle: A Naïve Fixed‑Window Counter

import time
from collections import defaultdict

class FixedWindowLimiter:
    def __init__(self, max_req, window_sec):
        self.max_req = max_req
        self.window = window_sec
        self.hits = defaultdict(int)   # ip -> count
        self.reset = defaultdict(float) # ip -> next reset time

    def allow(self, ip):
        now = time.time()
        if now > self.reset[ip]:
            self.hits[ip] = 0
            self.reset[ip] = now + self.window

        self.hits[ip] += 1
        return self.hits[ip] <= self.max_req

Problem: If a client makes 10 requests at 0:59 and another 10 at 1:01, they’ve effectively doubled the allowed rate within a two‑second span. The limiter can’t smooth bursts.

The Victory: Token Bucket in Python

import time
import math

class TokenBucket:
    def __init__(self, rate_per_sec, capacity):
        """
        rate_per_sec : tokens added each second (can be fractional)
        capacity     : max tokens the bucket can hold
        """
        self.rate = rate_per_sec
        self.capacity = float(capacity)
        self.tokens = float(capacity)   # start full
        self.timestamp = time.time()

    def _refill(self):
        now = time.time()
        delta = now - self.timestamp
        if delta > 0:
            new_tokens = delta * self.rate
            self.tokens = min(self.capacity, self.tokens + new_tokens)
            self.timestamp = now

    def allow(self, cost=1):
        """
        Try to consume `cost` tokens.
        Returns True if allowed, False otherwise.
        """
        self._refill()
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

Why this feels like a win

• Burst handling – Start with a full bucket; a client can blast through up to capacity requests instantly.
• Smooth throttling – After the burst, the token count drains at rate_per_sec, preventing overload.
• Fractional tokens – Using floats lets us model rates like 2.5 req/s without rounding tricks.
• Minimal state – Only three floats per key (ip, user, API key, etc.). Easy to drop into Redis or an in‑memory dict.

Common Pitfalls (the “traps”)

Trap	What happens	How to avoid
Using integers for rate	If rate_per_sec is 0.2, integer division gives 0 → bucket never refills.	Keep rate as a float; store tokens as float.
Forgetting to update timestamp	The bucket thinks time hasn’t passed, leading to over‑counting or under‑counting.	Always set self.timestamp = now after refilling.
Allowing negative tokens	A buggy cost larger than current tokens can dip below zero, breaking the invariant.	Clamp: only subtract if tokens >= cost.

A Quick Demo

limiter = TokenBucket(rate_per_sec=5, capacity=10)  # 5 req/s, burst up to 10

for i in range(15):
    if limiter.allow():
        print(f"Request {i+1}: ✅")
    else:
        print(f"Request {i+1}: ❌ (rate limited)")
    time.sleep(0.1)  # 100ms between tries

Output (first 10 get through, then it starts to throttle):

Request 1: ✅
...
Request 10: ✅
Request 11: ❌
Request 12: ❌
...

Watch how after the initial burst, the limiter settles into a steady ~5 req/s pace — exactly what we wanted.

Why This New Power Matters

Now you have a lightweight, understandable rate limiter that you can drop into any service — API gateways, webhooks, micro‑service chats, or even a CLI tool that talks to external endpoints. Because the algorithm is stateless aside from a tiny record per key, you can scale it horizontally by sharding the key space across Redis instances or using a consistent‑hashing layer.

The real magic is the mental model: thinking in terms of “tokens flowing in and out” makes it trivial to tweak behavior for different scenarios. Need a stricter limit for a particular endpoint? Just instantiate another bucket with a lower rate. Want to allow a massive burst for a batch job? Increase the capacity. The same code adapts.

And the best part? You didn’t need to pull in a heavyweight dependency or read a 50‑page whitepaper. You built it yourself, tested it, and felt that click — the same rush Neo gets when he finally sees the Matrix for what it is.

Your Turn

Grab your favorite language, implement a token bucket (or adapt the snippet above), and throw it at a toy API simulator. Try varying the rate and capacity, then plot the allowed requests over time with a quick script. Share your results, or better yet, tweak it to support per‑user limits with a sliding‑window fallback for audit logs.

What’s the coolest rate‑limiting trick you’ve discovered? Drop it in the comments — let’s keep the quest going!