A Token Bucket for Free-Tier LLM Endpoints: Rate Limiting Without a Fancy Gateway

The server wasn't down, and the model wasn't slow — my client was flooding the endpoint with more concurrent requests than the free tier allowed. A token bucket queue fixed it in an afternoon, and this article shows you the exact implementation so you don't repeat my mistake. I learned this the hard way when a parallel summarization script fired twenty requests at once and got back a wall of confusing errors.

The 429 That Wasn't a 429

My first clue was a mix of HTTP 500s and timeouts, not the clean 429 that the documentation promised. I assumed the server was having a bad day, so I added a random sleep between requests and hoped for the best. The errors kept coming, which is when I realized I was guessing instead of measuring.

I wrote a probe script that sent one request, then two, then four, doubling the concurrency until something broke. That experiment revealed the real constraint: the endpoint handled roughly five concurrent requests smoothly, but beyond that, failures spiked hard and fast. Why did I trust a random sleep in the first place? Because it felt like progress, and it was actually just noise.

Step 1: Probe the Real Limits Before You Optimize

The probe is simple but essential, because documentation lies and reality is the only source of truth. I used Python's concurrent.futures to ramp up concurrency and recorded the success rate at each level.

from concurrent.futures import ThreadPoolExecutor, as_completed
import requests

def probe(concurrency, total=20):
    def call(i):
        r = requests.post(url, json=payload, timeout=30)
        return i, r.status_code
    with ThreadPoolExecutor(max_workers=concurrency) as ex:
        futures = [ex.submit(call, i) for i in range(total)]
        results = [f.result() for f in as_completed(futures)]
    ok = sum(1 for _, code in results if code == 200)
    print(f"concurrency={concurrency}: {ok}/{total} ok")

Run this for concurrency values of 1, 2, 4, 8, and 16, and you will see a cliff. That cliff is your real limit, and it is the number you design around — not the number in the docs, not the number in a blog post, but the number your actual endpoint tolerates.

Step 2: Replace Sleep with a Token Bucket

A random sleep is a guess, and a fixed sleep is a guess with a schedule. A token bucket, on the other hand, gives you a hard guarantee: no more than N requests per second, with room for short bursts. The math is simple, and the implementation fits in about twenty lines.

import time, threading

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate
        self.capacity = capacity
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
            self.updated = now
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

The bucket refills at a steady rate, so a burst of five requests passes immediately, but the sixth waits for the next token. This smooths out spikes without the wasted idle time of a naive sleep, and it turns a probabilistic failure into a deterministic queue.

Step 3: Add a Priority Queue for Important Work

Not all requests are equal, and a FIFO queue treats a health check the same as a production batch job. I wrapped the bucket in a priority queue so urgent tasks jump the line while background jobs fill the remaining capacity. This is the piece that turned my script from a fragile hack into something I could trust with real workloads.

import heapq, time

class PriorityQueue:
    def __init__(self, bucket):
        self.bucket = bucket
        self.heap = []

    def push(self, priority, task):
        heapq.heappush(self.heap, (priority, task))

    def run_next(self):
        while self.heap:
            _, task = heapq.heappop(self.heap)
            if self.bucket.acquire():
                task()
                return
            heapq.heappush(self.heap, (_, task))
            time.sleep(0.1)

The priority value can be as simple as an integer: 0 for interactive requests, 1 for batch work, 2 for exploratory probes. The queue never blocks a high-priority task behind a long backlog, which is exactly the behavior you want when a user is waiting on a response.

Step 4: Keep a Quota Ledger

The free tier gives you a token budget, but you cannot manage what you do not measure. I added a small logger that appends every call's token usage to a JSONL file, so I can see exactly how much quota remains at any moment. A quick awk command sums the totals and tells you whether you have room for another batch run or whether you should stop and wait.

import json, time

def log_call(model, prompt_tokens, completion_tokens):
    entry = {
        "ts": time.time(),
        "model": model,
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "total": prompt_tokens + completion_tokens,
    }
    with open("quota_ledger.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

awk -F'"' '{sum += $NF} END {print sum}' quota_ledger.jsonl

The ledger also reveals which experiments are worth repeating and which are burning tokens on dead ends. After a week of logging, I noticed that my probe scripts consumed almost as much quota as my actual workloads, so I cut their frequency and saved a third of my budget.

Where the Free Server Fits

I tested this whole setup against the open-source MonkeyCode project's free server option, which gives you a no-credit-card endpoint for exactly this kind of experimentation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The advertised 10M token quota at the time of writing was generous enough for my probe runs and queue experiments, but the quota is only half the story — the token bucket is what kept me from burning it all on failed requests.

Limitations and Who Should Skip This

This pattern assumes you control the client, which is true for scripts and batch jobs but false for a public API you expose to others. It also assumes the endpoint enforces limits client-side, which is a polite fiction — the server may have its own stricter limits, and your bucket should be conservative, not optimistic. And if you are running a single request at a time, the whole queue is overkill; a simple retry loop will serve you better.

The next time you see a wall of timeouts, ask yourself one question before you blame the server: how many requests did I actually fire at once? The answer might surprise you.

MonkeyCode provides free models that can run this workflow.