Choosing the right throttling algorithm for a multi‑tenant TypeScript API

in a SaaS platform that exposes a public GraphQL endpoint, each tenant is allowed a fixed number of requests per minute. the naive implementation—just checking a counter at the start of each resolver—collapsed under burst traffic, causing 429 spikes and downstream database overload.

Token bucket vs. leaky bucket

the core problem is two‑fold: we need a deterministic way to limit traffic per tenant, and we need that limiter to be cheap enough to run inside the request path without adding noticeable latency. two classic algorithms fit the bill:

• token bucket – fills a bucket with tokens at a steady rate; a request consumes a token, allowing short bursts as long as tokens remain.
• leaky bucket – treats incoming requests as water flowing into a bucket that leaks at a constant rate; excess traffic is discarded, producing a smoother request stream.

Both can be implemented in pure TypeScript with an in‑memory map or a distributed cache for horizontal scaling. the choice hinges on the desired traffic shape and the cost of state synchronization.

interface Bucket {
  tokens: number;
  lastRefill: number;
}

const LIMIT = 120; // requests per minute per tenant
const REFILL_RATE = LIMIT / 60_000; // tokens per ms
const buckets = new Map<string, Bucket>();

function allow(tenantId: string): boolean {
  const now = Date.now();
  const bucket = buckets.get(tenantId) ?? { tokens: LIMIT, lastRefill: now };

  // refill based on elapsed time
  const elapsed = now - bucket.lastRefill;
  bucket.tokens = Math.min(LIMIT, bucket.tokens + elapsed * REFILL_RATE);
  bucket.lastRefill = now;

  if (bucket.tokens >= 1) {
    bucket.tokens -= 1;
    buckets.set(tenantId, bucket);
    return true;
  }
  // out of tokens – reject request
  return false;
}

The snippet shows a simple in‑memory token bucket. for a distributed system you would replace the Map with a Redis hash and use Lua scripts to keep the refill‑and‑check operation atomic.

Trade‑offs and edge cases

• Burst tolerance – token bucket permits short spikes, which is useful for UI‑driven interactions but can still overwhelm downstream services if the burst size exceeds their capacity. leaky bucket smooths traffic but may reject legitimate user bursts, hurting perceived responsiveness.
• Clock drift – both algorithms rely on timestamps. if nodes have unsynchronized clocks, token counts diverge. using a single source of truth (e.g., Redis server time) eliminates the drift at the cost of an extra network hop.
• State overflow – a naive map grows with every tenant that ever hits the endpoint. you need eviction (e.g., setTimeout to purge idle buckets) or a TTL in the cache to keep memory bounded.
• Distributed consistency – in a multi‑instance deployment, race conditions can allow a tenant to exceed its limit if two instances read the same token count concurrently. atomic scripts or compare‑and‑set primitives guard against this.

When to pick which?

if your service can absorb short bursts and you value lower latency, token bucket is the default. if downstream APIs are fragile or you need to guarantee a smooth request rate, leaky bucket (or a hybrid that caps burst size) is safer.

how do you handle throttling in your services, and which algorithm has proven most reliable under real load?