This post was created with AI assistance and reviewed for accuracy before publishing.
Rate limiting a normal API protects your servers. Rate limiting an LLM API protects your bank account. The difference matters, because it changes what you are counting and where the limit has to live.
A conventional web endpoint costs you CPU time measured in milliseconds. A single model call can cost real money, and an agent loop can issue hundreds of them without a human ever clicking anything. One customer leaving a misconfigured retry loop running overnight is a bill, not a blip.
Count tokens, not just requests
The instinct is to limit requests per minute. That is necessary but not sufficient, because requests vary in cost by three orders of magnitude. A one-line prompt and a 200-page document summarisation are both "one request".
Track both, and enforce whichever binds first:
// Two windows, different units, same tenant.
await Promise.all([
consume(`rl:${tenantId}:req`, 1, 60), // 60 requests / minute
consume(`rl:${tenantId}:tok`, tokens, 3600), // 200k tokens / hour
]);
The request limit stops runaway loops quickly. The token limit is what actually caps spend. You need the second one before you need the first.
Key design is the whole security model
Every limit key must contain the tenant identifier, and that identifier must come from your authenticated session, never from the request body.
// Correct: identity comes from the verified session.
const key = `rl:${session.tenantId}:${operation}`;
// Wrong: caller controls their own bucket, so they control their own limit.
const key = `rl:${req.body.tenantId}:${operation}`;
That second version is not a rate limit. Anyone who reads their own network traffic can send a different tenant id and get a fresh quota, or send someone else's and exhaust theirs.
Include the operation in the key too. Embedding calls and chat completions have wildly different cost profiles, and a shared bucket means a bulk embedding job starves interactive users of the quota they are paying for.
A sliding window that does not drift
Fixed windows have a well-known flaw: a caller can send a full quota at 11:59:59 and another full quota at 12:00:00, which is double the intended rate across a two-second span.
A sorted set gives you a true sliding window. Each request is a member scored by timestamp; expired entries are trimmed on read.
-- KEYS[1] window key, ARGV: now_ms, window_ms, limit, member
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1] - ARGV[2])
local used = redis.call('ZCARD', KEYS[1])
if used >= tonumber(ARGV[3]) then
return {0, used}
end
redis.call('ZADD', KEYS[1], ARGV[1], ARGV[4])
redis.call('PEXPIRE', KEYS[1], ARGV[2])
return {1, used + 1}
Running this as a Lua script matters. The trim, the count, and the insert have to be one atomic unit, otherwise two concurrent requests can both read used = limit - 1 and both proceed. Redis executes a script without interleaving other commands, which closes that race without a distributed lock.
The PEXPIRE on every call is deliberate. It means idle tenants expire out of memory on their own, so your key count tracks active tenants rather than every tenant you have ever had.
Reserve before the call, reconcile after
Token counts create an ordering problem: you cannot know the true cost until the response comes back, but you have to decide before sending it.
Reserve an estimate up front, then correct it:
const estimate = countPromptTokens(prompt) + maxOutputTokens;
const ok = await consume(`rl:${tenantId}:tok`, estimate, 3600);
if (!ok) throw new QuotaExceeded();
const res = await model.complete(prompt);
// Give back what was reserved but not used.
const actual = res.usage.inputTokens + res.usage.outputTokens;
await refund(`rl:${tenantId}:tok`, estimate - actual);
Reserving maxOutputTokens is pessimistic on purpose. Most responses come in well under the ceiling, and the refund returns the difference. The alternative, charging only actual usage after the fact, lets a tenant exceed their quota by the size of whatever is in flight when they hit the limit.
Fail closed on spend, open on latency
When Redis is unreachable you have to choose, and the right answer differs by limit type.
| Limit | Redis down | Why |
|---|---|---|
| Requests per minute | Allow | Protects servers, and your servers are fine |
| Token or spend quota | Deny | Protects money, and money does not recover |
An abuse-prevention limit failing open for thirty seconds is an acceptable risk. A spend cap failing open for thirty seconds during an agent loop is a genuinely expensive outage. Encode the difference rather than applying one blanket policy.
Watch the limiter's own latency
Every model call now waits on a Redis round trip first. That is cheap, but it is not free, and it sits directly in your user-facing path.
Two things keep it cheap. Put Redis in the same region as the service calling it, because a cross-region check can cost more than the limit saves. And pipeline the request and token checks into a single round trip rather than issuing them serially, as in the Promise.all above.
Measure the p99 of the check itself, separately from the model call. If it drifts upward, the usual cause is a sorted set that has grown large because the window is long and the traffic is heavy. A one-hour window on a busy tenant holds every request in memory for the full hour.
What to build first
Start with a per-tenant token quota on an hourly window, failing closed. That single control stops the failure mode that actually hurts, which is unbounded spend from one customer. Add per-operation buckets when a bulk job first starves your interactive traffic, and add request-per-minute limits when you see abuse rather than before.
Everything here depends on the key containing an identity the caller cannot choose. Get that right and the rest is tuning.
Top comments (1)
"Rate limiting an LLM API protects your bank account" is the right frame, and taking the tenant id from the verified session instead of the body is what separates a limit from decoration — the second version isn't a weaker limit, it's an absent one.
The token side has one wrinkle I'd add: a bucket debited after the response is a lagging control, because the expensive call has already happened. For a model with a large max_tokens, reserving the worst-case estimate at admission and reconciling against actual usage at completion is what keeps the spend cap honest; otherwise a burst of long completions blows past the hourly bucket before any of them meters. Same reason the request limit has to sit in front of the provider call, not behind it.
On the sliding window: one zset member per request is exact but you're storing every event. Do you trim with ZREMRANGEBYSCORE on write only, or also expire the whole key on idle — a tenant that goes quiet for a week otherwise holds a dead zset forever.