Stop Counting Requests: The Case for Token-Based Quotas in LLM SaaS

If you are building a multi-tenant AI application, your current rate-limiting strategy is likely broken. Traditional "requests per minute" (RPM) metrics, inherited from standard REST APIs, fail catastrophically when applied to Large Language Models. This isn't just an imprecision; it is a fundamental architectural error that systematically throttles efficient short queries while allowing high-cost massive contexts to bypass limits undetected.

Why RPM Metrics Fail for LLMs

The core issue is cost variance. In traditional APIs, the computational cost of a request is relatively uniform. In LLMs, cost is driven by token count, not frequency. A single request with a 100k-token context window can cost hundreds of times more than a simple 50-token query. If you limit users by request count, you inadvertently penalize lightweight users while giving heavy users free rein to drain resources until the provider’s account-level quota is exhausted.

This leads to the "noisy neighbor" effect. Most LLM providers enforce rate limits at the organization or account level, not per user. Without internal controls, one tenant’s batch processing spike can throttle API access for every other customer on your platform, causing widespread service degradation.

Tokens ≠ Costs: The Complexity Layer

Even shifting to token counts introduces inaccuracies because tokens do not map linearly to dollars. Three factors complicate this relationship:

1. Input vs. Output Pricing: Output tokens are often 3 to 5 times more expensive than input tokens.
2. Model Variance: Tenants using premium models can incur 3 to 4 times the costs of those using cheaper models, even with identical usage patterns.
3. Prompt Caching: Repeated context blocks may be charged at a fraction of the regular price upon cache hits, meaning two requests with the same token count can have vastly different costs.

Therefore, financial control requires managing token credits rather than raw token counts. You must normalize costs across different models and pricing tiers into a common currency for your internal ledger.

The Reservation Pattern: Solving the Temporal Problem

A critical challenge in LLM billing is temporal uncertainty. You only know the exact cost after generation completes. If you wait for completion to charge, a user could trigger a long-running stream that exceeds their budget, resulting in negative balances.

The solution is a reservation pattern, similar to credit card pre-authorizations. Before calling the provider, your system should:

1. Estimate the maximum possible cost using max_tokens as a conservative upper bound.
2. Reserve these funds in an internal credit ledger (creditLedger.reserve).
3. Execute the API call.
4. Settle the exact cost based on the provider’s actual usage report, releasing any excess reservation.

If the call fails, the reservation is released immediately. This ensures that no tenant can spend more than they have authorized, even during streaming processes.

Implementation Strategy: Two-Tier Limits

To maintain stability, implement a two-tier limiting structure:

• Internal Per-Tenant Sub-Limits: Use a token bucket algorithm tied to the user’s credit balance to prevent individual overspending.
• Global Provider Quota Checks: Monitor the aggregate usage against the provider’s account limits to prevent the noisy neighbor effect.

Crucially, always base final cost tracking on the actual usage data returned in the provider’s API response. Internal estimates are useful for reservations, but discrepancies between your bookkeeping and provider invoices signal bugs in your cost logic, not mere estimation errors. Real-time monitoring of usage fidelity is essential for accurate billing.

By shifting from request counting to token-based credit management with reservation patterns, you align your architecture with the economic reality of LLMs, ensuring both financial accuracy and service reliability for all tenants.