LLM Rate Limits Across OpenAI, Anthropic, and Gemini

TL;DR

• Production AI applications encounter HTTP 429 errors because foundation model providers enforce rate limits across multiple concurrent axes, including requests per minute, input tokens, and daily spend.
• OpenAI, Anthropic, and Google Gemini implement distinct tracking algorithms, header specifications, and reset mechanisms that break simplistic client-side sleep-and-retry logic.
• Building a client that never encounters a 429 error requires client-side token bucket scheduling, exact response header parsing, proactive queuing, and semantic caching.
• Deploying an infrastructure gateway such as Bifrost allows engineering teams to pool multi-provider quotas, automate weighted key balancing, and fall back across model families with 11 microseconds of overhead.

Production applications calling foundation model APIs encounter HTTP 429 rate-limit errors whenever request volume or token throughput exceeds provider-enforced concurrency thresholds. Bifrost, an open-source AI gateway built in Go by Maxim AI, provides dynamic rate limiting, token bucket tracking, and provider failover to eliminate client-side throttling. While single-turn chat scripts rarely trigger API caps, autonomous multi-agent pipelines, parallel extraction workloads, and real-time user traffic routinely overwhelm default tier allocations. This article analyzes how rate limits operate across OpenAI, Anthropic, and Google Gemini, examines why standard exponential backoff fails under sustained load, and details the architectural patterns required to design an LLM client infrastructure that never drops a request to rate limiting.

How Modern LLM Rate Limits Work

Foundation model rate limits are multi-dimensional throughput constraints enforced at the project or organization layer. Unlike traditional REST APIs that calculate quotas purely on request counts per second, large language model providers evaluate consumption simultaneously across distinct metrics:

1. Requests per minute (RPM): The total number of distinct HTTP calls initiated within a rolling 60-second window.
2. Tokens per minute (TPM): The total volume of prompt (input) and completion (output) tokens processed across all concurrent requests within that window.
3. Requests per day (RPD): A macro ceiling that caps total daily operations, typically resetting at midnight UTC or Pacific time.
4. Tokens per day (TPD) or spend limits: Monetary or volume-based ceilings over fixed intervals (such as a rolling 10-minute window) designed to prevent runaway billing spikes.

A request is rejected with an HTTP 429 status code whenever any single metric breaches its ceiling, regardless of whether substantial headroom remains on the other dimensions. For example, a single prompt containing 150,000 tokens sent to an endpoint with an allowance of 200,000 TPM will succeed, but a second concurrent request with 60,000 tokens issued two seconds later will trigger an immediate 429 error, even if the application has only consumed 2 of its 500 allowed RPM.

Furthermore, providers do not evaluate per-minute limits as rigid batch counters that reset abruptly when the clock strikes the minute mark. Instead, they enforce continuous token bucket or leaky bucket algorithms with short-term rate smoothing. An account with 60 RPM is governed roughly as 1 request per second. Sending 15 concurrent calls in a single second will immediately exhaust the burst capacity of the token bucket and trigger HTTP 429 rejections, even if total volume over the previous 59 seconds was zero.

Provider Comparison: Rate Limits and Header Conventions

Understanding the mechanical differences across upstream providers is essential when writing client interceptors or configuring gateway routing policies. Each provider reports capacity metrics through non-standard response headers and evaluates resource exhaustion differently.

Provider Dimension	OpenAI	Anthropic	Google Gemini (AI Studio)
Primary Limits	RPM, TPM, RPD, TPD	RPM, ITPM (Input), OTPM (Output)	RPM, TPM, RPD, Spend per 10m
Token Tracking	Combined input and output	Separated into input and output pools	Evaluated primarily on input prompt tokens
Prompt Caching Impact	Cached prompt tokens count toward TPM	Cached read tokens do not count toward ITPM	Cached prompt tokens bypass standard TPM
Quota Scope	Organization or Project level	Workspace level per model family	Cloud Project level across all keys
Standard Status Code	HTTP 429	HTTP 429	HTTP 429 (RESOURCE_EXHAUSTED)
Key Status Headers	x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens	anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining	No active remaining count headers returned by default
Reset Header Format	Duration string (e.g., 1s, 6m0s)	RFC 3339 timestamp string	retry-after seconds in error response

OpenAI Rate Limit Architecture

The OpenAI API rate limit system measures throughput across requests per minute and combined tokens per minute for each model family. When sending a request, OpenAI checks whether the estimated prompt tokens plus requested max_tokens fit within the remaining TPM pool. Setting an artificially high max_tokens parameter on completion requests reserves that budget in advance, causing premature 429 errors even when the generated output is short.

OpenAI supplies diagnostic response headers on every HTTP 200 and HTTP 429 response:

• x-ratelimit-limit-requests: The maximum RPM ceiling for the configured tier.
• x-ratelimit-limit-tokens: The maximum TPM ceiling for the configured tier.
• x-ratelimit-remaining-requests: The integer count of requests permitted in the current window.
• x-ratelimit-remaining-tokens: The integer count of tokens remaining before throttling occurs.
• x-ratelimit-reset-requests: The time remaining until the request counter replenishes (expressed as a string duration such as 20ms or 1s).
• x-ratelimit-reset-tokens: The time remaining until the token pool fully replenishes (for example, 6m0s).

Under an HTTP 429 condition, OpenAI includes an error payload containing a diagnostic string and a retry-after header specifying the duration in seconds the client must pause before retrying.

Anthropic Rate Limit Architecture

The Anthropic Messages API rate limit model decouples token tracking into distinct Input Tokens per Minute (ITPM) and Output Tokens per Minute (OTPM) constraints alongside standard RPM. This distinction reflects the asymmetrical computational cost of LLM generation: generating tokens autoregressively is far more compute-intensive than ingesting input context.

Anthropic isolates limits by model family: requests to Claude 3.5 Sonnet draw from a different bucket than Claude 3.5 Haiku or Claude 3 Opus. Crucially, Anthropic implements prompt caching mechanics where cache read tokens do not consume ITPM allowance. For long conversational agents, establishing a stable cache prefix dramatically reduces rate limit pressure.

Anthropic returns the following headers:

• anthropic-ratelimit-requests-remaining: Remaining allowed HTTP calls.
• anthropic-ratelimit-input-tokens-remaining: Remaining input token allowance for the model family.
• anthropic-ratelimit-output-tokens-remaining: Remaining output token generation allowance.
• anthropic-ratelimit-requests-reset: RFC 3339 timestamp (such as 2026-10-15T10:05:00Z) indicating when request limits reset.
• anthropic-ratelimit-input-tokens-reset: RFC 3339 timestamp for input token replenishment.

Google Gemini Rate Limit Architecture

The Google Gemini API rate limit framework operates strictly at the Google Cloud project layer rather than per individual API key. This means that all background jobs, local developer environments, and production microservices referencing the same project share an identical quota bucket.

Gemini enforces RPM, TPM, and daily RPD quotas, alongside a rolling 10-minute spend limit on paid tiers to guard against unintended billing spikes. When Gemini quotas are exhausted, the API does not return a custom header suite on successful calls; instead, it raises a gRPC-style RESOURCE_EXHAUSTED error wrapped in an HTTP 429 response. The client must inspect the error response payload or extract the standard retry-after HTTP header to determine backoff duration.

Why Naive Exponential Backoff Fails in Production

The standard recommendation found in client SDK documentation is to wrap requests in an exponential backoff loop with jitter. A client encounters an HTTP 429, sleeps for $t = \min(t_{\max}, 2^{\text{attempt}} \times \text{base}) + \text{jitter}$, and retries up to three or four times. While this approach prevents single-threaded command-line scripts from crashing, it fails under sustained production workloads for three mathematical reasons:

1. The Herd Problem and Amplified Concurrency

When a high-concurrency fleet (such as 50 microservice pods or Celery worker nodes) hits a shared TPM ceiling, all 50 workers receive a 429 simultaneously. Even when randomized full jitter is introduced, the fleet remains in lockstep. As soon as the jitter window expires, 50 retries descend upon the upstream provider at the exact moment the token bucket partially replenishes. This immediately exhausts the newly replenished tokens, generates another wave of 429s, and doubles the total request queue.

2. Tail Latency Explosion and Upstream Timeouts

Exponential backoff absorbs rate limits by transforming quota deficits into client-side wait time. If an upstream provider requires a 30-second backoff to replenish a 400,000-token prompt quota, downstream HTTP clients, API gateways, or edge load balancers (such as Cloudflare or AWS ALB) will drop the connection with a 504 Gateway Timeout long before the retry succeeds.

3. Blind Execution Without Token Awareness

Standard retry algorithms do not calculate token costs. If an application attempts to process a 60,000-token context document when the upstream provider only has 10,000 tokens remaining in its rolling bucket, retrying three seconds later will fail with mathematical certainty. The client needs proactive flow control, not reactive apologies.

Designing a Client-Side Rate-Limiting Engine

To design an application client that never generates an HTTP 429 status code, engineering teams must invert the control flow: rate limits must be tracked, budgeted, and scheduled locally before any network packet is dispatched.

The Hierarchical Token Bucket Algorithm

The core engine relies on a localized, thread-safe token bucket algorithm that tracks both requests and tokens. A token bucket holds up to a maximum capacity $C$ and continuously refills at rate $r$ tokens per second. Before an outbound request is dispatched:

1. The client calculates the exact prompt token count using a local tokenizer (such as tiktoken for OpenAI models or provider-specific tokenization endpoints).
2. The client checks both the Request Bucket ($C_{\text{rpm}}, r_{\text{rpm}}$) and the Token Bucket ($C_{\text{tpm}}, r_{\text{tpm}}$).
3. If insufficient tokens exist, the thread is not dispatched; it is parked in a prioritized FIFO queue with an exact wake-up timer calculated as: $$\Delta t = \frac{\text{Tokens Required} - \text{Tokens Available}}{r}$$
4. Upon receiving the API response, the client parses the returned rate-limit headers to calibrate its local clock against the provider's server-side state.

The following Python implementation demonstrates a production-grade proactive client governor:

import time
import threading
from typing import Optional
import tiktoken
import httpx

class ProactiveRateLimiter:
    """
    A proactive dual-constraint token bucket governor (RPM and TPM)
    designed to eliminate client-side HTTP 429 errors.
    """
    def __init__(self, max_rpm: int, max_tpm: int):
        self.max_rpm = max_rpm
        self.max_tpm = max_tpm

        self.rpm_fill_rate = max_rpm / 60.0
        self.tpm_fill_rate = max_tpm / 60.0

        self.current_requests = float(max_rpm)
        self.current_tokens = float(max_tpm)

        self.last_update = time.monotonic()
        self.lock = threading.Lock()

    def _replenish(self) -> None:
        now = time.monotonic()
        elapsed = now - self.last_update
        self.last_update = now

        self.current_requests = min(
            float(self.max_rpm),
            self.current_requests + elapsed * self.rpm_fill_rate
        )
        self.current_tokens = min(
            float(self.max_tpm),
            self.current_tokens + elapsed * self.tpm_fill_rate
        )

    def acquire(self, estimated_tokens: int) -> None:
        """Blocks execution until both request and token capacity are available."""
        while True:
            with self.lock:
                self._replenish()
                if self.current_requests >= 1.0 and self.current_tokens >= estimated_tokens:
                    self.current_requests -= 1.0
                    self.current_tokens -= estimated_tokens
                    return

                # Calculate required sleep duration
                req_wait = (1.0 - self.current_requests) / self.rpm_fill_rate if self.current_requests < 1.0 else 0.0
                tok_wait = (estimated_tokens - self.current_tokens) / self.tpm_fill_rate if self.current_tokens < estimated_tokens else 0.0
                sleep_duration = max(req_wait, tok_wait, 0.05)

            time.sleep(sleep_duration)

    def synchronize_from_headers(self, remaining_requests: Optional[int], remaining_tokens: Optional[int]) -> None:
        """Calibrates local state against upstream provider response headers."""
        with self.lock:
            if remaining_requests is not None:
                self.current_requests = min(float(remaining_requests), self.current_requests)
            if remaining_tokens is not None:
                self.current_tokens = min(float(remaining_tokens), self.current_tokens)

class Zero429Client:
    def __init__(self, api_key: str, max_rpm: int, max_tpm: int):
        self.api_key = api_key
        self.limiter = ProactiveRateLimiter(max_rpm=max_rpm, max_tpm=max_tpm)
        self.client = httpx.Client(timeout=60.0)
        self.tokenizer = tiktoken.get_encoding("cl100k_base")

    def post_chat_completion(self, model: str, messages: list[dict], max_tokens: int = 1000) -> dict:
        # Pre-calculate token consumption
        prompt_text = "".join([m["content"] for m in messages])
        prompt_tokens = len(self.tokenizer.encode(prompt_text))
        total_reserved = prompt_tokens + max_tokens

        # Block locally until capacity exists
        self.limiter.acquire(total_reserved)

        response = self.client.post(
            "https://api.openai.com/v1/chat/completions",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"model": model, "messages": messages, "max_tokens": max_tokens}
        )

        # Synchronize limits from actual provider headers
        rem_req = response.headers.get("x-ratelimit-remaining-requests")
        rem_tok = response.headers.get("x-ratelimit-remaining-tokens")
        self.limiter.synchronize_from_headers(
            int(rem_req) if rem_req and rem_req.isdigit() else None,
            int(rem_tok) if rem_tok and rem_tok.isdigit() else None
        )

        response.raise_for_status()
        return response.json()

Eliminating 429s at the Infrastructure Layer with Bifrost

While client-side token bucket wrappers work well for standalone processes, distributed architectures across multiple application containers break isolated client governors. If 10 container instances run in Kubernetes, each maintaining its own token bucket without inter-process coordination, the aggregate fleet will routinely exhaust upstream provider quotas.

Moving rate limit enforcement to a centralized infrastructure layer solves this problem. Bifrost serves as a unified entry point that intercepts LLM calls, arbitrates rate limits across multiple keys and providers, and exposes an OpenAI-compatible API that functions as a drop-in replacement requiring only a single base URL change in your application.

Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, ensuring that centralized rate management does not introduce network latency bottlenecks into your inference pipeline.

Dynamic Virtual Keys and Budget Enforcement

Bifrost decouples upstream vendor API credentials from internal application consumers through virtual keys. Rather than distributing raw OpenAI or Anthropic API keys to application microservices, teams issue virtual keys (sk-bf-*) governed by precise hierarchical controls:

Customer (independent budget)
  └── Team (independent budget)
        └── Virtual Key (independent budget + rate limits)
              └── Upstream Provider Config (independent budget + rate limits)

At the gateway layer, Bifrost enforces:

• Local request and token limits: Virtual keys can be capped at custom RPM and TPM thresholds to isolate runaway development environments from production capacity.
• Upstream provider quotas: Bifrost monitors upstream rate limits and tracks capacity per provider account.
• Prioritized request shedding: When total application demand approaches provider limits, Bifrost queues requests or returns managed responses to lower-priority keys while keeping mission-critical services operating uninterrupted.

Multi-Key Load Balancing and Key Pooling

Enterprises frequently provision multiple API keys across distinct provider accounts or projects to multiply their aggregate rate limits. For example, two Tier 4 Anthropic workspaces offer double the cumulative RPM and ITPM of a single workspace.

Bifrost automates load balancing across API keys using weighted routing strategies. If Key A begins returning response headers indicating fewer than 10% remaining tokens, Bifrost routes subsequent calls to Key B in sub-microsecond time without the client application ever knowing.

Cross-Provider Fallback and Model Routing

Even with multiple API keys, large flash-traffic spikes or regional cloud outages can cause a provider to throttle traffic. Bifrost addresses this through automatic fallbacks.

When an upstream provider returns an HTTP 429 status code or a 5xx gateway error, Bifrost retries the request against a pre-configured secondary model or provider. For instance, if an incoming prompt routed to claude-3-5-sonnet encounters a rate limit on the primary Anthropic account, Bifrost converts the payload into an OpenAI-compatible format and dispatches it to gpt-4o on Azure OpenAI or AWS Bedrock with zero client downtime.

{
  "provider": "anthropic",
  "model": "claude-3-5-sonnet-20241022",
  "fallbacks": [
    {
      "provider": "openai",
      "model": "gpt-4o"
    },
    {
      "provider": "bedrock",
      "model": "anthropic.claude-3-5-sonnet-20241022-v2:0"
    }
  ]
}

The gateway handles request translation, streaming SSE conversion, and token attribution automatically across its matrix of supported providers.

Beyond routing, Bifrost applies governance and security controls (virtual keys, budgets, guardrails, audit logs) centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines, with endpoint enforcement on each device.

Reducing Rate-Limit Consumption with Semantic Caching

The most efficient way to avoid rate limits is to never contact the upstream LLM provider at all. Bifrost integrates native semantic caching, storing previous request prompts and their corresponding completions in an in-memory or vector database store.

When a subsequent request arrives with high semantic similarity (such as recurring retrieval-augmented generation queries or repetitive customer support prompts), Bifrost serves the response directly from cache. This eliminates both the request count and the token consumption against your upstream provider limits while returning answers in single-digit milliseconds.

Architectural Checklist for Zero-429 Systems

To achieve zero 429 errors in high-throughput enterprise deployments, teams should implement the following architectural checklist:

1. Calculate tokens prior to transmission: Run local BPE tokenization to budget token allocations instead of guessing based on character counts.
2. Never send unbounded max_tokens: Always cap max_tokens (or max_completion_tokens) to realistic generation boundaries to prevent providers from over-reserving token capacity against your TPM pool.
3. Parse provider rate-limit headers: Ingest remaining request and token counts on every successful HTTP 200 response to dynamically recalibrate local bucket fill rates.
4. Deploy a centralized AI gateway: Offload multi-key pooling, cross-provider fallbacks, and shared token tracking to an infrastructure layer such as Bifrost.
5. Implement prompt prefix caching: Structure agent prompts with stable system instructions to ensure prompt-cache hits bypass input token per minute (ITPM) quotas.
6. Decouple interactive and background jobs: Isolate batch tasks (data extraction, evaluation pipelines, offline embeddings) onto dedicated API keys or distinct virtual keys with strict lower-priority rate limits.

Frequently Asked Questions

What is an HTTP 429 status code in LLM APIs?

An HTTP 429 status code indicates that an application has exceeded provider rate limits across requests per minute, tokens per minute, or daily usage quotas. Large language model providers return this error to protect shared inference clusters from compute starvation and to enforce commercial account tier thresholds.

Why does exponential backoff fail during LLM traffic spikes?

Standard exponential backoff fails during traffic spikes because retried requests re-enter the queue without token awareness. When multiple concurrent clients hit a rate limit simultaneously, backoff causes them to retry in lockstep, immediately exhausting newly replenished tokens and triggering cascading 429 errors across the application fleet.

How do prompt tokens affect tokens per minute (TPM) limits?

Prompt tokens immediately consume capacity from the provider's rolling TPM token bucket upon request receipt. If the combined prompt tokens and requested completion tokens exceed the currently available bucket allowance, the provider rejects the call with an HTTP 429 error before generating any output.

How does Bifrost prevent HTTP 429 errors?

Bifrost prevents 429 errors by centralizing rate limiting across application services, pooling multiple API keys with weighted load balancing, and executing automatic fallbacks to secondary models or cloud providers whenever an upstream rate limit is approached. It also reduces quota consumption by serving repeated queries through semantic caching.

What is the difference between RPM and TPM?

Requests per minute (RPM) measures the raw count of HTTP API requests initiated within a 60-second window, regardless of payload size. Tokens per minute (TPM) measures the aggregate volume of input and output text tokens processed across those requests. A system can trigger a rate limit on TPM while remaining far below its RPM limit.

Does prompt caching help reduce rate limits?

Yes, prompt caching significantly reduces rate limit pressure on supported platforms such as Anthropic and Google Gemini. Anthropic does not count cache read tokens toward Input Tokens per Minute (ITPM) limits, allowing applications with stable prompt prefixes to process higher throughput without triggering 429 errors.

Can an AI gateway route traffic across multiple API keys?

Yes, an AI gateway such as Bifrost pools multiple API credentials for the same provider and balances requests using round-robin or weighted algorithms. This aggregates the rate limits of multiple accounts, allowing high-throughput applications to scale without encountering individual key throttling.

Getting Started with Resilient LLM Routing

Designing client applications that never fail to HTTP 429 errors requires moving beyond reactive retries and implementing proactive, token-aware traffic orchestration. By pairing client-side rate governors with a high-performance infrastructure gateway, engineering organizations can scale LLM workloads across multiple providers without downtime. Teams looking to eliminate rate-limit errors and unify multi-provider governance can book a Bifrost demo or inspect the open-source implementation in the Bifrost GitHub repository.

Sources

• OpenAI Rate Limits Documentation - Official reference for OpenAI request tiers, TPM calculation, and diagnostic response headers.
• Anthropic Messages API Rate Limits - Official reference covering Anthropic workspace tiers, RPM/ITPM/OTPM tracking, and cache exclusion rules.
• Google Gemini API Rate Limits - Official Google AI Studio documentation covering project-level limits, RPM/TPM thresholds, and spend quotas.
• Bifrost Architecture and Benchmarks - Published performance analysis detailing Bifrost's 11-microsecond gateway latency and routing architecture.