Implementing Rate Limits and Retries in LLM-Powered Backends

LLM-powered applications depend on external inference services that enforce request and token limits, experience temporary outages, and occasionally return network or server errors. Without proper rate limiting and retry mechanisms, these failures can increase latency, trigger retry storms, exhaust API budgets, and affect application availability. Building a reliable backend requires a coordinated strategy that combines request and token-based rate limits, concurrency controls, exponential backoff, and selective retries.

Applications must also respect provider-supplied retry instructions, enforce execution deadlines, and prevent duplicate operations. These mechanisms help developers build production-ready AI systems that handle unpredictable traffic without overwhelming external model providers or compromising application reliability.

Why Do LLM-Powered Backends Need Rate Limits and Retries?

Traditional backend applications generally operate within infrastructure controlled by the development team. LLM-powered applications introduce an additional dependency: external model providers with independent infrastructure, usage policies, and capacity restrictions.

A single user interaction can trigger several model requests, particularly in applications involving retrieval-augmented generation, AI agents, and multi-step workflows.

Consider an enterprise AI assistant that performs the following operations:

• Receives a question from a user.
• Generates an embedding for semantic search.
• Retrieves relevant documents.
• Sends contextual information to an LLM.
• Validates the generated response.
• Produces a final answer.

One interaction can consume multiple API requests and thousands of tokens. When hundreds of users execute similar workflows simultaneously, traffic can quickly exceed provider limits.

Two mechanisms address this challenge.

Rate limiting controls how frequently requests reach a service, while retry handling determines whether failed operations should be attempted again and when.

Both mechanisms are necessary. Retries without rate limiting can amplify traffic during outages, while rate limiting without retries can expose users to temporary provider failures that could otherwise be recovered.

Rate Limits in LLM APIs

Rate limits define how much traffic an application can send to an LLM provider within a specific period. These restrictions vary depending on the provider, model, account tier, organization, and endpoint.

Common rate-limit measurements include:

• Requests per minute (RPM): Maximum permitted API requests within a period.
• Tokens per minute (TPM): Maximum permitted token consumption.
• Input tokens per minute: Restrictions on tokens submitted to a model.
• Output tokens per minute: Restrictions on generated tokens.
• Concurrent requests: Limits on simultaneously active operations, where applicable.

HTTP 429 Responses

HTTP 429 indicates that a request has been rejected because of a rate or usage restriction. However, not every 429 response represents temporary throttling.

An application might receive this response because it:

• Exceeded its request allowance.
• Consumed its available token capacity.
• Reached an account usage restriction.
• Exhausted API credits.
• Exceeded an enforced spending limit.

Temporary throttling can often be addressed by reducing request frequency and retrying after an appropriate delay. Billing and quota errors require administrative intervention instead.

OpenAI's error documentation distinguishes temporary rate-limit conditions from exhausted credits and spending limits. This distinction should be reflected in application-level error handling.

Inspecting Provider Response Headers

Response headers provide useful information for managing traffic.
For example, OpenAI documents headers such as:

• x-ratelimit-limit-requests
• x-ratelimit-remaining-requests
• x-ratelimit-limit-tokens
• x-ratelimit-remaining-tokens
• x-ratelimit-reset-requests
• x-ratelimit-reset-tokens
• Retry-After

Developers can use these values to update local admission policies and determine when another request should be attempted.

Why Does Traditional Retry Logic Fail?

A basic retry implementation might look like this:

for attempt in range(3):
    try:
        return await call_llm()
    except Exception:
        continue

Although straightforward, this approach creates several reliability problems.

• Immediate retries: Requests are repeated without allowing the provider time to recover.
• Unnecessary retries: Authentication failures and invalid requests are treated like temporary network problems.
• Unbounded execution: Operations can remain active longer than the application's acceptable response time.
• Retry amplification: Multiple application instances can repeat failed requests simultaneously, increasing pressure on an already overloaded provider.
• Duplicate operations: A timeout does not necessarily mean that the original operation was never processed.

Designing a Rate-Limiting Architecture

A production-ready LLM backend should enforce traffic controls before requests reach the external model provider.

A practical architecture separates application-level protection from provider-level traffic management.

Each component has a defined responsibility:

• API gateway: Authenticates users and applies general traffic restrictions.
• Tenant rate limiter: Enforces individual customer usage policies.
• Token admission controller: Estimates whether requests fit available token capacity.
• Concurrency controller: Restricts simultaneous model calls.
• Request queue: Smooths traffic bursts and manages background workloads.
• Retry manager: Handles eligible temporary failures.
• Monitoring layer: Tracks throttling, retries, latency, and token consumption.

This separation is particularly useful in enterprise AI applications where multiple customers share infrastructure and provider credentials.

Implementing Request and Token-Based Rate Limits

Selecting an appropriate rate-limiting algorithm is an important architectural decision.

Fixed-Window Counters

A fixed-window limiter counts requests within a defined interval. For example, an application might permit 100 requests per minute for each tenant. Once the counter reaches its threshold, additional requests are rejected until the next window.

Sliding-Window Rate Limiting

A sliding-window limiter calculates request activity over a continuously moving interval. It provides smoother enforcement than fixed-window counters but requires additional state and processing. It is useful when consistent traffic distribution and tenant fairness are important.

Token-Bucket Rate Limiting

Token buckets are particularly useful for LLM applications because they allow controlled bursts while maintaining an average request rate.
A bucket has a maximum capacity and refills tokens at a configured rate.

For example:

• Bucket capacity: 20 tokens.
• Refill rate: 2 tokens per second.
• Request cost: 1 token.

Requests are admitted when sufficient tokens are available. Otherwise, they are delayed or rejected.

Implementing Distributed Rate Limits With Redis

An in-memory rate limiter works for a single application process but becomes unreliable when multiple backend instances share provider credentials.

Suppose an application runs five instances, each allowing 100 requests per minute. Without shared coordination, the combined system could admit substantially more than the intended organization-wide allowance.

Redis provides shared state for distributed rate limiting. Its documentation describes using atomic operations and Lua scripts to implement token buckets without race conditions between concurrent requests.

The following Lua script demonstrates the core token-bucket operation:

local key = KEYS[1]

local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local bucket = redis.call(
    "HMGET",
    key,
    "tokens",
    "last_refill"
)

local tokens = tonumber(bucket[1])
local last_refill = tonumber(bucket[2])

if tokens == nil then
    tokens = capacity
    last_refill = now
end

local elapsed = math.max(0, now - last_refill)

tokens = math.min(
    capacity,
    tokens + elapsed * refill_rate
)

local allowed = 0

if tokens >= requested then
    tokens = tokens - requested
    allowed = 1
end

redis.call(
    "HSET",
    key,
    "tokens", tokens,
    "last_refill", now
)

redis.call("EXPIRE", key, 3600)

return {allowed, tokens}

The script refills the bucket, checks available capacity, and updates its state atomically. For production deployments, use a consistent time source such as Redis server time rather than relying on potentially different application-server clocks. Validate configuration values, use tenant-scoped keys, and implement explicit Redis failure handling.

Implementing Exponential Backoff With Jitter

Exponential backoff increases the waiting period after each unsuccessful attempt.

A typical sequence might be:

• First retry: 1 second.
• Second retry: 2 seconds.
• Third retry: 4 seconds.
• Fourth retry: 8 seconds.

A maximum delay prevents waiting periods from growing indefinitely.
However, exponential backoff alone does not prevent synchronized retries. If hundreds of application instances receive an error simultaneously, they might all retry after the same interval.

Why Jitter Matters

Jitter introduces randomness into retry delays so that clients do not repeatedly send requests at identical times.

A commonly used approach is full jitter:

import random

def calculate_backoff(
    attempt: int,
    base_delay: float = 1.0,
    max_delay: float = 30.0
) -> float:

    upper_bound = min(
        max_delay,
        base_delay * (2 ** attempt)
    )

    return random.uniform(0, upper_bound)

For example, a retry with an eight-second upper bound might wait 2.3 seconds instead of always waiting exactly eight seconds.

AWS engineering guidance explains how jitter distributes retry traffic and reduces synchronized requests against overloaded services.

Handling HTTP 429, 5xx, and Timeout Errors

Not every failed request should trigger another attempt. A reliable backend should classify errors before applying its retry policy.

HTTP 429: Too Many Requests

Temporary rate-limit responses are generally retryable.

The application should inspect the response, check for a valid Retry-After header, and wait for the specified period before attempting another request.

If the requested delay exceeds the application's execution deadline, the operation should be deferred or returned as a controlled failure.

HTTP 500 and 503: Server Errors

Temporary server errors may justify retries. However, repeated failures should trigger backoff and potentially a circuit breaker to avoid sending additional traffic to an unhealthy dependency.

HTTP 408 and Network Timeouts

Temporary server errors may justify retries. However, repeated failures should trigger backoff and potentially a circuit breaker to avoid sending additional traffic to an unhealthy dependency.

HTTP 408 and Network Timeouts

Connection interruptions and timeouts may indicate temporary network problems. However, the backend cannot automatically assume that the original operation was never processed. This is especially important for AI agents that execute external operations.

Errors That Generally Should Not Be Retried

HTTP 400: Invalid request.
HTTP 401: Authentication failure.
HTTP 403: Permission or access restriction.
HTTP 404: Missing resource.
Billing-related HTTP 429: Exhausted credits or spending limits.

The exact classification should consider provider-specific error codes and response bodies rather than relying exclusively on HTTP status codes.

Implementing an Asynchronous Retry Mechanism

The following example uses Python 3.11+ and httpx to demonstrate an application-managed retry policy.

import asyncio
import random
import time
import httpx

MAX_ATTEMPTS = 5
BASE_DELAY = 1.0
MAX_DELAY = 30.0
TOTAL_TIMEOUT = 60.0

RETRYABLE_STATUS_CODES = {
    408, 429, 500, 502, 503, 504
}

def get_retry_after(response):
    value = response.headers.get("Retry-After")

    if value is None:
        return None

    try:
        return max(0.0, float(value))
    except ValueError:
        return None


def is_retryable(response):
    if response.status_code not in RETRYABLE_STATUS_CODES:
        return False

    if response.status_code == 429:
        try:
            error = response.json().get("error", {})
            code = error.get("code")

            if code in {
                "credit_balance_exhausted",
                "organization_spend_limit_exceeded",
                "project_spend_limit_exceeded",
                "organization_usage_limit_exceeded"
            }:
                return False

        except (ValueError, AttributeError):
            pass

    return True


async def call_llm_with_retries(
    client, url, headers, payload
):
    deadline = time.monotonic() + TOTAL_TIMEOUT

    for attempt in range(MAX_ATTEMPTS):
        remaining = deadline - time.monotonic()

        if remaining <= 0:
            raise TimeoutError(
                "LLM request deadline exceeded"
            )

        try:
            response = await asyncio.wait_for(
                client.post(
                    url,
                    headers=headers,
                    json=payload
                ),
                timeout=remaining
            )

        except (
            httpx.TimeoutException,
            httpx.TransportError
        ):
            if attempt == MAX_ATTEMPTS - 1:
                raise

            upper_bound = min(
                MAX_DELAY,
                BASE_DELAY * (2 ** attempt)
            )

            delay = random.uniform(0, upper_bound)

        else:
            if response.is_success:
                return response.json()

            if not is_retryable(response):
                response.raise_for_status()

            if attempt == MAX_ATTEMPTS - 1:
                response.raise_for_status()

            server_delay = get_retry_after(response)

            upper_bound = min(
                MAX_DELAY,
                BASE_DELAY * (2 ** attempt)
            )

            delay = random.uniform(0, upper_bound)

            if server_delay is not None:
                delay = max(delay, server_delay)

        if delay >= deadline - time.monotonic():
            raise TimeoutError(
                "Insufficient time for another retry"
            )

        await asyncio.sleep(delay)

    raise RuntimeError("Retry policy exhausted")

This example demonstrates selective retries, bounded attempts, exponential backoff, jitter, and an overall deadline. It deliberately uses direct HTTP requests instead of an SDK with automatic retries to make retry behavior explicit.

Managing Concurrency and Request Queues

Rate limits control how frequently requests are submitted. Concurrency limits control how many requests can execute simultaneously.

An application might remain below its requests-per-minute allowance but still overwhelm its infrastructure because too many long-running LLM requests are active.

Python's asyncio.Semaphore provides a simple way to restrict concurrent operations:

import asyncio

MAX_CONCURRENT_REQUESTS = 10

llm_semaphore = asyncio.Semaphore(
    MAX_CONCURRENT_REQUESTS
)

async def execute_llm_request(client, payload):
    async with llm_semaphore:
        return await client.create(**payload)

However, distributed deployments require shared coordination because every application instance maintains its own semaphore.

Queues are also useful for background workloads such as document summarization, embedding generation, batch classification, and knowledge-base indexing.

A production queue should support bounded capacity, task expiration, dead-letter handling, and backpressure.

Preventing Duplicate Requests and Retry Storms

Retries introduce an important distributed-systems challenge: the client might not know whether the original operation succeeded.

For example, an AI agent might request a payment operation through an external tool. The downstream service completes the payment, but the response is lost because of a network timeout.

If the agent repeats the operation without duplicate protection, the payment could execute twice.

Use Idempotency Keys

An idempotency key identifies a logical operation across repeated requests.

import uuid

operation_id = str(uuid.uuid4())

headers = {
    "Idempotency-Key": operation_id
}

The backend should persist the operation identifier and processing status.

A reliable workflow should:

• Receive the operation and its idempotency key.
• Check whether the key already exists.
• Return the previous result if processing has completed.
• Avoid starting another execution if the operation is in progress.
• Execute and persist the result if the operation is new.

Use database constraints or atomic operations to prevent concurrent requests from creating duplicate records.

An idempotency key is effective only when the receiving service enforces it. Sending a header alone does not guarantee duplicate protection.

Introduce Retry Budgets and Circuit Breakers

A retry budget limits how much additional traffic the system can generate through retries.

Useful controls include:

• Maximum retries per operation.
• Retry limits per tenant.
• Global retry budgets.
• Reduced retry activity during provider outages.
• Circuit breakers for unhealthy dependencies.
• Fallback models or controlled degraded responses.

Circuit breakers generally operate in three states:

• Closed: Requests are allowed normally.
• Open: Requests are temporarily rejected or redirected.
• Half-open: A limited number of test requests determine whether the dependency has recovered.

Monitoring Rate Limits and Retry Behavior

A rate-limiting system cannot be considered production-ready without observability. Developers should monitor:

• Rate-limit metrics: Requests admitted, requests rejected, HTTP 429 responses, remaining token allowances, and rate-limit wait duration.
• Retry metrics: Total attempts, retries per successful operation, retry exhaustion, backoff duration, and retry success rate.
• Performance metrics: End-to-end latency, provider response time, queue waiting time, active requests, and timeout frequency.
• Cost metrics: Input tokens, output tokens, retry-related consumption, and usage by tenant or workflow.

Production Deployment and Security Considerations

Moving rate limiting and retries into production requires additional operational safeguards.

Enforce Tenant-Level Isolation

Shared provider credentials can create resource contention between customers.

Maintain individual tenant budgets alongside organization-wide limits. Derive rate-limit keys from authenticated tenant identities rather than untrusted request parameters.

Protect Provider Credentials

Store API credentials in a secrets manager or secure environment-specific configuration. Never expose provider keys through frontend applications or client-visible errors.

Define a Redis Failure Policy

A distributed rate limiter introduces another infrastructure dependency.

If Redis becomes unavailable, the application must decide whether to fail closed, fail open, or apply conservative local limits.

For enterprise systems with strict spending requirements, unrestricted fail-open behavior can create unexpected API consumption.

Enforce Execution Deadlines

Define separate limits for connection establishment, provider requests, queue waiting, retry delays, and total workflow execution.

A request should not remain active indefinitely because each retry receives a fresh timeout.

Validate Failure Scenarios

Before deployment, test simulated HTTP 429 responses, server errors, connection failures, billing-related errors, Redis outages, duplicate operations, retry exhaustion, and cancellation during backoff.

Multi-instance load testing should verify that distributed request and concurrency limits remain consistent under realistic traffic.

Conclusion

Rate limits and retries are essential components of reliable LLM infrastructure. Combining token-aware admission control, distributed rate limiting, exponential backoff, jitter, concurrency management, and idempotency helps prevent failures and excessive API consumption. Production systems should also enforce deadlines, retry budgets, and comprehensive monitoring.

FAQs

1. What is the difference between rate limiting and throttling in LLM applications?

Rate limiting defines permitted request or token consumption. Throttling delays or rejects requests when traffic exceeds available capacity.

2. Should LLM applications retry every HTTP 429 response?

No. Temporary throttling can be retried, but billing, credit, and usage-limit errors require administrative intervention.

3. Why is exponential backoff with jitter recommended?

Exponential backoff increases waiting periods, while jitter randomizes delays. Together, they reduce synchronized retries and pressure on overloaded providers.

4. Should rate limits be implemented at the application or provider level?

Both. Provider limits protect external infrastructure, while application-level limits enforce tenant quotas, control costs, and protect shared resources.

5. How can developers prevent duplicate AI agent tool executions?

Use idempotency keys, persistent operation records, atomic state transitions, and downstream duplicate protection.