Fixing "TooManyRequests" From Azure OpenAI Under Load

#ai #debugging #llm #machinelearning

HTTP 429 from Azure OpenAI is four different problems sharing one status code. Three of them are fixed by backing off and one is not, and the response headers distinguish them in about a line of code. Most teams skip that line and file a quota increase for a condition that would have cleared on its own.

The error

The SDK surfaces it as a rate-limit error — openai.RateLimitError in Python, a RequestFailedException with Status == 429 in .NET. The message text is the first discriminator, and Microsoft documents the indicator phrases rather than a single fixed string:

"Requests to … have been limited" or "Rate limit is exceeded"
"The service is temporarily unable to process your request" or "System is experiencing high demand"

Those two groups mean opposite things. The first is your allocation; the second is Azure’s capacity. Log the message body on every 429 — without it you are guessing. Microsoft, Manage Azure OpenAI quota.

Four causes wearing one status code

Rate limit exceeded. Your traffic genuinely passed the deployment’s TPM or RPM allocation. Remedy: raise the deployment’s TPM, rebalance quota from an underused deployment, or request an increase.
System capacity throttling. Backend capacity is constrained. Documented as often transient. Remedy: retry after the delay the service gives you. A quota increase does nothing here.
Temporary rate limit adjustment. The one worth knowing about. Standard and Global Standard deployments share a resource pool across customers, and Microsoft documents that when demand approaches capacity limits the system may temporarily reduce your deployment’s effective rate limit to keep the pool reliable. Your configured quota has not changed. The adjustment typically resolves within a few hours.
Token budget consumed by parameters. The rate-limit calculation includes max_tokens and the prompt estimate, not the tokens actually generated. A request with a large max_tokens spends that budget whether or not it uses it.

Two more mechanics explain 429s that look impossible. RPM is enforced over short windows — Microsoft documents evaluation typically at 1 or 10 seconds — so a 600-RPM deployment throttles if more than about 10 requests arrive in one second, even though the minute total is fine. And unsuccessful requests still count toward your per-minute rate limit, which is why an aggressive retry loop makes throttling worse rather than better.

The headers that tell you which

Azure OpenAI returns rate-limit information on every call. The documented headers:

x-ratelimit-limit-requests      # e.g. 60      requests/min for this deployment
x-ratelimit-limit-tokens        # e.g. 150000  tokens/min for this deployment
x-ratelimit-remaining-requests  # e.g. 59
x-ratelimit-remaining-tokens    # e.g. 149984
x-ratelimit-reset-requests      # e.g. 10      until the request limit resets
x-ratelimit-reset-tokens        # e.g. 300     until the token limit resets
retry-after-ms                  # e.g. 2000    on 429s: recommended wait, in ms

Note the unit on the last one. The header Microsoft documents for Azure OpenAI is retry-after-ms and its value is milliseconds. Client code written against the more familiar seconds-valued Retry-After convention will sleep 2,000 seconds where it meant to sleep two, or two milliseconds where it meant two seconds — read whichever header is present and convert explicitly rather than assuming.

And here is the diagnostic that resolves cause three. Compare x-ratelimit-limit-tokens against the TPM you configured on the deployment. If the header is lower, a temporary protective reduction is active — that is the documented way to detect it. Retry with backoff and wait it out; a quota request will not shorten it.

Monitoring x-ratelimit-remaining-tokens in normal operation lets you throttle before you are throttled, which is strictly better than discovering the limit by hitting it. Microsoft’s guidance is explicit on what each signal is for: use token usage metrics in Azure Monitor to understand billed consumption, and use the HTTP status codes and these response headers to detect and respond to rate-limit enforcement in real time. They will not reconcile, and expecting them to is the root of most of the confusion in this area.

One more documented asymmetry explains 429s that survive a quota increase. Approved quota is a subscription-and-region pool; the rate limit that rejected your request is the TPM assigned to the specific deployment receiving traffic. Quota approved but never allocated changes nothing. TPM moves between deployments of the same model freely, in increments of 1,000, so rebalancing from an idle deployment is usually faster than any request form.

Retrying without making it worse

The simplest correct answer is the SDK’s own retry. The OpenAI Python SDK from v1.0 has built-in automatic retry with exponential backoff for 429 and transient errors, and Microsoft documents the default as two retries:

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://my-aoai.openai.azure.com/",
    api_key="<key>",
    api_version="2024-10-21",
    max_retries=5,   # default is 2
)

If you need custom behaviour — logging, a circuit breaker, selective handling — reach for a retry library, and then heed the documented warning that catches almost everybody:

client = AzureOpenAI(
    azure_endpoint="https://my-aoai.openai.azure.com/",
    api_key="<key>",
    api_version="2024-10-21",
    max_retries=0,   # REQUIRED when wrapping with tenacity/Polly
)

@retry(
    wait=wait_random_exponential(min=1, max=60),
    stop=stop_after_attempt(6),
    retry=retry_if_exception_type(openai.RateLimitError),
    reraise=True,
)
def chat_with_backoff(**kwargs):
    return client.chat.completions.create(**kwargs)

Without max_retries=0, each of your six outer attempts triggers up to two more SDK retries underneath. Six becomes eighteen, every one of them counts against the per-minute limit, and you have built an amplifier for the condition you were trying to survive.

The rest of the documented backoff recipe: honour retry-after-ms where present, otherwise exponential backoff with random jitter so clients do not resynchronise, and always a maximum attempt count.

When retrying is not the answer

Before adding retries, remove the cause:

Shrink max_tokens to the smallest value that serves the case. Microsoft’s first listed optimisation, and the highest-leverage one, because the estimate is what the limiter counts. Set best_of to 1 unless you truly need it — each increment multiplies the counted tokens.
Check the allocation is where the traffic is. Approved quota at the subscription level does nothing if it is not assigned to the deployment receiving requests. TPM is modified in increments of 1,000 and moves between deployments freely.
Flatten the burst. RPM enforcement expects requests evenly distributed across the minute. Ramp new workloads gradually; a queue with a paced consumer converts a spike into a shape the limiter accepts.
Spread across deployments or regions if the workload needs more throughput than one deployment supports, and check first whether your deployments share a quota pool — see how Global Standard pools quota, because a second Global Standard deployment of the same model may add no headroom at all. There are ceilings on this remedy too: Microsoft documents a limit of 30 Azure OpenAI resources per Azure subscription and a maximum of 32 standard deployments per resource, so “add another deployment” is a finite strategy rather than an unbounded one.
Move to provisioned throughput for latency-critical work. Microsoft’s escalation table is direct: sustained 429s in production below approved quota is a support request; frequent 429s on a mission-critical workload is a signal to buy dedicated capacity.

Spreading one workload across two deployments is straightforward until you look at the details: each deployment has its own key or its own token audience, its own rate-limit header values, and possibly its own model version. Something has to hold the per-deployment budget, read the headers from whichever one answered, and decide where the next request goes. That routing and normalisation layer is what an LLM gateway is; building it inside your application means every service that calls a model gets its own copy of the logic and its own bugs in it.

Header names, the SDK retry defaults and the documented root-cause taxonomy are Microsoft’s current published guidance. Verify against the quota management article before you encode a header name.