Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.

LLM API Error Handling: Retries, Rate Limits, and Backoff (2026)

The first time you put an LLM feature in front of real users, the model quality is rarely what breaks. What breaks is the network in between: a 429 during a traffic spike, a 529 when the whole platform is busy, a connection reset on a 4-minute generation. If your code treats every non-200 response the same way, you will either retry things that can never succeed (wasting money and time) or give up on things that would have worked on the second try.

Reliability lives in an unglamorous layer underneath the model call: which errors to retry, how long to wait, and what the official SDKs already do for you so you do not reinvent it badly. Get those three right and the same flaky network becomes invisible to your users.

Start by sorting errors into three buckets

Every HTTP error from an LLM API falls into one of three categories, and the right action is completely different for each. The single most useful habit is to classify the status code before deciding to retry.

Bucket	Example codes	What it means	Action
Don't retry (you broke it)	400, 401, 403, 404, 413, 422	Bad request, bad key, no permission, payload too large	Fix the request or config; retrying repeats the same failure
Retry with backoff	408, 409, 429, 500, 502, 503, 504, 529	Transient: timeout, conflict, rate limit, server error, overload	Wait, then retry a bounded number of times
Special handling	429 vs 529, streaming refusals	Needs logic beyond "wait and retry"	See sections below

The reason this matters: a 401 authentication_error will never succeed no matter how many times you retry it, and burning five attempts on it just delays the error your user sees. A 529 overloaded_error, on the other hand, almost always succeeds within a few seconds. Same "request failed" feeling for the user, opposite correct response from your code.

The Anthropic error shape, concretely

Anthropic's API uses a predictable set of codes and type names: 400 invalid_request_error, 401 authentication_error, 402 billing_error, 403 permission_error, 404 not_found_error, 413 request_too_large, 429 rate_limit_error, 500 api_error, 504 timeout_error, and 529 overloaded_error. Errors always arrive as JSON with a top-level error object holding a type and message, plus a request_id:

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "Number of request tokens has exceeded your per-minute rate limit."
  },
  "request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
}

Catch the SDK's typed exception classes, not the message string. In the Anthropic Python SDK the mapping is 400 BadRequestError, 401 AuthenticationError, 403 PermissionDeniedError, 404 NotFoundError, 409 ConflictError, 422 UnprocessableEntityError, 429 RateLimitError, >=500 InternalServerError, and connection failures to APIConnectionError. Handle the most specific classes first. String-matching the message field is fragile because, per Anthropic's versioning policy, the human-readable text can change at any time.

429 and 529 are not the same problem

These two codes both mean "try again later," but they have different root causes, and confusing them leads to the wrong fix.

• 429 rate_limit_error is about your usage. You exceeded one of your own limits. The remedy is on your side: slow down, spread out, or raise your tier.
• 529 overloaded_error is about the platform. Anthropic documents that 529s "can occur when the API experiences high traffic across all users" — it is not caused by your own usage. The remedy is simply to back off and retry; there is nothing in your code to "fix."

There is one nuance worth knowing: in rare cases a sharp, sudden spike in your organization's usage can produce 429s due to acceleration limits, even when you are under your steady-state limit. Anthropic's guidance is to ramp traffic up gradually rather than going from zero to full throttle.

Rate limits: the three dimensions you can blow through

Anthropic enforces limits on the Messages API across three dimensions per model class, and exceeding any one of them returns a 429:

• RPM — requests per minute
• ITPM — input tokens per minute
• OTPM — output tokens per minute

This is why "I'm only sending 10 requests a minute" can still get rate-limited: each request might carry a 150K-token context, so you hit ITPM long before RPM. Limits use a token-bucket algorithm that continuously replenishes capacity rather than resetting on a fixed clock, so short bursts can trip a limit even when your per-minute average looks fine.

Limits scale by usage tier, and your organization moves up tiers automatically over time:

Tier	Claude Opus 4.x RPM	Opus 4.x ITPM	Opus 4.x OTPM
Start	1,000	2,000,000	400,000
Scale	10,000	10,000,000	2,000,000

A high-leverage detail: for most current Claude models, only uncached input tokens (input_tokens + cache_creation_input_tokens) count toward ITPM. Tokens served from cache (cache_read_input_tokens) do not count. So prompt caching does not just cut your bill — it raises your effective throughput against the rate limit. With an 80% cache hit rate, a 2M ITPM limit effectively processes around 10M total input tokens per minute.

Read the headers; don't guess the wait

When you hit a 429, the response tells you exactly how long to wait. Use it instead of guessing.

Header	Use it for
retry-after	Seconds to wait before retrying; earlier retries fail
anthropic-ratelimit-requests-remaining	How many requests you have left this window
anthropic-ratelimit-input-tokens-remaining	Input-token headroom (rounded to nearest 1,000)
anthropic-ratelimit-input-tokens-reset	When input-token capacity refills (RFC 3339)
anthropic-ratelimit-output-tokens-reset	When output-token capacity refills (RFC 3339)

The *-remaining headers let you do proactive throttling: if you see remaining tokens getting low, slow down before you get a 429 at all. The retry-after value should always win over any computed backoff for rate-limit errors — the server knows precisely when its bucket refills, and retrying earlier than retry-after is guaranteed to fail.

Exponential backoff with jitter, and why jitter isn't optional

For the retryable bucket, the wait strategy is settled engineering practice, and both OpenAI and AWS document the same shape: exponential backoff plus random jitter.

• Exponential means the delay grows each attempt (e.g. 1s, 2s, 4s, 8s) so early retries are fast but repeated failures back off hard.
• Jitter means you add randomness so that many clients that failed at the same instant do not all retry at the same instant. This synchronized-retry failure is widely known as the thundering herd problem: without jitter, every client that errored together retries together, recreating the exact overload that caused the errors.

AWS's recommended "full jitter" picks a random delay uniformly between 0 and the (capped) exponential value. Their analysis confirms it "spreads out the spikes to an approximately constant rate" instead of clustering retries into waves. OpenAI's cookbook says the same thing in plainer words: "Adding random jitter to the delay helps retries from all hitting at the same time."

A minimal, correct implementation:

import random, time

def backoff_delay(attempt, base=1.0, cap=20.0):
    # full jitter: uniform between 0 and the capped exponential value
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)

# On 429, prefer the server's retry-after over computed backoff:
delay = retry_after if retry_after is not None else backoff_delay(attempt)
time.sleep(delay)

Cap the backoff (AWS SDKs cap around 20 seconds) and cap the number of attempts. Unbounded retries turn a transient blip into a self-inflicted outage.

What the SDKs already do — so you don't double up

Before you write any of the above, know that the official SDKs already retry for you, and stacking your own retry loop on top can multiply the real attempt count without you realizing it.

Behavior	Anthropic Python SDK	OpenAI Python SDK
Default retries	max_retries=2	max_retries=2
Backoff	Short exponential backoff	Short exponential backoff
Retried automatically	Connection errors, 408, 409, 429, ≥500	Connection errors, 408, 409, 429, ≥500
Configure globally	Anthropic(max_retries=0)	client init max_retries
Configure per request	client.with_options(max_retries=5)	with_options()
Default timeout	10 minutes (raises APITimeoutError, retried twice)	configurable

Both SDKs default to two automatic retries with exponential backoff for exactly the transient bucket above. So if you wrap a 3-attempt loop around an SDK call, you can end up with up to nine real attempts. The cleaner pattern is to set the SDK's max_retries to the value you want and let it own the retry logic, or set it to 0 and own retries entirely yourself — but not both at once. The Anthropic SDK also sets a 10-minute default timeout and a TCP keep-alive socket option, which is why it recommends streaming or the Batch API for anything that legitimately runs longer.

For a deeper look at how token usage drives both your bill and your ITPM ceiling, see our LLM API cost control guide and the Claude vs GPT vs Gemini API comparison.

Refusals are not errors — and not naively retryable

One failure mode looks like a success at the HTTP layer but needs special handling. When Anthropic's streaming classifiers detect a policy violation, the API returns stop_reason: "refusal" on a 200 response. This is not retryable as-is: retrying with the same conversation history just triggers another refusal. You must reset or prune the context — remove or rephrase the offending turn, or clear history — before retrying. The silver lining: when a refusal arrives before any output is generated, you are not billed for that request.

The order to handle a failed request

When a request fails, walk this in order:

1. Read the status code, not the message. Classify into don't-retry / retry / special.
2. If it's in the don't-retry bucket (400/401/403/404/413/422), surface a clear error and stop. No backoff will help.
3. If it's 429, honor the retry-after header verbatim. Consider proactive throttling using the *-remaining headers.
4. If it's 408/409/500/502/503/504/529, retry with capped exponential backoff plus full jitter.
5. Bound everything — cap attempts and cap delay so a transient problem can't become a permanent one.
6. Don't double-retry. Decide whether the SDK or your code owns retries; configure max_retries accordingly.
7. Log the request_id on every failure so support can trace a specific call.

Source map

Most of the specifics here come straight from Anthropic's API documentation: the status codes and type names, the JSON error shape and request_id, the stop_reason: "refusal" behavior, the RPM/ITPM/OTPM dimensions, the rate-limit header names, the usage-tier numbers, and the rule that only uncached input counts toward ITPM are all from the linked Errors and Rate-limits pages. The max_retries=2 default, the retried-status list, and the 10-minute timeout are from the Anthropic and OpenAI Python SDK references. The case for exponential backoff plus full jitter — including the verbatim quotes — is drawn from OpenAI's cookbook and the AWS Architecture Blog; "thundering herd" is the standard industry term for the failure mode jitter fixes, rather than a phrase drawn verbatim from that post. One number to treat as perishable rather than fixed: usage-tier limits shift by model and account over time, so confirm yours on the Console Limits page before sizing capacity.

Sources

• Anthropic API — Errors (status codes, error shapes, request IDs, refusals)
• Anthropic API — Rate limits (RPM/ITPM/OTPM, tiers, headers, cache tokens)
• Anthropic Python SDK — error types, retries, and timeouts
• OpenAI Python SDK reference — automatic retries
• OpenAI Cookbook — How to handle rate limits
• AWS Architecture Blog — Exponential Backoff and Jitter