LLM APIs fail far more often than a typical database or internal service. You'll see rate limits (429), timeouts on long generations, occasional 5xx errors, and responses that arrive but aren't usable. If your app assumes every call succeeds, your users will find every one of those failures for you.
Here are the five patterns I put around every production LLM call.
1. Always set a timeout
The default in many HTTP clients is "wait forever". For a user-facing feature, a request hanging for two minutes is worse than a fast failure.
import httpx
client = httpx.AsyncClient(timeout=httpx.Timeout(20.0, connect=5.0))
Pick the number from your real latency data: comfortably above your slowest normal response, well below the point where users give up.
2. Retry the right errors, with backoff and jitter
Retry errors that are likely to succeed on a second try, and nothing else. A 400 Bad Request will fail again no matter how many times you send it.
from tenacity import (
retry,
retry_if_exception,
stop_after_attempt,
wait_random_exponential,
)
def is_retryable(exc: BaseException) -> bool:
if isinstance(exc, httpx.TimeoutException):
return True
if isinstance(exc, httpx.HTTPStatusError):
return exc.response.status_code in {429, 500, 502, 503, 504}
return False
@retry(
retry=retry_if_exception(is_retryable),
wait=wait_random_exponential(min=1, max=20),
stop=stop_after_attempt(4),
reraise=True,
)
async def call_llm(payload: dict) -> dict:
response = await client.post(LLM_URL, json=payload)
response.raise_for_status()
return response.json()
The random jitter matters. Without it, every client that got rate-limited at the same moment retries at the same moment, and you hit the limit again.
3. Have a fallback
When retries run out, decide what the user sees. Options, roughly from best to worst:
- a secondary or smaller model
- a cached answer to the same or a similar question
- an honest "try again in a minute" message
async def answer(question: str) -> str:
try:
return await primary_model(question)
except Exception:
logger.exception("primary model failed")
cached = await cache.get(question_key(question))
return cached or "Sorry, I can't answer that right now. Please try again in a minute."
A clear message is always better than a spinner that never stops.
4. Cap your own concurrency
Most rate limits are hit by your own traffic spikes. A semaphore keeps bursts under the provider's limits instead of turning them into a wave of 429s:
import asyncio
limiter = asyncio.Semaphore(10)
async def limited_call(payload: dict) -> dict:
async with limiter:
return await call_llm(payload)
If you run several workers or servers, move the limit into something shared, such as Redis.
5. Measure all of it
Log latency, retry count and whether a fallback was used for every call. Those three numbers tell you when a provider is degrading, when your timeout is too tight, and whether your fallback is quietly handling 20% of traffic.
Wrapping up
Timeouts, targeted retries, fallbacks, concurrency limits and metrics are boring individually. Together they're the difference between an AI feature that demos well and one that survives real traffic.
What failure modes have bitten you in production? Let me know in the comments.
Top comments (0)