DEV Community

Deependra Rajput
Deependra Rajput

Posted on

Retries, Timeouts and Fallbacks: Making LLM Calls Reliable

LLM APIs fail far more often than a typical database or internal service. You'll see rate limits (429), timeouts on long generations, occasional 5xx errors, and responses that arrive but aren't usable. If your app assumes every call succeeds, your users will find every one of those failures for you.

Here are the five patterns I put around every production LLM call.

1. Always set a timeout

The default in many HTTP clients is "wait forever". For a user-facing feature, a request hanging for two minutes is worse than a fast failure.

import httpx

client = httpx.AsyncClient(timeout=httpx.Timeout(20.0, connect=5.0))
Enter fullscreen mode Exit fullscreen mode

Pick the number from your real latency data: comfortably above your slowest normal response, well below the point where users give up.

2. Retry the right errors, with backoff and jitter

Retry errors that are likely to succeed on a second try, and nothing else. A 400 Bad Request will fail again no matter how many times you send it.

from tenacity import (
    retry,
    retry_if_exception,
    stop_after_attempt,
    wait_random_exponential,
)


def is_retryable(exc: BaseException) -> bool:
    if isinstance(exc, httpx.TimeoutException):
        return True
    if isinstance(exc, httpx.HTTPStatusError):
        return exc.response.status_code in {429, 500, 502, 503, 504}
    return False


@retry(
    retry=retry_if_exception(is_retryable),
    wait=wait_random_exponential(min=1, max=20),
    stop=stop_after_attempt(4),
    reraise=True,
)
async def call_llm(payload: dict) -> dict:
    response = await client.post(LLM_URL, json=payload)
    response.raise_for_status()
    return response.json()
Enter fullscreen mode Exit fullscreen mode

The random jitter matters. Without it, every client that got rate-limited at the same moment retries at the same moment, and you hit the limit again.

3. Have a fallback

When retries run out, decide what the user sees. Options, roughly from best to worst:

  • a secondary or smaller model
  • a cached answer to the same or a similar question
  • an honest "try again in a minute" message
async def answer(question: str) -> str:
    try:
        return await primary_model(question)
    except Exception:
        logger.exception("primary model failed")
        cached = await cache.get(question_key(question))
        return cached or "Sorry, I can't answer that right now. Please try again in a minute."
Enter fullscreen mode Exit fullscreen mode

A clear message is always better than a spinner that never stops.

4. Cap your own concurrency

Most rate limits are hit by your own traffic spikes. A semaphore keeps bursts under the provider's limits instead of turning them into a wave of 429s:

import asyncio

limiter = asyncio.Semaphore(10)


async def limited_call(payload: dict) -> dict:
    async with limiter:
        return await call_llm(payload)
Enter fullscreen mode Exit fullscreen mode

If you run several workers or servers, move the limit into something shared, such as Redis.

5. Measure all of it

Log latency, retry count and whether a fallback was used for every call. Those three numbers tell you when a provider is degrading, when your timeout is too tight, and whether your fallback is quietly handling 20% of traffic.

Wrapping up

Timeouts, targeted retries, fallbacks, concurrency limits and metrics are boring individually. Together they're the difference between an AI feature that demos well and one that survives real traffic.

What failure modes have bitten you in production? Let me know in the comments.

Top comments (0)