The boring layer around your LLM call

Most of what I learned this year came from building the interesting parts. The retrieval, the prompts, the agent loop. The stuff that's fun to think about.

Then I put a FastAPI endpoint in front of one of my projects, showed it to a friend, and watched him break it in about four minutes. Not maliciously. He just asked something long and weird and the request sat there spinning until he closed the tab.

That's when I realised the model was maybe a third of the actual work. The rest is the stuff wrapped around it, which is boring and nobody writes tutorials about it because it isn't interesting to build.

This is what I've ended up with. I'm still a student, so treat this as notes rather than advice from someone who's run this at scale. I probably have some of it wrong.

The call has no timeout by default and that is a problem

This was the first one that bit me.

I assumed there was a sensible default somewhere. There sort of is, depending on your client, but it tends to be very long or effectively absent. So when a provider gets slow, your request doesn't fail. It just waits. And while it waits it's holding a worker that can't do anything else.

response = await litellm.acompletion(
    model="deepseek/deepseek-chat",
    messages=messages,
    timeout=30,
)

Thirty seconds felt aggressive to me at first, then I actually measured my p95 and realised nothing legitimate was taking longer than about fifteen. If a call is at thirty seconds it's already gone wrong and waiting longer doesn't help.

One thing that confused me for a while: if you're behind something with its own timeout (nginx, a cloud load balancer, an API gateway) and yours is longer than theirs, you get the worst version. The user gets a 504, and your call carries on running and carries on costing money for an answer nobody will ever see. Yours should be the shorter one.

Retries are easy to get half right

Everyone tells you to retry with exponential backoff. That part's fine, most libraries do it for you.

What I didn't think about was that different errors mean different things.

A 429 means slow down, you're going too fast, and backing off is exactly right. A 500 means the provider had a problem and retrying is reasonable. A 400 means your request was malformed and retrying it will produce the identical error every time while you pay for the privilege of finding out.

response = await litellm.acompletion(
    model="deepseek/deepseek-chat",
    messages=messages,
    timeout=30,
    num_retries=2,
)

Two retries, not five. I had a bug at one point where a retry loop and a validation failure fed each other and the same request went out something like a dozen times before I noticed. Nothing dramatic happened because my test corpus was tiny and it was pennies, but the same shape of bug on a real workload is how you end up explaining a bill to someone.

Also worth logging the requests that exhausted their retries and gave up. It's easy to only log errors that surface to the user, and the ones that quietly failed after three attempts are exactly the ones you want to know about.

Both ends need a token ceiling

I knew about max_tokens. Everyone knows about max_tokens. It bounds what comes back.

It took me longer to think properly about the input side, and in RAG that's where the risk actually is, because you're stuffing retrieved chunks into the prompt and you don't fully control what those chunks contain. Most of my documents were normal. One of them was enormous. It went through the same code path as everything else and cost roughly ten times what a typical request cost, and I only found it because I was staring at per-call costs for an unrelated reason.

So now I count tokens before sending, not after:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def cap_context(chunks, budget=6000):
    kept, used = [], 0
    for chunk in chunks:
        cost = len(enc.encode(chunk))
        if used + cost > budget:
            break
        kept.append(chunk)
        used += cost
    return kept

Crude, and it drops chunks by position rather than by relevance, which isn't ideal. But a crude ceiling you actually have beats an elegant one you're planning to add.

Fallbacks matter more than I expected

Providers go down. Not often, but they do, and when it happens there is nothing you can do except wait, which is a bad thing to discover during a demo.

LiteLLM makes this genuinely easy, which is most of why I use it:

response = await litellm.acompletion(
    model="deepseek/deepseek-chat",
    messages=messages,
    timeout=30,
    num_retries=2,
    fallbacks=["gemini/gemini-2.0-flash"],
)

The fallback doesn't have to be as good. That's the bit I initially misunderstood. It's the difference between a slightly worse answer and no answer at all, and users are much more forgiving of the first one.

Do actually test it though. I had a fallback configured for a while that would have failed if it ever triggered, because the model name was wrong and nothing had ever exercised that path. I found it by deliberately putting a garbage primary model name in and seeing what happened, which took two minutes and I should have done it immediately.

Validation tells you something broke, not what to do about it

Pydantic is great. You define the shape you want, you get a clean error when the model returns something else.

from pydantic import BaseModel

class Answer(BaseModel):
    text: str
    confidence: float
    sources: list[str]

What tripped me up is that catching the error is only half a decision. You still have to choose what happens next, and I didn't choose for a while, which meant my choice was "throw a 500 at the user" by default.

The options I've ended up thinking about:

Retry once, feeding the validation error back into the prompt. This works surprisingly often for small schema mistakes. It also costs you another call and another few seconds, so it's not free.

Fall back to something simpler. If the structured version keeps failing, take plain text and lose the structure rather than losing the response.

Fail properly, with a real message. Sometimes this is right. But "sorry, something went wrong" is much better than a stack trace, and it's better than silently returning an empty object that breaks something three layers up.

None of these is correct in general. The point is just to pick one on purpose.

A spend ceiling, because dashboards tell you afterwards

This is the one I'd add first if I were starting again.

Provider dashboards are good. They are also retrospective. They tell you what you spent after you spent it, and if something loops overnight you find out in the morning.

So I keep a counter in the process:

class SpendGuard:
    def __init__(self, ceiling_usd):
        self.ceiling = ceiling_usd
        self.spent = 0.0

    def record(self, response):
        cost = response._hidden_params.get("response_cost", 0)
        self.spent += cost
        if self.spent > self.ceiling:
            raise RuntimeError(
                f"Spend ceiling hit: ${self.spent:.2f} of ${self.ceiling:.2f}"
            )

It's twenty lines and it's naive. It resets when the process restarts, and it won't help you across multiple workers unless you move the counter somewhere shared like Redis. But it turns "unbounded" into "bounded", and that's the part that actually matters. My entire dissertation experiment ran on about four pounds of compute, and knowing a bug couldn't turn that into four hundred let me iterate a lot more freely.

If you're on a hosted provider, set a hard billing limit in their console too. Belt and braces.

What I'd do differently

Honestly, I'd write all of this before writing any of the interesting parts.

Every item here I added after something surprised me, which meant each one arrived as a small panic rather than as a decision. It's not much code. It's a timeout, a retry cap, two token ceilings, a fallback, a validation branch, and a counter. Maybe an hour to put in place at the start, versus finding each one individually the hard way.

The model is the part everyone talks about. The stuff around it is what determines whether the thing survives contact with an actual user.

If you're further along than me and I've got something wrong here, I'd genuinely like to know.

Comments

[Luis Cruz]

I appreciated the point about setting a timeout for LLM calls, as it's easy to overlook this detail and end up with requests waiting indefinitely. The example of setting a 30-second timeout with await litellm.acompletion(..., timeout=30) is a good one, and I've found that measuring the p95 of my own requests has helped me determine a suitable default. Have you considered implementing a circuit breaker pattern to handle cases where the LLM provider is experiencing issues, or do you rely on retries and timeouts to handle these situations?

[Veera Venkata Satyanarayana Gannamraju]

I've been leaning on retries and the fallback, the reason I didn't reach for one is that I've only ever run this at small scale, where an outage means my own requests hang rather than a queue backing up. At real traffic the difference is much bigger.
Did you end up implementing one yourself, or using something off the shelf? I've seen pybreaker mentioned but haven't tried it.

[Luis Cruz]

Thanks for sharing your perspective! Yes—I’ve implemented circuit breaker patterns in a few production AI systems, although the implementation depends on the workload and reliability requirements.

For smaller projects, retries, sensible timeouts, and provider fallbacks are often enough. As traffic grows, though, a circuit breaker becomes valuable because it prevents repeatedly calling a degraded provider, reduces cascading failures, and gives fallback models or cached responses a chance to take over until the service recovers.

I've built lightweight implementations as well as integrated them with resilience libraries, and I usually combine them with metrics, health checks, and cost monitoring to make routing decisions dynamically rather than relying on a fixed fallback order.

I haven't used pybreaker extensively either, but it looks like a solid option if you don't want to maintain the state machine yourself. I'd be interested to hear how your architecture evolves as your project scales. Feel free to connect if you'd ever like to exchange ideas—or collaborate on AI infrastructure or agent engineering in the future.

[Reid Marlow]

Circuit breakers are one of the first things I add once traffic is high enough that retries can stampede. For a small tool, timeout plus retry cap plus fallback is usually enough. The bit I would add before scale is a clear open state so the app can fail fast instead of quietly burning workers.

[Veera Venkata Satyanarayana Gannamraju]

"Fail fast instead of quietly burning workers" - that's the actual failure mode I had, better put than I managed. And an open state is barely more code than the retry cap I already have. Will consider adding it, thanks.