DEV Community: Veera Venkata Satyanarayana Gannamraju

The boring layer around your LLM call

Veera Venkata Satyanarayana Gannamraju — Wed, 05 Aug 2026 14:45:12 +0000

Most of what I learned this year came from building the interesting parts. The retrieval, the prompts, the agent loop. The stuff that's fun to think about.

Then I put a FastAPI endpoint in front of one of my projects, showed it to a friend, and watched him break it in about four minutes. Not maliciously. He just asked something long and weird and the request sat there spinning until he closed the tab.

That's when I realised the model was maybe a third of the actual work. The rest is the stuff wrapped around it, which is boring and nobody writes tutorials about it because it isn't interesting to build.

This is what I've ended up with. I'm still a student, so treat this as notes rather than advice from someone who's run this at scale. I probably have some of it wrong.

The call has no timeout by default and that is a problem

This was the first one that bit me.

I assumed there was a sensible default somewhere. There sort of is, depending on your client, but it tends to be very long or effectively absent. So when a provider gets slow, your request doesn't fail. It just waits. And while it waits it's holding a worker that can't do anything else.

response = await litellm.acompletion(
    model="deepseek/deepseek-chat",
    messages=messages,
    timeout=30,
)

Thirty seconds felt aggressive to me at first, then I actually measured my p95 and realised nothing legitimate was taking longer than about fifteen. If a call is at thirty seconds it's already gone wrong and waiting longer doesn't help.

One thing that confused me for a while: if you're behind something with its own timeout (nginx, a cloud load balancer, an API gateway) and yours is longer than theirs, you get the worst version. The user gets a 504, and your call carries on running and carries on costing money for an answer nobody will ever see. Yours should be the shorter one.

Retries are easy to get half right

Everyone tells you to retry with exponential backoff. That part's fine, most libraries do it for you.

What I didn't think about was that different errors mean different things.

A 429 means slow down, you're going too fast, and backing off is exactly right. A 500 means the provider had a problem and retrying is reasonable. A 400 means your request was malformed and retrying it will produce the identical error every time while you pay for the privilege of finding out.

response = await litellm.acompletion(
    model="deepseek/deepseek-chat",
    messages=messages,
    timeout=30,
    num_retries=2,
)

Two retries, not five. I had a bug at one point where a retry loop and a validation failure fed each other and the same request went out something like a dozen times before I noticed. Nothing dramatic happened because my test corpus was tiny and it was pennies, but the same shape of bug on a real workload is how you end up explaining a bill to someone.

Also worth logging the requests that exhausted their retries and gave up. It's easy to only log errors that surface to the user, and the ones that quietly failed after three attempts are exactly the ones you want to know about.

Both ends need a token ceiling

I knew about max_tokens. Everyone knows about max_tokens. It bounds what comes back.

It took me longer to think properly about the input side, and in RAG that's where the risk actually is, because you're stuffing retrieved chunks into the prompt and you don't fully control what those chunks contain. Most of my documents were normal. One of them was enormous. It went through the same code path as everything else and cost roughly ten times what a typical request cost, and I only found it because I was staring at per-call costs for an unrelated reason.

So now I count tokens before sending, not after:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def cap_context(chunks, budget=6000):
    kept, used = [], 0
    for chunk in chunks:
        cost = len(enc.encode(chunk))
        if used + cost > budget:
            break
        kept.append(chunk)
        used += cost
    return kept

Crude, and it drops chunks by position rather than by relevance, which isn't ideal. But a crude ceiling you actually have beats an elegant one you're planning to add.

Fallbacks matter more than I expected

Providers go down. Not often, but they do, and when it happens there is nothing you can do except wait, which is a bad thing to discover during a demo.

LiteLLM makes this genuinely easy, which is most of why I use it:

response = await litellm.acompletion(
    model="deepseek/deepseek-chat",
    messages=messages,
    timeout=30,
    num_retries=2,
    fallbacks=["gemini/gemini-2.0-flash"],
)

The fallback doesn't have to be as good. That's the bit I initially misunderstood. It's the difference between a slightly worse answer and no answer at all, and users are much more forgiving of the first one.

Do actually test it though. I had a fallback configured for a while that would have failed if it ever triggered, because the model name was wrong and nothing had ever exercised that path. I found it by deliberately putting a garbage primary model name in and seeing what happened, which took two minutes and I should have done it immediately.

Validation tells you something broke, not what to do about it

Pydantic is great. You define the shape you want, you get a clean error when the model returns something else.

from pydantic import BaseModel

class Answer(BaseModel):
    text: str
    confidence: float
    sources: list[str]

What tripped me up is that catching the error is only half a decision. You still have to choose what happens next, and I didn't choose for a while, which meant my choice was "throw a 500 at the user" by default.

The options I've ended up thinking about:

Retry once, feeding the validation error back into the prompt. This works surprisingly often for small schema mistakes. It also costs you another call and another few seconds, so it's not free.

Fall back to something simpler. If the structured version keeps failing, take plain text and lose the structure rather than losing the response.

Fail properly, with a real message. Sometimes this is right. But "sorry, something went wrong" is much better than a stack trace, and it's better than silently returning an empty object that breaks something three layers up.

None of these is correct in general. The point is just to pick one on purpose.

A spend ceiling, because dashboards tell you afterwards

This is the one I'd add first if I were starting again.

Provider dashboards are good. They are also retrospective. They tell you what you spent after you spent it, and if something loops overnight you find out in the morning.

So I keep a counter in the process:

class SpendGuard:
    def __init__(self, ceiling_usd):
        self.ceiling = ceiling_usd
        self.spent = 0.0

    def record(self, response):
        cost = response._hidden_params.get("response_cost", 0)
        self.spent += cost
        if self.spent > self.ceiling:
            raise RuntimeError(
                f"Spend ceiling hit: ${self.spent:.2f} of ${self.ceiling:.2f}"
            )

It's twenty lines and it's naive. It resets when the process restarts, and it won't help you across multiple workers unless you move the counter somewhere shared like Redis. But it turns "unbounded" into "bounded", and that's the part that actually matters. My entire dissertation experiment ran on about four pounds of compute, and knowing a bug couldn't turn that into four hundred let me iterate a lot more freely.

If you're on a hosted provider, set a hard billing limit in their console too. Belt and braces.

What I'd do differently

Honestly, I'd write all of this before writing any of the interesting parts.

Every item here I added after something surprised me, which meant each one arrived as a small panic rather than as a decision. It's not much code. It's a timeout, a retry cap, two token ceilings, a fallback, a validation branch, and a counter. Maybe an hour to put in place at the start, versus finding each one individually the hard way.

The model is the part everyone talks about. The stuff around it is what determines whether the thing survives contact with an actual user.

If you're further along than me and I've got something wrong here, I'd genuinely like to know.

Where to cut a prompt so the cache actually hits

Veera Venkata Satyanarayana Gannamraju — Wed, 05 Aug 2026 14:29:31 +0000

I spent a while assuming prompt caching was something you turn on. It isn't. It's a decision about where you cut your prompt, and if you cut it in the wrong place the feature quietly does nothing while your code still looks correct.

This is what I learned building TechnoRAG, a hybrid RAG service over a daily-refreshed news corpus. I ended up sustaining a 57 to 60% input cache hit rate, which made cached calls roughly 10x cheaper. Most of that came from one structural change, not from tuning anything.

The thing nobody tells you upfront

Prompt caching is prefix caching. The provider hashes your input from the very first token and matches the longest identical run against what it already has stored.

That word "prefix" is doing all the work. It means the cache breaks at the first byte that differs, and everything after that point is a miss, even if 90% of it is identical to last time.

So this prompt:

You are a research assistant.
Current time: 2026-08-05T14:22:31Z
[2000 tokens of instructions and output schema]
[retrieved context]
[user question]

caches nothing. Not the instructions, not the schema. The timestamp on line two poisons every token after it. I had something close to this for longer than I'd like to admit, looked at my hit rate sitting near zero, and assumed caching just wasn't working on my provider.

Sorting by how often things change

The fix is boring. Order your prompt by rate of change, most stable first:

[1] role and task definition         never changes
[2] output schema and format rules   never changes
[3] query-type instructions          4 variants, changes per request type
[4] retrieved chunks                 changes every request
[5] user question                    changes every request

Layers 1 and 2 are the same on every single call, so they sit at the front and cache permanently. Layers 4 and 5 are different every time and can never cache, so they go at the back where they can't hurt anything.

Layer 3 is the interesting one. TechnoRAG routes queries into four intent types (conceptual, factual, comparative, temporal) and each gets slightly different instructions. My first instinct was that this breaks caching, since the text varies. It doesn't. It just means you maintain four cache prefixes instead of one, and each of them still gets reused across every request of that type. Four warm caches is fine. What isn't fine is putting that block above the schema, because then the schema has to be re-processed four different ways for no reason.

That's the whole idea. Stable stuff first, volatile stuff last, and volatile stuff should never sit above stable stuff.

Things that broke my cache without looking like they would

A few of these cost me real time.

Timestamps anywhere in the system prompt. Obvious in hindsight. If your pipeline is time-aware (mine has a temporal freshness layer) the instinct is to tell the model what time it is up front. Move it down next to the retrieved context instead.

Retrieved chunks in non-deterministic order. My reranker was returning ties in whatever order they came out of the fusion step. Same chunks, different sequence, different hash. This only matters if any of that content sits above something you want cached, but it's worth knowing your ordering is stable.

JSON key ordering. If you're serialising part of your prompt from a dict, Python preserves insertion order but the thing constructing that dict might not be deterministic across runs. sort_keys=True costs nothing.

Trailing whitespace. Two prompt templates that differ by a single newline are two different prefixes. I found one of these by diffing two raw request bodies byte for byte, which is not a fun evening.

Minimum cacheable size

Worth checking your provider's floor before you spend an afternoon restructuring. Caches work in blocks and there's usually a minimum number of tokens below which nothing gets stored at all.

If your static prefix is short, you may get nothing back no matter how well you order it. In that case you either accept it, or you find that padding the stable section with genuinely useful content (fuller output examples, more explicit format rules) pushes you over the line and improves output quality at the same time. That second option felt like cheating until I realised the examples were making the responses better anyway.

How I actually verified it

I didn't trust the hit rate until I could see it per call. LangFuse gives you cached versus uncached input tokens on each generation, so I logged both and watched the ratio over a few hundred requests.

Two things showed up immediately. The rate was near zero on cold start and climbed as prefixes warmed, which is expected but looks alarming if you only check once. And it dropped sharply whenever I edited the system prompt, since every edit invalidates everything downstream of the change. If you're iterating on prompt wording, your hit rate during that session tells you nothing useful.

What I'd tell myself at the start

Draw the line before you write the prompt. Decide which parts are allowed to change per request, put everything else above them, and don't let a single dynamic value sneak into the stable section for convenience.

It took me one restructuring pass to go from a hit rate near zero to the high fifties. The prompt content barely changed. Only the order did.

Building TechnoRAG (hybrid RAG service) and HopLens (step level evaluation for multi hop agentic RAG). Code on GitHub.