The cheapest LLM call is the one you don't make: a caching layer that actually pays off

YaFei — Wed, 19 Aug 2026 02:34:43 +0000

In the last post I wrote about routing across providers to cut our bill ~40%. Caching was the second lever — and honestly the more underrated one. Here's what we learned shipping it.

Routing gets most of the attention because it's sexy: traffic dancing across providers, failover kicking in, dashboards lighting up. But the single biggest cost lever we pulled after routing wasn't smarter routing. It was not calling the model at all.

Why caching gets ignored

When people talk about LLM cost, they picture the per-token price. That's the wrong unit. The question is how many of your calls are genuinely new information versus repeats wearing a costume.

We were shocked at the overlap. Once we started measuring, a large share of production traffic was re-asking near-identical things:

The same system prompt + near-identical user input, re-embedded every time.
The same retrieval-augmented question asked by different users within minutes.
Deterministic pre/post-processing steps recomputed on every request.

None of that needs a fresh model call. It needs a cache with a brain.

Three layers that actually paid off

1. Exact cache (the boring one that works immediately)

Hash the full request (system + messages + params). If you've seen it, return the stored completion. Obvious, but most teams skip it because "our prompts are dynamic." They usually aren't that dynamic.

import hashlib, json

def cache_key(req):
    return hashlib.sha256(json.dumps(req, sort_keys=True).encode()).hexdigest()

def complete(req):
    k = cache_key(req)
    hit = store.get(k)
    if hit:
        return hit  # zero tokens spent
    out = model_call(req)
    store.set(k, out, ttl=300)
    return out

This alone killed a chunk of bill on our highest-traffic endpoints.

2. Semantic cache (the one people underestimate)

Exact matching misses the real win: similar prompts returning similar answers. Embed the user turn, store embeddings in a vector index, and on each request check for a neighbor above a similarity threshold (we use ~0.92). If found, reuse the prior completion.

The catch: semantic caching is only safe for deterministic-ish tasks (classifications, extractions, stable Q&A). Don't cache creative generation — you'll serve stale voices. We scope it tightly and it still covers a surprising volume.

3. Deterministic-step cache

A lot of "LLM calls" are actually deterministic work wrapped in a prompt: parsing, normalization, format conversion. We moved those to pure functions computed once and reused. It's not even a model cache — it's just not pretending the model is needed.

Tuning without breaking things

TTL by volatility. Stable reference answers: long TTL. Fast-moving data: short or none.
Token budget for the lookup. An embedding + vector search costs tokens too. Make sure the cache check is cheaper than the miss — for us it is, by a wide margin.
Measure hit rate, not just savings. Hit rate tells you when caching stopped helping (prompt drift, new use cases) so you can re-scope.

The numbers

Cache hit rate across cached endpoints: ~35%.
Additional bill reduction on top of routing: meaningful — combined with routing we're now well past the original 40% on the endpoints that use both.
p95 latency on cached hits: sub-50ms instead of hundreds of ms. Users notice the speed more than the savings.

None of this is exotic. It's the same caching discipline people have applied to databases for decades, applied to model calls where the per-hit savings are bigger.

Where this fits with the rest

Routing moves traffic to the cheapest healthy provider (how we cut the bill with routing). A circuit breaker keeps a flaky provider from turning an outage into a bill explosion (the pattern we use). Caching is the layer underneath both: the call you skip is the call you never have to route or protect.

If you're optimizing the same thing

Getting reliable, affordable model access set up for a team has its own headaches — provider quotas, region limits, payment friction. If any of that sounds familiar, I'm happy to compare notes. Find me here or DM me; no pitch, just war stories.

How we cut our LLM bill 40% with multi-provider routing

YaFei — Mon, 10 Aug 2026 08:31:12 +0000

Running LLM features in production is expensive in ways the pricing page doesn't show. Here's what actually moved the needle for us.

We ship LLM features to production. For the first few months, our bill looked reasonable — until it didn't. The model call price was never the problem. The problem was everything around it.

The real cost isn't the happy path

When a provider starts rate-limiting or flaking, two things happen that the naive cost estimate ignores:

Retry storms. The same prompt gets hammered 3–5 times before anything gives up. Each retry is a full-price call.
Tail latency. Requests queue behind slow providers, users time out, and you retry again.

In one bad week, our effective cost per successful request was ~2.4× the listed price. Nobody budgets for that.

What we changed

1. Route across providers by latency + price

Instead of pinning one model to one provider, we route each request to the cheapest healthy endpoint that meets our latency budget.

A request that needs a fast answer goes to the lowest-latency provider currently under quota.
A batch job goes to the cheapest one.
Same model family, different provider — the caller doesn't care.

This alone removed most of the "stuck behind a throttled provider" tax.

2. Cache embeddings and deterministic steps

A surprising amount of our traffic re-embeds the same inputs. We added a cache layer:

Embeddings for repeated prompts → served from cache.
Deterministic pre/post-processing → computed once, reused.

That cut a meaningful chunk of repeat calls without touching quality.

3. Health-checked failover

We keep a health-checked pool of upstreams. A provider that returns N consecutive 5xx/429 gets pulled from rotation for ~60s. Retries use exponential backoff with jitter.

Result: when one provider degrades, traffic moves before users notice. No dropped requests as long as one upstream is alive.

The numbers

Bill per successful request: down ~40%.
Retry calls: down ~60%.
2am incidents where we manually switched providers: roughly zero.

None of this required an exotic stack. It's routing, caching, and a health check — the boring infra stuff that nobody writes blog posts about until the bill arrives.

DEV Community: YaFei

The cheapest LLM call is the one you don't make: a caching layer that actually pays off

Why caching gets ignored

Three layers that actually paid off

1. Exact cache (the boring one that works immediately)

2. Semantic cache (the one people underestimate)

3. Deterministic-step cache

Tuning without breaking things

The numbers

Where this fits with the rest

If you're optimizing the same thing

How we cut our LLM bill 40% with multi-provider routing

The real cost isn't the happy path

What we changed

1. Route across providers by latency + price

2. Cache embeddings and deterministic steps

3. Health-checked failover

The numbers

If you're in the same boat