How I Built a Cost-Aware Layer for LLM APIs: Caching, Routing, and Fallback

Last month my API spend roughly tripled while traffic grew about 40%. Nothing was broken — no runaway loop, no leaked key. The bill was just... honest. I was paying the strongest model to answer "what's the status of order 12345," re-paying for the same prompt hundreds of times a day, and quietly paying again every time a request timed out and got retried.
This post is about the three layers I added to fix that. None of them are exotic, and all of them are things you can drop into an existing codebase this afternoon.
Why the bill grows faster than the traffic
Four things compound, and they're all invisible until you look:

1. Everything goes to the strongest model. A default model="gpt-4o" set once and never revisited turns your entire product into a premium-tier product.
2. Identical prompts get billed repeatedly. Support bots, batch jobs, CI runs — a large share of production traffic is the same request wearing a different timestamp.
3. Failures still cost money. A timeout at 30 seconds may have already consumed the tokens. Retry it and you pay twice for one answer.
4. Long static prefixes are charged every single call. A 3,000-token system prompt is billed on every request, forever. The fix isn't "use a cheaper model." That just moves the cost into quality complaints. The fix is three layers, in this order: cache → route → fall back. Layer 1: Caching There are two kinds, and most people only know one. Provider-side caching is automatic on most major APIs now. If the prefix of your request matches a recent one, the cached portion is billed at a steep discount and returns faster. You don't enable it — you qualify for it by structuring your prompts correctly. And here's the part almost everyone gets wrong: the static part must come first. # Wrong — the user's question sits in front of the docs, # so every request has a different prefix and never hits cache messages = [ {"role": "user", "content": user_question}, {"role": "system", "content": long_policy_docs}, ]

Right — stable prefix first, dynamic input last

messages = [
{"role": "system", "content": long_policy_docs},
{"role": "user", "content": user_question},
]
That one reordering is usually worth more than any other single change in this post. Put system prompts, tool definitions, and reference documents at the top. Put anything that changes per request at the bottom.
Your own cache handles the cases provider caching can't — exact repeats of a full request. Here's a small one with no dependencies:
import hashlib
import json
import time

class ResponseCache:
def init(self, ttl: int = 3600, max_items: int = 10_000):
self.ttl = ttl
self.max_items = max_items
self._store: dict[str, tuple[float, str]] = {}

@staticmethod
def key(model: str, messages: list, **kwargs) -> str:
    payload = json.dumps(
        {"model": model, "messages": messages, **kwargs},
        sort_keys=True,
        ensure_ascii=False,
        default=str,
    )
    return hashlib.sha256(payload.encode()).hexdigest()

def get(self, key: str) -> str | None:
    hit = self._store.get(key)
    if hit and time.time() - hit[0] < self.ttl:
        return hit[1]
    self._store.pop(key, None)
    return None

def set(self, key: str, value: str) -> None:
    if len(self._store) >= self.max_items:
        self._store.pop(next(iter(self._store)), None)  # drop oldest
    self._store[key] = (time.time(), value)

Two rules for using it safely:

• Only cache deterministic calls. If temperature is above 0, or the prompt contains a timestamp, a request ID, or anything time-sensitive, don't cache it. You'll serve stale or wrong answers and won't notice for weeks.
• Start with the boring workloads. CI suites, eval runs, and batch jobs are where the hit rate is highest and the risk is lowest. Caching those costs you nothing and can remove a surprising line item. Layer 2: Routing by complexity Not every request deserves your most expensive model. The trick is picking a tier without spending a call to decide. import re

TIERS = {
"cheap": "gpt-4o-mini",
"balanced": "claude-3-5-sonnet",
"strong": "gpt-4o",
}

_STRONG_SIGNALS = re.compile(
r"\b(analyze|design|refactor|architecture|trade-?off|prove|debug why|optimize)\b",
re.I,
)

def route(prompt: str, has_tools: bool = False, turns: int = 1) -> str:
"""Pick a model tier from cheap signals — no extra LLM call required."""
if len(prompt) < 120 and not _STRONG_SIGNALS.search(prompt) and not has_tools:
return TIERS["cheap"]
if _STRONG_SIGNALS.search(prompt) or turns > 6:
return TIERS["strong"]
return TIERS["balanced"]
Short, no tools, no reasoning keywords → cheap tier. Explicit reasoning language or a long multi-turn conversation → strong tier. Everything else lands in the middle.
You can use a small model as a classifier instead of regex, and it works better on messy input. But be honest about the math: the classifier call costs money too. It only pays off when the price gap between your tiers is wide and your traffic is high enough to amortize it. For most teams, start with heuristics and only upgrade when you can see the heuristics failing.
One thing worth doing from day one: log which tier handled each request. That distribution is the single most useful artifact you'll have when someone asks "why is the bill going up?"
Layer 3: Fallback
Fallback gets filed under reliability, but it belongs in a cost post too — because the cheap fallback is almost always better than a failed request.
def call_with_fallback(client, messages, chain, **kwargs):
"""Try each model in order. First success wins."""
last_error = None
for attempt, model in enumerate(chain, start=1):
try:
resp = client.chat.completions.create(
model=model, messages=messages, **kwargs
)
resp._attempt = attempt # handy for the metrics layer below
return resp
except Exception as e:
last_error = e
continue
raise RuntimeError(f"all models failed: {chain}") from last_error

FALLBACK_CHAIN = ["gpt-4o", "claude-3-5-sonnet", "deepseek-chat"]
Keep retry and fallback as separate concepts, because they solve different problems:
Retry
Fallback
Same model?
Yes
No
Solves
Transient errors (429, 502, blip)
Sustained unavailability
Budget
2 attempts, with backoff
Walk the chain once
Retrying the same model three times against a provider that's genuinely down just burns money and latency. Detect the difference by error class: rate limits and 5xx are worth retrying; a persistent 400 or a hard outage is worth falling back on.
Putting it together
Here's the wrapper I actually run. It does all three layers and records enough to answer "where did the money go":
import os
import time
from dataclasses import dataclass, field
from openai import OpenAI

@dataclass
class CallRecord:
model: str
prompt_tokens: int
completion_tokens: int
cached: bool
attempt: int
latency_ms: int

class CostAwareClient:
def init(self, chain: list[str], cache_ttl: int = 3600):
self.client = OpenAI(
api_key=os.getenv("LLM_API_KEY"),
base_url=os.getenv("LLM_BASE_URL", "https://easy88ai.com/v1"),
)
self.chain = chain
self.cache = ResponseCache(ttl=cache_ttl)
self.records: list[CallRecord] = []

def complete(self, messages: list, **kwargs):
    model = route(messages[-1].get("content", ""))
    started = time.perf_counter()

    # 1. cache
    ck = self.cache.key(model, messages, **kwargs)
    if (hit := self.cache.get(ck)) is not None:
        self.records.append(CallRecord(model, 0, 0, True, 0, 0))
        return hit

    # 2. route + 3. fall back
    chain = [model] + [m for m in self.chain if m != model]
    try:
        resp = call_with_fallback(self.client, messages, chain, **kwargs)
    except Exception:
        raise

    text = resp.choices[0].message.content
    self.cache.set(ck, text)

    usage = getattr(resp, "usage", None)
    self.records.append(
        CallRecord(
            model=resp.model,
            prompt_tokens=getattr(usage, "prompt_tokens", 0) if usage else 0,
            completion_tokens=getattr(usage, "completion_tokens", 0) if usage else 0,
            cached=False,
            attempt=getattr(resp, "_attempt", 1),
            latency_ms=int((time.perf_counter() - started) * 1000),
        )
    )
    return text

def report(self) -> dict:
    total = len(self.records)
    if not total:
        return {}
    cached = sum(1 for r in self.records if r.cached)
    by_model: dict[str, int] = {}
    for r in self.records:
        by_model[r.model] = by_model.get(r.model, 0) + 1
    return {
        "calls": total,
        "cache_hit_rate": round(cached / total, 3),
        "tier_distribution": by_model,
        "retry_rate": round(
            sum(1 for r in self.records if r.attempt > 1) / total, 3
        ),
        "prompt_tokens": sum(r.prompt_tokens for r in self.records),
        "completion_tokens": sum(r.completion_tokens for r in self.records),
    }

How to measure whether it worked
I'm not going to paste a pricing table here — per-token prices change often, vary by tier and region, and any numbers I write will be stale within a few weeks. Pull current rates from your provider's pricing page, or better, read actual charges off your console.
What matters is the metric, not the unit price:
Cost per resolved task = total spend ÷ number of tasks actually completed successfully.
That's the number that moves when you do this right. Everything else is diagnostic:
Metric
Healthy
What a bad value tells you
Cache hit rate
20–40% on support/batch workloads
Your static prefix isn't actually first, or TTL is too short
Cheap-tier share
50%+ of calls
Your router is too conservative
Retry rate
< 5%
Timeouts are set too tight, or you're retrying non-retryable errors
Fallback rate
< 2%
A provider is degrading and you haven't noticed
Track these before you change anything. Without a baseline you can't tell a 40% saving from a slow week.
What I'd skip

• Semantic caching, at first. Embedding-similarity caching sounds great and is genuinely powerful, but tuning the similarity threshold takes longer than the money it saves at low volume. Exact-match caching gets you most of the value with none of the false-positive debugging. Add semantic later, once you know your hit rate ceiling.
• An LLM-based router on day one. Start with heuristics. You can always upgrade, and by then you'll have the logs to prove it's worth it.
• Trimming prompts until quality breaks. Cutting a system prompt from 3,000 tokens to 400 saves real money until the model starts ignoring your output format. Measure quality alongside cost, not instead of it. Three things I'd tell myself at the start
• Prompt order is free money. Moving static content to the front of your messages costs ten minutes and can be the single largest win available to you.
• Route before you optimize. Knowing which tier handled each request turns cost conversations from guesswork into a pie chart.
• Retry and fallback are different tools. Retry absorbs blips; fallback absorbs outages. Conflating them is how you end up paying for three failed attempts. None of this requires a framework or a vendor. It's a cache, a regex, and a loop — and it's the difference between a bill that scales with your product and one that scales with your inattention.

I'm building easy88ai, a unified API gateway that routes GPT, Claude, Gemini and 200+ models through one OpenAI-compatible endpoint — which is what I use as the base_url in the examples above. Happy to swap notes on LLM tooling in the comments.