How I Cut Our LLM Bill 95% — A Backend Engineer's Notes for 2025

How I Cut Our LLM Bill 95% — A Backend Engineer's Notes for 2025

I'll be honest with you — when I first looked at our AI infrastructure bill, I nearly choked on my coffee. The team had been spinning up GPT-4o for everything, including a glorified FAQ bot that mostly answered "what's your refund window?" ten different ways. We're talking thousands of dollars a month for what was effectively string.contains() with extra steps.

This is the story of how I went from "what the hell is this charge" to running a tiered routing system that handles 85% of requests on a $0.01/M model. Fwiw, I didn't invent any of this — I just stole ideas from the database query optimizer playbook and applied them to LLM calls. If you've ever set up a read replica or a CDN, you already understand 80% of what's coming.

Let me walk you through the seven moves that took us from reckless burning to a budget I'd actually defend in a planning review.

---

The uncomfortable truth about model selection

Most teams I talk to pick their default model the same way they pick a default text editor — once, early on, and never revisit it. Then someone wires it into twelve services, the cost grows linearly with usage, and by the time anyone notices, you're paying GPT-4o rates to summarize customer reviews.

Under the hood, what you're actually buying is capability gradient. Different tasks need different floors of competence. A sentiment classifier doesn't need a PhD; it needs basic pattern recognition. A code refactoring agent arguably needs more.

Here's the matrix I built after auditing our actual workloads:

Task profile	What we used to pay	What we pay now	Net savings
Casual chat / FAQ	GPT-4o ($10/M out)	DeepSeek V4 Flash ($0.25/M)	97.5%
Classification / tagging	GPT-4o-mini ($0.60/M)	Qwen3-8B ($0.01/M)	98.3%
Code generation	GPT-4o ($10/M)	DeepSeek Coder ($0.25/M)	97.5%
Long doc summarization	GPT-4o ($10/M)	Qwen3-32B ($0.28/M)	97.2%
Translation	GPT-4o ($10/M)	Qwen-MT-Turbo ($0.30/M)	97%

Look at those numbers. The "expensive choice" column should make you physically uncomfortable if you shipped any of them to a public endpoint without a rate limiter.

Here's the routing table I keep in a models.yaml and load at boot:

# models.py
from openai import OpenAI

client = OpenAI(
    base_url="https://global-apis.com/v1",
    api_key=os.environ["GLOBAL_API_KEY"],
)

MODEL_MAP = {
    "chat_simple":  "deepseek-v4-flash",    # $0.25/M
    "code":         "deepseek-coder",       # $0.25/M
    "classify":     "Qwen/Qwen3-8B",        # $0.01/M
    "summarize":    "Qwen/Qwen3-32B",       # $0.28/M
    "translate":    "Qwen-MT-Turbo",        # $0.30/M
    "reasoning":    "deepseek-reasoner",    # $2.50/M — last resort
}

def pick_model(user_input: str) -> str:
    bucket = classify_complexity(user_input)
    return MODEL_MAP[bucket]

resp = client.chat.completions.create(
    model=pick_model(user_input),
    messages=[{"role": "user", "content": user_input}],
)

That classify_complexity function is embarrassingly simple — a keyword/regex check, basically — and it saved us more than every other optimization combined. Imo, this single change is worth the entire rest of this article.

---

Tiered routing: the cascade pattern

Once I'd accepted that not every request needs a frontier model, the next move was obvious: build a waterfall. Try cheap, fail fast, escalate when needed.

This is basically how compiler optimization works — cheap passes first, expensive ones only if the cheap pass can't prove correctness. Same idea, different domain.

def cascading_generate(prompt: str, budget_usd: float = 0.50) -> str:
    """
    Try the cheapest model that can plausibly handle the request.
    Escalate only when quality is insufficient.
    """

    # Tier 1 — ultra-budget. Handles ~80% of traffic.
    tier1 = call_model("Qwen/Qwen3-8B", prompt)        # $0.01/M
    if quality_score(tier1) >= 0.8:
        return tier1

    # Tier 2 — standard. Handles ~15% of traffic.
    tier2 = call_model("deepseek-v4-flash", prompt)    # $0.25/M
    if quality_score(tier2) >= 0.9:
        return tier2

    # Tier 3 — premium. The remaining ~5%.
    return call_model("deepseek-reasoner", prompt)     # $2.50/M

The quality_score function is the secret sauce. For us it was a tiny classifier model that looked at things like response length, presence of refusal phrases, and a lightweight embedding-distance check against an expected answer distribution. Took me a weekend to wire together and saved roughly $390/month on our customer support workload — went from $420/month to $28/month by routing 85% of queries through Qwen3-8B.

The lesson here isn't "always use cheap models." The lesson is "default to cheap, prove you need expensive."

---

Caching: the thing everyone forgets

I cannot tell you how many times I've joined a team and discovered their app re-asks GPT the exact same question 4,000 times a day. "What is the return policy?" doesn't need to hit an LLM if your return policy is in your database and never changes.

LLM caches come in three flavors, in increasing order of complexity:

1. Exact-match cache — hash the (model, messages) tuple, store the response for N seconds. Solves 50-80% of common queries (FAQ, docs, static content).
2. Semantic cache — embed the query, look up the nearest neighbor in a vector store, return cached response if cosine distance is below a threshold. Handles paraphrases.
3. Speculative cache — pre-compute likely responses ahead of time (think: cron job generating answers to expected questions).

Here's the exact-match version, which is what I'd recommend you start with:

import hashlib
import json
import time

_cache: dict[str, dict] = {}

def cached_chat(model: str, messages: list, ttl_seconds: int = 3600):
    key = hashlib.md5(
        json.dumps({"model": model, "messages": messages}, sort_keys=True).encode()
    ).hexdigest()

    hit = _cache.get(key)
    if hit and (time.time() - hit["ts"]) < ttl_seconds:
        metrics.counter("llm.cache.hit").inc()
        return hit["response"]

    response = client.chat.completions.create(
        model=model, messages=messages
    )
    _cache[key] = {"response": response, "ts": time.time()}
    metrics.counter("llm.cache.miss").inc()
    return response

Add a TTL because model behavior drifts over time. Add an LRU eviction because caches aren't free. Add observability because you need to know your hit rate before you start arguing with finance about the bill.

---

Prompt compression: the under-appreciated lever

Token costs are sneaky. People look at output prices and ignore input prices, and then they ship a 4,000-token system prompt and wonder why their monthly invoice looks like a phone number.

Compressing prompts isn't glamorous but it pays rent. The general idea: summarize long context with a cheap model before passing it to the expensive one.

def compress_prompt(text: str, target_ratio: float = 0.5) -> str:
    if len(text) < 500:
        return text  # Don't bother with short prompts

    target_chars = int(len(text) * target_ratio)
    summary = call_model(
        "Qwen/Qwen3-8B",  # $0.01/M — almost free
        f"Summarize the following in under {target_chars} characters, "
        f"preserving all factual details:\n\n{text}",
    )
    return summary

Let me give you the concrete math that got our PM to actually approve this work:

A 2,000-token system prompt compressed to 400 tokens saves $0.024/request on DeepSeek V4 Flash. At 10,000 requests/day, that's $240/day → $87,600/year. From a single compression step. I didn't even have to break out the spreadsheet to convince anyone after that.

The trick is to compress with a model much cheaper than the target. Using GPT-4o to compress prompts for GPT-4o is a net loss. Using Qwen3-8B at $0.01/M to compress for DeepSeek V4 Flash at $0.25/M? Now you're playing the optimizer game correctly.

---

Batch processing: amortize the fixed cost

Every LLM call has overhead — network round-trip, JSON parsing, prompt re-tokenization. If you're sending N similar requests back-to-back, you can collapse them into one call with a structured prompt and a single response.

This is straight out of the database playbook. If you've ever written WHERE id IN (...) instead of N separate queries, you already get it.

# ❌ Before: N calls, N round-trips, N × input token cost
def classify_legacy(items: list[str]) -> list[str]:
    results = []
    for item in items:
        r = client.chat.completions.create(
            model="deepseek-v4-flash",
            messages=[{
                "role": "user",
                "content": f"Classify sentiment as positive/negative: {item}"
            }],
        )
        results.append(r.choices[0].message.content)
    return results

# ✅ After: 1 call, 1 round-trip, batched prompt
def classify_batched(items: list[str]) -> list[str]:
    numbered = "\n".join(f"{i}. {it}" for i, it in enumerate(items))
    r = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{
            "role": "user",
            "content": (
                f"Classify each line as POSITIVE or NEGATIVE. "
                f"Return one label per line, in order:\n{numbered}"
            ),
        }],
    )
    return r.choices[0].message.content.strip().splitlines()

You lose some reliability on huge batches (the model can drift or skip items past ~50-100 entries), so split into chunks. Fwiw, my sweet spot has been batches of 20-30 for classification, 10-15 for generation.

Expect 10-20% savings from batching alone, mostly from amortized input tokens and reduced network chatter. Bigger wins come from larger batches, but you trade off against response latency.

---

Streaming, truncation, and stopping the model early

Most teams forget that output tokens cost more than input tokens — and they also don't realize they can cut off generation mid-flight. If you're summarizing an article and the model decides to write a closing paragraph, you can stop it.

Two patterns I use constantly:

# 1. Streaming + early termination on a sentinel token
def summarize_with_stop(text: str) -> str:
    stream = client.chat.completions.create(
        model="Qwen/Qwen3-32B",
        messages=[{"role": "user", "content": f"Summarize: {text}"}],
        stream=True,
        stop=["\n\n---", "In summary,", "Conclusion:"],
    )
    chunks = []
    for chunk in stream:
        if chunk.choices[0].finish_reason:
            break
        chunks.append(chunk.choices[0].delta.content or "")
    return "".join(chunks)

# 2. Hard max_tokens cap, tuned per task
client.chat.completions.create(
    model="Qwen3-8B",
    messages=[...],
    max_tokens=64,  # classification doesn't need paragraphs
)

Sounds trivial, but capping max_tokens to the true minimum (instead of the default 256 or 512) routinely shaves 15-25% off output spend for short-form tasks. It's the equivalent of LIMIT in SQL — boring, essential.

---

Fine-tuning small models for your domain (the power move)

This is the strategy with the biggest upfront cost and the biggest long-term payoff. If you have a high-volume, narrow task — intent classification, entity extraction, support routing — fine-tuning a small open model on your labeled data will obliterate your per-request cost.

Here's the rough economics as I've seen them play out:

• Pre-trained Qwen3-8B at $0.01/M handles ~75% of intent classification correctly.
• Fine-tuned Qwen3-8B on ~5k labeled examples handles ~94% correctly.
• Suddenly you don't need to escalate those queries to GPT-4o at all.

The fine-tuning cost is a one-time hit, maybe a few hundred dollars in GPU time and a week of your time cleaning labels. The breakeven point is somewhere around 1-3 months of production traffic, depending on volume.

I won't dump a full training script in here because (a) it's outside the API cost scope and (b) it's a project unto itself, but the tl;dr is: collect failures from your cascade system, label them, fine-tune, redeploy. That's the loop.

---

Speculative execution and prefix caching

Two more tricks worth mentioning briefly because they compound with everything above:

Prefix caching. Most LLM providers (and Global API's underlying infrastructure) cache the KV state for long common prefixes. If your system prompt is stable, leave it identical across calls — don't randomize whitespace or timestamps. Anthropic and OpenAI both discount repeated prefixes heavily; Global API's setup plays nicely with this too.

Speculative execution. When latency matters more than marginal cost, you can fire off the cheap model immediately and the expensive model in parallel, return whichever wins. Sounds wasteful, but for user-facing UIs with tight latency budgets, it's often a net win because user-perceived speed converts to retention, which converts to revenue, which makes the LLM bill a rounding error.

---

Putting it all together: the actual stack

Here's what my services/llm.py looks like in production now, give or take:

python
import os
import hashlib
import json
import time
from openai import OpenAI

client = OpenAI(
    base_url="https://global-apis.com/v1",
    api_key=os.environ["GLOBAL_API_KEY"],
)

TIER_CONFIG = {
    "ultra":  {"model": "Qwen/Qwen3-8B",     "cost_per_m": 0.01},
    "std":    {"model": "deepseek-v4-flash", "cost_per_m": 0.25},
    "heavy":  {"model": "Qwen3-32B",         "cost_per_m": 0.28},
    "premium":{"model": "deepseek