How I Cut My LLM API Bill by 95% Without Breaking Things

How I Cut My LLM API Bill by 95% Without Breaking Things

Last quarter, our finance team pinged me with a screenshot of an invoice that made me physically recoil. We'd burned through $14,800 in a single month on LLM API calls for what I thought was a "modest" chatbot service. I'm a backend engineer. I write Go for a living. I had assumed someone on the AI team had thought about routing, but when I dug in (under the hood, as we say), I discovered we were doing the equivalent of using a sledgehammer to crack every nut, including the peanuts.

This post is the war diary of that six-week optimization project. fwiw, none of this is theoretical. Every number below is something I measured with a prom-client counter and a spreadsheet. I'm not going to bore you with vendor whitepapers or "industry benchmarks" — just what worked in production, with code you can paste into your repo tonight.

Before we get into it, one quick clarification on terminology. When I say "M tokens" I mean one million tokens. Token pricing is the metric that actually matters here, not "per request," because request sizes vary wildly.

---

First, Audit What You're Actually Paying For

Before optimizing anything, you need visibility. I added a thin middleware wrapper around every LLM call that logged the model, token counts (both input and output), latency, and cost to a Postgres table. The query that changed everything for me was this:

SELECT
    model,
    COUNT(*)                       AS calls,
    SUM(total_tokens)              AS total_tokens,
    SUM(cost_usd)                  AS spend,
    AVG(cost_usd)                  AS avg_cost_per_call
FROM llm_usage_log
WHERE created_at > NOW() - INTERVAL '30 days'
GROUP BY model
ORDER BY spend DESC;

What I found was embarrassing, in a productive way: 71% of our spend was on gpt-4o for tasks that a $0.01/M model could have handled without anyone noticing the difference. That's the use. The remaining strategies are mostly ways to claw back the other 29%.

---

Strategy 1: Stop Blindly Defaulting to the Top-Tier Model

This is the single biggest lever. imo, it's not even close. Most teams pick one "good" model during the prototype phase and never revisit the choice. Months later, they're paying for frontier reasoning on tasks that need basic text generation.

Here's the comparison matrix I built and taped to my monitor. These are the exact models and exact prices I benchmarked against:

Task Type	What We Were Using	What We Switched To	Savings
Simple chat / FAQ	GPT-4o ($10/M out)	DeepSeek V4 Flash ($0.25/M)	97.5%
Classification	GPT-4o-mini ($0.60/M)	Qwen3-8B ($0.01/M)	98.3%
Code generation	GPT-4o ($10/M out)	DeepSeek Coder ($0.25/M)	97.5%
Summarization	GPT-4o ($10/M out)	Qwen3-32B ($0.28/M)	97.2%
Translation	GPT-4o ($10/M out)	Qwen-MT-Turbo ($0.30/M)	97%

The model map now lives in a config file and the routing function reads from it. Here's roughly what that looks like in Python, using Global API's OpenAI-compatible interface (which is what we standardized on because the SDK drop-in just works):

from openai import OpenAI

client = OpenAI(
    api_key="sk-global-...",
    base_url="https://global-apis.com/v1",
)

MODEL_MAP = {
    "chat":      "deepseek-v4-flash",     # $0.25/M output
    "code":      "deepseek-coder",        # $0.25/M
    "simple":    "Qwen/Qwen3-8B",         # $0.01/M  ← yes, really
    "reasoning": "deepseek-reasoner",     # $2.50/M
    "translate": "Qwen-MT-Turbo",         # $0.30/M
    "summarize": "Qwen/Qwen3-32B",        # $0.28/M
}

def route(user_input: str) -> str:
    task = classify_complexity(user_input)   # your classifier here
    return MODEL_MAP[task]

resp = client.chat.completions.create(
    model=route(user_input),
    messages=[{"role": "user", "content": user_input}],
)

That's it. One change, and the bill dropped by ~70% in a week.

---

Strategy 2: Cache Aggressively (RFC 7234 Would Approve)

If you've ever written an HTTP cache layer, you already understand the intuition here. RFC 7234 — the HTTP caching RFC, in case you're new to caching semantics — describes exactly the same hierarchy of considerations: freshness, validation, and the staleness tradeoff. The same principles apply to LLM responses.

The win is that a huge percentage of real-world traffic is repetitive. People ask the same FAQs, hit the same documentation lookups, send the same boilerplate prompts. We measured a 62% cache hit rate in our chatbot after deploying a deterministic cache. That alone cut spend by another ~30% on top of Strategy 1.

Here's the implementation. It's about 30 lines of Python and uses an in-memory dict for simplicity, but you should swap in Redis for anything running more than one process:

import hashlib
import json
import time
from typing import Any

_CACHE: dict[str, dict[str, Any]] = {}

def cached_chat(
    client,
    model: str,
    messages: list[dict],
    ttl_seconds: int = 3600,
) -> dict:
    """Cache key = hash(model + canonical-messages)."""
    payload = json.dumps(
        {"model": model, "messages": messages},
        sort_keys=True,
        separators=(",", ":"),
    ).encode("utf-8")
    key = hashlib.sha256(payload).hexdigest()

    now = time.time()
    entry = _CACHE.get(key)
    if entry and (now - entry["ts"]) < ttl_seconds:
        return entry["response"]   # ← cache hit, $0 cost

    response = client.chat.completions.create(
        model=model,
        messages=messages,
    )
    _CACHE[key] = {"response": response, "ts": now}
    return response

A few production hardening notes I learned the hard way:

1. Normalize your messages before hashing. Trailing whitespace, different quote styles, and reordered system prompts will all produce different cache keys and destroy your hit rate. I wrote a canonicalize() helper that strips whitespace and sorts the system message to a fixed position.
2. Set a sensible TTL. One hour is a good default. Anything longer and you'll serve stale answers to questions whose answers have changed.
3. Use semantic caching for fuzzy matches. Embedding-based similarity caches can catch "how do I reset my password?" vs "I forgot my password, how do I reset it?" — which pure hash-based caching misses. We'll touch on that in a future post; it's a different beast.

---

Strategy 3: Compress Your Prompts Before You Send Them

Here's the part that surprised me most. I'd assumed prompts were short. They weren't. Our average system prompt was 1,400 tokens because some product manager had decided the LLM needed "comprehensive brand voice guidelines, ten example responses, and a full glossary of internal product names."

Most of that was noise. A cheap model can summarize a long preamble down to its essentials, and the expensive downstream model performs just as well (sometimes better, because it has less to wade through).

The math on this one is what got my director's attention:

• Original system prompt: ~2,000 tokens
• Compressed: ~400 tokens (5× reduction)
• Per-request savings on DeepSeek V4 Flash: $0.024
• At 10,000 requests/day: $240/day
• Annualized: ~$87,600/year

Yes, you pay a small amount to run the compressor. But the compressor is Qwen3-8B at $0.01/M tokens, and it summarizes a 2,000-token block for roughly $0.00002. The math isn't even close.

def compress_prompt(client, text: str, target_ratio: float = 0.5) -> str:
    """Use a cheap model to compress long prompts."""
    if len(text) < 500:
        return text

    target_chars = int(len(text) * target_ratio)
    summary = client.chat.completions.create(
        model="Qwen/Qwen3-8B",
        messages=[{
            "role": "user",
            "content": (
                f"Compress the following into roughly {target_chars} "
                f"characters. Preserve all factual content and constraints. "
                f"Do not add new information:\n\n{text}"
            ),
        }],
    )
    return summary.choices[0].message.content

One caveat: don't compress the user's message. Compress only the static system context, the retrieval-augmented context, and any history. Compressing the actual question the user just asked is a great way to make your support bot confidently wrong.

---

Strategy 4: The Escalation Ladder (Tiered Routing)

This is the strategy that pushed our total savings above 95%. The idea is borrowed from the classic web architecture pattern: try the cheap tier first, escalate only on failure. Like a circuit breaker, but for quality.

The setup:

• Tier 1 (Ultra-budget, $0.01/M): Handles 80%+ of traffic. Use Qwen3-8B or similar.
• Tier 2 (Standard, $0.25/M): Handles ~15% of traffic. Use DeepSeek V4 Flash.
• Tier 3 (Premium, $2.50/M): Handles the remaining ~5%. Use deepseek-reasoner.

The trick is the quality check between tiers. You need a fast, deterministic signal that "the cheap answer was probably wrong." I used a combination of:

• Self-scoring: ask the cheap model to rate its own confidence.
• Heuristic length/format checks.
• A small classifier trained on past examples where humans flagged bad answers.

def smart_generate(client, prompt: str, max_budget_usd: float = 0.50):
    resp = call_with(client, "Qwen/Qwen3-8B", prompt)
    score = quality_check(resp)
    if score >= 0.8:
        return resp, "tier-1"

    # Tier 2
    resp = call_with(client, "deepseek-v4-flash", prompt)
    score = quality_check(resp)
    if score >= 0.9:
        return resp, "tier-2"

    # Tier 3 — the expensive one
    return call_with(client, "deepseek-reasoner", prompt), "tier-3"

The headline result: our customer support chatbot went from $420/month to $28/month. Same answer quality on the satisfaction surveys. Finance team stopped cc'ing me on worried emails.

---

Strategy 5: Batch the Easy Stuff

The last easy win is batching. If you have a stream of independent requests (think: nightly document summarization, bulk classification of support tickets, log analysis), you almost certainly shouldn't fire them one at a time. You're paying for the input prompt to be re-tokenized for every call.

A simple Python rewrite usually looks like this:

# ❌ Before: N calls, N× input overhead
for q in questions:
    client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content": q}],
    )

# ✅ After: 1 call, single shared system prompt
batch_prompt = "\n".join(f"{i+1}. {q}" for i, q in enumerate(questions))
resp = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[
        {"role": "system", "content": "Answer each numbered question on its own line."},
        {"role": "user",   "content": batch_prompt},
    ],
)
answers = parse_numbered(resp.choices[0].message.content)

This typically buys you 10–20% savings on batched workloads, and you also get a latency win because one network round-trip beats a hundred. If you need true async batching, most providers (including Global API) expose a /v1/batches endpoint that's even better.

---

Bonus: Streaming and Token Discipline

Two smaller wins that are easy to skip:

• **Stream long complet