I Cut My AI API Bill By 95% — Here's What Actually Worked

Look, i Cut My AI API Bill By 95% — Here's What Actually Worked

Six months ago I opened our monthly cloud bill, took a long sip of coffee, and nearly spit it across my keyboard. We were spending more on LLM inference than on our entire Postgres cluster. Fwiw, this is a pretty common story for backend teams right now — every product manager wants AI sprinkled in, but nobody asks who's paying for it until the invoice lands.

So I spent the next few weekends going down a rabbit hole. I read pricing docs, ran benchmarks, broke things, rebuilt them, and slowly turned a $400+/month LLM line item into something I could actually justify. This post is my playbook — the techniques that moved the needle, ranked by impact, with real code snippets you can paste into a Python service today.

Before we dive in, one quick note: every example below uses global-apis.com/v1 as the OpenAI-compatible base URL. It's the cheapest unified gateway I've found for routing across GPT-4o, DeepSeek, Qwen, and friends — and since it speaks the OpenAI SDK protocol, you barely change your code. More on that at the end.

---

The 800-Pound Gorilla: Stop Sending Everything to GPT-4o

The single biggest mistake I see in code reviews is hardcoding gpt-4o as the default model for every task. Need to classify a product review? GPT-4o. Need to summarize a support ticket? GPT-4o. Need to translate a single string? You guessed it — GPT-4o.

Here's the thing. Most of those tasks don't need a frontier reasoning model. They need an LLM, any LLM, the kind you'd happily hand to a script in 2019 if that script could talk. Routing them to GPT-4o is like using a Cray to add 2+2. Technically correct. Financially criminal.

Let me show you what I mean. Here's the cost-per-million-tokens table I taped above my monitor after one too many 2am debugging sessions:

Task	"Easy" Choice	Cost/M (out)	"Right" Choice	Cost/M (out)	Savings
Simple chat	GPT-4o	$10.00	DeepSeek V4 Flash	$0.25	97.5%
Classification	GPT-4o-mini	$0.60	Qwen3-8B	$0.01	98.3%
Code generation	GPT-4o	$10.00	DeepSeek Coder	$0.25	97.5%
Summarization	GPT-4o	$10.00	Qwen3-32B	$0.28	97.2%
Translation	GPT-4o	$10.00	Qwen-MT-Turbo	$0.30	97%

Yeah. 97% savings. Not a typo. Not "if you squint" — literally an order of magnitude.

Now, how you decide which model to call is a problem in itself. Imo, the cleanest approach is to wrap your LLM calls behind a thin router. Something like this:

import openai

client = openai.OpenAI(
    api_key="YOUR_GLOBAL_API_KEY",
    base_url="https://global-apis.com/v1",
)

MODEL_MAP = {
    "chat":      "deepseek-v4-flash",     # $0.25/M
    "code":      "deepseek-coder",        # $0.25/M
    "simple":    "Qwen/Qwen3-8B",         # $0.01/M
    "summarize": "Qwen/Qwen3-32B",        # $0.28/M
    "translate": "Qwen-MT-Turbo",         # $0.30/M
    "reasoning": "deepseek-reasoner",     # $2.50/M
}

def route(task: str, user_input: str) -> str:
    model = MODEL_MAP[task]
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": user_input}],
    )
    return resp.choices[0].message.content

The taxonomy will vary per app, but the principle holds: pay for what you use, not for the brand name on the box.

---

Strategy 2: Tiered Routing — Why Pay Premium for Easy Questions?

Once you have a router, the next step is to stop even pretending every query deserves the same model. This is the pattern that gave us our biggest single win.

The idea is embarrassingly simple. Try the cheapest model first. If its answer is good enough, ship it. Only escalate to a smarter (more expensive) model when the cheap one flounders.

def smart_generate(prompt: str, max_budget: float = 0.50) -> str:
    """
    Tier 1: Ultra-budget ($0.01/M) — handles ~80% of traffic
    Tier 2: Standard ($0.25/M)     — handles ~15%
    Tier 3: Premium ($0.78–$2.50/M) — handles ~5%
    """

    cheap = call_model("Qwen/Qwen3-8B", prompt)
    if quality_check(cheap) >= 0.8:
        return cheap

    # Tier 2 — DeepSeek V4 Flash
    mid = call_model("deepseek-v4-flash", prompt)
    if quality_check(mid) >= 0.9:
        return mid

    # Tier 3 — fall back to reasoning model
    return call_model("deepseek-reasoner", prompt)

In our support chatbot, this exact pattern took monthly spend from $420 down to $28 — about 93% savings, almost entirely because 85% of incoming tickets were routine ("where's my order?", "how do I reset my password?") that Qwen3-8B answered perfectly fine.

A few caveats from running this in production:

• The quality_check function is the whole game. If it's too lax, customers get garbage. If it's too strict, you burn money on the next tier for no reason. We landed on a small classifier-as-judge with a calibrated threshold; more on that in a future post.
• Watch out for latency cliffs. A tier-3 fallback mid-conversation can blow your p99 if you're not careful. We cache the tier decision per conversation thread.
• Log everything. If you can't answer "what % of tier-1 attempts actually escalated?" on a dashboard, you're flying blind.

RFC 7807 — the Problem Details for HTTP APIs spec — has nothing to do with this, but it's a great read if you want to standardize how you surface failures from these routing layers. Sorry, couldn't resist the tangent.

---

Strategy 3: Response Caching — The Free Lunch

I love caches. They turn compute into memory, and memory is cheap. Caching LLM responses is one of those rare wins that improves latency and cost simultaneously.

The simplest version hashes your request and stores the response. If a user asks "What's your refund policy?" twice in a day, you serve the same answer twice for the price of one.

import hashlib, json, time

cache = {}

def cached_chat(model: str, messages: list, ttl: int = 3600):
    key = hashlib.md5(
        json.dumps({"model": model, "messages": messages}, sort_keys=True).encode()
    ).hexdigest()

    if key in cache:
        entry = cache[key]
        if time.time() - entry["time"] < ttl:
            return entry["response"]  # cache hit — $0 cost

    resp = client.chat.completions.create(model=model, messages=messages)
    cache[key] = {"response": resp, "time": time.time()}
    return resp

The hit rate depends entirely on your workload. For an FAQ-style assistant or a docs lookup bot, 50–80% hit rates are common. For a creative writing tool where every prompt is unique, maybe 2%. Be honest with yourself about the distribution before you get too excited.

Two practical notes from the trenches:

• Use semantic caching, not just exact-match, when prompts are paraphrases of the same underlying intent. Embed the prompt, look up nearest neighbors above a cosine threshold, and you're golden.
• TTL matters. One hour is a sensible default for conversational UIs. Don't cache user-specific data for long, or you'll leak PII across sessions — which is, you know, bad. (See: basically every privacy regulation ever written.)

---

Strategy 4: Prompt Compression — Pay for Tokens You Don't Need

Long prompts are the silent killer of LLM budgets. People paste 4,000-token system prompts full of "you are a helpful assistant who must always…" filler and then wonder why their bill looks like a phone number.

The fix: compress. A cheap model can summarize 2,000 tokens of context into 400 tokens for roughly the price of one premium call. The downstream model sees a tighter prompt, you pay less, and latency drops because attention is O(n²). Under the hood, you're saving on three axes simultaneously.

def compress_prompt(text: str, target_ratio: float = 0.5) -> str:
    if len(text) < 500:
        return text  # already short, don't bother

    target_chars = int(len(text) * target_ratio)
    summary = call_model(
        "Qwen/Qwen3-8B",
        f"Summarize this in {target_chars} chars: {text}",
    )
    return summary

The math on this one is genuinely wild. A 2,000-token system prompt compressed to 400 tokens saves roughly $0.024 per request on DeepSeek V4 Flash. Multiply by 10,000 requests per day and that's $240/day, or about $87,600/year — saved by adding ~15 lines of Python.

Imo, this is the most underused optimization on the list. People obsess over model selection and ignore the fact that they're shipping a 5KB marketing pitch to the LLM every single request.

---

Strategy 5: Batch Processing — Amortize the Overhead

LLM pricing has two components: tokens and, effectively, round-trip overhead. If you're firing 50 small requests in a loop, you're paying that overhead 50 times. Batch them into one prompt and you pay it once.

Before:

# 3 separate API calls = 3x round trips
for question in questions:
    response = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content": question}],
    )

After:

# 1 batched call — single round trip
batch_prompt = "\n\n".join(
    f"[Item {i}] {q}" for i, q in enumerate(questions)
)

response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{
        "role": "user",
        "content": (
            "Answer each numbered item below. "
            "Format: '1. <answer>', '2. <answer>'.\n\n"
            f"{batch_prompt}"
        ),
    }],
)

# parse the structured response back out
answers = parse_numbered_response(response.choices[0].message.content)

Typical savings: 10–20%, plus a serious latency win when you're calling from a serverless function with cold-start overhead. Don't batch if individual items are huge — at that point you've just traded one problem for another.

---

Putting It All Together: What the Numbers Actually Looked Like

A quick before/after from the system that prompted this whole investigation:

Technique	Monthly Cost Before	Monthly Cost After	Reduction
GPT-4o everywhere	$420	—	—
+ Tiered routing	—	$28	93%
+ Response caching	—	$18	96%
+ Prompt compression	—	$14	96.7%
+