Cutting AI API Bills From Scratch: What Nobody Tells You

Here's the thing: cutting AI API Bills From Scratch: What Nobody Tells You

I'll never forget the Slack message from our finance lead last March. "Why did our OpenAI line item jump 8x this month?" Fair question. I'd been shipping features, not watching the meter. After a week of digging through logs, I realized something embarrassing: roughly 60% of our spend was GPT-4o calls answering questions a $0.01/M model could have handled just fine. Fwiw, that was the day I started taking LLM cost engineering seriously.

This is the playbook I wish someone had handed me twelve months earlier. I'll walk through the seven techniques that took our monthly bill from roughly $18,400 down to about $1,950 without anyone in the product noticing. Same outputs, better prompts, smarter routing. Under the hood, nothing about the user experience changed — only the line items.

The uncomfortable math nobody talks about

Most engineering teams I've worked with (including mine, prior to last year) treat model selection as a one-time decision. Pick the best, ship it, move on. That works fine until the bill arrives. Here's the thing: the gap between the cheapest and most expensive viable model is often two orders of magnitude. Not 2x. Not 5x. Two orders of magnitude. Once you internalize that, the rest of this guide becomes obvious.

Let me set the stage with the numbers I keep taped to my monitor. All figures are output token pricing per million tokens, which is where most teams bleed cash without realizing it (IMO, input token pricing is the bait — output is where the real money evaporates).

Use Case	The Convenient Pick	The Actually-Appropriate Pick	What You Save
Casual chat	GPT-4o ($10.00/M)	DeepSeek V4 Flash ($0.25/M)	97.5%
Classification	GPT-4o-mini ($0.60/M)	Qwen3-8B ($0.01/M)	98.3%
Code generation	GPT-4o ($10.00/M)	DeepSeek Coder ($0.25/M)	97.5%
Summarization	GPT-4o ($10.00/M)	Qwen3-32B ($0.28/M)	97.2%
Translation	GPT-4o ($10.00/M)	Qwen-MT-Turbo ($0.30/M)	97%

If you're using GPT-4o as your default for anything listed above, you're leaving a small fortune on the table. The drop-in replacements produce comparable quality for those workloads. I know this because I ran the evaluations myself, blind A/B with our customer support transcripts. Side note: there's a reason OpenAI charges 40x more — it's not because they're 40x better at summarizing a refund policy.

Tier 1: Pick the right model for the job

The biggest lever is also the easiest one to ignore. People default to flagship models because they feel safe. I get it. But "safe" is a poor cost-control strategy.

Here's the routing table I landed on after months of iteration. It's deliberately boring:

MODEL_ROUTER = {
    "trivial":     "Qwen/Qwen3-8B",        # $0.01/M — FAQ, classification, intent
    "moderate":    "deepseek-v4-flash",    # $0.25/M — chat, summaries, translations
    "code":        "deepseek-coder",       # $0.25/M — code gen, refactors
    "reasoning":   "deepseek-reasoner",    # $2.50/M — math, planning, hard logic
    "premium":     "gpt-4o",               # $10.00/M — only when we truly need it
}

The classification step is the secret sauce. You need a small classifier that decides which bucket a request falls into. I use the same Qwen3-8B model for that classification step, which costs basically nothing:

from openai import OpenAI

client = OpenAI(
    base_url="https://global-apis.com/v1",
    api_key=os.environ["GLOBAL_API_KEY"],
)

def classify_complexity(user_input: str) -> str:
    """Pick a tier based on what the user actually needs."""
    classifier_prompt = f"""Classify this request into one of:
- trivial: factual lookups, simple Q&A, classification, formatting
- moderate: chat, summarization, translation, code edits
- code: generating new code, debugging, refactoring
- reasoning: math, multi-step planning, logic puzzles

Request: {user_input[:500]}
Tier:"""

    resp = client.chat.completions.create(
        model="Qwen/Qwen3-8B",
        messages=[{"role": "user", "content": classifier_prompt}],
        max_tokens=10,
        temperature=0,
    )
    return resp.choices[0].message.content.strip().lower()

def route_and_respond(user_input: str) -> str:
    tier = classify_complexity(user_input)
    model = MODEL_ROUTER.get(tier, "deepseek-v4-flash")

    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": user_input}],
    )
    return response.choices[0].message.content

This single change — literally swapping the default model — accounts for the bulk of our savings. Around 90% on its own, by my back-of-envelope math.

Tier 2: Escalate only when necessary

Once you have routing in place, the next refinement is escalation. Don't pick one model and commit. Try the cheap one, evaluate the output, and escalate only if the cheap one flunked.

The pattern looks like this:

def cascading_generate(prompt: str, budget_ceiling: float = 0.50) -> str:
    """Cheap-first, escalate-on-failure. Handles 80%+ at the bottom tier."""

    # Tier 1 — ultra-budget
    cheap_resp = call_model("Qwen/Qwen3-8B", prompt)  # $0.01/M
    if quality_score(cheap_resp, prompt) >= 0.8:
        return cheap_resp

    # Tier 2 — mid-budget
    mid_resp = call_model("deepseek-v4-flash", prompt)  # $0.25/M
    if quality_score(mid_resp, prompt) >= 0.9:
        return mid_resp

    # Tier 3 — premium only when forced
    return call_model("deepseek-reasoner", prompt)  # $2.50/M

def quality_score(output: str, original_prompt: str) -> float:
    """Heuristic: length sanity + a cheap-model self-eval."""
    if len(output.strip()) < 20:
        return 0.0
    eval_resp = client.chat.completions.create(
        model="Qwen/Qwen3-8B",
        messages=[{
            "role": "user",
            "content": f"Rate 0.0-1.0 how well this answers the question.\n"
                       f"Question: {original_prompt[:300]}\n"
                       f"Answer: {output[:300]}\n"
                       f"Score:"
        }],
        max_tokens=5,
        temperature=0,
    )
    try:
        return float(eval_resp.choices[0].message.content.strip())
    except ValueError:
        return 0.5

The distribution matters. In our deployment, roughly 85% of traffic resolves at the bottom tier. Another 12% needs the mid-tier. The remaining 3% hits premium. That 3% is where the quality-critical stuff lives — anything we'd be embarrassed to get wrong in front of a paying customer.

A concrete data point: our support chatbot went from $420/month to $28/month on the same volume. Same answers (or arguably better, since we tuned prompts more carefully), 15x cheaper. Nobody complained.

Tier 3: Cache everything cacheable

Caching is the technique that everyone says they're doing and almost nobody is actually doing right. The naive version is lru_cache on the function call. That's fine for unit tests. In production, you want keyed caching with TTLs, plus semantic similarity matching for the long tail.

Here's the deterministic version — identical inputs return cached outputs for a configurable window:

import hashlib
import json
import time
from typing import Any

_cache: dict[str, dict[str, Any]] = {}

def cached_completion(
    model: str,
    messages: list[dict],
    ttl_seconds: int = 3600,
    **kwargs,
):
    payload = json.dumps(
        {"model": model, "messages": messages, **kwargs},
        sort_keys=True,
    )
    key = hashlib.sha256(payload.encode()).hexdigest()

    if key in _cache:
        entry = _cache[key]
        if time.time() - entry["ts"] < ttl_seconds:
            return entry["response"]  # Free round-trip

    response = client.chat.completions.create(
        model=model,
        messages=messages,
        **kwargs,
    )
    _cache[key] = {"response": response, "ts": time.time()}
    return response

The hit rate depends entirely on your workload. For a customer support bot with a heavy FAQ tail, expect 50–80% cache hits once the cache warms up. For a creative writing tool, expect much less — those prompts are mostly unique. Don't get cocky and assume universal applicability.

For semantic caching (caching "What's your refund policy?" and "How do I get a refund?" as the same thing), I'll be honest — I've tried it. The embedding model costs offset a chunk of the savings, and the false-positive rate is a constant headache. Stick with deterministic caching unless you have very specific use cases where it pays off. YMMV, as the RFC folks say.

Tier 4: Compress your prompts

Here's the part where I think most engineers are leaving easy money on the table. Token costs are linear — a 2,000-token prompt costs exactly twice as much as a 1,000-token prompt. So why do I keep seeing prompts with three paragraphs of preamble that could be one sentence?

Let me show you what compression looks like in practice:

def compress_for_inference(context: str, target_chars: int | None = None) -> str:
    """Shrink long context windows before sending to the model."""
    if len(context) < 500:
        return context  # Not worth the round-trip cost

    target = target_chars or int(len(context) * 0.4)

    compression_prompt = (
        f"Compress the following into roughly {target} characters. "
        f"Preserve all facts, named entities, and numeric values. "
        f"Drop filler words and redundant phrasing.\n\n"
        f"---\n{context}\n---\n\nCompressed:"
    )

    resp = client.chat.completions.create(
        model="Qwen/Qwen3-8B",  # cheap model does the summarizing
        messages=[{"role": "user", "content": compression_prompt}],
        max_tokens=target // 3,
        temperature=0,
    )
    return resp.choices[0].message.content.strip()

The arithmetic matters here. A 2,000-token system prompt sent to DeepSeek V4 Flash costs roughly $0.0005 per request as input. Compressed to 400 tokens, that drops to $0.0001. Save $0.0004 per request. Sounds trivial, right?

Multiply by traffic. We were running about 10,000 inference calls per day at the time. $0.0004 × 10,000 = $4/day = $120/month = $1,460/year. From one compression job. Add a few more high-volume endpoints and the numbers climb fast. The math is not subtle.

Tier 5: Batch the small stuff

Batching is the technique most teams skip because the OpenAI Python SDK doesn't expose it natively. But if you're hitting the API from a backend, you almost certainly have opportunities to coalesce.

The before/after:

# Bad: N requests, N round-trips, N input token bills
results = []
for question in questions:
    resp = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content": question}],
    )
    results.append(resp.choices[0].message.content)

# Good: 1 request, shared system prompt, list-style output
batch_prompt = (
    "Answer each numbered question concisely. "
    "Return your answers in the same numbered format.\n\n"
    + "\n".join(f"{i+1}. {q}" for i, q in enumerate(questions))
)

resp = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": batch_prompt}],
    max_tokens=sum(estimated_len(q) for q in questions) // 2,
)

answers = parse_numbered_list(resp.choices[0].message.content)

Caveat: batching only works when you can defer the work. Real-time chat doesn't batch well. Nightly report generation absolutely does. Bulk classification, content moderation, analytics summarization — those are batching paradise.

The savings come from three places: shared system prompt tokens, shared boilerplate tokens, and one round-trip instead of N. In our analytics pipeline (running nightly summarization over ~8,000 support tickets), batching cut inference time by 60% and cost by roughly 18%. Not the biggest lever, but it's free money if your workload tolerates the latency.

Tier 6: Set token budgets explicitly

Most teams don't pass max_tokens to the API. That means the model decides when to stop. For most chat workloads, that's fine. For some workloads (summarization, classification, extraction), it's wasteful — the model happily writes three paragraphs when one sentence would do.

RESPONSE_BUDGETS = {
    "classification": 5,
    "extraction":     200,
    "summarization":  300,
    "chat":           800,
    "code":           1500,
    "reasoning":      2000,
}

def budgeted_call(model: str, messages: list[dict], tier: str):
    return client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=RESPONSE_BUDGETS.get(tier, 800),
    )

This is one of those changes that's almost embarrassing to write up because it's so obvious. But I've reviewed a lot of codebases that just leave max_tokens unset. Every team that fixes this saves 10-25% on output costs without changing anything else. Cumulatively across an org, that's real money.

Tier 7: Monitor or it didn't happen

You can't optimize what you can't measure. Wire up token-cost observability before you start tweaking. Otherwise you're flying blind, and you'll never know which optimizations actually worked.

Here's the snippet I drop into our middleware:

python
import logging

logger = logging.getLogger("llm.cost")

PRICING = {
    "Qwen/Qwen3-8B":       {"input": 0.0,   "output": 0.01},
    "deepseek-v4-flash":   {"input": 0.01,  "output": 0.25},
    "deepseek-coder":      {"input": 0.01,  "output": 0.25},
    "deepseek-reason