How I Cut Our AI API Bill by 95% Without Sacrificing p99 Latency

How I Cut Our AI API Bill by 95% Without Sacrificing p99 Latency

Six months ago I inherited a production workload at the company I work for. It was a customer-facing summarization service running on the most expensive model our team could name — GPT-4o — for every single request, no matter how trivial. The first invoice I pulled up nearly made me spill my coffee. We were pushing north of $14,000 a month for what was, fundamentally, a glorified summarization pipeline.

I spent the next quarter rebuilding the routing layer, the caching layer, and the prompt ingestion layer from scratch. By the time I was done, the bill had dropped to under $700 a month. Throughput went up. Our p99 latency actually got better, not worse, because we stopped hammering the slow tier for trivial work. This post is the architectural playbook I wish someone had handed me on day one.

Let me walk you through what changed.

Treat Model Selection Like a Routing Decision

In a multi-region deployment, you don't send every packet to your most expensive edge node. You route intelligently — static assets go to the CDN, dynamic writes to the primary region, reads to the replica in the closest zone. AI inference works the same way. The moment I started thinking of model choice as a routing problem rather than a "which tool do I like best" problem, everything clicked.

Look at the cost differentials in our internal benchmark. They're not marginal. They're absurd.

Task	What We Used	What We Use Now	Cost Delta
Casual chat	GPT-4o ($10/M output)	DeepSeek V4 Flash ($0.25/M)	-97.5%
Classification	GPT-4o-mini ($0.60/M)	Qwen3-8B ($0.01/M)	-98.3%
Code generation	GPT-4o ($10/M output)	DeepSeek Coder ($0.25/M)	-97.5%
Long-form summary	GPT-4o ($10/M output)	Qwen3-32B ($0.28/M)	-97.2%
Translation	GPT-4o ($10/M output)	Qwen-MT-Turbo ($0.30/M)	-97%

None of those workloads needed reasoning depth. They needed throughput, deterministic formatting, and reasonable language quality. The first move in any cost optimization is mapping task complexity to model tier — and ruthlessly stopping the bleed of expensive-tier calls on tasks that don't justify them.

Here's a simplified slice of the router I ended up shipping:

import openai

client = openai.OpenAI(
    api_key="sk-global-your-key-here",
    base_url="https://global-apis.com/v1"
)

MODEL_TIER = {
    "trivial": "Qwen/Qwen3-8B",          # $0.01/M
    "chat": "deepseek-v4-flash",          # $0.25/M
    "code": "deepseek-coder",          # $0.25/M
    "reasoning": "deepseek-reasoner",    # $2.50/M
}

def classify_load(user_input: str) -> str:
    # Your complexity classifier goes here.
    # In our case: keyword heuristics + length-based bucketing.
    ...

def route(user_input: str) -> str:
    tier = classify_load(user_input)
    return MODEL_TIER[tier]

def generate(user_input: str):
    model = route(user_input)
    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": user_input}],
    )

One file. One map. Swap providers by editing one constant. That's the entire philosophy.

Cascade Routing: The Pattern That Killed Our Burn Rate

Cascading tier routing is the single technique that recovered the most cost in our setup. The idea is borrowed from CDN origin-shield patterns: cheap tier first, expensive tier only on demand.

Most requests don't need the strongest model. Most requests are FAQ-style questions, account lookups, simple parsing, short rewrites. Those belong on a model that costs effectively nothing per million tokens. The remaining edge cases — the ones a budget model genuinely can't answer correctly — escalate to a stronger tier.

Here's the exact pseudocode we now run in production, lightly anonymized:

def cascade_generate(prompt: str, budget: float = 0.50):
    """
    Try cheap first. Escalate only when quality demands it.
    Designed for 99.9% uptime SLA — each tier has independent fallback.
    """

    # Tier 1: $0.01/M — handles the long tail of easy traffic
    cheap_resp = call_model("Qwen/Qwen3-8B", prompt)
    if confidence_score(cheap_resp) >= 0.80:
        return cheap_resp  # ~80% of traffic stays here

    # Tier 2: $0.25/M — for things that need coherence, not genius
    std_resp = call_model("deepseek-v4-flash", prompt)
    if confidence_score(std_resp) >= 0.90:
        return std_resp  # ~15% of traffic

    # Tier 3: $0.78–$2.50/M — reserved for actual reasoning
    return call_model("deepseek-reasoner", prompt)  # ~5% of traffic

The impact in plain numbers: our customer support chatbot went from $420/month down to $28/month, simply because 85% of inbound questions are now answered at the Qwen3-8B tier without ever touching anything more expensive. The answers are equally good for the user — which is the only metric that actually matters.

Edge Caching: Latency Goes Down, Cost Goes Down

Every cloud architect knows the rule: cache reads aggressively. When the read is hitting an external API that bills per token, the ROI on caching is even higher than the usual database case.

I run an in-memory LRU with a TTL, fronting every model call. The hit rate on common prompts — FAQ lookups, documentation Q&A, anything templated — sits between 50% and 80% in our production logs. Every cache hit is a request that costs us zero tokens and zero milliseconds of p99 latency exposure.

import hashlib
import json
import time

_cache = {}

def cached_call(model: str, messages: list, ttl: int = 3600):
    digest = hashlib.md5(
        json.dumps({"model": model, "messages": messages},
                   sort_keys=True).encode()
    ).hexdigest()

    if digest in _cache:
        entry = _cache[digest]
        if time.time() - entry["ts"] < ttl:
            return entry["response"]

    resp = client.chat.completions.create(
        model=model,
        messages=messages,
    )
    _cache[digest] = {"response": resp, "ts": time.time()}
    return resp

A few operational notes from the trenches:

• Use sort_keys=True in your hash input. Two semantically identical prompts with reordered keys must hash to the same value, or you'll fragment your cache.
• Memory bounds matter. Set an LRU cap. An unbounded dict on a long-running pod will eventually OOM at the worst possible time.
• TTL discipline is non-negotiable. Stale answers are a correctness bug, not a cost optimization. Pick TTLs based on how stale the underlying facts can safely get.

For multi-region deployments we also push the cache into Redis (or Memcached) so warm hits are shared across pods and across AZs. That's another layer — but the principle is identical.

Token Budget Engineering: The Cheap Model That Pays For Itself

This one surprised me the first time I measured it. We had a 2,000-token system prompt that got piped into every request. Two thousand tokens of carefully written context that the model technically needed but practically only referenced one-tenth of.

The fix was so simple I almost felt silly: run the long context through a cheap summarizer before it reaches the real model. The summarizer costs a tenth of a cent. The savings on the downstream call are an order of magnitude larger.

def compress_context(text: str, target_ratio: float = 0.5) -> str:
    if len(text) < 500:
        return text  # Already cheap enough — don't bother

    target_chars = int(len(text) * target_ratio)
    summary = call_model(
        "Qwen/Qwen3-8B",
        f"Summarize the following in roughly {target_chars} chars. "
        f"Preserve facts, drop filler:\n\n{text}",
    )
    return summary

I'll do the math on the napkin the way I did for my team. A 2,000-token prompt trimmed to 400 tokens on DeepSeek V4 Flash saves $0.024 per request. At 10,000 requests a day that's $240/day, which works out to about $87,600 a year — from one cleanup. The cost of the summarization round-trip is roughly a tenth of that, and it goes through the cheapest tier possible.

I run compression in a sidecar that batches and amortizes the summarization cost. In multi-region deployments you can co-locate this with your edge routing so the summarization hop doesn't add meaningful p99 latency.

Batching: Throughput Is a Latency Strategy

Last optimization I want to talk about is the unsexy one: batching. When you have ten customer support questions lined up in a queue, sending them one at a time is the cloud equivalent of making ten separate HTTP requests instead of one. Not catastrophic at small scale, but at our QPS it's the difference between autoscaling at 4 pods and 12.

The before/after is straightforward enough that I'll show it bare:

# Before — 3 separate round-trips, 3× the input tokens billed
answers = []
for q in questions:
    resp = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content": q}],
    )
    answers.append(resp)

# After — 1 round-trip, 1 shared system prompt, lower total tokens
joined = "\n".join(f"{i+1}. {q}" for i, q in enumerate(questions))
batch = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{
        "role": "system",
        "content": "Answer each numbered question in order. "
                   "Keep answers concise.",
    }, {
        "role": "user",
        "content": joined,
    }],
)
answers = parse_numbered(batch.choices[0].message.content)

In our logs the batched path completes 10 requests inside what a single request would have taken on the unbatched path. That's not a small win — it's a p99 improvement because tail latency lives in tail round-trips, and you've just made ten of them into one.

What I Wish I'd Done First

If I had to give a single piece of advice to anyone staring down a six-figure AI bill, it would be this: don't start with prompt engineering. Start with the routing layer. A perfect prompt on the wrong model is still expensive; an imperfect prompt on the right model is often free.

The full sequence I'd recommend is:

1. Stand up tier routing — even a if/elif chain on task type beats none.
2. Add caching on top, because it's almost zero implementation cost.
3. Compress prompt context as a steady-state cleanup, not a one-off rewrite.
4. Batch anything queue-like.
5. Observe, then iterate.

Throughout, keep p99 latency and 99.9% uptime as your north-star metrics. Cost optimization that breaks your SLA isn't optimization, it's a different kind of outage.

One More Thing

All the pricing and benchmarks I quoted here came from running the workloads against a single endpoint — Global API (base URL https://global-apis.com/v1). They handle the multi-region fan-out