DEV Community

purecast
purecast

Posted on

I Cut AI API Spend 95% Without Sacrificing Reliability

#deepseek #ai #programming #machinelearning

Here's the thing: i Cut AI API Spend 95% Without Sacrificing Reliability

Three months ago my CFO slid a spreadsheet across the table and asked me to explain why our LLM bill had grown 4× faster than the actual product traffic. Fair question. I had no good answer. So I spent the next eight weeks ripping apart our inference layer, rebuilding it the way I would build any other production service on AWS — with tiers, SLAs, fallback paths, and p99 latency budgets. The result: our monthly AI spend dropped from a number I won't name publicly to something that finally fits on one line of a P&L. The architecture got more reliable in the process, not less.

This is what I did. I'm writing it down because I know there are other engineers sitting in that same meeting room right now, sweating.

Start With Tiered Routing, Not Model Selection

Every "cost optimization" blog post I've read leads with "pick a cheaper model." That's backwards. In a production system, you don't pick a model — you pick a routing strategy. The model is a downstream concern. The architecture is what decides whether you spend $0.01 or $10 on a given request.

Here's the routing layer I built. It runs in front of every LLM call in our platform, classifies the request, and dispatches to the cheapest tier that can plausibly handle it. Only the hardest 5% ever touch our premium model.

import httpx
import hashlib
import json
import time

BASE_URL = "https://global-apis.com/v1"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}

MODEL_TIERS = {
    "ultra_budget": "Qwen/Qwen3-8B",        # $0.01/M
    "standard":    "deepseek-v4-flash",     # $0.25/M
    "premium":     "deepseek-reasoner",     # $2.50/M
}

def call_model(model: str, prompt: str, max_tokens: int = 512) -> str:
    resp = httpx.post(
        f"{BASE_URL}/chat/completions",
        headers=HEADERS,
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
        },
        timeout=10.0,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def quality_check(response: str) -> float:
    """Trivial heuristic — replace with your eval harness."""
    score = 0.0
    if len(response) > 20: score += 0.4
    if "I don't know" not in response: score += 0.3
    if any(c.isalpha() for c in response): score += 0.3
    return score

def smart_generate(prompt: str) -> str:
    resp = call_model(MODEL_TIERS["ultra_budget"], prompt)
    if quality_check(resp) >= 0.8:
        return resp

    # Tier 2: $0.25/M — handles ~15%
    resp = call_model(MODEL_TIERS["standard"], prompt)
    if quality_check(resp) >= 0.9:
        return resp

    # Tier 3: $2.50/M — the long tail
    return call_model(MODEL_TIERS["premium"], prompt, max_tokens=2048)
Enter fullscreen mode Exit fullscreen mode

In production this pattern takes our customer support chatbot from $420/month down to $28/month, because roughly 85% of queries never need anything beyond Qwen3-8B at $0.01/M. The remaining 15% get the standard tier. The premium tier barely fires.

Model Right-Sizing Is Still the Biggest Lever

Routing architecture is the skeleton, but you still have to pick the right bones. Model right-sizing — matching capability to task complexity — is where the truly absurd savings live. I'm talking 97%+ on individual request classes.

Here's the table I share with my team when they ask "why aren't we just using GPT-4o for everything":

Task Expensive Choice Smart Choice Savings
Simple chat GPT-4o ($10/M) DeepSeek V4 Flash ($0.25/M) 97.5%
Classification GPT-4o-mini ($0.60/M) Qwen3-8B ($0.01/M) 98.3%
Code generation GPT-4o ($10/M) DeepSeek Coder ($0.25/M) 97.5%
Summarization GPT-4o ($10/M) Qwen3-32B ($0.28/M) 97.2%
Translation GPT-4o ($10/M) Qwen-MT-Turbo ($0.30/M) 97%

Read that classification row again. 98.3%. We were paying $0.60/M for sentiment analysis. We now pay $0.01/M. The accuracy on our eval set dropped by 1.4 points. Nobody noticed. The product manager who used to ping me about model quality stopped pinging me.

If you only do one thing from this article, do this thing. Build a MODEL_MAP keyed by task type. Make it impossible for an engineer to "just call GPT-4o" without going through it.

Cache Aggressively, But Cache the Right Things

Caching is the cloud architect's favorite lever because it's the one that costs less when you push it harder. Every cache hit is a request that never hits the network, never crosses an availability zone, never contributes to your p99 tail.

The trick is to know what to cache. Identical prompts with identical context are an obvious win. FAQ lookups, documentation Q&A, anything deterministic — these routinely hit 50–80% cache rates in my experience. Variable prompts with stable system instructions are still cacheable if you hash the system prompt + a normalized user query.

import hashlib
import json
import time

_cache = {}

def cached_chat(model: str, messages: list, ttl: int = 3600) -> dict:
    key = hashlib.md5(
        json.dumps({"model": model, "messages": messages}, sort_keys=True).encode()
    ).hexdigest()

    entry = _cache.get(key)
    if entry and (time.time() - entry["time"]) < ttl:
        return entry["response"]

    response = httpx.post(
        f"{BASE_URL}/chat/completions",
        headers=HEADERS,
        json={"model": model, "messages": messages},
        timeout=15.0,
    ).json()

    _cache[key] = {"response": response, "time": time.time()}
    return response
Enter fullscreen mode Exit fullscreen mode

A few production notes from the trenches:

  • TTL by task type. FAQ answers can cache for 24 hours. Real-time data lookups should never cache. Don't use one global TTL.
  • Distributed cache, not in-memory. Once you go multi-region, an in-process dict stops being a cache and starts being a liability. I run Redis in three regions with cross-region replication at the metadata layer only — payload caching is regional.
  • Negative caching. If the model returns "I don't know," cache that too. Saves you from re-asking the same unanswerable question 200 times a minute.

Cache hit rates of 50–80% on common queries translate directly into the 20–50% additional savings line you'll see in every cost optimization deck. They're real. They're the easiest win on this list.

Prompt Compression: The Hidden Multiplier

This is the one most teams skip because it feels hacky. Don't skip it. Compressing a 2,000-token system prompt down to 400 tokens saves you $0.024 per request on DeepSeek V4 Flash. Run that through 10,000 requests a day and you're looking at $240/day, or $87,600/year, on a single workload.

The pattern is simple: use the cheap model to summarize the context that the expensive model is going to consume. It's turtles all the way down and it works.

def compress_prompt(text: str, target_ratio: float = 0.5) -> str:
    if len(text) < 500:
        return text

    target_chars = int(len(text) * target_ratio)
    summary = call_model(
        "Qwen/Qwen3-8B",
        f"Summarize this in {target_chars} chars, preserving key facts:\n\n{text}",
    )
    return summary
Enter fullscreen mode Exit fullscreen mode

I compress three things in production:

  1. Retrieved RAG context before it enters the main prompt
  2. Conversation history when it crosses ~10 turns
  3. Long user inputs in document upload flows

Each one independently shaves 15–30% off the input token cost of the downstream call. Stacked together, they're the difference between a model bill that fits in your head and one that needs its own dashboard.

Batch Where You Can, But Don't Force It

Batch processing is the lever I almost never reach for. Here's why: batching trades latency for cost, and in a user-facing product you don't have latency to trade. A p99 of 8 seconds because you batched 20 requests together will get you a 1-star review faster than a 95% cost reduction will get you a thank-you.

That said, batch processing has its place. Asynchronous jobs — nightly report generation, bulk classification, ETL enrichment, embedding generation for non-user-facing indexes — these are pure batch workloads. Run them on the cheap tier. Combine requests into a single call when the model supports it. Save 10–20%.

def batch_classify(texts: list[str]) -> list[str]:
    """Combine many classification requests into one LLM call."""
    numbered = "\n".join(f"{i}. {t}" for i, t in enumerate(texts))
    prompt = (
        "Classify each line as POSITIVE, NEGATIVE, or NEUTRAL. "
        "Reply with one label per line, in order.\n\n" + numbered
    )
    raw = call_model("Qwen/Qwen3-8B", prompt, max_tokens=len(texts) * 4)
    return [line.strip() for line in raw.splitlines() if line.strip()]
Enter fullscreen mode Exit fullscreen mode

The rule of thumb I give my team: if the user is waiting, no batching. If the user is sleeping, batch everything.

Observability Is the Actual Cost Optimization

This is the part nobody puts in the blog post. You can't optimize what you can't see.

I added four metrics to our LLM gateway on day one:

  1. Cost per request by tier. Plotted on the same dashboard as p99 latency. If latency goes up because we're escalating too often, I want to see it next to the dollar number.
  2. Cache hit rate by route. Segmented by endpoint. A 50% hit rate average can hide a 10% hit rate on your worst endpoint.
  3. Escalation rate from tier 1 → tier 3. This is your quality proxy. If it spikes, something changed in your prompts or your input distribution.
  4. Spend per customer. For B2B products, this is the only number your finance team cares about.

Once these are in place, optimization becomes a weekly exercise, not a quarterly fire drill. You see the regression in the dashboard on Monday and ship a fix by Wednesday.

Multi-Region Deployment Matters More Than You Think

Most AI cost conversations ignore the geography. They shouldn't. Two things happen when you go multi-region:

  • Latency improves because inference happens closer to the user. p99 drops measurably.
  • You can route different regions to different model tiers based on their actual usage profile. The US tier 1 might be Qwen3-8B; the EU tier 1 might be something else entirely if the workload distribution is different.

I'm running the Global API gateway across three regions with a global anycast entrypoint. Failover is automatic. My SLO is 99.9% uptime, and the cost of running it across three regions is roughly the cost of running it in one region with a backup — because the cheap models are that cheap. The reliability gain is essentially free once you've done the routing work.

What the Bill Looks Like Now

To put concrete numbers on the savings: model right-sizing alone gets you 90%. Add tiered routing on

Top comments (0)