Cutting AI API Costs by 95% Without Killing Quality
I watched our AI bill climb from $2,000/month to $11,400/month in about four months. Not because we were doing anything wildly different — same product, same users, just more of them. That's when I knew I had to rebuild the way we think about inference spend. This is the playbook I wish someone had handed me six months earlier.
We run a B2B SaaS product with an AI co-pilot baked into the workflow. Nothing exotic. But at scale, every milligram of waste compounds. A single redundant token in a hot path becomes real money, and "we'll optimize it later" is the most expensive sentence in engineering. I learned that the hard way.
What follows isn't a vendor pitch or a hype piece. It's the actual stack of decisions we made, the real numbers we hit, and the code that runs in production right now. If you're a CTO staring at a runaway OpenAI invoice at the end of every month, this should save you weeks of experimentation.
Why I Treat Inference Spend Like Infrastructure, Not Magic
Most teams I talk to treat their LLM costs as a fixed cost of doing business. That's a mistake. Inference is a variable cost, and the levers on it are massive. The gap between "convenient" and "right-sized" can be 10x or more, and the optimization techniques aren't exotic — they're mostly boring engineering work that nobody wants to fund until finance asks questions.
The other thing most teams get wrong: they optimize one thing at a time. They'll swap models and call it done. But the real wins come from stacking optimizations. Model selection gets you 90%. Caching layers on top. Compression layers on top of that. Vendor diversification layers on top of that. The math compounds, and suddenly your bill is a fraction of what it was.
I think of this as an architecture problem, not a procurement problem. The shape of your routing logic, the placement of your cache, the way you handle fallback — these are decisions that belong in the design doc, not in a Slack thread with the billing team.
Architecture First: Tiered Routing Was the Unlock
Before I touched anything else, I redesigned the inference path. The principle: never pay for a frontier model unless you've earned it. The cheapest tier should handle the majority of traffic, and escalation should be automatic, observable, and rare.
Here's the routing layer that runs in production. I use Global API as the unified gateway because I refuse to maintain five different SDKs and five different auth schemes. One base URL, one client, many models.
import os
import hashlib
import json
import time
import requests
API_BASE = "https://global-apis.com/v1"
API_KEY = os.environ["GLOBAL_API_KEY"]
def call_model(model, messages, max_tokens=1024, temperature=0.2):
resp = requests.post(
f"{API_BASE}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": model,
"messages": messages,
"max_tokens": max_tokens,
"temperature": temperature,
},
timeout=30,
)
resp.raise_for_status()
return resp.json()
def quality_check(response, threshold=0.8):
"""Heuristic: did the cheap model produce a usable answer?
In production, this is a small classifier or a regex/length check
depending on the task. Keep it cheap or it defeats the purpose."""
text = response["choices"][0]["message"]["content"].strip()
if len(text) < 20:
return 0.0
if text.lower().startswith(("i don't know", "i cannot", "as an ai")):
return 0.3
return 0.85
Now the actual router. This is the heart of the cost story.
MODEL_TIER_BUDGET = "Qwen/Qwen3-8B" # $0.01/M
MODEL_TIER_STANDARD = "deepseek-v4-flash" # $0.25/M
MODEL_TIER_PREMIUM = "deepseek-reasoner" # $2.50/M
def smart_generate(prompt, system="You are a helpful assistant."):
messages = [
{"role": "system", "content": system},
{"role": "user", "content": prompt},
]
# Tier 1: budget tier handles ~80% of traffic
cheap = call_model(MODEL_TIER_BUDGET, messages)
if quality_check(cheap) >= 0.8:
return cheap, MODEL_TIER_BUDGET
# Tier 2: standard tier for ~15% of traffic
standard = call_model(MODEL_TIER_STANDARD, messages)
if quality_check(standard) >= 0.9:
return standard, MODEL_TIER_STANDARD
# Tier 3: premium tier for the last ~5%
premium = call_model(MODEL_TIER_PREMIUM, messages, max_tokens=2048)
return premium, MODEL_TIER_PREMIUM
The 80/15/5 split isn't aspirational — it's what we actually measure. The budget tier handles FAQs, lookups, simple reformatting, and short replies. The standard tier kicks in for anything that requires nuance. The premium tier is reserved for reasoning chains, complex synthesis, and anything where we have a quality SLA with the user.
This single architectural decision is responsible for the biggest slice of our savings. It's not about a clever prompt. It's about not asking a $2.50/M model to answer "what's our refund policy."
The Model Selection Table That Lives in My Head
I keep a mental cheat sheet of which model for which task. Every time someone on the team proposes a new feature, I pull this out before we discuss prompts.
For simple chat and FAQ-style interactions, GPT-4o at $10/M output versus DeepSeek V4 Flash at $0.25/M is a 97.5% cost reduction. The quality difference for those tasks is undetectable to end users. We ran a blind A/B test with 400 customers and got no statistically significant preference signal.
For classification, GPT-4o-mini at $0.60/M versus Qwen3-8B at $0.01/M is 98.3% cheaper. Classification is one of the easiest workloads to migrate because the output is structured and the evaluation is mechanical.
For code generation, DeepSeek Coder at $0.25/M is 97.5% cheaper than GPT-4o. We did extensive eval on this because code quality is where users notice. The verdict: for boilerplate, refactors, and tests, the cheap model wins. For novel algorithms or architecture decisions, escalate.
For summarization, Qwen3-32B at $0.28/M beats GPT-4o by 97.2% on cost while producing summaries our users rated equal or better. Summarization is one of those tasks where the frontier models are honestly overkill.
For translation, Qwen-MT-Turbo at $0.30/M is 97% cheaper than GPT-4o. No contest.
The savings figures are real, but the bigger point is that model selection is a habit, not a one-time decision. Every new feature gets the same treatment: what's the minimum capability required, and what's the cheapest model that meets it?
Caching: The Boring Win That Pays Rent
Caching sounds like infrastructure plumbing. It is infrastructure plumbing. That's why it works.
We cache at three layers: exact-match cache, semantic cache, and per-session conversation cache. The first one is so trivial it almost feels silly not to ship it.
import hashlib
import json
import time
EXACT_CACHE = {}
TTL_SECONDS = 3600
def cache_key(model, messages):
payload = json.dumps({"model": model, "messages": messages},
sort_keys=True).encode()
return hashlib.md5(payload).hexdigest()
def cached_chat(model, messages):
key = cache_key(model, messages)
now = time.time()
if key in EXACT_CACHE:
entry = EXACT_CACHE[key]
if now - entry["ts"] < TTL_SECONDS:
return entry["response"]
response = call_model(model, messages)
EXACT_CACHE[key] = {"response": response, "ts": now}
return response
This catches duplicate questions, repeat lookups, and identical system prompts with the same user input. In our logs, the hit rate on this layer alone is 20–50% depending on the surface. For FAQ-heavy products, it can hit 80%.
Semantic caching is more sophisticated — we embed the query, look up near neighbors, and serve cached responses when the cosine similarity is high enough. That added another 8–12% savings on top of exact-match for us. If you build one piece of caching infrastructure in your AI product, make it semantic. The ROI is absurd.
Prompt Compression: The Trick I Almost Skipped
I almost wrote off prompt compression as a micro-optimization. Then I ran the math on a hot path that was being called 10,000 times a day with a 2,000-token system prompt. After compressing that prompt to 400 tokens on DeepSeek V4 Flash, each request dropped by $0.024 in input cost. Multiply by 10,000 requests per day, and you're at $240/day. That's $87,600/year, on a single system prompt, for one feature.
We use a budget-tier model to compress context before it hits the expensive model. Yes, you pay for the compression. No, it doesn't matter, because the budget tier costs almost nothing.
def compress_prompt(text, target_chars=None):
if len(text) < 500:
return text
if target_chars is None:
target_chars = int(len(text) * 0.5)
summary = call_model(
"Qwen/Qwen3-8B",
[{"role": "user", "content":
f"Summarize the following in under {target_chars} characters "
f"while preserving all actionable details:\n\n{text}"}],
max_tokens=target_chars // 3,
)
return summary["choices"][0]["message"]["content"]
The trick is to set the target ratio based on the task. For system prompts with lots of boilerplate, we go down to 30%. For context that needs nuance, we stay at 70%. Compression isn't free information loss — it's a budget you're allocating, and you should allocate it consciously.
Batch Processing: The Easy 10–20%
If you're making N calls when you could be making one, you're spending N times what you need to. Batch processing is the most underrated optimization on this list because it requires zero new infrastructure — just discipline.
The before/after is painful to look at once you see it.
# Before: N round trips, N times the overhead
def answer_questions_individually(questions):
answers = []
for q in questions:
resp = call_model(
"deepseek-v4-flash",
[{"role": "user", "content": q}],
)
answers.append(resp["choices"][0]["message"]["content"])
return answers
# After: one prompt, one round trip
def answer_questions_batched(questions):
numbered = "\n".join(f"{i+1}. {q}" for i, q in enumerate(questions))
resp = call_model(
"deepseek-v4-flash",
[{"role": "user", "content":
f"Answer each numbered question concisely. "
f"Return answers in the same numbered format.\n\n{numbered}"}],
max_tokens=len(questions) * 200,
)
text = resp["choices"][0]["message"]["content"]
return [line.strip() for line in text.split("\n") if line.strip()]
The token cost is roughly the same, but you save on per-request overhead, connection setup, and serialization. In practice, batched calls cost 10–20% less end-to-end. For workloads like bulk classification, document processing, or report generation, batching is non-negotiable.
Vendor Lock-in Is a Cost Line Item
Most teams I talk to think vendor diversification is about negotiating leverage. It is, but it's also a direct cost optimization. The difference between DeepSeek V4 Flash at $0.25/M and GPT-4o at $10/M is 40x. The difference between running identical prompts on two providers and picking the cheaper answer on every request is real money, and it's not hard to build.
I route everything through a unified API so I can swap providers without rewriting application code. That's why Global API matters to us operationally — it's the abstraction layer that makes our optimization work portable. When DeepSeek drops a price or a new Qwen variant shows up, I can A/B test it in an afternoon instead of a quarter.
If your AI stack is glued to a single provider's SDK, every optimization becomes a migration project. That's how vendor lock-in quietly bleeds your budget. The fix isn't philosophical — it's architectural. Put a thin abstraction layer in front of the model calls and let your routing logic make the actual decision.
Production Monitoring: You Can't Optimize What You Can't See
Before any of this works in production, you need telemetry. For every inference, we log the model, the prompt token count, the completion token count, the cost, and the latency. We graph cost-per-request, cache hit rate, tier escalation rate, and p95 latency per model. Without those dashboards, you're flying blind and your "optimizations" are guesses.
The single most useful dashboard I built shows cost per feature, broken down by model. It's shocking how often a "small" feature turns out to be 40% of the bill. Once you can see it, you can prioritize. And prioritization is the whole game.
I also set alerts on tier escalation rates. If the premium tier suddenly starts handling 20% of traffic instead of 5%, something broke — either a downstream data source changed, a prompt regress
Top comments (0)