I Cut My AI API Bill By 95% — Here's Everything I Wish I Knew Sooner
I still remember the Slack message. Our VP of Engineering pinged me at 11pm on a Thursday: "Why did our OpenAI bill just triple this month?" I opened the dashboard, and my stomach dropped. Three separate product teams had started shipping features that hit GPT-4o for every single request. Nobody had set up budgets. Nobody had set up alerts. Nobody had even thought about p99 latency on the upstream provider. We were just... spending.
That night kicked off a six-week sprint where I rebuilt our entire inference layer from the ground up. By the end, we were running at roughly 5% of our previous spend — with better p99 latency, 99.9% availability across multi-region failover, and a routing layer I'm honestly a little proud of.
Here's what I learned. Everything below uses real numbers from that project.
Start With the Cost Dashboard Before You Touch Anything Else
Before optimizing anything, I had to know where the money was actually going. I built a small middleware that wrapped every model call and logged: model name, input tokens, output tokens, latency, region, and a tag for which feature triggered it. If you skip this step, you'll optimize the wrong things. I promise you.
Once that was live, I pulled the numbers. About 85% of our spend was going to GPT-4o at $10/M output tokens. The remaining 15% was split across GPT-4o-mini at $0.60/M and a few smaller experiments. None of it was being measured at p99 latency. None of it had a failover plan.
That visibility is what made the rest possible.
Step 1: Compress Prompts at the Edge
The first thing I tackled was prompt bloat. Our RAG pipeline was stuffing 2,000-token context windows into every single request. Most of that context was repetitive system instructions, examples, and tool descriptions that never changed.
Here's the approach I landed on — a cheap summarization pass before the real call:
import openai
client = openai.OpenAI(
api_key="YOUR_GLOBAL_API_KEY",
base_url="https://global-apis.com/v1"
)
def compress_prompt(text, target_ratio=0.5):
"""Shrink long prompts with an ultra-cheap model before the real call."""
if len(text) < 500:
return text
summary = client.chat.completions.create(
model="Qwen/Qwen3-8B", # $0.01/M
messages=[{
"role": "user",
"content": f"Summarize this in {int(len(text)*target_ratio)} chars: {text}"
}]
)
return summary.choices[0].message.content
The math on this one stunned me. A 2,000-token system prompt compressed to 400 tokens saves $0.024 per request on DeepSeek V4 Flash. At 10,000 requests per day, that's $240/day, which compounds to roughly $87,600/year. From one function.
I deployed this at our edge layer with a 50ms p99 budget for the compression step. Anything that took longer got passed through uncompressed. Multi-region caching of compressed prompts was a free win — once you've compressed a prompt, you don't need to compress it again.
Step 2: Cache Aggressively, Invalidate Carefully
Next up was caching. The classic move is to hash the exact prompt and serve it from memory when it matches. But from an architect's perspective, you need to think about TTL, cache stampede, and what happens when your cache layer goes down.
Here's the version I shipped:
import hashlib, json, time
cache = {}
def cached_chat(model, messages, ttl=3600):
key = hashlib.md5(
json.dumps({"model": model, "messages": messages}).encode()
).hexdigest()
entry = cache.get(key)
if entry and (time.time() - entry["time"]) < ttl:
return entry["response"] # Cache hit — zero cost
response = client.chat.completions.create(model=model, messages=messages)
cache[key] = {"response": response, "time": time.time()}
return response
For our FAQ and documentation lookup features, this hit 50-80% cache rates. That's an extra 20-50% savings on top of everything else.
The reliability trick: I ran the cache as a separate service in two regions with eventual consistency. If both regions went down, the system gracefully degraded to direct API calls. No hard dependency. p99 latency on cache hits was under 5ms — your users will never notice.
Step 3: Stop Using GPT-4o for Everything
This was the big one. The single largest lever in the whole project.
I sat down with each product team and asked: "What does this feature actually need?" The answers were humbling. Our internal chatbot didn't need a frontier model. Our classification pipeline didn't need a frontier model. Our translation feature definitely didn't need a frontier model.
Here's the matrix I built and shared with leadership:
| Task | Expensive Choice | Smart Choice | Savings |
|---|---|---|---|
| Simple chat | GPT-4o ($10/M) | DeepSeek V4 Flash ($0.25/M) | 97.5% |
| Classification | GPT-4o-mini ($0.60/M) | Qwen3-8B ($0.01/M) | 98.3% |
| Code generation | GPT-4o ($10/M) | DeepSeek Coder ($0.25/M) | 97.5% |
| Summarization | GPT-4o ($10/M) | Qwen3-32B ($0.28/M) | 97.2% |
| Translation | GPT-4o ($10/M) | Qwen-MT-Turbo ($0.30/M) | 97% |
Once the teams saw those numbers, the conversation basically ended itself. Nobody argued for paying 40x more for equivalent output on a classification task.
This single change — matching the model to the task — saved us roughly 90% on the affected workloads. And honestly, the smaller models often had better p99 latency because they're lighter and the upstream providers have more capacity headroom.
Step 4: Build a Tiered Routing Layer
This is the architect's move. Don't pick one model. Build a cascade.
The idea: try the cheapest model first. If the response is good enough, return it. If not, escalate. And only escalate to the expensive reasoning model when you really need it.
def smart_generate(prompt, max_budget=0.50):
"""Cascade from cheap to expensive based on quality."""
resp = call_model("Qwen/Qwen3-8B", prompt)
if quality_check(resp) >= 0.8:
return resp # Handles 80%+ of traffic
# Tier 2: Standard at $0.25/M
resp = call_model("deepseek-v4-flash", prompt)
if quality_check(resp) >= 0.9:
return resp # Handles another 15%
# Tier 3: Premium at $0.78–$2.50/M
return call_model("deepseek-reasoner", prompt) # Only 5% of traffic
The quality_check function is the secret sauce. For our use case it was a simple classifier — another Qwen3-8B call — that scored whether the response was coherent, complete, and on-topic. We A/B tested the whole cascade for two weeks against the old single-model setup. Quality metrics were statistically indistinguishable. Cost was down 95%.
From a reliability standpoint, this design is also beautiful. If the cheap model goes down, traffic naturally flows to Tier 2. If Tier 2 has an incident, Tier 3 picks up. No manual failover needed. p99 latency actually improved because 80% of requests were terminating at the cheapest, fastest model.
Our customer support chatbot — the one I mentioned earlier — went from $420/month to $28/month. Same quality scores, same SLA, 99.9% availability maintained.
Step 5: Batch What You Can
The last lever is batching. If you're sending three separate requests to the same model, send one request with three prompts. You're charged for the combined input tokens anyway, but you avoid three separate round trips and three separate output token minimums.
# Before: 3 calls, 3x output token minimums
for question in questions:
response = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[{"role": "user", "content": question}]
)
# After: 1 call, shared context
batch_prompt = "\n\n".join([f"{i+1}. {q}" for i, q in enumerate(questions)])
response = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[{
"role": "user",
"content": f"Answer each question separately:\n{batch_prompt}"
}]
)
This isn't appropriate for every workload — latency-sensitive user-facing requests shouldn't be batched — but for our nightly report generation and bulk classification jobs, it saved another 10-20%.
The Reliability Layer Nobody Talks About
Here's the part most cost optimization guides skip: when you start running across multiple models and providers, you need to think about uptime. Single-provider architectures have single points of failure. We learned this the hard way when our primary provider had a regional incident and our entire product went dark.
What I shipped after that:
- Multi-region deployment with active-active routing between two cloud regions
- Per-model health checks with circuit breakers — if a model's p99 latency spikes above 3x baseline, traffic auto-routes to the next option
- A 99.9% uptime SLA committed internally, backed by the redundancy
- Auto-scaling on the routing layer itself, because the cascade means traffic patterns shift when models go down
This is the unsexy work that makes the cost optimizations actually safe to ship. Without it, you're one upstream incident away from an outage.
Putting It All Together
After six weeks, here's where we landed:
- Smart model selection: ~90% reduction on matched workloads
- Tiered routing on top of that: pushed us to ~95% on the chatbot specifically
- Response caching: 20-50% additional savings depending on the feature
- Prompt compression: 15-30% per request on long-context workloads
- Batching: 10-20% on bulk jobs
The bill went from "alarming" to "basically nothing." More importantly, we hit our 99.9% SLA, our p99 latency actually dropped (because cheap models tend to be faster), and our engineering team finally had visibility into what every dollar was buying.
If you're staring at your own AI bill right now and feeling that same 11pm-Slack-message panic, I feel for you. The good news is that all of this is achievable in a single sprint with one engineer. The model landscape has matured enough that you can route aggressively without quality tradeoffs.
One thing that helped me a lot was routing everything through Global API (https://global-apis.com/v1). Having a single OpenAI-compatible endpoint across DeepSeek, Qwen, and the rest made the cascade layer way cleaner to build and way easier to monitor. Worth checking out if you want to skip some of the integration plumbing.
Good luck out there. And set up the budget alerts before you go to bed tonight.
Top comments (0)