I Cut Our AI API Bill by 95% in 90 Days: A CTO's Playbook
Three months ago our LLM line item showed up in the monthly close and I nearly spit coffee on my keyboard. We were spending more on tokens than on our entire AWS bill. That's not hyperbole — that's me refreshing the dashboard at 11pm wondering where our runway went.
Here's the thing nobody tells you when you start building with AI APIs: the default path is the expensive path. Most teams I talk to are paying 5–10x what they should, and they don't know it because the bills are noisy and the per-request cost feels "small." At scale, those tiny costs compound into payroll. So I rebuilt our inference layer from scratch, and what follows is the playbook I wish someone had handed me on day one.
If you're optimizing for ROI, fast iteration, and not getting trapped in vendor lock-in, this is for you.
The 5–10x Tax Nobody Warned Me About
When I joined my current company, the previous engineer had wired up every feature against the "good" model because, well, it worked. GPT-4o for chat, GPT-4o for summarization, GPT-4o for classification. Default to the expensive option. It shipped fast. It also burned cash.
The dirty secret of LLM costs is that model selection alone can swing your bill by 90%. Not 30%, not 50% — ninety. The second dirty secret is that the cheaper models are not the terrible models you might be imagining. They've gotten production-ready for a huge swath of real workloads.
So step one is admitting you have a selection problem.
Step 1: Stop Picking One Model For Everything
The single biggest architectural decision you can make is to treat the model as a swappable component matched to task complexity. This is the opposite of what most teams do. Most teams pick one vendor, one model, and shove every prompt through it. That's not an architecture — that's a habit.
Here's the table I ended up with after benchmarking our actual workloads:
| Task | What We Used To Use | What We Use Now | Savings |
|---|---|---|---|
| Simple chat | GPT-4o ($10/M out) | DeepSeek V4 Flash ($0.25/M) | 97.5% |
| Classification | GPT-4o-mini ($0.60/M) | Qwen3-8B ($0.01/M) | 98.3% |
| Code generation | GPT-4o ($10/M) | DeepSeek Coder ($0.25/M) | 97.5% |
| Summarization | GPT-4o ($10/M) | Qwen3-32B ($0.28/M) | 97.2% |
| Translation | GPT-4o ($10/M) | Qwen-MT-Turbo ($0.30/M) | 97% |
I want to highlight that classification line because it changed my mental model permanently. We were spending $0.60 per million output tokens on GPT-4o-mini to do sentiment tagging. Sentiment tagging. Qwen3-8B at $0.01/M does it at parity. That's a 98.3% reduction on a workload that fires thousands of times per hour.
The trick is classification. You don't need a frontier model to label an email as "spam" or "not spam." You need a model that's good enough, fast enough, and cheap enough to run at scale. Once you internalize that, the rest of this article falls into place.
Implementation looks like this — note the base URL:
from openai import OpenAI
client = OpenAI(
base_url="https://global-apis.com/v1",
api_key=os.environ["GLOBAL_APIS_KEY"],
)
MODEL_MAP = {
"chat": "deepseek-v4-flash", # $0.25/M
"code": "deepseek-coder", # $0.25/M
"simple": "Qwen/Qwen3-8B", # $0.01/M
"reasoning": "deepseek-reasoner", # $2.50/M
}
def route_to_model(user_input: str) -> str:
complexity = classify_complexity(user_input)
return MODEL_MAP[complexity]
model = route_to_model(user_input)
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": user_input}],
)
That global-apis.com/v1 base URL is doing more work than it looks like. By routing through a unified API, I've already decoupled my code from any single provider. If DeepSeek raises prices next quarter, I swap a string. If a new model launches that beats Qwen3-8B on classification, I swap a string. Vendor lock-in avoidance is a feature, not a luxury, and it costs nothing extra to build in from day one.
Step 2: Tiered Routing — Pay For Intelligence Only When You Need It
Once you're picking models per task, the natural next move is tiered routing. Try the cheap model first. If the answer is good enough, return it. If not, escalate. The architecture is dead simple and the wins are enormous.
Here's the function I run in production today:
def smart_generate(prompt: str, max_budget: float = 0.50):
"""
Try cheap first. Escalate only when quality demands it.
Distribution in our traffic: ~80% tier 1, ~15% tier 2, ~5% tier 3.
"""
resp = call_model("Qwen/Qwen3-8B", prompt)
if quality_check(resp) >= 0.8:
return resp
# Tier 2: standard at $0.25/M
resp = call_model("deepseek-v4-flash", prompt)
if quality_check(resp) >= 0.9:
return resp
# Tier 3: premium at $0.78–$2.50/M
return call_model("deepseek-reasoner", prompt)
The real-world result from our customer support chatbot: monthly spend dropped from $420/month to $28/month. That's a 93% reduction. Same quality. Same SLA. The reason is that 85% of support queries are things like "where's my order" and "how do I reset my password" — pattern matches that any small model handles fine. The 5% that hit tier 3 are the gnarly multi-step reasoning queries, and they're worth paying for.
The quality_check is just an LLM-as-judge pattern. Use a small model to grade the output. If it scores above a threshold, ship it. If not, escalate. It's a few hundred microseconds of latency and pays for itself instantly.
Step 3: Cache Like You Mean It
Caching is the unsexy work that quietly buys you back 20–50% on top of everything else. A surprising amount of traffic is repetitive. FAQ lookups. Documentation questions. "What are your hours." If you're hitting the API for those every time, you're leaving money on the floor.
import hashlib
import json
import time
cache: dict = {}
def cached_chat(model: str, messages: list, ttl: int = 3600):
key = hashlib.md5(
json.dumps({"model": model, "messages": messages}).encode()
).hexdigest()
if key in cache:
entry = cache[key]
if time.time() - entry["time"] < ttl:
return entry["response"] # cache hit — $0 cost
response = client.chat.completions.create(
model=model, messages=messages
)
cache[key] = {"response": response, "time": time.time()}
return response
On our support traffic we hit cache rates between 50% and 80% for the FAQ surface. That's free money. Every cached response is one less token billed. At our scale, this single change returned roughly $1,400/month.
If you want to go further, swap the dict for Redis and add semantic caching — a vector lookup for "is this query close enough to one I've already answered." That bumps hit rates even higher. We haven't needed it yet, but it's in the backlog.
Step 4: Compress Your Prompts Before You Send Them
Long prompts are an under-discussed cost driver. Every system prompt you ship, every context block you prepend, every RAG chunk you stuff in — it all multiplies by request volume. A 2,000-token system prompt that runs 10,000 times a day is 20M input tokens per day. At $0.25/M, that's a real number.
The fix is to use a cheap model to summarize context before you ship it to the expensive one:
def compress_prompt(text: str, target_ratio: float = 0.5) -> str:
if len(text) < 500:
return text # already short, leave it
summary = call_model(
"Qwen/Qwen3-8B",
f"Summarize this in {int(len(text) * target_ratio)} chars: {text}",
)
return summary
The math from our own logs: a 2,000-token system prompt compressed to 400 tokens saves about $0.024 per request on DeepSeek V4 Flash. At 10,000 requests per day, that's $240/day — or $87,600/year — from one prompt change.
I want to pause on that number. $87,600. From compressing one prompt. That's an entire engineer's salary. This is the kind of thing that gets lost in the noise of a $10,000/month AI bill, and it's why I tell every founder I know to dig into prompt length before they dig into anything else.
Step 5: Batch The Small Stuff
For workloads where you can tolerate a few hundred milliseconds of extra latency, batching is a free 10–20%. Instead of making 100 calls to summarize 100 documents, make one call that summarizes all 100.
Before:
results = []
for doc in documents:
resp = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[{"role": "user", "content": f"Summarize: {doc}"}],
)
results.append(resp)
After:
batched_prompt = "\n\n".join(
f"[DOC {i}]\n{doc}\n[/DOC {i}]" for i, doc in enumerate(documents)
)
resp = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[{
"role": "user",
"content": (
f"Summarize each document below. Return one summary per doc, "
f"preserving the [DOC N] markers.\n\n{batched_prompt}"
),
}],
)
You collapse 100 requests into 1. You share the system overhead once. The model gets a chance to amortize attention across similar tasks, which actually improves quality in my experience. We use this for nightly report generation and backfill jobs. Anything user-facing stays synchronous; anything batch-y goes through here.
Step 6: Build The Abstraction Layer Now, Not Later
I saved the most important step for last. None of the optimizations above matter if your code is hard-wired to one provider. Vendor lock-in is the silent killer of long-term ROI. The day a vendor raises prices — and they will, every vendor does — you should be able to swap them out in an afternoon, not a quarter.
That's why everything above routes through https://global-apis.com/v1. A unified API gives me one client, one auth flow, and a menu of models from every major provider. When I want to A/B test Qwen against DeepSeek against Llama against GPT, I change one string. When I want to migrate 30% of traffic to a new model that just dropped, I change one string. When a vendor has a bad day and their uptime tanks, I failover by changing one string.
This is the architecture-decision mindset I wish more teams adopted on day one. The abstraction layer isn't overhead — it's leverage. It costs you maybe a day to set up and it pays back forever.
Step 7: Instrument Everything
You can't optimise what you can't measure. Every request should log: model, prompt tokens, completion tokens, latency, cost, cache hit (yes/no), tier. Pipe it to your warehouse. Build a dashboard. I have a Grafana panel called "cost per resolved task" and I look at it every morning like other people look at their steps counter.
Once you have the data, you'll find surprises. We discovered that 12% of our spend was going to one prompt that fired every time a user opened the app — a "welcome message" that was static. We cached it, deleted it from the API path entirely, and reclaimed roughly $300/month from one decision.
The Math, All Together
Let me stack these up the way I presented it to my board:
- Smart model selection: 90% reduction baseline
- Tiered routing on top: another 5–10% (95% total)
- Caching: additional 20–50% on cacheable workloads
- Prompt compression: 15–30% on long-context workloads
- Batching: 10–20% on async workloads
Combined, our blended reduction across all traffic sits around 95%. Our token bill went from "alarmingly large" to "a line item we don't worry about." More importantly, our cost-per-resolved-task dropped to a number that makes our unit economics work. That's the real ROI — not the percentage, but the fact that the product is now fundable.
The Mindset
If there's one thing I want you to take away, it's this: AI API costs are an engineering problem, not a procurement problem. Every optimization above is something a competent engineer can ship in a week. None of them require a new vendor contract, a new tool, or a new hire. They require the willingness to treat model selection and inference architecture with the same seriousness you'd treat database indexing or cache eviction.
Pick the right model. Route by complexity. Cache what repeats. Compress what doesn't. Batch what you can. Abstract the provider. Measure everything.
Do that, and your AI bill stops being a scary line item and starts being a competitive advantage.
If you want a fast way to test these patterns without committing to a dozen vendor accounts, take a look at Global API. I use their unified endpoint
Top comments (0)