Our LLM feature worked.
That was the problem.
It worked well enough that usage kept growing, nobody wanted to turn it off, and the inference bill quietly became an engineering problem of its own.
The first reaction was predictable:
Should we switch to a cheaper model?
We eventually did use smaller models for some requests.
But that wasn't where we started.
We started with measurement.
And once we measured where tokens and model calls were actually going, we discovered that our problem wasn't simply an expensive model.
We were paying for:
- repeated context
- oversized prompts
- unnecessary output
- simple tasks running on expensive models
- duplicate requests
- workloads that could have been batched
- LLM calls that didn't need an LLM at all
So instead of making one dramatic architecture change, we optimized the inference pipeline one step at a time.
This post walks through that process.
Important: the numbers below are an illustrative benchmark showing the measurement methodology and compounding effect of the changes. They are not presented as telemetry from a specific undisclosed production system. If you apply this process, measure against your own workload.
Our example baseline:
1,000,000 requests / month
Average input: 2,400 tokens
Average output: 420 tokens
Normalized monthly inference cost:
$10,000
After all optimizations:
$10,000 → ~$3,000
≈ 70% lower inference cost
The interesting part isn't the final percentage.
It's where the money disappeared.
First Rule: Don't Optimize $/Token
A lot of LLM cost discussions begin with provider pricing.
Model A:
$X / 1M input tokens
$Y / 1M output tokens
Model B:
$A / 1M input tokens
$B / 1M output tokens
That comparison matters.
But it doesn't tell us the actual cost of our feature.
The number we really care about is closer to:
Cost per successful task
not:
Cost per token
Imagine Model A costs half as much as Model B but requires more retries or produces outputs that fail our quality requirements.
The cheaper token may create a more expensive task.
So our optimization target became:
Total inference spend
Cost/task = ─────────────────────────────
Successfully completed tasks
And every optimization had to satisfy two conditions:
Cost ↓
Quality ≈
If quality dropped significantly, we didn't count it as a successful optimization.
Step 0: Build a Cost Baseline
Before changing anything, instrument the pipeline.
At minimum, log something like:
{
"request_id": "...",
"feature": "support_assistant",
"model": "...",
"input_tokens": 2381,
"output_tokens": 417,
"cached_tokens": 0,
"latency_ms": 1840,
"estimated_cost": 0.0124,
"success": true
}
Then aggregate by:
model
feature
endpoint
customer
request type
prompt version
That last one is surprisingly useful.
If you change a prompt and costs suddenly increase 20%, you want to know exactly which version caused it.
Our conceptual dashboard became:
┌───────────────────────────────────┐
│ LLM COST DASHBOARD │
├───────────────────────────────────┤
│ Requests │
│ Input Tokens │
│ Output Tokens │
│ Cached Tokens │
│ Cost / Request │
│ Cost / Successful Task │
│ P50 / P95 Latency │
│ Quality Score │
│ Cache Hit Rate │
│ Model Distribution │
└───────────────────────────────────┘
Without this baseline, a statement like:
“Prompt caching saved us money.”
isn't an engineering result.
It's an assumption.
Change #1: Stop Sending Tokens Nobody Needs
Illustrative cost:
Before: $10,000
After: $8,900
Savings: 11%
Our first target wasn't the model.
It was the prompt.
Production prompts have a tendency to grow.
They start like this:
You are a customer support assistant...
Then somebody adds formatting rules.
Then policy.
Then examples.
Then tool instructions.
Then edge cases.
Then documentation.
Eventually every request may carry thousands of tokens of instructions and context.
The first question we asked was:
Does every request actually need every token we're sending?
Usually the answer is no.
Remove Redundant Instructions
A prompt can easily contain rules that say effectively the same thing multiple times.
Instead of:
Always answer concisely.
Do not produce unnecessarily long answers.
Keep your response brief.
Avoid excessive explanations.
we can say:
Answer concisely unless the user requests detail.
Small change?
Yes.
But multiply it across millions of requests.
Token efficiency is an infrastructure concern at scale.
Don't Send Full Conversation History
This was another easy source of waste.
Naive implementation:
messages = entire_conversation_history
Better:
messages = relevant_conversation_context
For long conversations, summarize older context or retain only turns relevant to the current request.
Conceptually:
20-message conversation
↓
Relevance / summarization
↓
6 useful messages
↓
LLM
This isn't just cheaper.
It can also reduce distraction from stale context.
Don't Retrieve More RAG Context Than You Need
RAG systems can accidentally become token machines.
documents = retrieve(query, top_k=20)
followed by:
context = "\n".join(documents)
may send huge amounts of partially relevant text to the model.
We prefer a pipeline closer to:
Retrieve candidates
↓
Rerank
↓
Select answer-bearing chunks
↓
Build compact context
↓
Generate
The goal isn't maximum context.
It's sufficient evidence.
Change #2: Control Output Length
Illustrative cumulative cost:
Original: $10,000
Previous: $8,900
Now: $7,950
Total reduction: ~20.5%
Input optimization gets a lot of attention.
Output tokens deserve just as much scrutiny.
If a task requires:
{
"category": "billing",
"priority": "high"
}
don't allow the model to generate a five-paragraph explanation.
For structured tasks, constrain the output.
Return only:
{
"category": string,
"priority": string
}
For summaries:
Maximum 5 bullets.
For extraction:
Return JSON only.
For routing:
Return one label.
The cheapest token is still the one you don't generate.
Change #3: Cache Repeated Work
Illustrative cumulative cost:
Original: $10,000
Previous: $7,950
Now: $6,850
Total reduction: ~31.5%
Then we looked at repetition.
Our requests weren't as unique as they appeared.
Many shared the same:
- system prompt
- tool definitions
- policies
- schemas
- documentation
- examples
That creates opportunities for prompt or prefix caching where supported.
Think about a request like:
[8,000-token shared system context]
+
[150-token user query]
Without caching, the large shared prefix may repeatedly be processed.
With caching:
First request:
Process prefix + query
Later compatible requests:
Reuse cached prefix + process new query
That can change the economics substantially for applications with large stable prefixes.
Application-Level Caching
We also looked above the model API.
Some requests were effectively duplicates.
For deterministic or low-volatility tasks:
key = hash(normalize(request))
if cache.exists(key):
return cache.get(key)
result = call_llm(request)
cache.set(key, result, ttl=...)
return result
This works well for appropriate workloads such as repeated classification or transformations where the underlying answer isn't expected to change.
But cache carefully.
You need to think about:
user-specific data
permissions
freshness
model version
prompt version
knowledge-base version
A cached incorrect or unauthorized answer is worse than an expensive one.
Change #4: Stop Using the Best Model for Every Request
This was the largest architectural change.
Illustrative cumulative cost:
Original: $10,000
Previous: $6,850
Now: $4,500
Total reduction: 55%
Originally the routing strategy was essentially:
Request
↓
Best Model
↓
Response
That was simple.
It was also wasteful.
Our application had tasks ranging from:
Classify this ticket
to:
Analyze these documents and identify
contradictory contractual obligations.
Those are not equivalent workloads.
Why should they use the same model?
Route by Complexity
We moved toward:
┌── Simple ──► Small Model
│
Request ─► Router ├── Medium ──► Mid-Tier Model
│
└── Complex ─► Strong Model
For example:
def choose_model(task):
if task.type in {
"classification",
"intent_detection",
"simple_extraction"
}:
return SMALL_MODEL
if task.complexity < COMPLEXITY_THRESHOLD:
return MID_MODEL
return STRONG_MODEL
Of course, production routing shouldn't rely on vibes.
We created evaluation sets for each task category and asked:
What is the cheapest model that still clears our quality threshold?
That is the important question.
Not:
What's the cheapest model?
And not:
What's the smartest model?
But:
What's the cheapest model that reliably solves this task?
This model-agnostic, evaluation-driven approach is also useful when designing broader large language model development architectures: model selection should be tied to the actual workload rather than assuming every request needs maximum capability.
Change #5: Escalate Instead of Starting Expensive
Model routing became even more useful when we added escalation.
Instead of:
Request
↓
Expensive Model
we could use:
Request
↓
Smaller Model
↓
Validation
↙ ↘
PASS FAIL
↓ ↓
Return Stronger Model
This is particularly useful when outputs can be validated.
Imagine a structured extraction task.
result = small_model.extract(document)
if schema_valid(result) and confidence_ok(result):
return result
return strong_model.extract(document)
Now the expensive model handles only the difficult tail of requests.
The important part is the validator.
Without reliable evaluation or validation, routing can quietly turn cost savings into quality regression.
Change #6: Remove LLM Calls That Should Have Been Code
This was one of my favorite findings.
We had tasks that looked vaguely “AI-like,” so they had been implemented using the LLM.
But some didn't need probabilistic generation at all.
Example:
Convert a date into ISO format.
Do we need an LLM?
Probably not.
datetime.strptime(...)
may be enough.
Other examples:
Validate JSON
→ JSON schema validator
Calculate price
→ application logic
Check permissions
→ authorization service
Look up exact ID
→ database query
Simple regex extraction
→ regex/parser
Our rule became:
Use an LLM when the problem requires language understanding or generation — not merely because an LLM can solve it.
Every avoided inference call has a very attractive price:
$0
Change #7: Batch Anything That Doesn't Need Immediate Results
Illustrative cumulative cost:
Original: $10,000
Previous: $4,500
Now: $3,750
Total reduction: 62.5%
Not every workload is interactive.
Users expect immediate responses from:
chat
search assistants
copilots
But other jobs can wait:
nightly classification
document enrichment
offline summaries
evaluation runs
bulk extraction
analytics pipelines
Those workloads are good candidates for batching.
Instead of:
request
request
request
request
request
independently, the serving layer can process compatible workloads more efficiently.
For self-hosted inference, batching can increase accelerator utilization.
For API-based workloads, providers may also offer batch-oriented pricing or processing modes.
The important distinction is:
Interactive → optimize latency
Offline → optimize throughput + cost
Don't pay interactive-serving economics for a job nobody needs until tomorrow morning.
Change #8: Make the Router Cache-Aware
Once caching and routing both existed, we found an interesting interaction.
Imagine:
Request A → Server 1
Request B → Server 2
Request C → Server 3
If all three requests share a large prefix, spreading them across workers can reduce cache reuse.
A better architecture for some self-hosted workloads can be:
Incoming Request
↓
Prefix / Cache-Aware Router
↓
Worker likely to have reusable KV state
↓
Inference
This is where cost optimization stops being only an application concern.
At scale, routing, scheduling, KV-cache reuse, and serving infrastructure begin interacting.
Change #9: Quantization — But Only After Measurement
If you're using hosted APIs, this may not be under your control.
If you're self-hosting, quantization becomes another lever.
Conceptually:
FP16/BF16
↓
FP8 / INT8 / INT4
depending on model, hardware, serving stack, and acceptable quality.
Lower precision can reduce memory requirements and potentially improve throughput.
But:
Lower precision ≠ automatic free performance
You need to benchmark:
quality
tokens/sec
TTFT
memory
throughput
cost/request
Research on inference efficiency has shown that precision, batching, and workload characteristics interact; a configuration that saves memory isn't automatically the configuration that produces the best end-to-end efficiency.
So we treat quantization as an experiment.
Not a checkbox.
Change #10: Add Budget-Aware Generation
Illustrative final cost:
Original: $10,000
Previous: $3,750
Final: ~$3,000
Total reduction: ~70%
The final optimization wasn't another model.
It was making cost visible to the application.
Instead of letting every request consume arbitrary resources, each task could have a rough budget.
budget = CostBudget(
max_input_tokens=4000,
max_output_tokens=500,
preferred_model="small",
allow_escalation=True
)
Different features can have different budgets.
Intent classification
Budget: tiny
Support response
Budget: medium
Complex document analysis
Budget: high
This turns cost from:
Something finance discovers later
into:
An architectural constraint
which is where it belongs.
The Full Measurement
Here's the illustrative progression.
| Change | Monthly Cost | Reduction vs. Baseline |
|---|---|---|
| Baseline | $10,000 | 0% |
| Prompt/context cleanup | $8,900 | 11% |
| Output control | $7,950 | 20.5% |
| Caching | $6,850 | 31.5% |
| Model routing | $4,500 | 55% |
| Batching | $3,750 | 62.5% |
| Budget + remaining optimizations | ~$3,000 | ~70% |
Notice something important.
We did not calculate:
11% + 10% + 15% + 35% + ...
Savings compound against a changing baseline.
That's why every change needs to be measured independently.
Our Final Architecture
The initial system looked roughly like:
User
↓
Prompt
↓
Large Model
↓
Response
The optimized design is closer to:
┌─────────────────┐
Request ────────►│ Cost / Task Log │
└────────┬────────┘
↓
┌─────────────────┐
│ Need an LLM? │
└────┬───────┬────┘
│ │
NO YES
│ │
Code ↓
│ Cache Check
│ ↓
│ Context Builder
│ ↓
│ Complexity Router
│ ↓
│ Cheapest Viable Model
│ ↓
│ Validate
│ ↙ ↘
│ PASS FAIL
│ ↓ ↓
│ Return Escalate
│ ↓
└──────────► Response
For offline workloads, add:
Batch Queue
For self-hosted systems, the serving layer can additionally contain:
Quantization
Continuous Batching
KV Cache
Prefix-Aware Routing
GPU Scheduling
This is why reducing LLM cost isn't really a “pick a cheaper model” problem.
It's systems engineering.
Quality Was a Hard Constraint
There is a dangerous way to achieve a 70% cost reduction:
Make the product 70% worse.
That's easy.
We wanted:
Cost ↓↓↓
Quality ───
not:
Cost ↓↓↓
Quality ↓↓↓
So every experiment went through the same framework.
Candidate Change
↓
Offline Evaluation
↙ ↘
FAIL PASS
↓ ↓
Reject Shadow Test
↓
Canary
↓
Production
↓
Cost + Quality
Monitoring
Metrics varied by feature.
For structured extraction:
field accuracy
schema validity
missing-field rate
For RAG:
retrieval quality
answer correctness
groundedness
citation support
For support:
resolution quality
escalation rate
user feedback
For agents:
task completion
tool-call accuracy
steps per task
cost per successful task
This is also why cost controls should be connected to evaluations rather than applied independently. A production-oriented generative AI consulting and architecture process should treat model choice, evaluations, monitoring, and cost controls as related system-design decisions rather than isolated optimizations.
The Metric That Changed How We Thought About Cost
Initially we monitored:
Total monthly LLM spend
Useful, but too broad.
Then:
Cost / request
Better.
Eventually the metric we cared about most became:
Cost / successful task
Consider:
Model A
Cost/request: $0.004
Success rate: 70%
versus:
Model B
Cost/request: $0.006
Success rate: 98%
Looking only at request cost makes Model A look cheaper.
But retries, fallbacks, failures, and human intervention can completely change the economics.
Optimization needs to follow the unit that creates product value.
What Didn't Work
Not every experiment saved money.
Blindly shrinking prompts
At some point, removing context starts removing information the model actually needs.
Token reduction has diminishing returns.
Sending everything to a tiny model
Great benchmark price.
Bad production strategy if the workload exceeds the model's capabilities.
Aggressive semantic caching
Near-duplicate questions aren't necessarily equivalent questions.
This becomes particularly dangerous with:
financial data
user-specific information
permissions
time-sensitive answers
Huge top_k reduction in RAG
Lower token count looked great until retrieval coverage dropped.
Quantization without workload benchmarks
Lower precision doesn't automatically produce the best cost/latency/quality combination.
Every optimization has a boundary.
The job is finding it.
If I Had to Do It Again
I'd optimize in roughly this order:
1. Instrument everything
Know:
tokens
requests
models
latency
quality
cost/task
2. Remove obvious waste
Shorten redundant prompts, unnecessary context, and excessive output.
3. Eliminate unnecessary LLM calls
Use normal software where normal software is better.
4. Cache safe repetition
Start with repeated static context and clearly cacheable workloads.
5. Route by task difficulty
Use the cheapest model that passes your own evaluations.
6. Batch asynchronous work
Separate latency-sensitive and throughput-sensitive workloads.
7. Optimize serving
For self-hosted models:
quantization
continuous batching
KV caching
prefix-aware scheduling
hardware utilization
8. Re-measure
Because the bottleneck after optimization probably isn't the bottleneck you started with.
The Bigger Lesson
When we first looked for ways to reduce LLM cost, we expected to find one expensive component.
We didn't.
We found dozens of small decisions multiplying together.
An unnecessary 500 tokens here.
An expensive model handling an easy task there.
A cache miss.
An overly long response.
A duplicate request.
An offline job processed synchronously.
One mistake isn't catastrophic.
At scale:
Small waste
×
millions of requests
=
large bill
The reverse is also true.
Prompt efficiency
×
caching
×
routing
×
batching
×
serving efficiency
=
very different economics
That is why the most important part of our optimization process wasn't any individual technique.
It was this:
Change one thing. Measure cost. Measure quality. Keep it only if both numbers make sense.
A 70% reduction shouldn't come from a clever trick.
It should be the cumulative result of many boring, measurable improvements.
And those are usually the optimizations you can trust in production.
Top comments (0)