This article was originally published at sivaro.in

How to Reduce Cost of LLM Inference in Production

Last month a Series B fintech called me in a panic. They'd shipped an AI feature in March, it worked beautifully, and by August their inference bill had crossed $220K/month against a forecast of $40K. Their CFO thought they'd been hacked. They hadn't. They'd just done what everyone does — shipped on top of a frontier API with zero cost discipline, then watched usage compound.

I've seen this movie a dozen times now. So let's talk about how to reduce cost of LLM inference in production, not as a list of tips, but as an actual buying decision you're making across four fronts: which models you run, where you run them, how you route traffic, and what you cache.

Here's the thing most teams get wrong: they treat this as an optimization problem. It's not. It's an architecture problem, and the best time to solve it was three decisions ago.

The Real Cost Curve Nobody Shows You

A token on a frontier model costs roughly 100x to 400x what the same token costs on a well-tuned open model running on your own hardware. That's not a typo. As of September 2026, GPT-5-class and Claude-class API pricing sits in the $10–15 per million output tokens range, while a quantized Llama or Qwen variant on an H100 you already own can land closer to $0.05–0.20 per million. The gap is enormous, and it doesn't go away with "we'll negotiate enterprise pricing."

But — and this is the part consultants leave out — self-hosting that open model badly can cost more than the API. I've watched a team burn $40K/month on idle GPU reservations because they overprovisioned for peak load they saw twice a week. So the decision isn't "API vs. self-host." It's a portfolio question.

Think of it as a buying guide with four levers, ranked by how much money they typically move:

1. Model selection and tiering — biggest lever, 60–90% of savings
2. Caching (prompt + semantic) — 20–60% savings, near-free to implement
3. Routing and fallback logic — 15–40% savings
4. Serving infrastructure (batching, quantization, hardware) — 30–70% savings if you're already self-hosting

Let's go through each.

Tiering Your Models Is the Single Highest-Leverage Move

Most people think you need one model for your product. You don't. You need a routing policy that sends each request to the cheapest model that can handle it correctly.

At SIVARO we built a support-ticket classifier last year that started on a frontier model at $85K/month. We moved it to a tiered setup:

• Tier 0 (60% of traffic): Regex + small fine-tuned classifier (BERT-class, ~$0.001/request)
• Tier 1 (30%): Open 8B model self-hosted (~$0.01/request)
• Tier 2 (10%): Frontier API for edge cases (~$0.30/request)

Total cost dropped to $9K/month. Same accuracy within 0.4 points. The trick wasn't a magic model — it was accepting that 60% of "AI inference" isn't AI at all.

The buying decision here: don't pick a model. Pick a ladder. For each use case, ask "what's the cheapest thing that gets me to acceptable quality," and build one tier below where you think you need to be.

from enum import Enum

class Tier(Enum):
    REGEX = 0
    SMALL = 1
    FRONTIER = 2

def route(request, complexity_score: float) -> Tier:
    if request.matches_regex():
        return Tier.REGEX
    if complexity_score < 0.7:
        return Tier.SMALL
    return Tier.FRONTIER

That's it. Complexity scores can come from token length, presence of rare entities, or a cheap classifier. You don't need a perfect scorer — you need one that's directionally right.

Caching: The Free Money You're Leaving on the Table

I'll be blunt. If you're not caching prompts and responses, you're burning cash. Anthropic, OpenAI, and Google all offer prompt caching at 50–90% discount on cached input tokens. Anthropic's prompt caching docs and OpenAI's cached input pricing both publish these discounts, and they've been stable since early 2025.

For a support bot with a 4,000-token system prompt and 200-token user turn, you flip the cost ratio. The 200-token user turn is what matters; the 4,000-token prompt is nearly free after the first hit.

But prompt caching only helps with identical prefixes. Real savings come from semantic caching — hashing-and-embedding the full request and serving near-duplicate queries from a vector store. For customer support, FAQ-style products, and code assistants, hit rates of 25–45% are common in my experience. That's a quarter of your traffic that never touches a model.

import hashlib
import numpy as np
from your_vector_db import upsert, query

def semantic_cache_lookup(user_prompt: str, threshold: float = 0.94):
    embedding = embed(user_prompt)
    hits = query(embedding, top_k=1)
    if hits and hits[0].score >= threshold:
        return hits[0].response  # cached
    return None  # miss — call the model

Tune the threshold carefully. Too loose and you serve wrong answers. Too tight and you never hit. Start at 0.95 for factual domains, 0.90 for chitchat.

One warning: invalidate aggressively. I've seen teams serve stale pricing answers for weeks because their cache never expired. Add a TTL tied to your content refresh cadence.

Self-Hosting: When It Wins, When It Wrecks You

I used to be religious about self-hosting. Then I ran the numbers on a low-traffic product and realized the API was cheaper. Self-hosting wins when you have sustained, high-volume, latency-tolerant traffic on a model that fits well on commodity GPUs.

Rough break-even math as of September 2026: an H100 rental is roughly $2–3/hr on spot, so about $1,500–2,200/month. If your workload runs at 30%+ sustained utilization on that GPU with a 70B quantized model, you beat API pricing. Below that, you don't.

The toolkit that's matured fast: vLLM for continuous batching, SGLang for structured generation, and TensorRT-LLM if you're all-NVIDIA. Continuous batching alone can 3–8x throughput versus naive serving — vLLM's own benchmarks show this clearly.

Key cost levers when self-hosting:

• Quantization (FP8, INT8, AWQ, GPTQ) — 2–4x throughput at small quality cost
• Batch size tuning — larger batches raise throughput but hurt p99 latency
• Speculative decoding — use a small draft model to accelerate a big one
• Spot instances + checkpointing — 40–60% cheaper, but you need graceful degradation

# vLLM launch with aggressive batching
vllm serve Qwen/Qwen3-32B-AWQ \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 256 \
  --quantization awq

That --max-num-seqs 256 number is where most people leave money. Default is much lower and your GPU sits idle between batches.

Routing, Fallback, and Load Shedding

If you're running across multiple providers (which you should), routing logic becomes a cost lever. The idea: monitor latency, quality, and price per provider in real time, and shift traffic to the cheapest provider that's healthy.

Caveats. Router complexity has a real maintenance cost. Every additional provider is another API contract, another SDK quirk, another auth scheme. If you're under 1M requests/day, a static routing table probably beats a dynamic router.

Load shedding is the underrated one. When your queue depth exceeds a threshold, degrade gracefully — shorter max_tokens, cheaper model tier, or a cached response. Your users would rather get a slightly worse answer in 400ms than a great answer in 12 seconds or an error.

async def call_with_shedding(req):
    if queue.depth() > 500:
        req.max_tokens = min(req.max_tokens, 256)
        req.model = "small-tier"
    return await router.dispatch(req)

Fine-Tuning vs. Prompt Engineering: A Cost Decision, Not a Quality One

Everyone frames fine-tuning as a quality play. It's mostly a cost play.

A fine-tuned 7B model can match a prompted 70B on narrow tasks. That's a 10x inference cost reduction, permanently. The upfront cost — data collection, training, eval, deployment — usually pays back inside 3–6 months for any product with steady traffic.

When it's worth it: you have >50K labeled examples, a bounded domain, and stable requirements. When it's not: your task changes every sprint, your data is thin, or quality bar is "roughly right." I've been burned fine-tuning too early twice. Don't do that.

An Honest Comparison Table

Approach	Typical Savings	Time to Ship	Best For
Model tiering	40–80%	Days	Every product with mixed request complexity
Prompt caching	20–50%	Hours	Long system prompts, repeated context
Semantic caching	15–40%	Weeks	FAQ, support, code assistants
Self-hosting (vLLM + AWQ)	30–70% vs. API	Months	Sustained high volume, latency-tolerant
Fine-tuning	60–90% (long-term)	1–3 months	Narrow, stable, data-rich tasks
Quantization	2–4x throughput	Days	Any self-hosted workload
Load shedding	Variable	Days	Anything with spiky traffic

None of these are mutually exclusive. The teams that win stack three or four.

What I'd Actually Do If I Were Starting Today

If I were rebuilding an inference-heavy product from scratch in September 2026, here's the sequence:

Week 1: Ship on a frontier API. Don't optimize prematurely. Measure real traffic patterns.

Week 2–3: Add prompt caching. Add a semantic cache in front of your most repeated endpoint. Instrument hit rates.

Month 2: Build routing tiers. Move the obvious 40–60% of traffic to a small self-hosted model.

Month 3–4: Evaluate fine-tuning for your top two use cases based on actual data.

Month 6+: Revisit self-hosting economics. If sustained utilization crosses 30%, migrate.

The order matters because each step gives you the data for the next. Teams that jump straight to self-hosting without usage data usually regret it.

Frequently Asked Questions

Is it always cheaper to self-host?
No, and this myth costs companies millions. Below 20–30% sustained GPU utilization, API wins. The crossover point moves with model size — a 7B model needs less traffic to justify self-hosting than a 70B model, but even a 7B needs steady load.

Does quantization hurt quality meaningfully?
For most classification, extraction, and RAG tasks, INT8 and FP8 quantized models perform within 1–2% of full precision. For creative writing and complex reasoning, you'll notice. Test on your specific eval set before shipping. Hugging Face's quantization docs have good starting benchmarks.

What's the fastest win for reducing inference costs?
Prompt caching. It's a config change, available on all major providers, and typically cuts 20–50% off your bill the same week. No architecture change required.

Can I use multiple providers without a router?
Yes — a static dictionary mapping task types to providers works fine up to about 1M requests/day. Dynamic routers add value at scale but also add failure modes.

How do I know if my semantic cache is helping or hurting?
Track two things: hit rate and quality-adjusted error rate on cached responses. If quality drops after adding cache, your threshold is too loose. Below 10% hit rate, the cache isn't paying for itself.

Is fine-tuning really a cost lever?
Yes, and it's underrated. A well fine-tuned 7B model replaces a prompted 70B for narrow tasks, which is a permanent 5–10x inference cost reduction. The catch is that it requires stable requirements and real labeled data.

What about smaller open models like 3B and 1B?
For high-volume classification, extraction, and routing decisions, 1B–3B models are shockingly good now. Qwen3-4B and Llama-3.2-3B handle a lot of "AI" traffic that teams are currently paying frontier prices for.

Should I care about batch APIs?
If your workload is async (nightly jobs, batch scoring, content generation pipelines), batch APIs are often 50% cheaper than real-time endpoints. Free money if latency doesn't matter.

The Thing Nobody Says Out Loud

Most inference cost problems aren't technical. They're organizational. The team that shipped on GPT-5 in March didn't ask "what's the cheapest way to do this" because nobody owned the question. The CFO saw the bill. The engineers saw a green checkmark in the demo. Nobody was measuring cost-per-request-per-user-segment.

So before you rewrite your serving stack, put a number on the dashboard. Cost per 1,000 requests, broken out by endpoint. That single metric changes behavior faster than any blog post, including this one.

Here's what I've learned after eight years of building production AI systems: how to reduce cost of LLM inference in production is 20% engineering, 80% deciding that it matters. The techniques — tiering, caching, routing, quantization, fine-tuning — are all well-documented and cheap to adopt. The teams that struggle are the ones that never made it a first-class metric.

Start with the metric. Then start with caching. Then tier. Everything else follows.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.