This article was originally published at sivaro.in
What Is Cost Efficient Architecture for AI Systems
Slug: what-is-cost-efficient-architecture-for-ai-systems
Last month a Series B founder showed me his inference bill. $340K in August 2026. For a product doing maybe 40 million requests a month. My first thought wasn't "your prompts are too long." It was: you built a Ferrari engine and bolted it to a shopping cart.
I've been building production AI systems since 2018. At SIVARO we've run workloads at 200K events/sec, shipped ML serving across three continents, and watched dozens of teams torch their gross margins on architecture they didn't need. So let me be blunt about what is cost efficient architecture for AI systems: it's not the cheapest stack. It's the architecture where every dollar of compute maps to a unit of user value you can actually measure.
Most teams think cost efficiency is a pricing problem. They're wrong. It's a design problem. You can't negotiate your way out of a bad architecture — I watched a company in 2025 cut their OpenAI bill by 40% and then lose it all again when QPS jumped 3x, because nothing autoscaled and everything was synchronous.
This guide is a buying guide. I'm going to compare the real architectural decisions — inference serving, training, vector storage, orchestration — the way I'd walk a peer through a purchase. You'll learn what actually drives cost, where the cheap wins hide, and how to implement autoscaling for cost efficient ML serving without breaking p99 latencies.
Let's get into it.
The Four Layers That Decide Your Bill
Every AI system has four cost centers. Miss one and your savings somewhere else evaporate.
Training and fine-tuning. Spiky, expensive, and usually not where the bleeding is — unless you're a foundation lab.
Inference. This is the killer. In 2026, most teams spend 60–85% of their AI budget here. It's the rent you pay every second.
Data and retrieval. Vector DBs, feature stores, embedding pipelines. They look cheap until you're storing 800M embeddings and paying for RAM you never touch.
Orchestration and control plane. Gateways, routers, observability. Often 5–10% of the bill — and often the thing that saves you 30% by routing smartly.
Here's the contrarian take: most people optimize the wrong layer. They obsess over model choice and ignore the serving topology. We tested this at a client in early 2026. Swapping from a fine-tuned 70B to a 8B model saved them 18%. Restructuring their serving layer for continuous batching and autoscaling saved them 54% off the same spend. The topology beat the model.
Inference Serving: Where Buying Guides Usually Lie
When you're evaluating inference options, most people hand you a table: serverless vs. dedicated vs. self-hosted. That table is useless without the numbers behind it. So here's the version with actual numbers.
Serverless Inference (Bedrock, Vertex, Together, Fireworks)
Pay per token. Zero ops.
Great for: spiky traffic, prototypes, low volume, hard-to-forecast load. A startup doing 2M tokens/day for a chatbot should be here. Full stop.
Bad for: sustained high volume. Token pricing has a markup measured in multiples, not percentages. At 500M tokens/day, you're paying for someone else's GPU utilization — a markup that compounds faster than any discount you'll ever negotiate.
Dedicated Endpoints (SageMaker, Vertex, Bedrock Provisioned)
You buy GPU-hours. Predictable cost, fixed capacity.
Great for: steady traffic, latency-sensitive workloads, compliance constraints.
The trap: you provision for peak and pay for it 24/7. If your peak is 4x your median, you're burning three-quarters of your money on idle silicon. This is the single most common waste I see in 2026 — teams pull a number from a load test, provision it, and never revisit.
Self-Hosted (vLLM, TGI, SGLang, TensorRT-LLM)
You rent GPUs (or buy them) and run the serving engine. Maximum control, maximum ops burden.
Great for: teams with real MLOps chops and steady, high-volume traffic. The break-even against dedicated endpoints usually lands somewhere around 40–60% sustained GPU utilization — below that, you're paying for complexity you don't use.
The cost efficient ML serving truth in 2026: utilization is the metric. Not price per GPU-hour. Not tokens per second. GPUs sitting at 12% utilization are the most expensive compute on earth, whether you rent them from AWS at $2.40/hr or bought them outright.
The Serving Engine Comparison
If you're self-hosting, the engine matters more than the GPU generation:
| Engine | Strength | Where It Costs You |
|---|---|---|
| vLLM | PagedAttention, huge ecosystem, continuous batching | Memory-hungry at long context |
| SGLang | RadixAttention, best prefix-cache reuse | Smaller ecosystem, more setup work |
| TGI | Hugging Face integration, mature | Slower on aggressive batching |
| TensorRT-LLM | Fastest per-token on NVIDIA | Compilation time, NVIDIA lock-in |
If you have heavy shared system prompts — and most RAG apps do — SGLang's radix cache will beat vLLM on cost by 15–30% in my experience. We measured this on a support agent with a 2K-token system prompt and saw TTFT drop 40%.
How to Implement Autoscaling for Cost Efficient ML Serving
This is the section most guides skip because it's hard. Autoscaling inference is not autoscaling a web server. Requests take seconds to complete, GPUs take minutes to warm, and cold-starting a 70B model can take 3–6 minutes if it's not pre-cached.
If you copy a standard HPA-on-CPU config, you'll thrash. I've seen it. Pods spin up, don't finish loading before the next scaling event, and you pay for 8 GPUs that serve zero requests.
Here's what actually works.
Scale on queue depth, not CPU
CPU utilization on a GPU pod tells you almost nothing. Queue depth and time-to-first-token are what your users feel.
# KEDA ScaledObject — scale on vLLM queue depth via Prometheus
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-inference-scaler
namespace: llm-serving
spec:
scaleTargetRef:
name: vllm-deployment
minReplicaCount: 2
maxReplicaCount: 12
cooldownPeriod: 300
pollingInterval: 15
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
metricName: vllm_num_requests_waiting
query: |
avg(vllm:num_requests_waiting{namespace="llm-serving"})
threshold: "8"
Eight waiting requests per replica is a starting threshold. We tune it per workload — for a chat product with p95 latency SLO of 3s, we often sit around 4–6. For batch summarization, 20+ is fine.
Pre-warm with model caching
If you're on Kubernetes, bake the model into a node-local cache or use a sidecar that keeps weights hot on NVMe. Pulling 140GB from S3 on every scale-up is a money bonfire.
Use predictive scaling for known patterns
Support bots spike every weekday at 9am local. Don't wait for KEDA to react — pre-scale at 8:50am with a cron trigger. We cut cold-start waste 60% on one client by layering a scheduled pre-scale on top of the reactive one.
Cap the ceiling honestly
Every scaling config needs a max. Teams resist this because they're scared of throttling. But an uncapped scaler on a runaway loop in 2025 cost one company $90K in six hours. Set the ceiling. Alert when you hit it.
The Routing Layer Nobody Budgets For
Here's where a small investment saves a fortune. A model router in front of your inference tier — think LiteLLM, OpenRouter, or a custom gateway like what we build at SIVARO — lets you send each request to the cheapest model that can handle it.
Not every query needs GPT-5-class reasoning. A password reset question doesn't. Route by complexity:
# Minimal complexity-based routing with a classifier
from functools import lru_cache
def estimate_complexity(prompt: str, history_len: int) -> str:
tokens = len(prompt.split())
if tokens < 30 and history_len < 3 and "?" in prompt:
return "small" # 8B class — ~10x cheaper
if tokens < 300:
return "medium" # 30B class
return "large" # frontier model
ROUTE_MAP = {
"small": "llama-3.1-8b-instruct",
"medium": "qwen-2.5-32b-instruct",
"large": "claude-sonnet-4-5",
}
@lru_cache(maxsize=2048)
def cached_route(prompt_hash: str) -> str:
return prompt_hash
In practice, 60–70% of production traffic hits the small bucket. On one deployment, this alone cut inference spend 47% with no measurable quality drop on user CSAT.
But — and this is the honest trade-off — routing adds a hop of latency and complexity. If your product is a latency-obsessed real-time app, sometimes you eat the cost. Don't route everything for its own sake.
Data and Retrieval: The RAM Tax
Vector databases are the quiet bleed. Everyone underestimates this.
A managed vector DB like Pinecone at scale charges you for storage and for RAM-resident indexes. If you've got 200M embeddings and you're not actually querying 80% of them frequently, you're paying memory rent on cold data.
Real comparison for 100M embeddings of 1536-dim vectors:
- Pinecone serverless: ~$50–70/mo baseline but query-based pricing scales hard at high QPS
- pgvector on Postgres: flat infra cost, great up to ~50M vectors, then index build and query latency get ugly
- Qdrant self-hosted: best cost-per-query at scale, real ops cost
- Turbopuffer: S3-backed, cheap storage, high latency on cold queries
My take: if you're under 20M vectors and already run Postgres, use pgvector. Don't buy a separate vector DB. You'll save $2–5K/mo and avoid an entire category of sync bugs.
If you're over 100M vectors and QPS is high, self-hosted Qdrant or Turbopuffer wins on raw cost — but budget for a real engineer.
Orchestration and MLOps: The Cost Efficiency Multiplier
This is the layer where cost efficient MLOps practices live or die. Orchestration doesn't usually generate revenue. But it's the difference between a 30% margin and a 70% margin.
What actually matters
Request batching at the gateway. Layer concurrency control so the backend can batch effectively. Continuous batching gets you 3–8x throughput per GPU when configured right.
Caching, aggressively. Semantic cache hits should be free-ish. We've seen 30% cache hit rates on enterprise support workloads — that's 30% of your spend returned with zero quality cost. Redis + embedding similarity is enough for most teams.
# Semantic cache with Redis (simplified)
import numpy as np, redis, json
from sentence_transformers import SentenceTransformer
r = redis.Redis()
embed = SentenceTransformer("all-MiniLM-L6-v2")
def semantic_query(prompt: str, threshold: float = 0.92):
vec = embed.encode(prompt).astype(np.float32).tobytes()
# In production use RediSearch vector index, not KEYS scan
for key in r.scan_iter("cache:*", count=200):
cached = json.loads(r.get(key))
if cosine(vec, cached["vec"]) > threshold:
return cached["response"]
return None
Observability that costs, not just reports. Trace every request's token count and route. If you can't attribute cost per customer, you can't price your product correctly. Tools: Helicone, Langfuse, or your own OpenTelemetry pipeline.
Smaller, distilled models where they fit. Distillation in 2026 is a solved recipe. Losing 3% accuracy for a 90% cost reduction is usually the right trade.
What sounds good but doesn't pay
Full-featured MLOps platforms you're not using. I've watched companies pay $80K/year for a platform where they use 12% of the features. Buy the piece you need.
The Comparison Table You Actually Need
| Scenario | Recommended Serving | Est. Monthly Cost (50M req) | Notes |
|---|---|---|---|
| Early stage, spiky traffic | Serverless (Fireworks/Together) | $4–12K | Overpay, but zero ops |
| Steady 100+ QPS, cost-sensitive | Self-hosted vLLM on L40S | $8–15K | Break-even at ~50% util |
| Latency-critical, predictable | Dedicated SageMaker endpoints | $12–20K | Pay for stability |
| Heavy shared prompt / RAG | Self-hosted SGLang | $6–12K | Prefix cache pays off |
| Highly regulated, on-prem | Bare-metal + TensorRT-LLM | $20K+ amortized | Compliance premium |
Numbers are directional. Your mileage depends on context length, model size, batch patterns, and — mostly — utilization.
Common Architecture Traps I Keep Seeing
The "one big model for everything" trap. You don't need Claude Sonnet for a sentiment classifier. You need a 400M parameter model and a Tuesday.
The "provision for peak" trap. If peak is 4x median, autoscale the replicas, don't buy for peak. Save 40%+.
The "managed everything" trap. Managed is worth it until it isn't. The crossover usually lands around 20–30K requests/hour. Past that, self-hosted pays for itself in 4–6 months.
The "no cache" trap. Every AI system I've audited since 2024 has an obvious cache it isn't using. Every single one.
The "one provider" trap. Multi-provider isn't about resilience. It's about arbitrage. Bedrock and Vertex have material price differences on the same open models. Abstract them.
FAQ
What is cost efficient architecture for AI systems in one sentence?
It's a system design where every layer — training, serving, retrieval, orchestration — is sized to actual measured demand, not peak guesses or vendor defaults.
How much can I realistically save?
Across about 30 audits we've run since 2024, the median first-year saving is 45%. The lowest was 12% (already efficient team). The highest was 78% (peak-provisioned, zero caching, single-provider).
Is serverless ever the wrong choice?
Yes — above roughly 200–300M tokens/day, serverless markups compound past the point where a single GPU-hosted replica would be cheaper, even with ops overhead included.
Do I need a vector database?
Not if you're under ~20M vectors and already run Postgres. Use pgvector. Buy a dedicated vector DB when query patterns or scale genuinely demand it.
How do I start implementing autoscaling for cost efficient ML serving?
Start with KEDA on queue-depth or TTFT metrics. Don't touch HPA on CPU. Set a max replica cap on day one and alert when you hit it.
Does quantization help?
Almost always yes on cost, sometimes on quality. INT8 usually costs <1% quality. INT4 varies wildly by model — benchmark on your eval set, not a leaderboard.
What's the single highest-ROI change?
Semantic caching plus complexity-based routing. Between them, we typically see 35–55% inference cost reduction with no user-visible quality change.
How often should I re-audit?
Quarterly, minimum. Model prices change monthly in 2026. An architecture that was optimal in January is often leaving 15%+ on the table by June.
What I'd Actually Do If I Were You
Start with observability. You cannot fix what you can't see. Get per-request cost attribution in place before you touch a single GPU.
Then add caching. Then routing. Then autoscaling on real metrics. Then revisit your vector layer. In that order. The compounding effect is what turns a failing margin into a healthy one.
At SIVARO, we've rebuilt serving layers for teams burning $400K/mo down to $90K — same traffic, same SLOs, same product. The trick is never one silver bullet. It's a hundred small, boring decisions that together define what is cost efficient architecture for AI systems: demand-matched, measurable, and honest about trade-offs.
Don't chase the cheapest infra. Chase maximum utilization at the demand you actually have. That's the whole game.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.
Top comments (0)