TL;DR — Serving an open-weight model can cost 5-8x more or less depending entirely on the runtime, not the weights. This episode breaks down the four cost levers — quantization, batching, speculative decoding, prefix caching — and gives a concrete framework for deciding when self-hosting actually beats an API.
Every open-weight model release gets judged on benchmarks. Almost none get judged on the thing that actually determines whether a team can afford to run it: the serving stack underneath it. The same 70B model, on the same GPU, can cost eight times more per request depending entirely on how you configure the runtime. That gap is not a rounding error. It is the difference between a side project and a production line item, and it has nothing to do with which model card you picked.
Two numbers, one diagnosis
Inference has two distinct phases with opposite bottlenecks. Prefill — reading the prompt — is parallel and compute-bound. Decode — generating tokens one at a time — is sequential and bound by memory bandwidth, because every step has to stream the model weights and the growing KV cache off GPU memory. Before touching any optimisation, instrument time-to-first-token and inter-token latency separately. If your GPU utilisation is low and inter-token latency is fine, you have a batching problem. If time-to-first-token dominates, you have a prefill or prefix-caching problem. Guessing which lever to pull without this split is how teams spend a month on speculative decoding when the real fix was batch size, a diagnosis laid out well in a recent field guide on serving cost.
The four levers, and what they actually buy you
Quantization shrinks weights and the KV cache — the two things you stream on every decode step. FP16 to FP8 roughly halves weight memory and is now near-lossless on Hopper-class GPUs and newer; INT4 via AWQ or GPTQ quarters it, at a real but usually small quality cost. Crucially, KV-cache quantization is the lever most teams skip: storing keys and values in INT8 instead of FP16 roughly halves cache memory, and combined with PagedAttention-style memory management the effective KV footprint reduction lands in the 4–8x range, per the same serving cost analysis. Continuous batching and prefix caching are the free levers — they don't touch model quality, they just stop you wasting GPU memory and recompute. Speculative decoding sits in between: it's quality-neutral by design but adds engineering complexity. Stack all four and the reported result, across more than one source, is roughly 5–8x better cost-efficiency than a naive FP16-plus-static-batching baseline on an H100-class card, with more aggressive INT4-plus-prefix-caching stacks cutting cost 60–80% against that same baseline.
A production deployment writeup from Tensoria makes the AWQ case concretely: 4-bit weight quantization drops VRAM use 60–70% versus BF16 while keeping quality degradation under 2% on general benchmarks — and on narrow tasks like extraction or classification, the gap is often undetectable. That's the one lever that can hurt output quality, so it's the one that needs an eval gate before you ship it, not the others.
Self-host or API: the actual decision
This is the question every team eventually asks, and the honest answer is a volume threshold, not a philosophy. Analysis from Prompt20's inference cost guide puts the API-to-self-hosted crossover at roughly $5–10M/year in inference spend for a typical SaaS workload — below that, API simplicity wins; above it, self-hosting starts paying for itself. A more granular breakdown from Tensoria's field notes puts the practical break-even at 50–200M tokens per month, with the range depending heavily on which API you're replacing and which open-weight model clears your quality bar. The arithmetic: at $3 per million tokens (a typical mid-range API price), 100M tokens/month costs $300, while a single A100 80GB cloud instance runs $1,500–2,500/month — so you need 500–833M tokens/month before that GPU pays for itself, and the math shifts hard against self-hosting if the API you're replacing is already a cheap $0.50/M-token small model.
Concrete numbers help more than rules of thumb. Tensoria reports an L40S running Llama 3 8B at 10,000 requests/day landing at $0.15–0.35 per million tokens effective rate — 10–20x cheaper than a frontier API — but that comparison only holds once the fixed $900–1,500/month GPU cost is amortised across sustained volume. At 1,000 requests/day, the API wins comfortably; the fixed cost doesn't get spread thin enough. A related framing from the Honest Field Guide to Production Inference introduces "loaded cost per accepted result" as the real unit to track, and reports serverless open-weight deployment running 5–10x cheaper at total cost for non-reasoning workloads, within 5–15% quality of a closed API — a middle tier worth knowing about before committing to owned GPUs.
Use cases that clear self-hosting fast: high-volume classification or extraction pipelines with predictable, repeated prompt structure (prefix caching does most of the work); internal tools with steady 24/7 traffic where utilisation stays above 40%; and any workload where a smaller open-weight model already meets the quality bar, because self-hosting a frontier-sized model rarely beats API pricing on anything but scale. Use cases that should stay on an API: spiky or unpredictable traffic, low absolute volume, or tasks that genuinely need frontier reasoning depth — where a hard query can emit 5,000 thinking tokens against 200 for a normal chat turn, a 25x per-request cost swing that self-hosted infrastructure sized for average load will not absorb gracefully.
The pattern that actually ships
The deployments that hold up in practice are rarely all-in on one side. Tensoria describes a tiered pattern that recurs across teams: a self-hosted smaller model handles 80–90% of requests — routine extraction, classification, routing — running continuously on dedicated GPU, while a frontier API is called on demand for the harder cases that need real reasoning. That architecture is reported to cut total inference cost 60–70% versus routing everything through a frontier API, while keeping quality where it's actually needed. It also sidesteps the biggest planning mistake called out in the Prompt20 analysis: sizing for average load instead of P95 traffic. Self-hosted capacity that looks cheap on a spreadsheet built from mean throughput turns expensive fast the first time real traffic spikes and there's no elastic API fallback to absorb it.
None of this requires a new model. It requires an engine that supports continuous batching and KV-cache management — vLLM, SGLang, TGI, or TensorRT-LLM depending on hardware — and the discipline to measure cost-per-thousand-requests rather than tokens-per-second, because tokens-per-second is the means and the actual bill is the end.
Credits & sources
Serving-cost breakdowns, the prefill/decode diagnosis framework, and the quantization-plus-batching stacking figures are drawn from aitechconnect.in. Break-even math, engine comparisons, and AWQ quality figures come from tensoria.fr. Reasoning-token cost multipliers, the $5–10M/year crossover figure, and the four-lever cost stack come from blog.prompt20.com. The loaded-cost-per-accepted-result framing and serverless open-weight comparison come from sohailmo.ai. Additional engine-stack framing referenced from app.sourcethread.com.
Tomorrow's episode turns from the money layer to the model itself: IBM's Granite 4.1 8B, and what it's actually built to do.
Appendix — the field in one chart
Top comments (0)