vLLM deleted its most famous feature last week, and that's the strongest signal of production maturity the project has ever sent. The open-source inference engine removed its legacy PagedAttention implementation in v0.25.0, making Model Runner V2 the default execution backend for all dense models, per the v0.25.0 release notes. Most projects would keep a founding innovation around forever because ripping it out feels risky. vLLM cut it because the new architecture absorbed the concept into something better.
Here's why that matters for you: vLLM has become the default inference layer for serious AI infrastructure. It runs in production at Meta, Mistral AI, Cohere, and IBM, and comes pre-installed on AWS, Google Cloud, and Azure GPU instances, according to analysis of the v0.25 release. The question isn't whether to use it — it's whether to self-host or rent it through a managed API, and at what volume the math flips.
What vLLM Actually Does Better Than the Alternatives
The core value proposition hasn't changed even as the internals have: vLLM achieves up to 24x higher throughput than naive HuggingFace Transformers serving and 2-4x higher throughput than TGI on identical hardware, using PagedAttention and continuous batching, per a vLLM deployment guide. PagedAttention applies OS-style virtual memory paging to the KV cache, cutting memory waste from 60-80% down to under 4%. Continuous batching then inserts new sequences into the active batch at every token step rather than waiting for the longest request to finish.
The throughput gains are real and well-documented. But the more interesting story is what happened to the architecture underneath.
The Rust Frontend and the Real Bottleneck
vLLM 0.24.0 introduced a Rust frontend that matches or exceeds the throughput of 32 Python API server processes — 837 req/s vs 786 req/s — and reduces P50 time-to-first-token by 3.3x at high concurrency, per benchmark analysis. The lesson isn't that Rust is fast. It's that the bottleneck moved.
As GPU-side latency fell through continuous batching, prefix caching, and speculative decoding, the limiting factor at high concurrency became the frontend's CPU work: tokenization, request validation, JSON serialization — all running under Python's GIL and a saturated asyncio event loop. A single Rust process now saturates the engine because it eliminates the CPU-GPU synchronization point where the GPU sits idle waiting for the host to prepare the next batch.
This is what I call the Idle Tax: the gap between your advertised per-token rate and your effective production cost, driven by utilization inefficiencies you can't see in a benchmark. The Rust frontend attacks one source of that tax. Model Runner V2 attacks another by preparing step N+1 while the GPU executes step N.
The v0.25 Architectural Shift: Deleting Your Way to Maturity
The deletion of PagedAttention in v0.25.0 is the headline event, but the deeper story is what replaced it. Model Runner V2 is a ground-up rewrite built around three constraints: modular by design, GPU-native for bookkeeping, and async-first. The original execution path had grown past 6,700 lines. The largest file in MRv2 is under 1,300 lines.
What you'll find is that the ideas from PagedAttention survived — MRv2 still manages KV caches with paged block allocation. The original code didn't. That's the right outcome for production software. The project absorbed its founding innovation into a maintainable architecture and shifted optimization focus to the actual bottleneck: CPU-bound request preprocessing and idle GPU time, not GPU kernel efficiency.
The v0.25 release also shipped 558 commits from 232 contributors, including a new Streaming Parser Engine for unified tool-call and reasoning parsing, universal speculative decoding for heterogeneous vocabularies (meaning your draft model and target model no longer need the same tokenizer), and a Transformers modeling backend that finally matches native vLLM speed.
Day-0 Model Support: The Platform Play
vLLM's competitive moat isn't just throughput. It's the speed at which new models become deployable. The engine supports day-0 production deployment of new models including Kimi K3 (2.8T parameters, 1M context window) and TML Inkling (1T parameters, multimodal), per the vLLM blog. When Moonshot AI releases Kimi K3 weights, vLLM will have Docker images, deployment recipes, and production validation ready.
This matters because the model landscape is fragmenting fast. New architectures — Mamba hybrids, mixture-of-experts, diffusion-based generation, models with novel attention mechanisms like Kimi Delta Attention — all need custom serving code. If your inference engine doesn't support a model on day zero, you're either waiting weeks for a patch or rewriting kernels yourself. vLLM's contributor base (232 people on a single release) means the gap between model release and production readiness is shrinking to hours.
For teams evaluating alternatives, SGLang is the other serious open-source option, particularly for prefix-heavy agentic workloads where its RadixAttention caching outperforms. The choice between them increasingly depends on your traffic shape, not raw throughput.
The Real Cost: Where Per-Token Prices Hide the Bill
Production vLLM inference costs range from $0.09 to $7.30 per million tokens depending on deployment model, with self-hosted reserved instances reaching ~$0.44/1M tokens and managed APIs between $0.08-$0.18/1M tokens, per a pricing breakdown. That range is enormous — the difference between the cheapest and most expensive option can fund a developer's salary at moderate volume.
Here's the comparison that matters:
| Deployment Model | Cost per 1M tokens (7B model) | Strengths | Target Audience |
|---|---|---|---|
| Self-hosted vLLM (on-demand T4) | $7.30 | Full control, no vendor lock-in | Teams with DevOps capacity, unpredictable traffic |
| Self-hosted vLLM (1-yr reserved) | $0.44 | Lowest cost at scale | High-volume workloads with predictable load |
| Together AI (managed vLLM) | $0.09–$0.18 | No ops, auto-scaling | Startups under 10M tokens/month |
| DeepInfra (managed vLLM) | $0.08–$0.15 | Cheapest per token, long context | Cost-sensitive hobby projects, non-critical workloads |
| AWS Bedrock (built-in vLLM) | $0.40–$0.80 | Integrated with AWS, security | AWS-native teams with compliance requirements |
The trap is that the cheapest advertised rate routinely becomes the most expensive option in production. DeepInfra offers $0.08/1M tokens — the cheapest in any comparison — but provides no SLA guarantee, and p95 latency jumps from 400ms to 2.3 seconds during US business hours. For production workloads, those latency spikes trigger client-side retries and timeouts that silently multiply your token count.
The Hidden Cost Multiplier
Hidden costs including idle GPU time, prefix cache misses, and request retries can add 20-40% to raw vLLM compute bills, per cost analysis. Let's break down where that tax comes from:
- Prefix cache misses: Every request with a unique context prefix forces a full KV cache recomputation. At 512 tokens prefix length, a 10% miss rate adds 20% to latency and 15% to compute cost.
- Request retries: Client-side retries on 503s or timeouts silently double your token count. A 5% retry rate on a 1B-token workload adds 50M tokens.
- Idle GPU time: A single A100 running 24/7 but only processing requests 8 hours/day still costs $2,240/month. If you're not autoscaling, you're paying for empty GPUs.
- Model reloads: Switching LoRA adapters or uploading new model versions without warm pools costs 10-30 seconds of GPU time per reload.
This is why prompt caching and LLM routing deserve as much engineering attention as your model choice.
The Crossover Point: When Self-Hosting Wins
The break-even point between self-hosted vLLM and managed APIs like Groq is approximately 4.7 million tokens per month; above this volume, self-hosting becomes cheaper, per a cost comparison. Below that, the operational simplicity of a managed API wins — no GPU management, no autoscaling configuration, no 3 AM pages about OOM errors.
The crossover accelerates with two optimizations:
- Reserved instances: Switching from on-demand to 1-year reserved GPU instances cuts vLLM compute costs by 40-60% compared to on-demand pricing, per pricing analysis. This compresses the breakeven timeline from 3-6 months down to weeks for high-volume workloads.
- Continuous batching: Continuous batching in vLLM keeps GPU utilization above 80% under sustained load, compared to sub-40% utilization for static batching, per Kubernetes deployment analysis. Higher utilization means lower effective per-token cost on the same hardware.
The Utilization Trap
Here's the contradiction that catches most teams: dedicated hardware economics only work if you can sustain high utilization. A DigitalOcean H200 GPU Droplet costs $3.44/hr and breaks even with serverless inference at 72.2% sustained utilization; at 40% utilization, dedicated hardware is 80% more expensive than serverless despite lower per-token rates, per DigitalOcean's cost framework.
Most teams overestimate their utilization. Chat applications with daily peaks, agentic pipelines with bursty traffic, and RAG systems with variable query complexity all spend significant time below 50% utilization. If your traffic is bursty, paying the on-demand premium and using autoscaling can be cheaper overall than over-provisioning reserved GPUs.
Quantization can help close the gap. AWQ, GPTQ, and NVFP4 reduce LLM memory footprint by 2-4x, enabling more concurrent requests on the same GPU, per deployment analysis. A 70B model at FP16 needs 140 GB VRAM; at Q4_K_M the same model fits in 40 GB, within reach of a single L40S. More concurrent requests per GPU means higher effective utilization, which means the reserved instance math works at lower traffic thresholds.
Your Decision Framework
Budget for LLM inference using a utilization-adjusted cost model, not a per-token rate card. Here's how I'd approach it:
Under 5M tokens/month: Use a managed API. The operational overhead of self-hosting isn't worth the savings. DeepInfra or Together AI at $0.08-$0.18/1M tokens is fine for prototyping and low-traffic applications. Just don't put production-critical workloads on a provider with no SLA.
5M to 100M tokens/month: The crossover zone. Self-hosting on reserved instances becomes cheaper, but only if your traffic is predictable enough to sustain above 60% GPU utilization. If it's bursty, stay managed. The hidden costs of the inference serving stack — idle GPUs, cache misses, retries — will eat any per-token savings you think you're getting.
Above 100M tokens/month: Self-host on reserved GPUs with continuous batching and quantization. At this volume, the 55% savings against managed APIs funds the DevOps investment. Monitor prefix cache hit rates, retry rates, and GPU utilization as primary metrics. If utilization drops below 50%, autoscale down or move to spot instances.
The cheapest advertised per-token rate — whether DeepInfra's $0.08/1M or a reserved A100's $0.44/1M — routinely becomes the most expensive option once hidden operational taxes are applied. The gap widens dramatically above 100M tokens/month. Your job is to figure out which tax you're paying and whether you can engineer it away, not to chase the lowest number on a pricing page.
Originally published at SaaS with Alex
Top comments (0)