vLLM hit the front page of Hacker News this week with a deep dive into its architecture. If you're running production LLM workloads, understanding how vLLM achieves its throughput is essential. Here's what makes it special.
The Problem vLLM Solves
Traditional LLM inference is memory-bound, not compute-bound. The GPU spends most of its time waiting for model weights to arrive from memory, not actually computing. This is why you can't just throw more GPUs at the problem — the bottleneck is memory bandwidth, not FLOPs.
vLLM attacks this problem from multiple angles: PagedAttention for memory efficiency, continuous batching for throughput, and optimized CUDA kernels for speed.
PagedAttention: The Key Innovation
The biggest innovation in vLLM is PagedAttention, which manages the KV cache (the memory where the model stores intermediate states during generation) the same way an operating system manages virtual memory.
In traditional inference, each request gets a contiguous block of memory for its KV cache. This means:
- You pre-allocate a large block (wasting memory if the request is short)
- You can't serve more requests than fit in your pre-allocated memory
- Fragmentation kills throughput
PagedAttention breaks the KV cache into fixed-size pages (blocks). Each request's KV cache is a collection of pages, not a contiguous block. This means:
- Memory is allocated on-demand (no waste)
- You can serve more concurrent requests
- Fragmentation is minimal
The result: vLLM can serve 2-4x more concurrent requests than HuggingFace's default inference engine, with the same GPU.
Continuous Batching
Traditional inference uses static batching: you wait until you have N requests, batch them together, generate one token for all, and repeat. If one request finishes early, it waits for the others. If a new request arrives mid-batch, it waits for the next batch.
Continuous batching is dynamic: requests join and leave the batch at any token boundary. When a request finishes, it leaves immediately (no waiting). When a new request arrives, it joins the current batch (no waiting for the next one).
This dramatically reduces latency for short requests and increases throughput for mixed workloads. In production, where request lengths vary wildly (a one-sentence answer vs. a 2000-word essay), continuous batching is the difference between 10 tokens/sec and 100+ tokens/sec aggregate throughput.
How to Use vLLM
Installation is straightforward:
pip install vllm
Serve a model with a single command:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--tensor-parallel-size 1 \
--max-model-len 8192
This gives you an OpenAI-compatible API on port 8000. Point your existing OpenAI client at http://localhost:8000/v1 and it just works.
When to Use vLLM vs Alternatives
Use vLLM when:
- You're serving models from 7B to 70B parameters
- You need maximum throughput on existing GPUs
- You want OpenAI API compatibility
- You're running in production with variable request patterns
Use Ollama when:
- You're running on consumer hardware (MacBooks, Raspberry Pi)
- You need to run small models (1B-3B)
- You want simplicity over raw throughput
- You're doing local development, not production serving
Use TGI (HuggingFace) when:
- You need fine-grained control over decoding parameters
- You're using models that vLLM doesn't support yet
- You're in the HuggingFace ecosystem
Use proprietary APIs (OpenAI, Anthropic) when:
- You need frontier model quality
- You don't want to manage infrastructure
- Your volume is low enough that API costs are reasonable
Performance Comparison
On a single A100 (80GB), serving Llama 3.1 8B with 1K context:
| Engine | Throughput (tok/s) | Concurrent Requests | Memory Efficiency |
|---|---|---|---|
| HuggingFace Transformers | ~500 | 8 | ~40% |
| TGI | ~2,000 | 32 | ~60% |
| vLLM | ~5,000+ | 64+ | ~90%+ |
Your mileage will vary based on model size, context length, and hardware, but the pattern holds: vLLM consistently delivers 2-5x better throughput than alternatives.
The Bigger Picture
vLLM represents a shift in AI infrastructure: the software layer matters as much as the hardware. You can buy an H100 for $30,000, but if your inference engine wastes 60% of its memory, you're getting the performance of a $12,000 GPU.
As inference costs dominate AI application expenses, efficient inference engines like vLLM are the difference between a viable business model and one that burns cash. The companies that win in AI won't just have the best models — they'll have the most efficient infrastructure for serving them.
Original analysis inspired by Inside vLLM by Aleksa Gordic (HN: 74 points)
Top comments (0)