SGLang vs vLLM mostly comes down to one question: how much of your prompt do your requests have in common? SGLang is built around a prefix cache called RadixAttention, and vLLM offers a similar feature called automatic prefix caching. Where requests share long openings, both skip repeated work. Where they share nothing, neither has much to skip. The headline speedups online come mostly from the first situation.
SGLang vs vLLM: how each tracks shared text
Reading a prompt is called prefill, and it must finish before the first word of a reply appears. If a new request starts with text the engine already processed, that part of prefill can be skipped.
SGLang. Per the LMSYS announcement and the SGLang paper, RadixAttention keeps the KV cache for prompts and generated answers in a radix tree, so prompts that begin the same way share a branch and the same stored memory. The tree lives on the CPU while the tensors stay on the GPU. A least-recently-used policy evicts old leaves, and a cache-aware scheduler orders requests so more of them find a match. LMSYS says the authors saw no noticeable overhead when there were no cache hits.
vLLM. The design documentation says vLLM hashes each block of the KV cache. A block's fingerprint combines the previous block's fingerprint, the tokens inside it and extras such as adapter IDs. Only full blocks are cached, and eviction is least-recently-used too. So a prefix that ends partway through a block cannot be reused beyond the last full block.
The vLLM feature page lists multi-turn chat and repeated questions about one long document as the cases where it helps. It also says prefix caching speeds up prefill only: it does nothing for long generated answers or for queries that share no prefix.
SGLang vs vLLM benchmarks: what they show
The SGLang paper reports up to 6.4x higher throughput than state-of-the-art systems across several tasks. The LMSYS post reports up to 5x against vLLM v0.2.5, Guidance and TGI, on Llama-7B with one A10G and on Mixtral-8x7B with eight.
Two cautions. Those baselines date from late 2023 and early 2024, and vLLM has since added automatic prefix caching. And "up to" means the best case among the workloads tested.
Newer comparisons are mostly single-author blog posts, and they conflict. One 2026 DEV Community post reports median time to first token of 380 ms on vLLM against 85 ms on SGLang for a shared-prefix test. A post from Spheron reports 1,920 tokens per second for SGLang against 1,850 for vLLM at 50 concurrent requests on an H100, close to a tie. A SwiftKV paper mentions 1.4x to 1.8x for SGLang over vLLM on Llama-3.1-8B.
Choosing
Prefix caching pays off when many requests begin with the same material: a long system prompt reused by every user, a tool list an agent resends on every step, or one document that many questions refer to. It pays off less when prompts are mostly unique or replies are very long.
To compare them for your case, take a sample of real traffic, run it through both engines on the same hardware with prefix caching switched on in vLLM, and compare time to first token and throughput at the concurrency you expect. The vLLM page says to set enable_prefix_caching=True and does not state the default, so check your version.
The original also covers memory pressure and the open research on hybrid models, with a short FAQ: https://syntaxradar.com/radar/radixattention-dynamic-prefix-caching-sglang-vllm-benchmarks
Top comments (0)