TL;DR — SGLang is a challenger inference engine to vLLmm that caches KV cache at the token level using a radix tree, called RadixAttention, giving huge speedups for agent loops, RAG, and chat workloads with repeated prefixes. It also compiles JSON schemas into finite-state machines for jump-forward decoding, cutting structured-output latency. This episode covers where it wins, where vLLM still wins, and how to choose.
Every inference engine promises to make your GPUs faster. Most of that is marketing. SGLang's pitch is narrower and more interesting: it noticed that a huge fraction of production LLM traffic is the same tokens over and over — the same system prompt, the same tool schema, the same document chunk — and built a cache that actually knows that.
RadixAttention: caching at the token, not the block
vLLM's PagedAttention manages the KV cache in fixed-size blocks and reuses them when prefixes align to block boundaries. SGLang, built by the LMSYS team behind Vicuna and Chatbot Arena, takes a different data structure entirely: every request's token sequence gets inserted into a radix tree, where each node stores the KV cache for the path from root to that node. A new request walks the tree, finds the longest matching prefix at the token level, and only computes the delta beyond that point, per the original LMSYS project writeup.
That precision matters. If your document boundaries don't line up with vLLM's block size, you lose reuse. A radix tree doesn't care about alignment — it matches exactly where the tokens diverge. LMSYS's original benchmarks, run on Llama-7B and Mixtral-8x7B on A10G GPUs, showed up to 5x higher throughput than vLLM and the Guidance library on programs with heavy KV reuse, with the biggest wins showing up in time-to-first-token when a cache hit lands.
The eviction policy is LRU with cache-aware scheduling, and it's on by default — there's a flag, --disable-radix-cache, purely for benchmarking what it's buying you, according to a 2026 engine comparison from dreaming.press. In practice, teams report cache hit rates approaching 95%+ for active multi-turn sessions once the conversation history is entirely a tree walk, per benchmarking from turion.ai — and on workloads with over 60% prefix reuse, that same analysis measured a 3–5x improvement in effective prefill latency switching from vLLM to SGLang. On workloads where every prompt is unique — translation, creative generation, one-off summarization — that advantage evaporates. The ablation studies in the original SGLang paper found no measurable overhead even when there's zero cache hit, which is the honest baseline: you don't pay a tax for trying.
Structured generation without paying per-token
The second half of SGLang's pitch is jump-forward decoding. When you force output into a JSON schema, most engines still generate token by token and mask invalid options at each step. SGLang instead compiles the grammar into a compressed finite-state machine, and when the next several tokens are fully determined by that grammar — a closing brace, a fixed key name, a quote character — it emits the whole span in one step instead of running the model forward for each token. Yotta Labs cites LMSYS's own reporting of up to 3x faster JSON decoding from this technique.
There's also a mechanical difference in where the CPU work happens. vLLM's guided decoding applies a grammar mask on the CPU during sampling, which becomes a bottleneck as batch size climbs — turion.ai's testing found noticeable throughput degradation at batch sizes of 8 and up. SGLang overlaps the grammar mask computation with the GPU forward pass on a separate thread, so the CPU cost is largely hidden even at batch sizes of 32+. Both projects have since adopted xgrammar as a shared default constraint backend, which narrows the gap — a comparison from convly.ai calls both engines production-ready but gives SGLang the edge specifically on heavily structured, high-volume extraction traffic.
Where this actually shows up in production
Strip the benchmarks and this is a checklist for whether SGLang is the right default for your service:
Multi-turn chat and assistants. After the first turn, the entire conversation history becomes a single radix tree walk. If your assistant carries a long system prompt plus growing history, SGLang's per-turn latency stays flat where a naive engine's would climb linearly with conversation length.
Agent and tool-calling loops. The system prompt plus every tool's JSON schema is identical across calls in the same session, and often across many sessions. SGLang caches that block once and every subsequent request only pays for the new user turn and the model's response — this is, per convly.ai, "exactly the traffic RadixAttention was built for."
RAG pipelines with shared documents. If ten users ask questions against the same retrieved chunk, SGLang reuses that chunk's KV cache across all ten requests without requiring the chunk boundaries to align with any fixed block size. vLLM can do this too, but only when the alignment cooperates.
High-volume structured extraction. Classification or extraction pipelines that force every response into a JSON schema — invoice parsing, log tagging, entity extraction at scale — benefit doubly: prefix caching on the shared instruction, and jump-forward decoding on the predictable parts of the output shape.
Batch summarization of unique documents. This is the honest counter-case. If every request is a different, non-overlapping document with no shared prefix beyond a short instruction, there's nothing for the radix tree to find. Both turion.ai and convly.ai land on the same verdict here: throughput is roughly even, and vLLM's broader tooling and deployment maturity make it the simpler default.
The pattern across all of these is prefix overlap. SGLang doesn't make the model smarter or the GPU faster in isolation — it makes redundant computation across requests disappear, and how much redundant computation you have is a property of your traffic shape, not your model choice.
Choosing between SGLang and vLLM
Both expose OpenAI-compatible endpoints, so switching is largely a base-URL change rather than a rewrite — which is why the sane strategy for an unsure team is to start on vLLM, then A/B SGLang against real request logs rather than synthetic benchmarks. That said, a few structural differences push the decision one way or the other regardless of prefix overlap. vLLm has broader hardware and backend coverage — TPUs, Inferentia, AMD, a wider range of quantization formats — and is the safer bet if you're serving many different models or swapping them frequently. SGLang is narrower in scope but deeper on the NVIDIA-centric, high-concurrency, structured-workload path.
If your production traffic is a chat API with a shared system prompt, a RAG pipeline hitting the same document set repeatedly, or a multi-agent system with fixed tool schemas, the RadixAttention cache is doing real work every single request, and the throughput numbers above aren't hypothetical. If your traffic is closer to a batch job over unique inputs, the cache has nothing to grab onto, and the choice comes down to which engine's operational model — deployment tooling, hardware support, model-swap frequency — fits your stack better.
Credits & sources
Architecture and RadixAttention details from the original LMSYS SGLang project blog. Engine comparison data and benchmark figures from turion.ai's 2026 vLLM vs SGLang comparison, convly.ai's serving engine guide, and dreaming.press's analysis of prefix reuse versus hardware reach. Jump-forward decoding explanation drawn from Yotta Labs' architecture writeup. Thanks to the maintainers of both the SGLang and vLLM projects for keeping their engines OpenAI-API-compatible enough that comparisons like this are even possible to run.
Tomorrow's episode steps back from any single engine to ask where the money in inference actually goes — and why the answer isn't just "tokens."
Appendix — the field in one chart
Top comments (0)