If vLLM already solved LLM serving, why did SGLang appear?

After the launch of ChatGPT and open-source models in 2022–2023, lots of companies tried hosting models on GPU infrastructure, but they encountered issues like GPU idle time and computational bottlenecks.

Then vLLM was launched by the Sky Computing Lab at UC Berkeley, which changed the game forever. Model serving became dramatically easier, and GPU utilization skyrocketed.

But, if vLLM solved all the problems, why was SGLang created by fellow graduates of UC Berkeley, Stanford, and Carnegie Mellon universities? Let's find out.

P.S. This blog is fairly technical and assumes basic familiarity with LLMs and transformers. If you're completely new to AI Infrastructure, I've also written a beginner-friendly version here for AI learners and non-technical folks: Coming Soon.

In the previous blog about vLLM, we saw how vLLM solved issues like memory fragmentation and static batching and as a result, the throughput improved dramatically and it is one of the most widely used serving stacks today.

So naturally it would occur to you, why did another inference engine suddenly become one of the hottest projects in AI Infrastructure? Let's find out.

The Change in Workload

Initially when open-source models launched, the nature of requests by all the users was very random and at a huge scale. For example, companies would rent GPUs and host models, and most users would ask queries that are independent in nature.

For example: "Translate this in Hindi…", "Why is the sky blue?", "Solve this math problem for me…".

But the workload gradually evolved with the advancements in the agentic domain and open source tools like LangChain.

Back in 2023 most workloads looked like

User → Prompt → Response

Then AI changed.

Now workloads look like

System Prompt → User → Reasoning → Tool → Reasoning → Tool → JSON → Reasoning → Answer

Notice the change? Same model. Different workload.

That's why another engine appeared, better optimized for the evolving workloads.

Patterns in Recent Workload

From the above example of evolving workloads, do you notice anything different than previous workloads?

First, we have moved towards multi-turn agentic workloads capable of handling longer context lengths. Second, the longer the context goes on, the more recomputation is required at each turn, increasing memory and KV cache calculation costs.

Even in the above example, you can see a pattern repeating, every next interaction has to recompute all tokens right from the system prompt to all reasonings, tool callings, and structured decoder outputs (JSON).

SGLang asked a very important question at this juncture:

Why are we recomputing all of this every iteration?

And hence, prefix reuse or prefix caching was the natural solution to solving this problem.

Prefix Reuse (Prefix Caching)

To learn prefix caching, let's understand with an example.

Imagine ChatGPT receives these requests:

Request 1

System: You are a helpful assistant.
User: Explain transformers.

Request 2

System: You are a helpful assistant.
User: Explain CNNs.

Request 3

System: You are a helpful assistant.
User: Explain RNNs.

Notice something? The beginning of every request is identical.

System: You are a helpful assistant.

This taps upon the similar problem we faced in the last section. The model has already processed those tokens once. Why should it process them again?

Without Prefix Reuse

Every request does

System → User → Attention → KV Cache

Even though the system prompt is identical. GPU repeats work.

With Prefix Reuse

The first request computes

System Prompt → KV Cache

The engine stores that KV cache. The second request arrives.

Instead of recomputing the system prompt, it simply says:

Already have those keys and values. I will just reuse them.

Now it only computes the parts that differ between two prompts, like:

User: Explain CNNs

Instead of re-computing the whole system + user prompts again and again.

Result

Less compute → Lower TTFT → Higher Throughput

This is especially valuable for:

• Chatbots
• Agents
• RAG
• Long system prompts
• Shared document context

Problem with Prefix Reuse

How do you know whether two prefixes are the same?

Let's imagine the following scenario:

• Request A - My name is Mayank Kulkarni
• Request B - My name is Anthony Gonsalves

They share - My name is

Should we reuse?

Yes. Obviously.

But wait… How do we know which parts of a request can actually be reused?

This is exactly the problem RadixAttention was designed to solve, also known as the heart of SGLang.

RadixAttention

One of the biggest innovations SGLang introduced is an algorithm that knows how much part of a query to reuse without needing to match the whole query.

To understand how RadixAttention works:

Let's consider an anology of a dictionary.

Words:

car, cart, care, carry

Instead of storing four separate trees, it stores car once.

And then for the subsequent words, it creates branches.

car
├── t
├── e
└── ry

That's a radix tree.

Now replace letters with tokens.

Example:

System: You are a helpful assistant.

Thousands of requests begin here.

Instead of storing:

System → You → are → ….

thousands of times, SGLang just stores it once.

You would be wondering, what actually happens when a request arrives?

It's simple, you just walk to the tree.

Token 1 → Token 2 → Token 3 → …

Keep walking until tokens differ. That's called the Longest Prefix Match.

Example:

• Request A - ABCDEF
• Request B - ABCDEH
• Common - ABCDE

Reuse all KV cache up to ABCDE.

And then it has to only compute:

F or H

Why is this better than hashing?

Hash lookup answers:

Exact match? → Yes/No

Radix tree answers → Longest reusable prefix.

Even partial overlap saves enough work.

This is why SGLang performs exceptionally well on agent workloads.

Now that we've eliminated unnecessary computation.

But modern agents still face another problem.

Modern applications don't just want essays anymore. They want JSON. They want tool calls. They want structured outputs that another application can consume directly.

Structured Generation

Now we move away from scheduling optimizations.

This is where SGLang's language/runtime really differentiates itself.

Normally LLMs generate "The weather today is sunny."

Just text. But many applications don't want text.

They want

{ "temperature": 31, "city": "Mumbai" }

The problem? LLMs are probabilistic.

Sometimes they output

{ temperature: 31,

Oops. Broken JSON.

JSON Generation

Instead of just hoping, if the decoder actually enforces that:

• "{" must come first.
• After "temperature," Only ":" is allowed. Not banana

This is called constrained decoding.

Tool Calling

Suppose the model should produce

{
  "name": "search",
  "arguments": {
    "query": "vLLM"
  }
}

Instead of again hoping for the token probabilities to align like: I think you should search…

The Decoder grammar forces exactly that structure even in tool calling use cases.

Hence, you encounter no malformed tool calls.

Why is SGLang famous for this?

Because SGLang treats structured outputs as a first-class problem.

Its runtime was designed specially for:

• Agents
• Tools
• Workflows
• JSON APIs

instead of "just generating text."

Many agent systems repeatedly need

JSON → Tool → JSON → Tool and so on….

In such sensitive cases, if JSON fails, the whole pipeline fails. SGLang makes this reliable.

Impact

Put together, the combination of prefix reuse, RadixAttention, and smarter scheduling substantially reduced latency metrics like Time to First Token (TTFT), especially for workloads where requests shared large portions of context. Combined with efficient paged KV cache management, SGLang is able to maintain high throughput while continually improving latency through ongoing optimizations.

Conclusion

To conclude, vLLM hasn't become obsolete. It solved the biggest problem of the initial generation.

SGLang simply solved the next biggest problem. The interesting part is that neither framework made transformers fundamentally faster.

They simply found smarter ways to avoid unnecessary work.

First, vLLM asked:

"Why waste GPU memory?"

Then SGLang asked:

"Why recompute tokens we've already seen?"

That's how inference engineering evolves.

Not by only changing the transformer architecture… but by changing the environment around it.

vLLM and SGLang Workloads

If you enjoyed this deep dive, stay tuned. In the next blog, we'll explore why two inference engines like vLLM and SGLang serving the same model on the same GPU can still produce very different benchmarking numbers and what that tells us about modern LLM serving.

---

Making AI infrastructure understandable for engineers and everyone who works with them - MK