If one tool already made serving open-source models efficient, why would anyone build a second one?
That's the question you might be pondering upon. So let's find out.
A while back, I wrote about a tool called vLLM: It made using open-source models efficient in terms of request handling and GPU hours.
If you haven't read that one, no worries, you don't need it for this.
Here's the short summary: Hosting AI models wasted a ton of resources like compute, power and memory. vLLM fixed that and it was considered a big win for the open-source industry.
So if that problem was already solved… Why did the smart people at Berkeley, Stanford, and CMU go build another one called SGLang?
Turns out, fixing throughput was only the beginning.
AI stopped just answering. It started iterating.
Back in 2023, using AI was fairly simple.
We asked simple questions like "Translate this." "Why is the sky blue?" "Solve this equation." One question, one answer, move on.
But somewhere along the way, AI stopped just answering questions. It started doing things.
Now a single request might include reasoning, use of tools (like Search Engine, Research etc.), reasoning again, and then finally you get an answer.
Same AI but doing a completely different job.
And here's the catch nobody talks about: every single time we move to the next step, it has to re-read everything it already read before — including the question, its own thoughts, the tool's response. All of it, and that too every single time.
You can imagine it like re-reading an entire book from page one, every time you wanted to read the next paragraph.
That's expensive, slow — and that's the problem SGLang was invented to solve.
Why do LLMs keep repeating the same thing?
Let's slow down and understand with an example.
Imagine you're chatting with an LLM, and it always starts by reading this line first:
"You are a helpful assistant."
This happens in every single conversation while talking to every single person. That exact same line, read fresh, every time.
Doesn't that feel… wasteful?
It's like if a customer service rep had to re-read their entire training manual before answering every single customer, instead of just remembering it.
SGLang looked at this and basically said:
"Hang on, we've already read this part. Why are we reading it again?"
So instead of starting from scratch every time, it remembers what it already read, and only pays attention to what's actually new.
Results in less repetition, faster replies. Simple idea but creates huge impact.
How does it know what to reuse?
This is where it gets clever. Say two people type:
"My name is Mayank"
"My name is Anthony"
Clearly, "My name is" is the same in both. Should the LLM reuse that part and only pay fresh attention to the name? Yes. Obviously.
But how does a computer know that, instantly, out of millions of requests?
This is the part SGLang nailed, and this implementation is also known as the heart of SGLang.
Think about words like: car, cart, care, carry.
Did you notice they all start with "car"? Instead of storing all four words separately from scratch, you could just store "car" once and then branch off into t, e, and ry. That branching idea is the intuition behind RadixAttention.
That's basically what SGLang does with everything the LLM reads. It stores the common starting part once, and only branches off when things start being different.
So when a new request comes in, the LLM doesn't ask "have I seen this exact thing before?" It asks "how much of this have I already seen?" — and reuses exactly that much.
That one shift is a big reason SGLang replies feel quicker, as it avoids computing the same parts repeatedly. Another big reason why SGLang is booming is because reusing work solved one part of the problem. Another part was making AI outputs reliable enough for applications.
Modern AI doesn't just talk anymore. It fills out forms.
So here's problem number two, and it's a fun one.
These days, a lot of apps don't want the LLMs to write a nice paragraph. They want the output to be in JSON format like:
{
"temperature": 31,
"city": "Mumbai"
}
Seems neat and structured — something that can directly be consumed by some other tool.
The problem? LLMs are unpredictable by nature. Ask it for that exact format, and sometimes it'll just… write you a paragraph instead. Or mess up the formatting halfway through.
Annoying, right? Especially if your app is expecting an exact format, gets a rambling essay instead, and breaks halfway through.
So SGLang basically forces the LLM to fill out the form correctly. Not with prompts like "please try to give me JSON" — but more like so that the AI literally can't type the wrong kind of character in the wrong place. If a curly bracket is open, then it'll only end with a curly bracket and nothing else.
No more broken outputs. No more guessing games.
This matters a lot for AI agents that need to talk to tools over and over — could be search engines, calculators, calendars, whatever. If even one of those steps breaks, the whole chain falls apart. SGLang was built with this exact headache in mind.
Which one's better, vLLM or SGLang?
Actually, neither. That's not really the point.
vLLM asked:
"Why are we wasting all this computer power?"
SGLang came along later and asked a different question:
"Why are we redoing work we've already done?"
In plain terms, vLLM focused on letting your AI juggle way more requests simultaneously (Improved throughput). SGLang ensured that every single user got that first word back much quicker (Time to First Token).
The two inference engines are optimised for volume versus immediate response time — both are equally critical. They simply address different parts of the same puzzle.
Neither one made the AI itself smarter. They just found smarter ways to use it — by having less waste, less repetition, and more reliability.
And honestly? That's most of what "AI infrastructure" actually is. Not reinventing the AI. Just being less wasteful about how we run it.
If this got you curious about the more technical side — prefix caching, RadixAttention, and why two serving engines can give completely different benchmark numbers on the exact same hardware — check out the full technical version here.
Making AI infrastructure understandable for engineers and everyone who works with them — MK
Top comments (0)