Daniel Kim

Posted on Aug 17

The Inference Engine Running Grok Has a Third of vLLM's GitHub Stars

#ai #machinelearning #llm #opensource

Open-weight models closed most of the quality gap to closed APIs this year, and GPU rental prices didn't drop nearly as fast as everyone hoped. The result is a lot of teams re-running the math on self-hosting, and running straight into the same question: which inference server do you actually put in front of the model?

There are effectively three serious answers if you're serving on your own infrastructure — vLLM, SGLang, and TensorRT-LLM — and the popularity numbers point one way while the production deployments point another. vLLM has roughly 89,000 stars on GitHub. SGLang has a bit over 32,000, a little more than a third of that. And yet SGLang is the engine behind xAI's Grok, serving it across more than 100,000 GPUs, and it's what Microsoft Azure uses to run DeepSeek R1 on AMD hardware. The most popular option by community size is not the one the two organizations running some of the largest inference fleets on the planet picked for their hardest workload.

That gap is worth sitting with before you pick a serving stack by star count or by whichever benchmark chart is making the rounds this week. Star count measures how many people evaluated a project or built a toy app on it. It does not measure what happens to a serving stack under real production concurrency, real prefix-sharing patterns, or a real GPU procurement budget. This piece looks at what vLLM, SGLang, and TensorRT-LLM actually are, how they're built differently, what the numbers say about each, and — the part vendor pages skip — what each one quietly costs you that isn't on the pricing page, because none of these three has a pricing page. They're all open source. The cost is entirely operational.

Why this decision is live right now

A year ago, "just call an API" was the default answer for most teams, and self-hosting was a niche concern for companies with either extreme scale or extreme data-sovereignty requirements. Two things changed that calculus. First, open-weight models — Llama 4, DeepSeek's V-series, and a wave of smaller fine-tunable models — now clear the quality bar for a meaningful share of production use cases, including agentic workloads that used to require a frontier closed model. Second, per-token API costs at scale stopped feeling negligible once agent loops started making five, ten, or fifty calls to accomplish one task. Self-hosting a 70B-class model for a high-volume internal workload increasingly pencils out, provided you can actually get GPU utilization high enough to matter, which is the entire job of the inference engine sitting between your model weights and your traffic.

That's the real stakes here. The inference engine is not a commodity wrapper around model.generate(). It determines your GPU utilization, your latency under concurrent load, how much hardware flexibility you retain, and how much engineering time you spend babysitting deploys. Pick wrong and you either overpay for GPUs you're not using efficiently, or you lock yourself into a single hardware vendor's roadmap.

There's also a second, less obvious pressure pushing this decision forward: agentic workloads changed what "one request" means. A chat completion used to be one prompt in, one completion out. An agent loop is a tool-definition preamble repeated on every step, a growing conversation history re-sent on every call, and often a RAG context block that barely changes between requests. That shape of traffic — long, largely repeated prefixes, high call volume per user session — is exactly the shape that makes the difference between inference engines stop being a rounding error and start being a line item your finance team asks about.

What each one actually is

vLLM started as a research project at UC Berkeley's Sky Computing Lab and is now maintained by a genuinely distributed community — many dozens of academic institutions and companies, over 2,000 contributors, no single corporate owner steering the roadmap. It's Apache 2.0 licensed and installs with pip install vllm. Point it at a Hugging Face model ID and you have an OpenAI-compatible endpoint running in minutes. It supports more than 200 Hugging Face model architectures — decoder-only LLMs, mixture-of-experts models, multimodal models, embedding models — and runs on essentially anything: NVIDIA and AMD GPUs, Intel GPUs, x86/ARM/PowerPC CPUs, Google TPUs, Intel Gaudi, even Apple Silicon.

SGLang grew out of LMSYS, the same non-profit research org behind Chatbot Arena, and has recently joined the PyTorch ecosystem, which shifts its governance toward a foundation-backed model rather than a single-company or single-lab project. It's also Apache 2.0. The headline claim from the project is that it now runs in production across more than 400,000 GPUs globally and processes trillions of tokens a day, with xAI, NVIDIA, AMD, Intel, LinkedIn, Cursor, Oracle Cloud, Google Cloud, Azure, and AWS all listed as contributors or production adopters, alongside Baidu, Alibaba, and Tencent.

TensorRT-LLM is NVIDIA's own inference library, built on top of the TensorRT compiler and now architected on PyTorch rather than a bespoke internal format. It's also Apache 2.0 licensed — NVIDIA opened the license a while back — but the hardware story is completely different from the other two: it targets NVIDIA GPUs specifically (H100, H200, L40S, RTX-class cards, and the newer Blackwell/GB200 generation) and nothing else. There is no AMD path, no CPU fallback, no TPU support. What you get in exchange is deep, hardware-specific compilation.

How they're actually built — the architecture difference that matters

The three projects optimize different bottlenecks, and that's the real reason they perform differently, not marketing.

vLLM's core contribution is PagedAttention, which manages the KV cache — the memory that stores each request's attention keys/values as it generates tokens — the way an operating system manages virtual memory: in fixed-size, non-contiguous pages instead of one large contiguous block per request. That eliminates the memory fragmentation that used to waste 60-80% of GPU memory in naive serving setups, which is what let vLLM push batch sizes and throughput up dramatically when it first shipped. It combines this with continuous batching (new requests join a running batch instead of waiting for the current batch to finish) and prefix caching.

SGLang's core contribution is RadixAttention, which takes prefix caching further by organizing cached KV entries in a radix tree structure so that any request sharing a prefix with a previous one — a system prompt, a few-shot template, a shared RAG context, the earlier turns of a multi-turn conversation — reuses that cached computation automatically, without the application needing to manage cache keys itself. This is the mechanism behind SGLang's biggest performance claims, and it's also the mechanism that explains exactly when those claims apply and when they don't.

TensorRT-LLM takes a fundamentally different approach: ahead-of-time compilation. Instead of interpreting the model graph at request time, it compiles a PyTorch model into an optimized TensorRT engine — fusing kernels, picking hardware-specific execution paths, tuning for the exact GPU SKU — before it ever serves a request. That compilation step is where TensorRT-LLM's speed comes from, and it's also where its cost comes from, because compiling is slow and has to happen again any time the model, its quantization, or the target hardware changes.

Quantization support is close to a solved problem across all three at this point, which is worth calling out because it used to be a real differentiator. vLLM supports FP8, the newer MXFP8/MXFP4 microscaling formats, NVFP4, INT8, INT4, GPTQ, AWQ, and GGUF. SGLang covers FP4, FP8, INT4, AWQ, and GPTQ, plus multi-LoRA batching for serving many fine-tuned adapters off one base model efficiently. TensorRT-LLM supports FP8, FP4, and INT4-AWQ, compiled directly into the engine rather than applied at load time. If your deciding factor used to be "which engine supports the quantization format I need," that's largely stopped being the question — all three cover the formats most teams actually use in production today. The real differentiator moved to the scheduling and memory-management layer, which is exactly where PagedAttention, RadixAttention, and AOT compilation diverge.

What changed versus a year or two ago

The trajectory of all three projects is instructive. vLLM has scaled its contributor base into the thousands and become something close to a default choice — the "pip install and go" option — without ever consolidating under one company's governance. SGLang went from a LMSYS research artifact to the engine choice of one of the largest and most aggressively-scaled AI labs in the industry, and recently formalized that trajectory by moving into the PyTorch ecosystem rather than staying a standalone project — a signal that its production-readiness bar has risen enough that a foundation was willing to take it on. TensorRT-LLM's biggest shift has been architectural: rebuilding on native PyTorch instead of a proprietary internal representation, which lowers (but doesn't eliminate) the customization barrier that used to make it feel like a black box you fed a model into and hoped for the best.

The practical upshot: all three are meaningfully more mature and more production-tested than they were two years ago, but they matured toward different things. vLLM matured toward breadth. SGLang matured toward hyperscale conversational and agentic throughput. TensorRT-LLM matured toward being a slightly less painful version of maximum-performance NVIDIA-only serving.

What the benchmarks actually show

Numbers here come from independent 2026 testing rather than vendor claims, and the caveats matter as much as the figures, because "fastest" changes completely depending on your traffic shape.

On Llama 3.1 8B with prefix-heavy traffic — the pattern you get from a chat product with a shared system prompt, or an agent framework reusing the same tool-definition preamble on every call — one widely-cited 2026 benchmark measured SGLang at roughly 16,200 tokens/sec against vLLM's roughly 12,500, a ~29% edge attributable almost entirely to RadixAttention catching those repeated prefixes. On unique, non-shared prompts using Llama 3.3 70B in FP8, that gap mostly closes: vLLM and SGLang land within a few percent of each other, with SGLang ahead by only 1-4% at low-to-mid concurrency. That's the single most important caveat in this entire comparison — SGLang's advantage is a function of how much your traffic actually shares prefixes, not a fixed multiplier you get for free.

Time-to-first-token tells a similar story with TensorRT-LLM in the picture. At ten concurrent requests, one benchmark put p50 TTFT at roughly 120ms for vLLM, 112ms for SGLang, and 105ms for TensorRT-LLM — TensorRT-LLM's compiled engine wins on raw latency, as you'd expect from hardware-specific compilation, but the margin over SGLang is single-digit milliseconds at that concurrency level, not a step change.

Deployment speed is where the real trade-off shows up, and it's not really a "benchmark" number so much as an operational fact. vLLM and SGLang both skip a compilation step entirely; vLLM's own JIT-warmed cache path can get a server ready in as little as 8 seconds once caches are populated, and full cold starts — pulling images, loading model weights — typically land somewhere in the range of a minute. TensorRT-LLM's engine build is a genuinely different order of magnitude: roughly 28 minutes to compile an engine the first time for a given model, quantization, and hardware configuration, though subsequent restarts that reuse an already-compiled engine reload in roughly 90 seconds. That 28-minute number isn't a one-time annoyance you pay once and forget — it recurs every time you bump the model version, change a quantization setting, adjust max_batch_size or max_seq_len, or move to a different GPU SKU. Get those build parameters wrong and the compiled engine quietly underperforms for a workload it wasn't tuned for, with no runtime error telling you so.

Why this should actually change what you build

Cost. The throughput differences translate directly into GPU-hours, and GPU-hours are the majority line item in any self-hosting budget. A 29% throughput edge on prefix-heavy traffic is a real, bookable cost reduction if that's your traffic shape — fewer GPUs needed for the same load. But TensorRT-LLM's raw-throughput lead on NVIDIA hardware only pays for itself if your deployment is stable enough to amortize the rebuild cost; a team that reconfigures batch sizes weekly while iterating on a new agent product will spend more engineer-hours fighting the build pipeline than they save in GPU-hours.

Latency. Single-digit-millisecond TTFT differences at low concurrency mostly don't matter for a chat UI. They matter a great deal for voice agents, real-time agentic loops with tight interaction budgets, and any system where TTFT compounds across a chain of calls.

Lock-in. This is the dimension vendor comparisons underplay most. License isn't the lock-in vector here — all three are Apache 2.0. Hardware is. vLLM and SGLang run across NVIDIA, AMD, Intel, TPUs, and more, which means you can shop GPU pricing across clouds and hardware generations without touching your serving code. TensorRT-LLM's performance is earned by compiling specifically for one hardware target, and every hour spent tuning that engine is stranded value the moment you want to run on anything that isn't an NVIDIA GPU — including, notably, a different NVIDIA GPU generation than the one you compiled for.

Maintainability. A community-governed project like vLLM spreads bug-fix and feature velocity across thousands of contributors, which is resilient but can mean slower, more negotiated responses to any single issue. A single-vendor project like TensorRT-LLM has one clear accountable party, but that party's incentives are selling NVIDIA hardware, not maximizing your portability. SGLang sits in between — foundation-adjacent governance, but still a smaller contributor surface than vLLM, meaning fewer battle-tested edge cases at the long tail even as its production track record at the top end (xAI, Azure) is arguably the strongest of the three.

Security surface. This gets skipped in most engine comparisons, but it's worth naming: all three run arbitrary model code pulled from Hugging Face or a local checkpoint, execute custom CUDA/Triton kernels, and expose an HTTP API you're presumably putting real traffic through. That's a broadly similar attack surface across all three — the differentiator isn't the engine's own code, it's the operational discipline of pinning model revisions, scanning checkpoints, and not treating "trusted enough to pip install" as "trusted enough to auto-pull whatever's newest on the Hub" in a production pipeline. None of the three projects solves that for you; it's an operational practice you own regardless of which engine you pick.

Migration cost. The good news buried in all of this: switching between vLLM and SGLang is not a rewrite. Both expose an OpenAI-compatible API, so an application built against either one is mostly portable at the app layer — what you actually re-do is re-benchmarking for your traffic shape, re-tuning batch and concurrency settings, and re-validating latency SLAs. TensorRT-LLM is the outlier here too: moving to or from it means the compile pipeline itself, not just a config swap, which is one more reason it suits a settled, slow-changing deployment far better than one still finding its shape.

Practical use cases

vLLM is the right default when you're serving a variety of models, running on mixed or opportunistic hardware (spot GPUs across clouds, a heterogeneous on-prem fleet), or want the fastest path from "we have model weights" to "we have a production endpoint." It's also the safer choice if you can't predict your future hardware — you're not betting the deployment on staying on NVIDIA forever.

SGLang earns its keep specifically on high-concurrency conversational and agentic workloads with heavy shared context: long system prompts, RAG pipelines reusing the same retrieved-context template, multi-turn chat where earlier turns act as a cached prefix for later ones, and structured-output-heavy pipelines (SGLang's compressed finite-state-machine approach to constrained generation is a meaningful feature on its own, separate from RadixAttention). If your traffic doesn't look like that — mostly unique, non-repeating prompts — you're not going to see the headline numbers.

TensorRT-LLM makes sense when you have a small number of stable, high-volume production models on a committed NVIDIA fleet, latency SLAs tight enough that single-digit milliseconds matter at massive request volume, and enough deployment stability that a 28-minute build amortizes over months rather than days. It also integrates natively with NVIDIA's Triton Inference Server, which matters if that's already your serving layer.

A few concrete scenarios make the split clearer than the abstract descriptions do. A team building an internal RAG tool that serves three different open-weight models to different departments, on whatever spot GPUs are cheapest that week, is a textbook vLLM case — the model-coverage breadth and hardware portability matter more than squeezing out the last 10% of throughput. A customer-support agent product with a long, mostly-static system prompt and multi-turn conversations that can run to twenty or thirty exchanges is a textbook SGLang case — every one of those turns after the first is largely a cache hit under RadixAttention, and that compounds fast at scale. A fintech or telco running one flagship 70B model at extremely high, extremely stable volume, already standardized on NVIDIA hardware and Triton for other workloads, is a textbook TensorRT-LLM case — the build cost is a rounding error against months of steady-state serving.

What the marketing leaves out

vLLM's breadth is also a diffusion problem: with support spread across NVIDIA, AMD, Intel, TPU, and CPU backends, the non-NVIDIA paths inevitably get less optimization attention than the primary one, and feature parity across backends lags. "Runs everywhere" doesn't mean "runs equally well everywhere."

SGLang's headline throughput numbers are prefix-cache numbers. On workloads without significant prefix sharing, the gap to vLLM nearly disappears, and the project's production pedigree — however impressive at xAI and Azure — still rests on a materially smaller contributor base than vLLM's, which matters for how quickly obscure bugs get found and fixed outside the handful of companies running it at that scale.

TensorRT-LLM's throughput claims — NVIDIA has cited figures north of 40,000 tokens/sec — are measured on its newest hardware generation, which most teams evaluating a serving stack today don't have racked yet. The hardware lock-in is close to absolute: an optimized engine is a compiled artifact tied to a specific GPU SKU, and "redeploy on new hardware" for TensorRT-LLM isn't a redeploy, it's a rebuild, with all 28 minutes of that rebuild back on the clock.

Comparison table

Dimension	vLLM	SGLang	TensorRT-LLM
License	Apache 2.0	Apache 2.0	Apache 2.0
Governance	Community (2,000+ contributors, no single owner)	LMSYS-originated, now part of the PyTorch ecosystem	NVIDIA-owned and maintained
GitHub stars	~89.3k	~32k+	~14.4k
Core technique	PagedAttention + continuous batching	RadixAttention (radix-tree prefix caching)	Ahead-of-time compiled TensorRT engines
Hardware support	NVIDIA, AMD, Intel, CPU (x86/ARM/PowerPC), TPU, Gaudi, Apple Silicon	NVIDIA, AMD, Intel Xeon, TPU, Ascend NPU	NVIDIA only (H100/H200/L40S/RTX/Blackwell)
Deploy model	`pip install`, no compile step	`pip install`, no compile step	AOT compile: ~28 min first build, ~90s reload
Best-case throughput edge	Baseline	Up to ~29% over vLLM on prefix-heavy traffic	Lowest measured TTFT at moderate concurrency
Throughput on unique prompts	Baseline	Within a few % of vLLM	Generally highest on NVIDIA hardware
Model coverage	200+ Hugging Face architectures	Broad, day-one DeepSeek/major open model support	Narrower; requires per-model engine builds
Known large-scale users	Broad community/enterprise use	xAI (Grok, 100k+ GPUs), Microsoft Azure (DeepSeek R1)	NVIDIA reference deployments, latency-critical enterprise
Primary lock-in vector	Low (broad hardware)	Low (broad hardware)	High (NVIDIA hardware + per-build tuning)
Best fit	Broad model/hardware coverage, fast iteration	High-concurrency, prefix-heavy conversational/agentic traffic	Stable, high-volume, NVIDIA-committed deployments

The independent read

None of these three is "the best inference engine" in the abstract, and any comparison that declares one is answering a question you didn't ask. The honest takeaway from the numbers is that engine choice is a function of your traffic shape and your deployment stability, not a function of which project has the most stars or the flashiest benchmark chart. SGLang's real advantage is prefix reuse, full stop — it's not a general 29% speedup, it's a 29% speedup specifically for the subset of teams whose traffic is dominated by repeated context, which happens to include a lot of today's highest-value workloads (agents, RAG, long conversations), which is exactly why xAI and Azure picked it for exactly those workloads. TensorRT-LLM's advantage is real but comes with an operational tax that only pays off with deployment stability most fast-moving product teams don't actually have. And vLLM's "no compile step, runs anywhere" simplicity is underrated by benchmark-driven comparisons precisely because it's not a benchmark number — it's an engineering-time number that only shows up months later when you're not the one debugging a stalled TensorRT build at 2am.

Who should pick what

If you're a small team or solo developer serving one or a handful of models, want the fastest path to a working endpoint, or need to stay hardware-flexible because your GPU budget depends on spot pricing across clouds — pick vLLM. It's the lowest-friction default and the safest bet if you're not certain what your infrastructure looks like in a year.

If you're building a conversational product or an agentic system with heavy shared context — long system prompts, RAG templates, multi-turn chat, structured-output pipelines — and you're running at real concurrency, benchmark SGLang against your own traffic before committing. The prefix-caching win is genuine, but it's traffic-dependent, so verify it on your actual request patterns rather than trusting a generic benchmark.

If you have a small number of stable, high-volume models, a committed NVIDIA fleet, and latency SLAs tight enough to justify engineering time on build pipelines — TensorRT-LLM will get you the lowest latency and highest raw throughput NVIDIA hardware can deliver, provided you can tolerate the compile cycle and accept that you're not going anywhere else for hardware.

What's your actual traffic pattern doing to these numbers — has anyone here run vLLM and SGLang side by side on their own production prefix-sharing rate instead of trusting a generic H100 benchmark, and did the gap look anything like 29%?

Sources:

DEV Community