Big models are where inference engines show off. There's a lot of room to win.
27B is harder. The model is small, the GPU isn't the bottleneck in the same way, and every engine starts to look alike.
So that's where we went next. And we brought SGLang along.
The setup
- Model: Qwen3.8-27B, BF16 weights with FP32 accumulation
- Hardware: one NVIDIA RTX PRO 6000 Blackwell
- Baselines: vLLM 0.33 and SGLang, both with speculative decoding on
- Workload: 1K and 8K input, 1K output, concurrency 1, 4, 16 and 64
- Harness: InferenceX, seed 0, so every engine got the same prompts
Output throughput
| Input | Concurrency | vs vLLM 0.33 | vs SGLang |
|---|---|---|---|
| 1K | 1 | +31% | +26% |
| 1K | 4 | +26% | +21% |
| 1K | 16 | +27% | +20% |
| 1K | 64 | +44% | +4% |
| 8K | 1 | +34% | +22% |
| 8K | 4 | +29% | +22% |
| 8K | 16 | +22% | +14% |
| 8K | 64 | +16% | +7% |
Ahead of both engines at every one of the eight points.
Latency
- Median time per output token: 4% to 34% better than vLLM, 4% to 27% better than SGLang
- Median end-to-end latency: 17% to 45% better than vLLM, 4% to 27% better than SGLang
- Every mean, p50, p90 and p99 end-to-end comparison went our way against both engines
- Median time to first token at 1K input and 64 concurrent requests: 44× faster than vLLM
The scorecard
8 workload points. 13 metrics each. 104 head-to-head comparisons per engine.
- vs vLLM 0.33: 104 of 104
- vs SGLang: 102 of 104
SGLang is a very good engine. Beating it on 102 of 104 at the size where engines bunch up the most is the result I'm proudest of from this round.
How we kept it full precision
Somebody will ask, so here it is.
We use a proprietary 4-bit draft head to propose tokens. The full BF16 model verifies every single one. Nothing gets through that the full model didn't sign off on.
Add an INT8-bound LM head and our own decode GEMM kernels. That's the engine.
One configuration for both comparisons. We didn't retune per competitor.
Read the full report
Configs for all three engines and the full run procedure are in the Qwen3.8-27B report on our blog.
Think your SGLang or vLLM setup does better? Bring it. We'll run it side by side.
Top comments (0)