Benchmarks of the same model on the same GPU across three serving stacks, then an FP8 pass on the winner. All numbers measured on our own hardware last week. Raw CSVs, the environment manifest and a one-command reproduction script exist for every figure; the script was re-run end to end after the report was written and all figures reproduced within 6 percent.
Setup
- GPU: RTX PRO 6000 Blackwell, 96 GB, sm_120 (workstation Blackwell, not the datacenter parts)
- Model: Qwen3-8B
- Stacks: vLLM 0.27.1, SGLang 0.5.9, llama.cpp (CUDA build)
- Method: identical prompt set and sampling settings across engines, greedy decoding, output token counts matched across engines before any timing was compared. Concurrency 32 for the aggregate numbers.
Matching token counts before comparing throughput matters more than it sounds. Engines disagree about stop conditions and detokenization, and a 10 percent difference in generated tokens quietly becomes a 10 percent throughput lie.
BF16, concurrency 32
| stack | aggregate tok/s | TTFT p50 | e2e p99 |
|---|---|---|---|
| vLLM 0.27.1 | 1,725 | 39 ms | 3.4 s |
| SGLang 0.5.9 | 1,327 | 42 ms | 5.0 s |
| llama.cpp (CUDA) | 428 | 316 ms | 16.3 s |
FP8 on the winner (vLLM, same settings)
| precision | single-stream tok/s | batch tok/s | latency p50 |
|---|---|---|---|
| BF16 | 86 | 1,725 | 0.74 s |
| FP8 (official checkpoint) | 130 | 2,597 | 0.49 s |
What the numbers say
Stack choice barely matters for one user. Single-stream decode sits in the 83 to 96 tok/s band on every stack. If your service has one concurrent user, pick whatever is easiest to operate.
Stack choice is a 4x decision under load. At concurrency 32 the spread is 428 to 1,725 tok/s for the same model on the same silicon. Time to first token spreads 4x as well (39 ms vs 316 ms p50), which dominates perceived snappiness for chat workloads.
FP8 was a clean 1.5x. 1,725 to 2,597 tok/s aggregate, 86 to 130 single-stream, p50 latency down a third. We ran a fixed 20-prompt factual check before and after: zero regressions. For an 8B model on a 96 GB card the VRAM saving is irrelevant, the win is throughput.
The sm_120 wrinkle
Workstation Blackwell is not datacenter Blackwell, and default paths tuned for the latter sometimes assert on the former. Getting FP8 to run here required routing around a kernel assertion in the default FP8 path (DeepGEMM asserting on sm_120, resolved by falling back to the CUTLASS path). Related example of the same hardware-class gap: vllm-project/vllm#53748, where the Triton MLA decode kernel overflows the 101,376-byte per-block shared-memory limit that GB10 and workstation Blackwell share, but datacenter parts do not.
If you are serving on RTX PRO 6000, GB10 or consumer Blackwell, budget time for this class of issue. The performance is there, the default configs often are not.
Method notes
- Greedy decoding for determinism; throughput measured after warmup
- Aggregate numbers are steady-state over the full prompt set, not peak
- Each engine got the same max batch and context settings where the knob exists
Full methodology and the case study writeup: conatus.jahn.ai/ai-engineering
Top comments (0)