One Knob, 19x Latency: What I Learned Benchmarking vLLM on a Single GPU

I built a small serving harness that measures latency, throughput, AND output drift per config. It found a 19x latency cliff on a single knob, a 3.6x model speed gap — and a reasoning model that can't reproduce its own outputs at temperature 0.

---

Scope note, first paragraph as promised: one RTX 3060, two models (Swallow-8B AWQ, llm-jp-1.8B bf16), short prompts, synthetic Poisson arrivals — not production traces. No human judged anything; drift is exact output-hash match.

Gentle benchmarks lie

First surprise: max-num-seqs 32 vs 128 showed identical numbers across fixed-concurrency and Poisson loads. The server never broke a sweat, so the knob looked irrelevant. It isn't — my load was just too polite. I added a stress arm (64 concurrent, 256 output tokens) and the same comparison became TTFT p50 8.3s vs 0.43s. Same models, same prompts, one knob: 19x. TPOT stayed flat throughout (16→28ms) — textbook prefill queueing. Lesson I'll reuse everywhere: config comparisons REQUIRE a stress arm; gentle benchmarks can't tell servers apart.

Small models win big

Same stress, different model: llm-jp-1.8B serves at 119ms TTFT vs Swallow-8B's 432ms, 28.5 vs 15.1 req/s. Unsurprising direction, useful magnitude — and a deployment footnote: llm-jp runs under vLLM's own kernels while stock transformers trips on its RoPE config. Know your stack's two paths.

Temperature 0 is not deterministic (for reasoning models)

The quality probe — 50 frozen prompts hashed per config, there to catch quality regressions — caught something else: the same Swallow server, identical requests, 10/50 outputs differ run to run (12/50 across restarts). A non-reasoning model on the same harness: identical ×3. Think-mode sampling leaks through temperature 0. If your production system assumes deterministic serving (caching, evals, replay), verify it per model — don't assume it.

The harness

Asyncio+aiohttp client, fixed + Poisson loads, TTFT/TPOT p50/p95, error rates, per-run JSONL, frozen quality set. Client-only venv; the server lives in sibling venvs. 3 tests against a fake SSE server run in CI (no GPU needed). All numbers, prompts, and hashes are linked below.

Comments

[Max Quimby]

The "gentle benchmarks lie" point deserves to be pinned somewhere. We lost a week once chasing a config that looked identical under fixed-concurrency load and only diverged once we added a bursty arrival arm — exactly your max-num-seqs story. TTFT is a queueing metric, so if your offered load never fills the queue you're benchmarking an idle server and calling it a config comparison.

Two things I'd add from running vLLM in anger: (1) p50 hides the cliff — the 8.3s number usually shows up in p95/p99 first, so watch the tail before the median moves; (2) the temperature-0 non-determinism on reasoning models bit us in eval replay too. We assumed a frozen seed + temp 0 = reproducible and built a cache key on it. It wasn't, for the same reason you found — sampling still leaks through think-mode. Did you see the drift correlate with output length, or was it uniform across the 50 prompts? Curious whether longer reasoning traces diverge more.