A post went around earlier this month, 82 to 381 tok/s on a single RTX 3090 running a 27B model. I have that card and it's never idle, a coder model lives on it for my agent sessions, so I installed the stack and measured it on my own bench. Twice, because the first attempt died on a BIOS setting.
The stack is HyperQwen (syv-ai on GitHub), an Apache-2.0 vLLM build tuned for Qwen3.8-27B at W4A16, about 19.5GB of weights, with DFlash2 speculative decoding. The baseline is the same 27B model as a Q4_K_M GGUF under llama.cpp, ~42 tok/s on the same card. That's the number from October 11.
The first attempt died in a BIOS menu
The box ran Windows in early October. The stack is Linux only, no native Windows path, so the way in was Docker Desktop plus WSL2. I staged everything in one night. WSL and VirtualMachinePlatform enabled, repo cloned, model downloading, Docker installed, a task armed to bring the server up after the next reboot, plus a watcher on a 10-minute loop that would park my inference crons and run the first bench automatically once the hypervisor appeared.
The dry run printed hypervisor present: False ... enable it in BIOS and stopped. Virtualization was off in firmware. Three separate checks agreed. Turning it on is a physical act: DEL, F7 for advanced, CPU config, SVM mode, enable, F10. Everything staged, armed, blocked on me walking over to a keyboard.
Then the box got wiped and rebuilt on Arch. The Windows staging died with the old OS, and the BIOS blocker died with it. Native Docker doesn't need a hypervisor.
Ten days later, a fourteen-minute window
The revival was mostly plumbing. Clone to ~/HyperQwen, pull a 9.5GB image, fetch the weights, key into .env, SPEC=dflash2, PREFIX_CACHE=1. The constraint is the card. A 24GB 3090 fits one 21GB-class model, and mine was occupied. Strata, my coder quant, holds 21.6GB with ttl 0 and llama-swap proxies it on :8080, so a second big model can only exist in a window.
So I checked traffic first. The only connection was cron traffic, no session held the model lock, safe to interrupt. I paused seven recurring agent crons, stopped the resident model, brought HyperQwen up on :18020, and benched it. Fourteen minutes from stopping the old tenant to numbers on disk, healthy in about three.
The numbers on my box
Greedy chat on thinking-heavy prompts, the worst case for speculative decoding: 118.1 tok/s mean across four prompts, 111 to 131. Sampled at temp 0.7: 123.3. The project's own figure for single-user chat is around 127. My box is in band.
Context quoting, a 29k-token prompt where the model reproduces a section verbatim: 237.6 tok/s decode. Verbatim check PASS. Prefill ~1,210 tok/s, about 24 seconds for that prompt, roughly five times what llama.cpp does here (~250). Boot to healthy: ~3 minutes with the first JIT compile, ~1 minute warm.
Against the GGUF baseline of 42: three times the speed on real chat, six times on quoting, both measured at a 420W power limit, driver 610.57, CUDA 13.3.
Where 381 comes from, and where it dies
The headline number needs a non-default setting, DFLASH_TOKENS=15, a longer draft block for the speculative decoder. I tried it. Chat stayed in the same band, 115.5 and 123.4. Then the quote request hit 20k context and the engine died. EngineDeadError, a marlin_gemm dispatcher allocation failure, aten::empty call failed on the int4 draft kernels. Every request after that 500s until the container restarts, and a restarted engine can still 500 until it's restarted again. It never recovers on its own. The trace is in the docker logs.
So the map is specific. The 82 in the post is a stock vLLM baseline, the 381 is the quoting workload only, and on my box the setting that produces it crashes at the context sizes quoting needs. I reverted to the default 7 drafts. That mode stays off-limits until upstream fixes the allocator.
The counter lied first
My first pass counted the stream chunks and read 43 tok/s. The measured mean is 118. The count was off by nearly three times, and it would have made the stack look slower than the GGUF it replaces.
Speculative decoding changes the shape of the stream. vLLM doesn't emit one token per SSE chunk, it emits whole accepted batches, about 2.5 tokens per chunk at 7 drafts. Count chunks and you undercount by that factor. The guard is reading usage.completion_tokens from the final chunk, with stream_options: {"include_usage": true} in the request. Same server, same prompt, the count matches the engine's own accounting.
There's a second trap in the same pipe. The model streams its thinking under delta.reasoning, not reasoning_content, and it thinks on every prompt. max_tokens headroom isn't optional. The thinking budget will eat the answer.
One tenant per 24GB card
Three operational rules, each one paid for in a lost afternoon.
Warm the server before trusting any number. The compose entrypoint doesn't auto-warm (WARMUP=0 is the default), and the first requests compile serving-path kernels, so cold numbers are junk. bench/warmup.sh after every boot.
Pause the frequent agent crons for any window. llama-swap burns a 420-second health-check timeout per request against a member that's dead, and my cron agents started erroring mid-run.
Then the rule that shapes everything: systemctl stop strata also stops llama-swap, because llama-swap carries Requires=strata.service. Starting llama-swap auto-starts Strata, and the proxy can never load a GGUF member while Strata runs, because Strata always holds the VRAM. HyperQwen windows bypass the proxy, direct on :18020, and :8080 is down for the duration. One big model at a time is the physical law of a 24GB card. The proxy config just makes it visible.
What runs now
Steady state: HyperQwen is down. Strata is back resident and serving, verified with a real completion before I called it done, all seven crons resumed, and the old Windows watcher cron from the first attempt is retired for good. The wire-in to make HyperQwen an on-demand llama-swap member is written and staged, waiting on a routing decision before it goes in.
Anyone chasing the 381 number should bring the workload they actually run. The quoting path flies, chat runs about three times the GGUF path, and the setting behind the headline crashed my engine at the exact context size the headline needs. Bring your own prompts and count tokens from usage.
Hardware, Ryzen 7 5800X, 64GB DDR4, RTX 3090 24GB, October 2026, single user, four chat prompts and one quoting workload, one sitting.
Top comments (1)
Thank you for benching the worst case instead of the demo case. The 381 → 118 gap isn't the stack lying — it's that speculative decoding's acceptance rate collapses on thinking-heavy, low-entropy-but-unpredictable output, which is exactly the traffic a coder model sees. Draft-token acceptance is the number people should report alongside tok/s, because it's what actually moved between the two measurements.
The "card was already occupied" constraint is the real-world part most writeups skip. A 24GB 3090 holding a 21.6GB resident model means the headline mode never had room, and that's not a footnote — it's the deployment reality for anyone self-hosting on one GPU. The 14-minute swap window you describe (pause crons, drop the resident model, bench, restore) is basically manual KV/VRAM scheduling by hand.
Did you measure acceptance rate directly, or infer it from the speedup? And on the crash in headline mode — was it an OOM at the KV ceiling, or did the speculative path itself fault? Those fail very differently and only one is fixable by quantizing harder.