I turned on vLLM speculative decoding on a Friday afternoon, ran my usual one-request benchmark, and watched tokens per second jump from about 48 to about 88. I wrote "1.8x free speedup" in the team channel. On Monday, with real traffic hitting the box at around 32 concurrent requests, aggregate throughput was lower than before I touched anything, and p50 inter-token latency had gone up.
Nothing was broken. Speculative decoding did exactly what it is designed to do. I just measured it in the one regime where it always looks good.
TL;DR
- Speculative decoding has a small draft model guess the next k tokens, then the big model checks all k in one forward pass and keeps the prefix it agrees with.
- It is fast at low concurrency because decoding is memory-bandwidth-bound: checking 5 tokens costs roughly the same as generating 1.
- The whole win depends on the acceptance rate. Below roughly 0.5 per token (k=4 with a 1B draft on an 8B target), the draft work eats the gain.
- Under load the GPU drifts toward compute-bound, so rejected draft tokens become real wasted FLOPs, and the draft model's weights and KV cache take memory from other requests.
- Measure acceptance on production traffic, use n-gram (prompt lookup) drafting for copy-heavy workloads, lower k, and keep speculation off for high-concurrency deployments.
What actually happened on my server?
The setup: Llama 3.1 8B Instruct as the target, Llama 3.2 1B Instruct as the draft (same tokenizer, which is required), one 48GB GPU, num_speculative_tokens set to 4.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{"model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 4}'
Two workloads share this box: a JSON extraction service (temperature 0, lots of text copied from the input) and an internal chat assistant (temperature 0.8). My Friday benchmark was a single extraction prompt. Monday was both workloads mixed at around 32 concurrent sequences.
Roughly, on my hardware:
| Scenario | Without spec decoding | With spec decoding |
|---|---|---|
| 1 request, extraction | ~48 tok/s | ~88 tok/s |
| 1 request, chat (temp 0.8) | ~48 tok/s | ~47 tok/s |
| ~32 concurrent, mixed | baseline | ~15% lower aggregate throughput |
The second row was the first clue. Even with a single user, chat gained nothing. To see why, you need the mechanism.
How does speculative decoding actually work?
Normal decoding runs the big model once per token. Speculative decoding splits each step in two:
- The draft model (small and cheap) generates k tokens autoregressively: t1, t2, t3, t4.
- The target model runs one forward pass over all k positions at once and computes its own distribution at each.
- Walk left to right. Accept t1 if the target agrees, then t2, and so on. At the first rejection, throw away that token and everything after it, and take the target's own token at that position instead.
So every step yields at least 1 token (the target's correction) and at most k+1 (all drafts accepted, plus one bonus token from the last position).
At temperature 0, "agrees" means the target's argmax matches the draft token. At temperature above 0, vLLM uses rejection sampling: accept with probability min(1, p_target / p_draft), and on rejection resample from the leftover distribution. This keeps the output distribution identical to the target model alone. Speculative decoding changes speed, never quality. That guarantee is what makes it easy to turn on without thinking, which is how I ended up here.
Why is speculative decoding faster at batch size 1?
Because generating one token for one user barely uses the GPU's math units. Every decode step has to stream all of the model's weights from memory. An 8B model in fp16 is about 16GB. On a GPU with around 860 GB/s of memory bandwidth, that is about 19ms per step as a floor, no matter how little math you do.
Verifying 5 positions instead of 1 reuses the same weight read. The extra matrix math is tiny next to the memory traffic, so the verification pass costs close to the same 19ms. The draft model reads its own 2.5GB of weights per token, about 3ms each.
That gives a simple cost model for one step with k=4:
- 4 draft forwards: ~12ms
- 1 target verification: ~19ms
- Total: ~31ms, versus 19ms for one plain token
You pay 31ms per step. You break even only if you get more than 31/19 ≈ 1.6 tokens per step.
What is the acceptance rate, and why does it decide everything?
Acceptance rate (α) is the probability the target accepts any given draft token. If you treat each position as independent, the expected tokens per step is:
E[tokens per step] = (1 - α^(k+1)) / (1 - α)
Plug in k=4 and the timings above:
def tokens_per_step(alpha, k):
return (1 - alpha ** (k + 1)) / (1 - alpha)
def speedup(alpha, k, t_target=19.0, t_draft=3.0):
step_ms = t_target + k * t_draft
return tokens_per_step(alpha, k) * t_target / step_ms
for a in (0.3, 0.5, 0.7, 0.9):
print(a, round(tokens_per_step(a, 4), 2), round(speedup(a, 4), 2))
| α | tokens per step | speedup |
|---|---|---|
| 0.3 | 1.42 | 0.87x |
| 0.5 | 1.94 | 1.19x |
| 0.7 | 2.77 | 1.70x |
| 0.9 | 4.10 | 2.51x |
Below about 0.45, you lose. My extraction traffic sat near 0.7: the output is mostly field values copied from the input, and a 1B model guesses those easily. My chat traffic at temperature 0.8 sat near 0.4. Higher temperature flattens both distributions, the draft and target disagree more often, and the sampled acceptance drops. That explains row two of my table without any load at all.
One trap in reading vLLM's own numbers: the /metrics endpoint exposes spec_decode_* counters (exact names shift between versions, so grep for spec_decode). The ratio accepted tokens / draft tokens is not α. With k=4 and α=0.7, you accept about 1.77 of 4 drafts on average, so that ratio reads 0.44 and looks terrible even though the speedup is 1.7x. Compute mean accepted length instead:
curl -s localhost:8000/metrics | grep spec_decode
# mean accepted per step = num_accepted_tokens / num_drafts
# tokens per step = that + 1
Then compare tokens per step against your own break-even (step cost divided by plain token cost).
Why does vLLM speculative decoding get slower under load?
Under load, the memory-bound assumption stops holding, and three costs show up at once.
1. Verification is no longer free. With 32 sequences and k=4, the target's verification pass processes 32 × 5 = 160 token positions per forward instead of 32. In fp16, each token does about 2 FLOPs per parameter while the weights cost 2 bytes each, so arithmetic intensity is roughly 1 FLOP per byte per token in the batch. A modern GPU's ridge point (peak FLOPs divided by bandwidth) sits in the hundreds. Real kernels hit the compute wall well before the theoretical ridge, so 160 positions per pass is already in the zone where extra positions cost real time. Every rejected draft token is now math you paid for and threw away, and that math was taken from other users.
2. The draft model scales too. Four sequential draft forwards per step don't disappear at batch 32. They're small, but they're four extra kernel rounds per step in the critical path for every sequence.
3. Memory goes to the draft. The draft's weights and its own KV cache come out of the same GPU memory pool. Fewer KV cache blocks means fewer sequences fit at once, so under real traffic more requests wait in the queue. Throughput can drop even before any compute math enters the picture.
Without speculation, batching already solved the memory-bandwidth problem. Thirty-two users share one weight read per step. Speculative decoding attacks the same inefficiency, and once batching has eaten it, you're left with only the overhead.
When should you use speculative decoding in vLLM?
Use it when concurrency is low and acceptance is high. In practice:
- Measure α on production traffic, not on your favorite prompt. My benchmark prompt was the best case I could have chosen by accident.
- Try n-gram drafting first for copy-heavy work. Extraction, code edits, and RAG answers that quote the context repeat input spans. Prompt lookup proposes tokens by matching n-grams from the prompt, so it needs no draft model, no extra weights, and no extra KV cache:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'
- Lower k when α is mediocre. Each additional draft position is worth α^i, which decays fast. At α=0.5, the 4th draft token adds about 0.06 expected tokens for a full draft forward of cost.
- Split deployments by traffic shape. I moved extraction to a small low-concurrency instance with n-gram speculation on, and kept the chat pool on plain decoding with a bigger batch. Some older vLLM versions had a setting to disable speculation above a batch size threshold; check what your version supports before relying on it.
- Look at trained draft heads like EAGLE or Medusa if you need speculation on chat-style traffic. They predict from the target's own hidden states and usually agree with it more often than a separate small model. Measure them the same way.
So, does vLLM speculative decoding make inference faster?
vLLM speculative decoding makes inference faster only when the GPU is memory-bound and the acceptance rate is high: low concurrency, low temperature, and outputs that repeat or closely follow the input. In that regime, verifying k draft tokens costs about the same as one normal step, so a per-token acceptance of 0.7 with k=4 gives roughly 1.7x. Under load, batching has already used up the idle memory bandwidth, verification turns into real compute, rejected drafts waste it, and the draft model takes KV cache memory from other requests, so aggregate throughput can drop. Measure mean accepted tokens per step from /metrics on real traffic, compare it to your break-even, and turn speculation on only where the numbers clear it.
Written by the developer behind Preterview, an interview prep platform.
Top comments (1)
This is why concurrency belongs in every inference-performance claim. A speedup that helps a single stream can lose to queueing, cache pressure, or scheduler overhead under load. Publishing throughput, tail latency, batch mix, and GPU utilization together would make the trade-off actionable for teams sizing real services.