Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

TL;DR

• Published speculative decoding speedups (2x to 3x) are almost always measured at low concurrency. The two vLLM posts I read this week report no concurrency sweep.
• Speculation spends spare compute to save memory traffic. When the verify pass stops being memory-bound, the spare compute is gone and the extra draft tokens become pure overhead.
• In an illustrative roofline model (27B dense verifier, H100-class GPU, 2k context), verifying 8 tokens per sequence turns compute-bound at about 30 concurrent sequences. Verifying 4 tokens per sequence holds out to about 85.
• The throughput-optimal speculation length falls from 8 at batch 1 to 3 at batch 128. A fixed num_speculative_tokens is the wrong setting for a server whose load changes during the day.
• The model overstates absolute speedups. Use it for the shape of the curve, then measure your own server with the sweep at the end.

Why do the headline speedups all come from batch 1?

Decode is memory-bound at small batch. Every step streams the full set of weights from HBM to produce one token per sequence, and the arithmetic units sit mostly idle. Speculative decoding exploits that idle compute: a cheap drafter proposes several tokens, and the large model verifies all of them in a single forward pass. Rejection sampling keeps the output distribution identical to ordinary decoding, so quality is unchanged (Leviathan et al., 2022).

The newest drafters make the drafting side cheaper. The vLLM Speculators team describes three parallel drafting methods: P-EAGLE, DFlash and DSpark. Instead of generating draft tokens one forward pass at a time like EAGLE-3, they predict a whole block of candidates in one pass, which decouples drafting latency from speculation length (vLLM blog, July 2026). The example launch command in that post looks like this:

vllm serve Qwen/Qwen3-30B-A3B \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --speculative-config '{
    "model": "RedHatAI/Qwen3-30B-A3B-speculator.dflash",
    "num_speculative_tokens": 7,
    "method": "dflash"
  }'

A follow-up post on AMD GPUs reports measured speedups on MI300X and MI355X: 2.87x for DFlash on Gemma-4-26B-A4B (MATH500) and 2.34x on Gemma-4-31B. The spread across tasks is wide. DSpark on Qwen3-8B ranged from 1.15x on MATH500 to 1.63x on GSM8K (vLLM blog, August 2026). That post also notes that a broader evaluation across concurrency levels and batch sizes would help, which is the gap this article tries to fill with arithmetic rather than a new benchmark.

None of this means the technique is overrated. It means the batch size at which a number was measured belongs next to the number.

What does one speculative iteration actually cost?

The original analysis gives the expected number of tokens produced per verify pass when each draft token is accepted independently with probability alpha and the drafter proposes gamma tokens:

E(alpha, gamma) = (1 - alpha^(gamma + 1)) / (1 - alpha)

With alpha = 0.8 and gamma = 7 that is 4.16 tokens per iteration. The iteration is not free. It costs one verify pass over (gamma + 1) tokens per sequence plus one drafter pass. Throughput speedup is therefore:

speedup(B) = E * t_base(B) / ( t_verify(B, gamma + 1) + t_draft(B, gamma) )

Everything interesting hides in how t_verify grows with batch size B. At small B it barely grows, because the step is bound by weight and KV-cache reads that do not depend on how many tokens you verify. At large B it grows almost linearly with (gamma + 1), because the step is bound by compute.

How does the roofline model work?

I built the smallest model that captures that transition. Treat every number below as illustrative, not as a measurement of any specific deployment.

Assumption	Value
Verifier	27B dense, BF16 (54 GB of weights)
GPU	H100 SXM class: 3.35 TB/s HBM3, 989 TFLOPS BF16 dense (NVIDIA H100)
Achievable fraction	80% of peak bandwidth, 50% of peak compute
KV cache	64 layers, 8 KV heads, head dim 128, BF16: 262,144 bytes per token
Context	2,048 tokens per sequence (0.54 GB of KV read per step per sequence)
Drafter	Parallel drafter, 4% of verifier parameters, one pass per iteration
Step time	max(memory time, compute time)

The core of the script is short:

def t_step(B, tokens_per_seq, frac=1.0):
    mem = (W * frac + B * KV_SEQ) / BW          # weights + KV reads
    cmp_ = 2 * P * frac * B * tokens_per_seq / FL
    return max(mem, cmp_)

def speedup(B, gamma, expected_tokens):
    base = t_step(B, 1)
    spec = t_step(B, gamma + 1) + t_step(B, gamma, 0.04)
    return expected_tokens * base / spec

One closed form falls out of the model. With weight-read time t_w (20.1 ms here), KV-read time per sequence k (0.20 ms), and compute time per token c (0.109 ms), the verify pass becomes compute-bound at:

B* = t_w / ((gamma + 1) * c - k)

gamma	verify turns compute-bound at
1	about 1,100 sequences
3	about 85 sequences
5	about 44 sequences
7	about 30 sequences

Past B*, every extra draft token costs real GPU time. Below it, extra draft tokens are almost free. That single threshold explains the rest of the article.

Why should you not trust an independent-acceptance model for long drafts?

The formula above assumes every position is accepted with the same probability. Real drafters degrade with distance. The AMD post reports, for Gemma-4-26B with Gemma 4 MTP at N=5, a 94% first-position acceptance, 66% at the fifth position, and 80.0% overall. Under independent acceptance at 80%, gamma = 7 gives 4.16 tokens per iteration. If acceptance declines linearly from 94% by 7 points per position (my interpolation between the two reported endpoints, treating each as a conditional acceptance), gamma = 7 gives 4.49 tokens, but the marginal gain from each extra position collapses:

gamma	expected tokens per iteration (decaying acceptance)
1	1.94
2	2.76
3	3.41
5	4.20
7	4.49
8	4.53

Going from gamma = 5 to gamma = 8 buys 0.33 tokens per iteration while adding 3 more verified tokens per sequence. At low batch that trade is free. At high batch it is not.

What does the speedup curve look like as concurrency grows?

I ran the model with the decaying-acceptance assumption. Values are throughput relative to plain decoding at the same batch size.

Concurrent sequences	gamma = 2	gamma = 4	gamma = 7	Best gamma (speedup)
1	2.63x	3.71x	4.28x	8 (4.32x)
8	2.48x	3.50x	4.04x	8 (4.08x)
32	2.17x	3.06x	3.39x	6 (3.45x)
64	1.95x	2.64x	2.13x	4 (2.64x)
128	1.75x	1.85x	1.49x	3 (1.90x)
256	1.45x	1.45x	1.16x	3 (1.49x)

Three things stand out.

1. The curves cross. The long draft (gamma = 7) wins until about 32 sequences, then falls behind the short one by 64. That matches the closed form: gamma = 7 turns compute-bound near 30 sequences.
2. The optimum moves. The best speculation length is 8 up to 16 sequences, 6 at 32, 4 at 64, and 3 from 128 up.
3. Speculation rarely turns negative in this model, but it can. With the decaying acceptance curve, gamma = 7 first drops below 1.0x at about 513 sequences and gamma = 8 at about 329. With independent acceptance at alpha = 0.6, gamma = 7 breaks even at only 84 sequences. A weaker drafter or a harder task moves the break-even left fast.

What does this model leave out?

Be suspicious of the absolute numbers. A 4.3x speedup at batch 1 is higher than the 2x to 3x measured in the posts above, because the model ignores scheduler overhead, drafter launch latency, sampling, and the real shape of acceptance across tasks. It also has these specific blind spots:

• Mixture-of-experts verifiers. A dense model touches all weights at every step. An MoE touches only the experts routed to. Verifying more tokens per sequence activates more distinct experts, so the verify pass reads more weight bytes as gamma grows. That shifts the break-even left compared with a dense model of the same total size. I did not model it.
• Longer context. The KV term k grows with context length. A larger k makes the base step more memory-bound for longer, which pushes B* to the right (the denominator in the closed form shrinks). Long-context workloads can tolerate longer drafts at a given batch size.
• Prefix caching and chunked prefill. Prefill chunks share each step with decode tokens and consume compute that the verify pass would otherwise use. The model treats decode in isolation.
• Drafter quality varies by traffic. Acceptance on code and math is usually higher than on open-ended chat. A single alpha across your traffic mix is a fiction.

How do you measure this on your own server?

Sweep concurrency with speculation on and off, and log acceptance per position. Use identical prompts and output lengths for both arms, and warm any prefix cache the same way before each run so one arm does not inherit a hotter cache. Flag names below match recent vLLM; check vllm bench serve --help on your build.

# Arm A: speculation off.  Arm B: same command plus --speculative-config.
for C in 1 8 32 64 128 256; do
  vllm bench serve --model "$MODEL" --dataset-name random \
    --random-input-len 1024 --random-output-len 256 \
    --num-prompts $((C * 8)) --max-concurrency $C \
    --save-result --result-filename "specoff_c${C}.json"
done

Then read the acceptance counters from the server's Prometheus endpoint. Metric names starting with vllm:spec_decode carry draft and accepted token counts, and recent builds expose accepted tokens per position:

curl -s localhost:8000/metrics | grep '^vllm:spec_decode'

Compute accepted tokens per position as a conditional rate (position k accepted given k-1 accepted) before fitting any model. If conditional acceptance at position 6 is below 50%, a gamma of 7 is probably costing you at any load above light.

Two more rules from earlier posts in this series apply here. Record the effective configuration the engine reports rather than the one you intended, and count failed or timed-out requests as outcomes instead of dropping them from the denominator.

What is a practical tuning rule?

1. Compute B* for your hardware and context length with the closed form, using your measured weight-read time per step and measured per-token compute time.
2. Pick gamma so that B* for that gamma sits above your p95 concurrency per replica, then verify with the sweep.
3. If load swings widely, either route low-concurrency traffic to a speculative replica and high-concurrency traffic to a plain one, or check whether your engine can shorten or skip speculation dynamically under load. Do not assume it does; look for it in the release notes of your version.
4. Re-measure whenever the drafter, the model or the traffic mix changes. Acceptance is a property of the traffic, not of the model card.

FAQ

Does speculative decoding change the output of the model?
No. With rejection sampling, the output distribution matches ordinary decoding exactly, so quality is mathematically identical (Leviathan et al.; the vLLM Speculators post makes the same statement). What changes is latency and throughput.

Why does speculative decoding help less at high batch size?
At high batch size the GPU is already busy producing tokens for many sequences, so there is little idle compute to spend on verification. Each extra draft token then costs real compute time while the number of accepted tokens per iteration saturates.

What speculation length should I use in vLLM?
Start from the value in the drafter's model card (the vLLM example above uses 7 for DFlash), then sweep lower values under your real concurrency. In the illustrative model the best value fell from 8 at batch 1 to 3 at batch 128.

Are parallel drafters like DFlash and P-EAGLE better than EAGLE-3 under load?
They make drafting cheaper because one drafter pass covers the whole block, which helps when drafting latency matters. They do not change the verify pass, so the compute-bound threshold for the verifier is unchanged. Compare them with your own concurrency sweep instead of relying on a single-batch number.

Can speculative decoding make throughput worse?
Yes, when the verify pass is compute-bound and acceptance is low. In this model, a weak drafter (alpha of 0.6) with gamma = 7 falls below plain decoding at about 84 concurrent sequences under independent acceptance.

Further reading

• Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off
• Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)

Sources

• Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding
• vLLM blog. Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding (July 28, 2026)
• vLLM blog. Exploring Speculative Decoding in vLLM on AMD GPUs (August 23, 2026)
• NVIDIA. H100 Tensor Core GPU (peak bandwidth and compute figures)
• All model outputs in this article are illustrative calculations from the stated assumptions, not measurements.