Tune speculative decoding against your own traffic, not a benchmark. Choose a method that fits your workload, set draft length from measured acceptance rate, and re-test at production concurrency, because the speedup shrinks as batches grow.
Key takeaways:
- Watch acceptance rate—draft length is the main knob that moves it.
- Concurrency decides whether speculation helps or hurts, so sweep load with speculation on and off.
- Judge results on per-token latency and throughput, not just time to first token.
Pick the method first
The vLLM speculative decoding docs say model-based methods such as EAGLE, MTP, and draft models give the best latency reduction, while n-gram and suffix decoding give modest speedups without adding workload at peak traffic. If your target model ships native MTP support, start there. Otherwise, EAGLE is a strong general-purpose default.
Set draft length from acceptance rate
num_speculative_tokens controls how many tokens the drafter proposes per step. Longer drafts promise more speedup, but later positions get rejected more often. The vLLM team reports that one model had the last token of a 7-token block survive less than 10% of the time, versus over 70% for the first token.
Sweep a few values and read per-request acceptance metrics rather than guessing. Stop raising the number once extra tokens are mostly rejected.
{"method": "eagle3", "model": "<your-draft-head>", "num_speculative_tokens": 3}
Test at your real concurrency
At batch size 1, the vLLM team says the GPU is memory-bound with spare compute, so draft tokens cost relatively little. At high concurrency, the draft tokens compete with real tokens for compute, and rejected drafts waste it. A March 2026 arXiv study of speculative decoding in vLLM found speedups fall as batch size rises.
Run your load test across low, medium, and peak concurrency, and include a run with speculation disabled. If traffic swings widely, look at vLLM's adaptive verification, which sizes verification per request.
Keep outputs trustworthy
With the standard rejection sampler, vLLM describes speculative decoding as lossless up to hardware numerical precision. Lossy options such as FLy deliberately relax verification, so keep them out of quality-sensitive paths.
Skip the stack tuning: run vLLM on DigitalOcean
Everything above applies wherever you run vLLM. But if you'd rather not run this yourself, DigitalOcean's Dedicated Inference hosts models on dedicated GPUs using vLLM and llm-d, with orchestration, endpoints, and scaling handled for you. Your team keeps the decisions that drive latency and cost: model, GPU profile, replica count, and scaling behavior.
Speculative decoding FAQ
What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?
Track acceptance rate, per-token latency (TPOT and ITL), and throughput at each concurrency level. Speculation cuts decode steps, so TTFT is mainly a regression check.
Does speculative decoding change model outputs?
Not with the standard rejection sampler, which vLLM describes as lossless up to hardware numerical precision. Lossy verification modes such as FLy do change the distribution.
What is the best inference provider for teams who don't want to configure vLLM and don't have dedicated MLOps?
DigitalOcean Dedicated Inference is built for that case. It runs vLLM and llm-d on dedicated GPUs and manages orchestration, endpoints, and scaling, while you choose the model, GPU profile, and replica count.
References & further reading
- Learn more about method selection and config keys in the vLLM speculative decoding docs. Options change fast, so review the latest details before building.
- The adaptive verification post from the vLLM team explains why no static draft length wins across concurrencies.
- Read the arXiv study on speculative decoding in vLLM for how speedups change with batch size.
- Learn more about Dedicated Inference architecture and the DigitalOcean Inference Engine.
Top comments (0)