Speculative decoding makes a large language model produce text faster by letting a cheaper process guess several tokens ahead while the big model checks the guesses all at once. The original paper, from Google researchers, reports a 2X to 3X speedup on a T5-XXL model with identical outputs. Later methods such as Medusa and EAGLE-3 report larger gains, under their own test conditions.
Why one token at a time is slow
A model predicts a token, adds it to the text, and predicts the next, so the steps are strictly sequential. The expensive part is moving data. According to the Medusa paper, each decoding step moves the full set of model parameters out of GPU memory, so the chip waits on data and most of its compute sits idle. Scoring several candidate tokens in one pass costs little extra, because the parameters are read once for the whole group.
How speculative decoding works
Introduced in Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman and Matias, November 2022):
- A small, fast draft model proposes the next few tokens.
- The large target model scores all of them in a single forward pass.
- The target keeps proposals up to the first disagreement and supplies its own token there.
When the draft is right, several tokens arrive for the price of one big-model step. When it is wrong, that step runs at roughly normal speed, and the proposals past the first miss are wasted compute.
The output does not change. The paper says the method samples without changing the distribution of results, because a draft token survives only if the target accepts it. The draft affects speed, not content.
Medusa and EAGLE-3
Medusa attaches extra decoding heads to the main model so each step proposes several future tokens, and tree-based attention verifies candidates together. No separate draft model is needed. Medusa-1 trains only the heads on a frozen backbone and reports over 2.2x speedup with no loss in quality. Medusa-2 trains heads and backbone together, reports 2.3x to 3.6x, and needs a special recipe to protect the model's abilities.
EAGLE-3 (paper) predicts tokens directly and fuses features from several layers of the target. The authors report up to 6.5x speedup across five tasks, about 1.4x better than EAGLE-2, and a 1.38x throughput gain in SGLang at batch size 64.
When speculative decoding gets slower
The gain depends on how often the target accepts the draft, which depends on how closely the draft matches, how many positions are drafted and the workload. The vLLM docs for v0.8.2 say speculative decoding there "is not yet optimized and does not usually yield inter-token latency reductions" for all prompt datasets or sampling parameters, and list restrictions around pipeline and tensor parallelism.
A 2026 vLLM post on AMD GPUs shows both sides. On gemma-4-26B-A4B-it, DFlash reached 2.87x throughput on MATH500 and EAGLE-3 reached 2.11x to 2.27x. On Qwen3-8B, EAGLE-3 with one drafted token ran at 0.44x of baseline on MATH500, which is slower than no speculation. The authors recommend measuring acceptance and end-to-end throughput on your own workload before choosing settings.
So if you self-host, the trade-offs are concrete: a separate draft model costs memory, trained heads cost training, and the best number of guessed tokens differed between models. Treat the paper numbers as best cases.
The original adds the vLLM details and what to watch next: https://syntaxradar.com/radar/speculative-decoding-medusa-eagle-multi-token-prediction-benchmarks
Top comments (0)