DEV Community

Cover image for Speculative decoding: how LLMs get 2-3x faster with identical outputs (and when it won't help you)
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Speculative decoding: how LLMs get 2-3x faster with identical outputs (and when it won't help you)

Tuesday AI Deep Dive. Most of the time you wait on an LLM, you're not waiting on math. You're waiting on memory. Generating each token means one full forward pass of the model, and those passes have to happen one after another. Speculative decoding is the trick behind a lot of "why is this model suddenly so fast" announcements: it lets a big model emit several tokens per forward pass without changing what it would have output.

Here's how it works, the one formula worth remembering, the variants you'll see in serving stacks, and when it won't help.

Why decoding is slow in the first place

From the Google paper that introduced the idea (Leviathan, Kalman, Matias, 2022): "decoding K tokens takes K serial runs of the model." Token 5 depends on token 4, so you can't parallelize across the output.

DeepMind's speculative sampling paper (Chen et al., 2023) adds the key hardware fact: transformer sampling is typically memory-bandwidth bound. For each new token, the accelerator streams the model's weights from memory, does relatively little arithmetic with them, and moves on. The Medusa paper puts it the same way: each step "necessitates moving the full model parameters from High-Bandwidth Memory (HBM) to the accelerator's cache."

The consequence: scoring several tokens in one forward pass costs about the same wall time as scoring one. Chen et al. build the whole method on that observation: the latency of scoring a short continuation in parallel "is comparable to that of sampling a single token from the larger target model."

So if someone could hand the big model a good guess of the next few tokens, it could check them all at once.

The algorithm: draft, verify, accept

You need two models:

  • a target model (the big one whose outputs you actually want), and
  • a draft model (small and fast, ideally similar in behavior).

Each iteration:

  1. Draft. The small model generates 纬 tokens autoregressively (cheap, because it's small).
  2. Verify. The target model runs one forward pass over the prefix plus all 纬 drafted tokens, getting its own probability for every position in parallel.
  3. Accept or reject, left to right. For each drafted token x, the target accepts it with probability min(1, p(x)/q(x)), where p is the target's probability and q the draft's. At the first rejection, it stops and samples a corrected token from the normalized residual max(0, p - q).
  4. If all 纬 are accepted, the target's pass already gives you the distribution for one more token, so you get a bonus token for free.

Every iteration produces at least one token from the target model, and up to 纬+1.

The important part is step 3. This modified rejection sampling is what makes the method exact: the paper proves the output follows the target model's distribution. With greedy decoding that means the same tokens you'd have gotten without the draft. Leviathan et al. show 2X-3X acceleration on T5-XXL "with identical outputs." Chen et al. report 2-2.5x decoding speedup on Chinchilla 70B in a distributed setup, preserving the target distribution "within hardware numerics."

That's the difference from just using a smaller model: the small model only proposes. The big model always has the final say.

The one formula: acceptance rate decides everything

Leviathan et al. define 伪, the expected acceptance rate, as a measure of how well the draft approximates the target (formally 伪 = E[min(p, q)] summed over tokens). Under a simplifying i.i.d. assumption, the expected number of tokens per target call is:

E[tokens per target pass] = (1 - 伪^(纬+1)) / (1 - 伪)
Enter fullscreen mode Exit fullscreen mode

Plugging numbers into that formula (my arithmetic, not benchmark results), with 纬 = 4 drafted tokens:

Acceptance rate 伪 Expected tokens per target pass
0.6 ~2.31
0.8 ~3.36

Two practical reads:

  • 伪 matters more than 纬. Drafting more tokens has diminishing returns, because every extra token must survive all the earlier ones. A bad draft model with a big 纬 just burns compute on tokens that get thrown away.
  • Tokens per pass isn't wall-clock speedup. You also pay for running the draft model. The paper's walltime analysis assumes you have spare compute for the extra concurrency.

The variants you'll actually meet

The classic two-model setup has a practical problem the Medusa authors call out: "acquiring and maintaining a separate draft model." It needs the same tokenizer, should behave like the target, and must be served alongside it. Newer methods attack exactly that:

  • Medusa (Cai et al., 2024). No separate model. Extra decoding heads on top of the target predict several future tokens, and tree-based attention verifies multiple candidate continuations in one step. Medusa-1 (heads trained on a frozen backbone) is lossless and reports over 2.2x speedup. Medusa-2 trains heads together with the backbone for higher speedup, but needs a careful training recipe.
  • EAGLE (Li et al., 2024). Drafts at the feature level (the second-to-top layer) instead of the token level, feeding in the token sequence advanced by one step to resolve uncertainty. For LLaMA2-Chat 70B it reports 2.7x-3.5x latency speedup and doubled throughput "while maintaining the distribution of the generated text."
  • Prompt lookup / n-gram drafting. The "draft model" is just string matching: propose the continuation that follows a matching n-gram in the prompt. It costs almost nothing and shines when outputs copy from inputs (summarizing a document, editing code, RAG answers quoting sources). The reference implementation describes itself as a simple speculative decoding technique integrated in vLLM and Hugging Face transformers.

When it won't help (or will hurt)

Speculative decoding sells spare compute for lower latency. That trade isn't always available:

  • High-batch, throughput-bound serving. The whole trick relies on decoding being memory-bound with idle compute. When you're already batching many requests, the GPU's compute is busier and verifying rejected draft tokens is real waste. Benchmark at your actual batch sizes and concurrency, not with one request.
  • Low acceptance. Creative, high-temperature output, or a draft model trained on different data, means low 伪 and small gains. Code, structured output and text that copies from the prompt tend to be more predictable.
  • Operational cost. A separate draft model means more GPU memory, matching tokenizers, and one more thing to upgrade when the target changes. Heads (Medusa/EAGLE) need training per target model.
  • "Lossless" depends on the implementation. The math is exact, but some variants trade exactness for speed. Medusa's paper, for example, proposes a "typical acceptance" scheme to boost acceptance while maintaining generation quality, which is a quality claim, not a distribution guarantee. Know which mode you've turned on.

What I'd do as an AI engineer

  1. Measure first. Log time to first token and inter-token latency per request. If your users are waiting on long generations at low concurrency, speculative decoding is a strong candidate.
  2. Start with the cheapest draft. For RAG, summarization and code edits, try n-gram/prompt lookup drafting before training anything.
  3. Track acceptance rate in production. It's the number that tells you whether the draft still fits your traffic. A drop after a prompt change or model upgrade explains a latency regression.
  4. Benchmark at real load. Compare p50 and p95 latency and tokens/sec with and without speculation at your typical and peak batch sizes.
  5. Verify outputs, don't assume. With greedy decoding, run a fixed eval set with speculation on and off and diff the outputs. They should match; if they don't, find out why before shipping.
  6. If you use hosted APIs, you can't control this, but it helps explain why output speed varies by provider for the same open model.

The takeaway

Speculative decoding works because a big model can check several guesses in the time it takes to write one token, and a clever acceptance rule makes those guesses invisible in the output. The speedup lives or dies on one number, the acceptance rate, and on whether your hardware has idle compute to spend.

What are you serving where latency per token matters most, chat, code, or agents?

References

Top comments (0)