TL;DR — Autoregressive LLM inference is memory-bandwidth-bound, not compute-bound — the GPU sits mostly idle waiting for weight reads. Speculative decoding breaks that bottleneck by drafting multiple candidate tokens cheaply and verifying them in a single parallel forward pass. This essay explains the mechanics, the acceptance-rate math, and the engineering tradeoffs that determine whether speculative decoding actually helps in production.
Every token an autoregressive LLM generates requires a full forward pass through the model. That sounds fine until you realize what that forward pass actually does at inference time: it reads billions of floating-point weights from GPU HBM, does a relatively modest amount of arithmetic, and produces exactly one token. The arithmetic intensity — FLOPs per byte of memory traffic — is so low that the GPU's compute units spend most of their time waiting for the memory bus. You paid for a machine with hundreds of teraFLOPS of compute, and you are bottlenecked on memory bandwidth.
This is the core problem speculative decoding solves. Not by making the big model faster, but by making the big model do more useful work per forward pass.
The Draft-Then-Verify Idea
Speculative decoding pairs two models: a small, fast draft model and the large target model you actually care about. The draft model autoregressively generates a short sequence of candidate tokens — say, four or five — very cheaply. Then the target model runs a single forward pass over the entire candidate sequence in parallel, producing logits for every position at once.
Here is the key insight: the target model's forward pass over a sequence of length k is only marginally more expensive than a forward pass over a sequence of length 1, because the attention and feed-forward layers are already batched across the sequence dimension. The memory reads — the bottleneck — happen once regardless of sequence length, up to the point where the KV cache starts to dominate.
So you get to evaluate k candidate tokens for roughly the price of one. If the draft model's guesses are good enough, you accept multiple tokens per target-model call and your effective throughput climbs.
The Acceptance Criterion
You cannot just accept whatever the draft model says — that would change the output distribution of the target model, which defeats the purpose. Speculative decoding uses a rejection-sampling scheme that provably preserves the target model's distribution.
For each candidate token at position i, compare the probability the target model assigns to that token against the probability the draft model assigned. If the ratio is at least 1 — the target model likes this token at least as much as the draft model did — accept it unconditionally. If the ratio is less than 1, accept it with probability equal to that ratio, and otherwise reject it and sample a correction token from a residual distribution derived from the difference between the two models' distributions.
The result is mathematically identical in distribution to sampling directly from the target model at every step. You are not approximating. You are accelerating exact inference.
The acceptance rate depends entirely on how well the draft model predicts what the target model would have chosen. A draft model that is too small or too different in training will have low acceptance rates, meaning you reject frequently, fall back to single-token generation, and gain little. A draft model that is a smaller version of the same model family — trained with the same tokenizer and similar data — tends to have high acceptance rates because the two models share strong distributional alignment.
Where the Speedup Actually Comes From
It helps to think in terms of tokens per second rather than latency per token. Without speculative decoding, each target-model call produces one token. With speculative decoding and a draft of length k, each target-model call produces somewhere between 1 and k+1 tokens depending on how many candidates are accepted. The expected number of accepted tokens per call — call it alpha — is a function of the per-token acceptance rate.
If per-token acceptance rate is p, the expected number of accepted tokens from a draft of length k is roughly (1 - p^(k+1)) / (1 - p). At high acceptance rates this approaches k+1; at low acceptance rates it collapses toward 1. In practice, well-matched draft and target models achieve acceptance rates high enough that a draft length of 4-6 tokens yields meaningful throughput gains — often in the range of 2x to 3x on latency-bound single-stream workloads.
Note that batch throughput is a different story. When you are running large batches, the forward pass is already compute-bound rather than memory-bandwidth-bound. Speculative decoding helps least in exactly the regime where you need help least. Its sweet spot is low-batch, latency-sensitive inference — the kind of workload you see in interactive applications and agentic pipelines where a single user is waiting on a single stream of tokens.
Practical Engineering Tradeoffs
Deploying speculative decoding in production introduces several complications that the research papers tend to understate.
- KV cache management doubles in complexity. You now maintain KV caches for both the draft model and the target model. When a candidate token is rejected, you must roll back the target model's KV cache to the last accepted position. Implementing this correctly without memory leaks or stale cache entries is non-trivial, especially in streaming inference servers.
- The draft model is not free. It consumes memory — potentially several gigabytes — that could otherwise go to a larger batch or a bigger target model. On memory-constrained hardware this is a real cost.
- Draft length is a tunable hyperparameter. Too short and you leave throughput on the table. Too long and you waste draft-model compute on tokens that will be rejected, and you increase the tail latency of calls where many tokens are rejected late in the draft sequence. The optimal draft length depends on the acceptance rate, which itself varies by prompt domain.
- Acceptance rates are not stationary. Code generation prompts have very different acceptance characteristics than open-ended creative writing. A draft length tuned for one task can be suboptimal for another. Some systems adapt draft length dynamically based on observed acceptance rates.
Variants Worth Knowing
The original formulation assumes a separate smaller model as the draft. Several variants relax this constraint.
Self-speculative decoding uses early exit from the target model itself as the draft. The first several transformer layers produce a draft prediction; the full model verifies it. This eliminates the need for a separate model but requires architectural support for early exit and careful calibration of which layers to exit at.
Medusa adds multiple parallel prediction heads to the target model, each predicting a token further ahead in the sequence. These heads are trained cheaply on top of a frozen base model. Verification uses a tree-based attention mask to evaluate multiple candidate continuations simultaneously. The tradeoff is that the heads must be trained and stored, and they are specific to the base model checkpoint.
Lookahead decoding takes a different approach entirely, using a Jacobi iteration over the token sequence to find fixed points without a draft model at all. It is less mature but interesting for cases where no suitable draft model exists.
When to Reach for It
Speculative decoding is the right tool when you have a latency-sensitive, low-batch workload and a draft model that is well-matched to your target model. If you are running a hosted API with large batch sizes, the gains will be modest and the operational complexity may not be worth it. If you are running an agentic loop where a single model is generating long reasoning traces and every token adds to end-to-end latency, speculative decoding can make a material difference to user experience.
The deeper lesson is architectural: the memory-bandwidth bottleneck is fundamental to how current transformer inference works, and any technique that amortizes weight reads across more useful output will help. Speculative decoding is the most practically deployed instance of that idea, but the same logic applies to continuous batching, prefix caching, and the emerging class of recurrent architectures that trade attention's quadratic memory for a fixed-size state. Understanding why speculative decoding works — not just that it works — puts you in a better position to evaluate all of them.
Top comments (0)