Speculative Decoding: Why Drafting Tokens in Parallel Speeds Up Autoregressive Generation
Large language models generate text one token at a time, and that sequential habit is expensive. Each new token needs a full forward pass through billions of parameters, yet it produces only a single token of output. Speculative decoding breaks that bottleneck without changing what the model would have generated. It drafts several tokens cheaply, checks them in parallel with the large model, and keeps the ones that match. The result is the same distribution, produced in fewer large-model passes.
The bottleneck is not computation, it is serial dependence
An autoregressive model defines a distribution over the next token given everything before it:
To generate a sequence of length T, the model must run T times, because token t+1 cannot be sampled until token t is known. On modern hardware, a single forward pass for one token is often memory-bound rather than compute-bound: loading the weights dominates the cost, while the arithmetic units sit mostly idle. Generating token by token therefore wastes the parallelism that GPUs are built for.
This is why simply buying a faster GPU gives limited relief for interactive generation. Latency scales with the number of sequential steps, not just with the size of each step. Any method that reduces the number of large-model passes, while preserving the output distribution, directly reduces latency.
Draft, then verify
Speculative decoding uses two models:
- A small, fast draft model that proposes tokens quickly.
- The large target model that verifies those proposals in a single parallel pass.
The procedure for one round is simple:
- The draft model generates k candidate tokens autoregressively, but cheaply.
- The target model scores all k candidates in one forward pass, using teacher forcing on the draft sequence.
- The target model accepts a prefix of the candidates, up to the first mismatch.
- From the first mismatch onward, the draft tokens are discarded, and the target model supplies the correct token at that position.
Because step 2 processes k tokens in parallel, it costs roughly one large-model pass rather than k passes. If most drafts are accepted, one large pass yields several tokens instead of one.
A minimal sketch of the loop looks like this:
def speculative_generate(prompt, draft, target, k=4):
tokens = list(prompt)
while not finished(tokens):
# Draft k tokens cheaply
drafts = draft.generate(tokens, n_tokens=k)
# Verify all k in one target pass
target_logits = target.logits(tokens + drafts)
accepted = []
for i, draft_token in enumerate(drafts):
target_token = sample(target_logits[i])
if target_token == draft_token:
accepted.append(draft_token)
else:
accepted.append(target_token)
break
else:
# All drafts accepted: sample one bonus token
accepted.append(sample(target_logits[k]))
tokens.extend(accepted)
return tokens
The bonus token in the all-accepted case matters. It means a fully accepted round can produce k + 1 tokens from a single target pass.
Why the output distribution is unchanged
A naive version that only keeps exact matches would bias the output toward tokens the draft model likes. Speculative sampling fixes this with an acceptance rule based on the ratio of the two distributions.
Let q be the draft distribution and p be the target distribution for the next token. A draft token x sampled from q is accepted with probability:
If it is rejected, a replacement token is sampled from the adjusted distribution proportional to:
This is the classic rejection sampling argument. Tokens that the draft model over-proposes relative to the target are down-weighted, and tokens the draft under-proposes get a second chance through the replacement step. The combined procedure samples exactly from p, not from q and not from a mixture of the two. That exactness is the key claim of speculative decoding: it is an acceleration method, not an approximation method.
In greedy decoding, the rule simplifies to exact matching, because p puts all its mass on one token. In temperature sampling, the ratio rule above is what preserves the target distribution.
How much speedup should you expect?
Let alpha be the average probability that a draft token is accepted. If drafts were independent, the expected number of tokens produced per target pass would be:
This formula has a clear intuition. The first draft token is accepted with probability alpha, the second only if the first two are accepted, and so on. The sum of those acceptance probabilities, plus the bonus token, gives the expectation.
Plug in some numbers. With k = 4 and alpha = 0.8, the expected yield is about 3.4 tokens per target pass. With alpha = 0.5, it falls to about 1.9 tokens. With alpha = 0.9, it rises to about 4.1 tokens. The lesson is blunt: speculative decoding lives or dies on draft quality. A draft model that agrees with the target model most of the time is worth far more than a slightly faster draft model that agrees rarely.
The wall-clock speedup is lower than the token yield, because drafting itself costs time and verification of k tokens costs slightly more than verification of one token. If drafting k tokens costs a fraction c of a target pass, the approximate speedup is:
A draft model that is 10 times faster than the target, with k = 4, makes c small. A draft model that is only 2 times faster can eat most of the gain. This is why draft models are usually much smaller versions of the same model family, or the same model with early exits, rather than a general-purpose small model trained from scratch.
Tree speculative decoding and other refinements
The basic scheme drafts a single chain of k tokens. If the second token is rejected, tokens three to k are wasted even if some of them would have been correct after the replacement token. Tree speculative decoding fixes part of this waste by drafting a tree of alternatives instead of a chain.
At each draft position, the draft model proposes several candidate continuations, not just its top choice. The target model verifies the whole tree in one pass using tree attention, which masks each branch so it only attends to its own ancestors. When a branch is rejected, another branch may still match. This raises the effective acceptance rate, at the cost of more draft computation and a more complex attention mask.
Other practical refinements include:
- Adaptive draft length: shorten k when acceptance is low and lengthen it when acceptance is high, rather than fixing k in advance.
- Draft model distillation: train the draft model on the target model's outputs so their distributions align, which directly raises alpha.
- Self-speculation: use the target model itself, with skipped layers or early exits, as the draft model, so no second model needs to be served.
- Prompt lookup decoding: when the output repeats text from the prompt, copy it as the draft instead of running a draft model at all. This works surprisingly well for code and retrieval-heavy generation.
Each refinement trades extra draft work for a higher acceptance rate. The right trade depends on the deployment: batch size, hardware, and how predictable the output is.
A worked intuition: code versus open-ended chat
Consider two workloads. In code completion, the next tokens are often highly predictable: a closing bracket, a familiar API call, a repeated variable name. A small draft model trained on code will agree with the target model often, so alpha is high and speculative decoding gives a large speedup.
In open-ended creative writing, the next token is far less constrained. The draft model may propose a plausible word that the target model would not have chosen, so rejection is frequent and the effective yield drops toward one token per pass. Speculative decoding still returns the exact target distribution in this case, but it adds draft overhead for little benefit.
This workload dependence explains why published speedups vary so widely, from near 1x to more than 2x or 3x. It is not a property of the method alone; it is a property of the method applied to a particular distribution of text.
Limitations and common pitfalls
Speculative decoding is exact, but it is not free, and several pitfalls catch first implementations:
Batching changes the arithmetic. With a large batch, the target model is already compute-bound rather than memory-bound, so verifying k tokens costs noticeably more than verifying one. The speedup shrinks, and at very large batch sizes it can disappear. Speculative decoding helps most for low-latency, small-batch serving.
KV cache management gets harder. The target model must keep key-value states for the accepted prefix and discard states for rejected suffixes. A bug here silently corrupts later tokens, because the model attends to a cache that does not match the accepted sequence. Rollback must be exact, not approximate.
Temperature sampling needs the ratio rule. Implementing only exact matching under sampling changes the distribution. It looks correct in greedy tests and fails in sampling tests, which is exactly the kind of bug that ships unnoticed. Test the sampler against the target distribution directly, not just against greedy outputs.
Draft quality is a training problem, not just a size problem. A draft model from a different family, or trained on different data, can have low acceptance even if it is fast. Distilling from the target model, or sharing its vocabulary and tokeniser exactly, usually matters more than shaving draft latency further.
Long drafts are not always better. Raising k increases the theoretical yield, but each extra draft token is less likely to be reached, and drafting costs grow linearly. There is an optimal k for a given alpha and draft cost, and it is usually small, often between 3 and 8.
When to use it
Use speculative decoding when latency matters, batch sizes are small, and you can obtain a draft model that agrees with your target model well. Interactive chat, code completion, and single-user serving are natural fits. It is especially attractive because it preserves quality exactly: there is no trade-off to justify to users, only an engineering cost to pay.
Skip it, or test carefully, when you serve large batches, when your draft model is poorly aligned with the target, or when your serving stack cannot handle KV cache rollback cleanly. In those settings, other optimisations such as quantisation, continuous batching, or a smaller target model may give more reliable gains.
Takeaway
Autoregressive generation is slow because it is serial, not because each step is intrinsically hard. Speculative decoding exploits the gap between cheap drafting and expensive verification: propose several tokens with a small model, check them all at once with the large model, and keep the prefix that matches. With the correct acceptance rule, the output distribution is unchanged. With a well-aligned draft model, the same text arrives in far fewer large-model passes. That combination, exactness plus speed, is why speculative decoding has moved from a research curiosity into production serving stacks.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Top comments (0)