There's a thread on the DeepSeek-R1 model page that's been sitting unresolved since March last year, and it bothered me enough to go read the papers.
Someone had tried speculative decoding and reported that the output got worse — "very low quality words for the given context, words it would never generate by itself". Someone else replied that this is impossible, because the main model verifies and corrects everything the draft model proposes. Neither budged.
They're both right. That took me a while to see.
The guarantee is real
The claim that speculative decoding can't hurt quality isn't marketing. It's proven, twice, independently.
Leviathan and colleagues at Google introduced the technique in 2022 and state it flatly: "A strong property of Algorithm 1 is that the output distribution is guaranteed to remain unchanged". Chen and colleagues at DeepMind arrived at the same place a few months later through a different proof — theirs is "a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics".
So the person insisting it cannot degrade quality has the papers behind them.
The guarantee is also narrower than it sounds
Here's the part almost every explainer skips, and it's sitting right there in the DeepMind paper:
because the different computation graphs lead to different numerics, we cannot not expect identical outputs
(The doubled "not" is theirs. I left it.)
Read that against the guarantee and the argument dissolves. The distribution is preserved. The sample is not the same sample. You're drawing from the same distribution through a different computation graph, with different floating-point behaviour in the batched verification pass than in a plain decode.
So on any single prompt, you can absolutely get different text. Sometimes worse-looking text. That isn't the method failing — it's what same-distribution-but-different-draw means in practice.
Which is exactly what the person reporting bad output experienced. And exactly why the person quoting the theory couldn't reproduce it.
Worth noting: Leviathan's own abstract does claim "identical outputs" for their T5X experiments. That's true for their setup — argmax sampling, one implementation. It's a property of that configuration, not a law.
It does more work, not less
This is the bit that genuinely surprised me.
I'd absorbed the idea that speculative decoding is a clever way to do less computation. It's the opposite. Leviathan has an entire section titled Number of Arithmetic Operations, and it says the number of concurrent arithmetic operations "grows by a factor of" gamma plus one. Every rejected draft token is compute you spent and threw away.
Total FLOPs go up.
It's faster anyway, because arithmetic was never what you were waiting on. From the same paper: inference from large models is "often not bottlenecked on arithmetic operations, but rather on memory bandwidth and communication". Chen puts it in five words — "Transformer sampling is typically memory bandwidth bound".
Your accelerator is sitting idle waiting on memory. Speculative decoding spends that idle compute to avoid serial steps. It's a trade, and it only pays because one side of the ledger was free.
That reframing changed how I think about the whole category. Same shape as FlashAttention, incidentally — a technique everyone describes as making attention cheaper, when it is really about memory movement.
What the numbers actually look like
Chen's Table 1, Chinchilla 70B, batch size 1, K=4:
| Result | Time per token | Speedup | |
|---|---|---|---|
| Autoregressive, XSum | 0.112 | 14.1 ms | 1x |
| Speculative, XSum | 0.114 | 7.52 ms | 1.92x |
| Autoregressive, HumanEval | 45.1% | 14.1 ms | 1x |
| Speculative, HumanEval | 47.0% | 5.73 ms | 2.46x |
Time per token roughly halves. Now look at the quality column — HumanEval goes up, 45.1 to 47.0.
Don't read that as an improvement. Elsewhere in the same table, XSum's greedy ROUGE-2 goes down, 0.157 to 0.156. Small movements in both directions across metrics is precisely what a distribution-preserving method should produce. If speculative decoding reliably improved scores, the theory would be wrong.
Where the speedup goes away
Both papers are more careful than the blog posts about them.
You need spare compute — Leviathan is explicit that the algorithm assumes "we have enough compute resources to support the increased concurrency". No headroom, no win.
You need a high acceptance rate. Every rejected token is wasted work.
And the draft model has to be genuinely cheap. This is the counterintuitive one: against an 11B target, Leviathan found T5-small (77M) beat both T5-base (250M) and T5-large (800M). The best draft model was the smallest one tested, not the most accurate.
One more caveat that gets dropped constantly: both papers measured at batch size 1. That's a latency benchmark. Chen notes that other techniques are the ones aimed at throughput "(at larger batch sizes)". If you're serving large batches, don't expect these numbers.
So who was right in that thread?
Both, and the disagreement was never really about speculative decoding.
The theory says your distribution is untouched. Your eyes say the text came out different. Both statements are true simultaneously, and the papers are clear about why, and nobody in the thread had read that far.
If your output is reproducibly worse across many prompts, that's an implementation problem — there's a 2025 paper showing some engines break the distribution guarantee outright once batch size exceeds one. If it's one bad generation, that's just a different draw.
I went through both papers line by line for a longer write-up at diffstudy.com, with every quote sourced. Papers are arXiv 2211.17192 (Leviathan et al.) and arXiv 2302.01318 (Chen et al.) if you'd rather go straight to them.
Top comments (0)