Speculative Decoding, Illustrated: Why Generating 7 Tokens Can Cost the Same as 1

Borui Cai — Wed, 05 Aug 2026 02:27:56 +0000

This is a plain-language, illustrated explainer. You don't need an inference background to read it: every idea arrives as an everyday metaphor first (a small model that guesses, a big model that grades; a warehouse, a truck, and a workshop) and as math second — and every formula that does appear gets a plain-English translation right next to it. In a few places I deliberately oversimplify to keep the intuition front and center; footnoted rigor is what the papers in the reference list are for. If you've bounced off the original papers, this is the on-ramp.

🔑 The one sentence that unlocks everything

Why is LLM inference slow, and why does speculative decoding make it fast? It comes down to one sentence: making the matrices bigger does add more multiplications for the GPU — but no matter how big the matrix is, one matrix multiplication loads the weights exactly once. More compute does not mean more loading. And in LLM inference, loading (moving weights from GPU memory into the compute units) is where nearly all the time goes, while compute is close to free. So processing a few extra tokens costs almost no extra wall-clock time. This is not an algorithmic trick — it is a physical fact of GPU hardware. Every bit of speedup speculative decoding delivers is this one hardware dividend being cashed in.

A concrete example: the sentence "the cat sat on the red mat" is a matrix. Each token is first turned into a row of numbers (its embedding, say d dimensions). Stack the 7 rows and you get a 7 × d matrix X:

        ┌                                ┐
  the   │  0.12  -0.83   0.05   0.47  …  │
  cat   │ -0.31   0.22   0.68  -0.10  …  │
  sat   │  0.55   0.09  -0.42   0.33  …  │
  on    │ -0.07   0.61   0.18  -0.25  …  │
  the   │  0.12  -0.83   0.05   0.47  …  │
  red   │  0.29  -0.14   0.52   0.40  …  │
  mat   │  0.44   0.37  -0.20   0.08  …  │
        └                                ┘
            = matrix X  (7 rows × d columns)

Rows = tokens (7 here). A longer sentence means more rows and more multiplications — but multiplying this entire block X by the weight matrix W loads W from memory exactly once. That is what "more compute, not more loading" looks like in the flesh. (Easter egg: the two "the" rows are identical — same token, same embedding row, before attention gets involved.)

Context: This is the technical deep-dive in a two-part series. Part 1 argued that speculative decoding is one of the most underrated skill niches for LLM inference roles in 2025–2026, and introduced the core idea ("a small model guesses, a big model grades"). This part covers the full math (including the rejection-sampling proof), the main research directions of the past two years, and a list of real interview questions.

How to read this: Section 1 is the foundation. Section 2 is the math (formulas, but every one gets a plain-English explanation). Section 3 maps the research landscape (use it as a study roadmap). Section 4 is a framing insight that stands out in interviews. Section 5 is the learning path and interview question bank.

1. Why LLM Inference Is Slow: Two Orthogonal Bottlenecks

To understand why speculative decoding works, you first have to see clearly why inference is slow at all. This section doesn't touch the algorithm yet, but it is the foundation for everything after — and explaining it well in an interview instantly separates you from candidates who merely memorized the algorithm.

1.1 Bottleneck A: The Serial Dependency of Autoregressive Generation

Mainstream LLMs are causal Transformers [15] that generate autoregressively:

P (x_{1}, x_{2}, \dots, x_{n}) = i = 1 \prod n P (x_{i} ∣ x_{< i})

In plain English: every token needs all previous tokens as input. Generating 100 tokens means 100 serial model calls.

This serial nature lives at the algorithm level and cannot be bypassed without changing the modeling paradigm — which is exactly where more radical approaches like Lookahead Decoding and diffusion language models come from.

1.2 Bottleneck B: Memory-Bandwidth Bound (the important one)

This one is counterintuitive and a favorite interview topic: the bottleneck of LLM inference is not GPU compute (FLOPs) — it is memory bandwidth.

GPU memory is hierarchical:

Tier	Capacity	Bandwidth	Role
HBM (device memory)	~80 GB	~3 TB/s	Where model weights live
L2 cache	~50 MB	~10 TB/s	Staging area
SRAM (on-chip)	~100 KB	~30 TB/s	Where matrix multiplies actually happen

GPU compute cores can only work out of SRAM. On every forward pass, all model weights must stream from HBM into SRAM to participate in the computation — this is dictated by GPU physics; no software layer can skip it.

Put a number on it: a 70B model in FP16 is roughly 140 GB of weights. At an H100's ~3 TB/s of HBM bandwidth, the floor for a single forward pass is about 47 ms — the physical lower bound on per-token latency.

But the more important fact is this: that 140 GB transfer cost barely changes with how many tokens you process. One token: load everything once. Seven tokens: still load everything once. Processing a few extra positions adds almost nothing to the time of that forward pass.

Why is "a few extra positions" nearly free? The core operation of a forward pass is the matrix multiply Y = X · W: input X has shape [N, d] (N positions, d dimensions each), weights W have shape [d, d']. Once the GPU has moved W from HBM into SRAM, it multiplies that same copy of W against all N rows of X.

To be precise: going from 1 row to 7 rows does increase the GPU's FLOPs — six extra rows of multiply-accumulate is not zero work. The key is that one matrix multiplication loads W exactly once, and the loading volume does not depend on the number of rows. Since LLM inference is memory-bound — compute is wildly abundant and bandwidth is the constraint — the extra compute hides inside the weight-loading time: the compute cores were sitting idle waiting for W to arrive anyway, so multiplying a few more rows costs almost no extra wall-clock time. "Extra compute is free" doesn't mean compute literally costs nothing — it is a direct consequence of the GPU's compute ≫ bandwidth characteristic.

1.3 Putting the Two Bottlenecks Together: The Core Pain

Bottleneck A (algorithm): N serial forward passes are mandatory.
Bottleneck B (hardware): each forward pass pays a fixed loading cost but produces only 1 token.

Together, they are the essence of slow inference: paying "move 140 GB of weights" per pass and getting exactly 1 token back is a terrible exchange rate.

The elegance of speculative decoding is that it uses B to partially defeat A — since extra tokens are free, find a way for one forward pass to verify many tokens. Hold on to this and every paper in the field snaps into place.

1.4 The Key Clarification: What Does One Forward Pass Actually Produce?

Time to puncture a very common misconception — getting this right is the key to everything that follows.

Most people think one Transformer forward pass computes 1 token. It doesn't.

💡 The key mechanism: one forward pass = one weight load + one matrix multiplication, and that single matrix multiplication computes every input position at once. Loading and computing are welded together — there is no such operation as "load once, then compute N separate times." "Computing N times" does not physically exist. What exists is "1 forward pass," "2 forward passes," … and each forward pass is an inseparable bundle of load once + compute all current positions.

Transformers [15] use causal self-attention: given an input sequence of length N, one forward pass outputs a "next-token distribution" at all N positions simultaneously. Every input position i gets a distribution P(x_{i+1} | x≤ᵢ). This is precisely why Transformers train so much faster than RNNs — all positions compute in parallel.

So why does ordinary decoding look like "1 token per pass"? Because:

Position 0 predicts "what comes after input token 1" → that's input token 2, which we already have
Position 1 predicts "what comes after input token 2" → that's input token 3, which we already have
…
Position N−1 predicts "what comes after the last input token" → the only genuinely new information

So the predictions at the first N−1 positions are entirely wasted — they predict input we've already seen. Ordinary decoding keeps only the last position's output, hence "one new token per pass" from the outside.

Speculative decoding exploits exactly this waste: let a small model pre-fill those "future positions" with guesses, and suddenly all N distributions from the big model's single forward pass become meaningful —

the first γ distributions verify the corresponding draft tokens ("do I, the big model, agree with the small model's guess?")
the last distribution yields a free bonus token (if all γ drafts were accepted)

Put ordinary and speculative decoding side by side and it becomes crystal clear (both producing 7 tokens):

Decoding	Forward passes	Weight loads	What those matrix multiplies compute
Ordinary	7	7 (≈ 7 × 47 ms)	Pass k computes k positions but keeps only the last one; the rest is wasted
Speculative	1	1 (≈ 47 ms)	One pass computes 7 positions, all of them used to verify drafts

Here is the counterintuitive part: ordinary decoding's 7th forward pass also computes 7 positions — it's just that the first 6 predict "input we already know" and get thrown away. So the total matrix-multiply work of the two approaches is roughly the same. The real difference isn't how much you compute — it's how many times you load: ordinary decoding loads the weights 7 times; speculative decoding compresses that to 1. And since loading dominates latency (~47 ms per load), compressing 7 loads into 1 is compressing the time itself.

Within that single verification pass, the costs are also wildly asymmetric:

Stage	Who	What	Cost
Verification forward	Big model (GPU)	One forward pass, one weight load, distributions at all positions in parallel	The main cost (~47 ms)
Accept/reject decisions	Algorithm layer (CPU or a tiny GPU kernel)	Table lookups over a batch of distributions + corrected rejection sampling	Practically free

⚠️ A common wrong formulation is "multiple verifications inside one forward pass" or "multiple forward passes." Both are imprecise. The forward pass is 1, the load is 1, and verification is many distributions from a single pass being consumed in parallel by the outer algorithm. It is a free dividend of the Transformer's causal structure.

Internalize this mechanism and the proof, the research landscape, and the interview questions in the next sections all become intuitive.

2. The Core Algorithm

2.1 The Procedure

Let:

Target model M_p with distribution p(· | context) (big, slow — say 70B)
Draft model M_q with distribution q(· | context), far smaller than M_p (small, fast — say 1B)

Each round of speculative decoding has 4 steps [1, 2]:

Draft: M_q autoregressively generates γ candidate tokens x₁, …, x_γ (typically γ = 4–7)
Parallel verification: feed all γ tokens plus the prefix to M_p in one forward pass, obtaining the conditional distributions p(· | x<ᵢ) at every position
Corrected rejection sampling: decide each xᵢ left to right —
- accept with probability min(1, p(xᵢ)/q(xᵢ))
- otherwise reject, resample a replacement from the corrected distribution p′(x) ∝ (p(x) − q(x))₊, and discard all subsequent drafts (where (·)₊ = max(0, ·))
Bonus token: if all γ are accepted, sample one extra token from p(· | x≤_γ) — that distribution was computed in the pass anyway, so it's free

2.2 The Key Question: Why Is the Output Exactly Equivalent to Sampling from M_p?

This is the most magical and most important property of speculative decoding: it is not "approximately lossless" — it is mathematically exact. Deeply counterintuitive, yet the proof is short. Deriving it live in an interview instantly separates "read the paper" from "understands it."

Goal: show the probability of outputting any token x is exactly p(x).

There are two paths to outputting x:

Path 1 (x is accepted):

Pr (output = x, accept) = q (x) \cdot min! (1, \frac{p ( x )}{q ( x )}) = min (p (x), q (x))

Path 2 (rejected, then x is resampled from the corrected distribution):

Pr (output = x, reject) = Pr (reject) \cdot \frac{( p ( x ) - q ( x )) +}{\sum _{y} ( p ( y ) - q ( y )) +}

Here we need one key identity:

y \sum (p (y) - q (y))_{+} = y \sum [p (y) - min (p (y), q (y))] = 1 - y \sum min (p (y), q (y)) = Pr (reject)

That is: the normalizing constant of the corrected distribution equals the rejection probability exactly. The two cancel:

Pr (output = x, reject) = (p (x) - q (x))_{+}

Combine both paths:

Pr (output = x) = min (p (x), q (x)) + (p (x) - q (x))_{+}

Check both cases:

If p(x) ≥ q(x): min = q(x), (p−q)₊ = p(x) − q(x), sum = p(x) ✓
If p(x) < q(x): min = p(x), (p−q)₊ = 0, sum = p(x) ✓

The output follows p exactly. ∎

Why this proof is beautiful: the normalizing constant Σ_y (p(y)−q(y))₊ would normally require summing over the entire vocabulary (tens of thousands of tokens). Through this identity, it cancels against the rejection probability automatically — the algorithm never has to compute it explicitly, yet the result is exactly correct. That is the elegance of corrected rejection sampling.

💡 Interview tip: learn these five lines cold and derive them live. Instantly upgrades you from "has read about it" to "understands it."

2.3 Efficiency: Acceptance Rate Sets the Speedup Ceiling

Define the acceptance rate:

α = E_{x \sim q} [min (1, p (x) / q (x))] = 1 - TV (p, q)

where TV is total variation distance. Intuitively, α measures the overlap between the target and draft distributions — the more alike they are, the higher the acceptance, the bigger the speedup.

Expected tokens per round [1]:

ParseError: KaTeX parse error: Expected 'EOF', got '#' at position 13: \mathbb{E}[#̲\text{tokens}] …

With draft-to-target cost ratio c (typically c ∈ [0.02, 0.1]), the per-round speedup is:

S (γ) = \frac{1 - α ^{γ + 1}}{( 1 - α ) ( 1 + γ c )}

Bottom line: as c → 0 with optimal γ, the speedup approaches 1 / (1 − α). vLLM reports up to 2.8× in production [3]; EAGLE-family methods reach 3–4×.

3. The Research Landscape (5 Subfields)

Once vanilla speculative decoding was established, the past two years produced several active branches. These are exactly the tech stacks that inference teams hire for.

3.1 Self-Speculation: Medusa & the EAGLE Family

Vanilla requires deploying a separate small model — real ops cost and memory overhead. Self-speculation lets the target model draft for itself:

Medusa [4]: attach K independent prediction heads after the target model's last layer, predicting tokens 1 through K ahead. Shares the backbone features; cheap to train, simple to deploy
EAGLE [5]: the core insight is that token-level drafting uncertainty mostly originates at the feature level, so speculate in hidden-state space (predict the next hidden state, then decode it to a token). Acceptance rates significantly beat token-level methods
EAGLE-2 / EAGLE-3 [6]: introduce the dynamic draft tree — current SOTA on academic benchmarks, integrated into vLLM [7]

EAGLE-family speedups are typically 3–4×, currently the mainstream production choice.

3.2 Token-Tree Verification: SpecInfer

Vanilla verifies one linear draft per round; a rejection at position 1 discards everything after it — wasteful.

SpecInfer [8] has the draft model generate multiple candidate branches organized as a tree, and the target model verifies the whole tree in one pass via custom tree attention, substantially raising expected accepted tokens. Later work (EAGLE-2 and others) adds dynamic depth adjustment.

💡 Note: SpecInfer was published at ASPLOS — a systems conference, not an ML one. That fact alone reflects the true nature of this field (see Section 4).

3.3 Multi-Token Prediction (MTP)

MTP [9, 10] takes a different route but is often discussed alongside spec decoding.

The idea is to change the training objective so the model predicts k future tokens at every position:

L MTP = - \sum i = 1^{N} j = 1 \sum k lo g P_{j} (x_{i + j} ∣ x_{\leq i})

MTP delivers two kinds of value:

High-quality drafts for spec decoding (a form of self-speculation)
Improved representation quality in the base model (empirically shown in Meta's paper) — a side benefit no other acceleration method offers

DeepSeek-V3 adopted MTP at scale [10]; it is one of the most watched directions in industry.

💡 Advanced insight: MTP and spec decoding attack different bottlenecks — spec decoding borrows the hardware dividend (Bottleneck B) to relieve the algorithmic one (A); MTP attacks A directly. If future hardware (compute-in-memory, etc.) erases B, spec decoding's dividend shrinks dramatically — but MTP still works. Articulating this distinction is a senior-level signal in interviews.

3.4 Lookahead Decoding

Lookahead Decoding [11] needs no draft model at all: view autoregressive decoding as solving a nonlinear system, update multiple positions in parallel with Jacobi iteration, and converge to output identical to standard autoregressive decoding.

Pro: zero auxiliary models, zero training.
Con: speedups usually below spec decoding (1.5–2×).

A good fit for resource-constrained deployments or teams that can't maintain an extra draft model.

3.5 Online Learning & Production Deployment

Online Speculative Decoding [12] observes that a draft model's acceptance rate drifts across query distributions — a draft trained on general text can lose half its acceptance rate on code-heavy production traffic. The fix: continuously fine-tune the draft model on live traffic during serving.

On the engineering side, vLLM Speculators [7] productizes spec-decoding training and deployment, supports mainstream variants like EAGLE-3, and is used in large production systems including Amazon Rufus and LinkedIn AI [13].

4. The Framing That Matters: This Is Software Engineering, Not Algorithmic Novelty

Understanding this makes you sound like someone who truly gets the field, not someone who just read the papers.

Step back and look at the whole area:

The model itself never changes. Vanilla spec decoding uses the target model exactly as-is — zero parameter updates, zero architecture changes
Asymptotic complexity never changes. Generating N tokens is still O(N) serial steps worst-case; only the constant factor drops severalfold
The core math is classical corrected rejection sampling — textbook sampling theory, not a new invention
The physical source of the speedup is a GPU-architecture dividend — the enormous gap between HBM bandwidth and SRAM speed opened a memory-bound window

Which implies:

It is tied to a hardware generation. If compute-in-memory (PIM/CIM) hardware erases the memory wall, the dividend shrinks dramatically
It is closer to a systems contribution. SpecInfer went to ASPLOS; vLLM went to SOSP [14] — the venues tell the story
Its algorithmic purity is limited — no new learning paradigm, no new representational capacity, no new objective

None of this diminishes its value — quite the opposite. It showcases what engineering insight does for LLM deployment: spotting wasted hardware capability and cashing it in with a clever scheme.

If you're job-hunting, this is great news. You don't need top-venue publications or deep ML theory. You need:

Solid systems knowledge (GPU architecture, memory hierarchy, concurrency)
The ability to read real code (navigate vLLM internals)
Hands-on projects with numbers (measured speedups, acceptance-rate distributions)

This is precisely the window of opportunity for strong engineers without elite-school or research pedigrees.

5. Learning Path & Interview Question Bank

5.1 Must-Read Papers (in order)

#	Paper	Focus
1	Leviathan et al. (2023) [1]	Algorithm 1 + appendix proof
2	Chen et al. (2023) [2]	Read against #1; two independent concurrent inventions
3	Cai et al. (2024) Medusa [4]	Method section
4	Li et al. (2024) EAGLE [5]	The motivation for feature-level speculation
5	Fu et al. (2024) Lookahead [11]	The Jacobi-iteration intuition

Advanced: MTP [9], the DeepSeek-V3 report [10], SpecInfer [8], EAGLE-3 [6].

5.2 Hands-On Projects (where the real differentiation happens)

Do at least one:

Run a spec-decoding demo with HuggingFace transformers: model.generate(..., assistant_model=...) — a few lines of code
Measure the acceptance rate α: swap draft models of different sizes (1B vs 3B) and watch α move
Read the vLLM spec-decoding source: github.com/vllm-project/vllm
Plot the acceptance-position distribution: which tokens get rejected? (Named entities and numbers get rejected often; "the/a/of" sail through)

One sentence in an interview — "I benchmarked the X + Y model pair, got a Z× speedup at α = …, and found that gets rejected most" — and you've pulled ahead of the pack.

5.3 Interview Question Bank

Warm-up (asked in every interview):

Explain speculative decoding in one sentence
Where does the speedup come from?
Why is the output exactly lossless? Derive the proof live
Define the acceptance rate α and what drives it

Mid-level (where most candidates get filtered):

If Transformers are parallel, why can an LLM only output 1 token per pass? (Key: each position predicts P(· | prefixᵢ))
How is the KV cache handled in spec decoding? How do rejected drafts get "rolled back" from the cache?
How do you choose the draft length γ? Why isn't bigger always better? Give the E[#tokens] formula and discuss
When does spec decoding fail to accelerate? (Large batches, low α, a draft model that's too heavy relative to the target — extremely common question)
What are Medusa's and EAGLE's core innovations, and what problems do they solve?

Senior (what separates top candidates):

The fundamental difference between spec decoding and MTP? When to use which?
If the workload becomes compute-bound (large batch, PIM hardware), does spec decoding still matter?
What does token-tree verification buy over single-branch verification?
What's the tension between speculative decoding and continuous batching?

Open-ended design:

Design a production spec-decoding system for a 70B model at 1000 QPS: draft selection, dynamic γ, memory budget
Acceptance rate is stuck at 50% — diagnose (tokenizer mismatch / domain drift / temperature / model-family mismatch)

5.4 Tricks for Interns and New Grads

Use the three key terms unprompted: memory-bound, acceptance rate α, corrected rejection sampling. Interviewers hear them and know you're real
Derive, don't recite: producing min(p,q) + (p−q)₊ = p live raises your level instantly
Volunteer the limitations: "In high-batch serving the gains shrink, because large batches are no longer memory-bound" — interviewers love hearing this
Nail spec decoding vs MTP: "one borrows the memory-bound dividend to attack Bottleneck A; the other attacks A directly" — top-candidate signal

Closing

Speculative decoding is not ML mysticism. It is a clear, learnable engineering optimization. The whole field fits in one sentence:

Paying "move 140 GB of weights" for 1 token is a terrible deal; let a small model guess several tokens, have the big model grade them all in one load, and use corrected rejection sampling to guarantee the output is exactly lossless.

Hold that thread and every variant — Medusa, EAGLE, SpecInfer, MTP, Lookahead — is just a different branch of the same tree.

I hope this pair of articles helps anyone preparing for LLM inference roles. This field genuinely isn't that hard — the key is building projects with measurable numbers.

And if you don't have an elite-school or research pedigree: this is a fully viable path. Companies desperately need engineers who understand inference optimization, and the barrier to entry here is far lower than "training large models." Work through these two articles carefully, build one or two small projects with real measurements, and you're already ahead of most candidates.

References

Academic Papers

[1] Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. Proceedings of the 40th International Conference on Machine Learning (ICML 2023). arXiv:2211.17192.

[2] Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv preprint arXiv:2302.01318.

[4] Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., & Dao, T. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024. arXiv:2401.10774.

[5] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.

[6] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv preprint arXiv:2406.16858.

[8] Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., et al. (2024). SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024.

[9] Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., & Synnaeve, G. (2024). Better & Faster Large Language Models via Multi-token Prediction. ICML 2024.

[10] DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437.

[11] Fu, Y., Bailis, P., Stoica, I., & Zhang, H. (2024). Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024.

[12] Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., & Zhang, H. (2024). Online Speculative Decoding. ICML 2024.

[14] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.

[15] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762.

Industry Blogs

[3] vLLM Team. (2024-10-17). How Speculative Decoding Boosts vLLM Performance by up to 2.8x. https://blog.vllm.ai/2024/10/17/spec-decode.html

[7] vLLM Team. (2025-12-13). Diving into speculative decoding training support for vLLM with Speculators v0.3.0. https://blog.vllm.ai/2025/12/13/speculators-v030.html

[13] vLLM Team. (2025-01-10). vLLM 2024 Retrospective and 2025 Vision. https://blog.vllm.ai/2025/01/10/vllm-2024-wrapped-2025-vision.html

If this helped, the best way to support the series is to share it with someone prepping for inference-engineering interviews. Questions and corrections welcome.

DEV Community: Borui Cai