Speculative Decoding: Why a Great Draft Model Still Caps Speedup

A 7B draft model that agrees with your 70B target 90% of the time sounds like it should make decoding roughly 10x faster. It won't. In practice you'll see something closer to 2-3x, and if your server runs large batches you may see nothing — or a regression. The gap between "the draft is usually right" and "the system is actually faster" is where most speculative decoding deployments quietly disappoint. This post is the math that explains why, and the two numbers you should actually measure before turning it on.

TL;DR

• Speculative decoding runs a cheap draft model to propose k tokens, then verifies all of them in one forward pass of the expensive target model. Output is provably identical to sampling from the target alone.
• The speedup is governed by the acceptance rate α, not draft accuracy. Expected accepted tokens per target step = (1 - α^(k+1)) / (1 - α) — a saturating curve, not linear.
• At α = 0.7 and k = 4 you accept ~2.6 tokens per target pass. That's your ceiling before you subtract draft and verification overhead.
• It works because decoding is memory-bound at low batch size: verifying 5 tokens costs almost the same wall-clock as generating 1. That free lunch vanishes as batch size grows and you become compute-bound.
• Speculative decoding is a latency optimization, not a throughput one. On a saturated server it can slow you down.

What is speculative decoding actually doing?

Speculative decoding trades cheap FLOPs for expensive memory bandwidth. A small draft model autoregressively generates k candidate tokens. The large target model then processes all k in a single forward pass — because with the candidate sequence in hand, you can compute the target's probability distribution at every one of those positions in parallel, exactly like prefill. You then verify the draft tokens left to right and keep the longest correct prefix.

The non-obvious part is that this is lossless. Naively you might "accept the draft token if the target agrees," but that changes the output distribution. The correct algorithm is modified rejection sampling: for a draft token x with draft probability q(x) and target probability p(x), accept with probability min(1, p(x)/q(x)). On rejection, resample from the normalized residual distribution (p(x) - q(x))₊. This provably yields samples distributed exactly as if you'd sampled from the target model directly. No accuracy trade-off, no eval regression — the outputs are statistically the target's.

# One verification step. draft_probs/target_probs are [k, vocab] tensors
# for the k drafted positions; draft_tokens is [k].
def verify(draft_tokens, draft_probs, target_probs, rng):
    accepted = []
    for i, tok in enumerate(draft_tokens):
        q = draft_probs[i, tok]
        p = target_probs[i, tok]
        if rng.random() < min(1.0, p / q):
            accepted.append(tok)              # keep and continue
        else:
            # reject: resample from (p - q)_+, renormalized
            resid = (target_probs[i] - draft_probs[i]).clamp(min=0)
            accepted.append(sample(resid / resid.sum(), rng))
            return accepted                   # stop at first rejection
    # all k accepted -> sample one bonus token from target's (k+1)th head
    accepted.append(sample(target_probs[-1], rng))
    return accepted

Note the bonus token: if all k drafts are accepted, the target already computed the distribution for position k+1 in the same pass, so you get one extra token for free. That's why the formula below runs to k+1.

Why won't a 90%-accurate draft give 10x?

Because rejection is a geometric process, and one wrong token throws away everything after it. Let α be the per-token acceptance probability (the expected min(1, p/q), averaged over the distribution — not the same as top-1 agreement). The number of accepted tokens in a block of k follows a truncated geometric distribution. The expected number of tokens produced per target forward pass is:

E[tokens] = (1 - α^(k+1)) / (1 - α)

Plug in real numbers:

α	k=3	k=5	k=7	k→∞ ceiling
0.6	2.18	2.72	3.10	2.50
0.7	2.53	3.19	3.61	3.33
0.8	2.95	3.69	4.16	5.00
0.9	3.44	4.10	4.70	10.00

Two things jump out. First, α=0.9 does have a 10x ceiling — but only at k→∞, and you never run infinite k because draft cost grows linearly with k. At a practical k=5 you get 4.1x, not 10x. Second, the curve saturates hard: going from k=3 to k=7 at α=0.7 moves you from 2.53 to 3.61, more than doubling draft work for a 43% gain. Every extra drafted token has lower marginal acceptance (α^i shrinks), so there's an optimal k, usually 3-5.

And "90% accurate" almost never means α=0.9. Top-1 agreement on easy tokens (whitespace, common words) is high, but α is the distribution-weighted acceptance under rejection sampling, dragged down by exactly the hard, high-entropy tokens where the draft and target disagree — and those are the tokens that matter. Real α for a well-matched draft/target pair tends to land in 0.6-0.8.

Where does the speedup actually come from?

From the roofline. During decode, generating one token requires reading the entire model's weights (and the KV cache) from HBM to compute a single column of activations. Arithmetic intensity is terrible — you're memory-bandwidth-bound, and the GPU's compute units sit mostly idle. Verifying k drafted tokens in one pass reads those same weights once and does k× the FLOPs. Since you had spare compute, those extra FLOPs are nearly free in wall-clock terms.

So the real speedup is:

speedup ≈ E[tokens] / (1 + c)

where c is the overhead ratio: draft generation time plus the marginally larger verification pass, expressed as a fraction of one target decode step. If your draft is 1/10 the size of the target and you run k=4, the k sequential draft steps plus overhead might cost c ≈ 0.3-0.5 of a target step. At α=0.7, k=4, E[tokens] ≈ 2.87, so speedup ≈ 2.87 / 1.4 ≈ 2.05x. That matches what people report. The draft isn't free, and it's sequential — those k draft tokens are themselves autoregressive, which is why models like Medusa (parallel heads) and EAGLE (feature-level drafting) exist: they cut c by drafting cheaply, not by raising α.

Why does speculative decoding fail at high batch size?

Because batching already fixed the problem speculative decoding solves. The free-FLOPs argument holds only while you're memory-bound. As you increase batch size, you amortize each weight read across many sequences, arithmetic intensity climbs, and you cross into the compute-bound regime. Now those extra verification FLOPs are not free — you're paying full price for every speculated token, most of which you'll reject and throw away.

At large batch, speculative decoding does strictly more compute for the same accepted output, so it becomes a throughput tax. This is the single most common reason teams turn it on, benchmark it under load, and see a regression. The mental model:

• Low batch / latency-critical (interactive chat, single-user agents, batch=1): memory-bound, speculative decoding shines, expect 1.5-3x.
• High batch / throughput-critical (offline batch jobs, saturated serving): compute-bound, speculative decoding hurts. Turn it off or use dynamic speculation that backs off as batch grows.

vLLM and TensorRT-LLM both expose this. A minimal vLLM config:

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",  # the draft
        "num_speculative_tokens": 4,                   # this is k
    },
)
# Serve interactive traffic through this; keep a non-speculative
# engine for bulk/offline batches. Measure acceptance rate in the logs
# before trusting the speedup.

Set num_speculative_tokens (k) to 3-5 and read the acceptance-rate metric vLLM emits. If α is under ~0.5, your draft is mismatched — a smaller, better-distilled, or same-family draft usually helps more than raising k.

Does temperature change the acceptance rate?

Yes, and it cuts both ways. At temperature 0 (greedy), acceptance collapses to "does the draft's argmax equal the target's argmax," which is often high on easy spans but brittle. As temperature rises, both p and q flatten, and the min(1, p/q) acceptance can actually increase because the two distributions overlap more — a flat target forgives a wrong draft. But high temperature also means more genuinely random high-entropy tokens where no draft can predict the sample. Net effect is workload-dependent, which is exactly why you must measure α on your real traffic and not trust a paper's number from a different sampling setup.

The bottom line

Speculative decoding is a lossless latency optimization that exploits the fact that LLM decode is memory-bound at low batch size: a cheap draft proposes k tokens and the target verifies them in one bandwidth-limited pass, with modified rejection sampling guaranteeing the exact target distribution. But the speedup is capped by (1 - α^(k+1))/(1 - α) divided by draft overhead, so a "90% accurate" draft realistically buys 2-3x, not 10x — because α is distribution-weighted, k has a sweet spot around 3-5, and the sequential draft isn't free. Most importantly, the entire benefit rests on being memory-bound: crank up your batch size into the compute-bound regime and speculative decoding turns from a speedup into a tax. Measure your acceptance rate and your batch size before you turn it on, and keep a non-speculative path for bulk work.