Speculative Decoding: Why 80% Acceptance Still Loses at Batch 64

A 1B draft in front of a 70B target. 80% token acceptance. Four speculative tokens per step. Your dashboard says the draft model is doing its job. Then you raise concurrency from 8 to 64 and throughput drops below the non-speculative baseline you were trying to beat.

Nothing broke. Speculative decoding is a latency optimization that you pay for in FLOPs, and at batch 64 you ran out of free FLOPs to spend. Acceptance rate was never the number that decided this.

TL;DR

• Acceptance rate is exactly 1 - TV(p, q), the total variation distance between target and draft distributions at that position. It is not "how often the draft is right."
• Expected tokens per step is a geometric sum, not γ × α. At α = 0.8 and γ = 4 you get ~3.36 tokens per verify step, not 3.2 accepted plus wishful thinking.
• Per-position acceptance decays because the draft conditions on its own drift. The 5th draft token is worth ~0.1 tokens of expected output while costing a full draft forward pass. γ = 3–4 is usually the peak.
• Speculative decoding trades memory bandwidth for compute. It wins only while decode is memory-bound. Batch B with γ speculative tokens puts B × (γ+1) tokens in flight; past roughly 300 on an H100 you are compute-bound and rejected tokens become pure waste.
• Measure tokens_out / target_forward_FLOPs, not acceptance. If that ratio drops, turn it off for that traffic shape.

If you call Claude Opus 4.5 or GPT-5.x over an API, this is your provider's problem. If you run vLLM, SGLang, or TensorRT-LLM yourself, it's yours.

What is the speculative decoding acceptance rate, exactly?

The acceptance rate is 1 - TV(p, q) where p is the target distribution and q is the draft distribution at the same position. That identity is exact, not an approximation.

The standard scheme (Leviathan et al., Chen et al., 2023) samples x ~ q(x) and accepts with probability min(1, p(x)/q(x)). Integrate over the draft's own sampling:

P(accept) = Σ_x q(x) · min(1, p(x)/q(x))
          = Σ_x min(q(x), p(x))
          = 1 - TV(p, q)

On rejection you resample from the normalized residual max(0, p - q). That correction is what makes the output distribution identical to the target's — speculative decoding is not an approximation, and you should refuse any implementation that claims a speed win by relaxing it.

Two consequences people get wrong:

Greedy is a special case, not the easy case. At temperature 0 both distributions collapse to one-hot, so acceptance equals top-1 agreement. Nothing about the tail matters. At T = 1 you are matching the whole distribution, and a draft that agrees on argmax 85% of the time can still have TV = 0.35. Acceptance is not monotone in temperature. Measure it per temperature; don't extrapolate from your greedy eval.

Sampler mismatch silently destroys both properties. If your stack applies top-p to the target but not the draft, or applies them in a different order relative to temperature, you get a different q than the acceptance test assumes. Acceptance falls, and the distributional guarantee is gone. This is the single most common real bug I see in hand-rolled speculative loops.

Why doesn't 80% acceptance mean a 5x speedup?

Because a run of accepted tokens terminates on the first rejection, so expected output per step is a geometric sum, and because the draft passes aren't free.

Let α_i be acceptance at draft position i. The target's verify pass always emits at least one token: either the correction at the first rejection, or the bonus token from the target's final position when all γ drafts are accepted. So:

def expected_tokens(alphas):
    """E[tokens emitted per verify step]. The leading 1.0 is the token the
    target always produces: correction on rejection, or bonus on full accept."""
    e, run = 1.0, 1.0
    for a in alphas:
        run *= a
        e += run
    return e

def speedup(alphas, c):
    """c = draft forward cost / target forward cost, in the memory-bound regime."""
    return expected_tokens(alphas) / (len(alphas) * c + 1.0)

With uniform α = 0.8, γ = 4: 1 + .8 + .64 + .512 + .4096 = 3.36 tokens per step. With a draft costing 10% of a target pass, the step costs 1.4 target-equivalents. Net: 2.40x, not 5x.

Forgetting the bonus token is a real regression, not a rounding error. It costs you a full token on every fully-accepted block — at α = 0.8, γ = 4 that is 41% of blocks and roughly 12% of total throughput.

Why does per-position acceptance decay make γ = 5 pointless?

Because the draft conditions on tokens it generated itself, so it drifts further from the target with each speculative step. Acceptance is a decaying sequence, and the marginal token requires every prior one to be accepted.

A realistic measured profile for a small draft against a large target looks like [0.80, 0.72, 0.66, 0.60, 0.55]. Run the model:

γ	E[tokens]	step cost (c=0.1)	speedup
2	2.376	1.2	1.98x
3	2.756	1.3	2.12x
4	2.984	1.4	2.13x
5	3.109	1.5	2.07x

The 4th draft token contributes 0.8 × 0.72 × 0.66 × 0.60 = 0.228 expected tokens for 0.1 target-equivalents of cost. The 5th contributes 0.125 and loses money. γ = 4 is the peak and γ = 3 is within noise of it, which means the cheaper config is the better config once you account for the extra KV memory the longer draft window holds.

One genuinely good piece of news: acceptance is bursty, not i.i.d. Long runs of easy tokens (JSON scaffolding, indentation, copied spans from the prompt) alternate with hard, high-entropy decisions. Σ α^i is convex in α, so by Jensen's inequality a bursty α with mean 0.7 beats a flat α of 0.7. Your average acceptance understates your real throughput. Log the distribution of accepted-run lengths, not the mean.

Why does speculative decoding lose at batch 64?

Because it converts a memory-bandwidth problem into a compute problem, and at batch 64 you no longer have spare compute.

Single-token decode is bandwidth-bound. You stream every weight from HBM to produce one token per sequence. Verifying γ+1 tokens in one forward pass reads the same weights, so the extra positions are nearly free. That is the entire trick — not "the draft is smart," but "the target's verify pass was already idle on FLOPs."

Now the roofline. An H100 SXM does roughly 990 TFLOP/s dense BF16 against about 3.35 TB/s of HBM3, so the crossover is near 300 FLOP per byte. A BF16 GEMM with B rows does about B FLOP per weight byte. You need on the order of 300 tokens in flight to saturate compute.

Speculative decoding multiplies tokens in flight by γ+1:

• Batch 8, γ = 4 → 40 tokens in flight. Deep in the bandwidth-bound regime. Free lunch.
• Batch 64, γ = 4 → 320 tokens in flight. Past the crossover. Verification now costs ~5x the FLOPs of a plain decode step, and at α = 0.8 roughly a third of those positions are discarded.

Continuous batching already gave you the throughput win that speculation was faking. Stacking them means paying twice for the same thing. GQA, MoE routing, and FP8 all move the crossover, so measure yours — but the shape holds everywhere.

The practical rule: speculative decoding is for low-concurrency, latency-sensitive traffic. Interactive coding agents, single-user local inference, tail-latency SLOs. It is not for batch summarization jobs.

How do you configure this in vLLM?

Keep the draft small enough that c stays under ~0.15, and gate speculation on concurrency.

from vllm import LLM

# Draft-model speculation: good for low-concurrency interactive serving.
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    max_num_seqs=16,                 # keep tokens-in-flight under the roofline
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 3,  # peak, not max
    },
)

# N-gram speculation: no draft model, no extra weights, no extra KV.
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 5,
        "prompt_lookup_max": 4,
    },
)

Flag names have moved across vLLM versions (the old --speculative-model family folded into speculative_config); check yours. What matters is that max_num_seqs × (num_speculative_tokens + 1) stays on the bandwidth-bound side of your hardware's crossover.

When should you use n-gram speculation instead of a draft model?

When the output copies substantially from the input. Then the best draft model is no model at all.

N-gram (prompt-lookup) speculation matches the last few generated tokens against the prompt and proposes the continuation verbatim. Zero draft parameters, zero draft KV, c ≈ 0. On workloads where the output quotes the input, acceptance runs long and the cost model has no denominator to speak of:

• RAG answers that quote retrieved passages
• Code edits that reproduce most of the original file
• Structured extraction echoing field values from a document
• Any "rewrite this with X changed" task

On free-form generation it does nothing, and because c ≈ 0 it costs nothing when it fails. That asymmetry makes it the default I'd reach for first. Only when it demonstrably doesn't fire do I add a draft model.

EAGLE-style methods sit between the two: they draft in the target's feature space and verify a tree of candidate branches instead of a chain, which raises acceptance for a given γ. Note the cost-model implication — tree verification puts even more tokens in flight per step, so it hits the compute roofline at a lower batch size than chain speculation does. Higher acceptance, narrower operating window.

Failure modes worth a runbook entry

• KV rollback off-by-one. Rejected positions leave KV entries that must be truncated before the next step. Off-by-one here produces fluent, subtly wrong output that no unit test catches.
• Draft/target tokenizer mismatch. Different vocabularies make p(x)/q(x) meaningless. Use a draft from the same family, or do explicit vocab mapping.
• Acceptance measured on the wrong traffic. Chat evals overstate acceptance for agentic tool-call traffic, where outputs are short, schema-constrained, and high-entropy at exactly the tokens that matter.
• TTFT regression. Draft weights and draft KV shrink the KV budget for real sequences, cutting how many requests fit and pushing queueing delay up. Watch TTFT, not just inter-token latency.

So does speculative decoding actually help?

Speculative decoding helps when decode is memory-bound and hurts when it is compute-bound, and acceptance rate tells you almost nothing about which side you're on. The acceptance rate is exactly 1 - TV(p, q); expected output per verify step is the geometric sum 1 + Σ Π α_i, which caps out near γ = 3–4 once per-position decay is accounted for; and the real speedup is that sum divided by γc + 1. The decisive variable is tokens in flight: batch × (γ+1). Below your hardware's roofline crossover — roughly 300 on an H100 — the extra verified positions ride along for free and you get a genuine 2–2.5x on inter-token latency. Above it, you are spending real FLOPs on tokens you will throw away, and 80% acceptance at batch 64 loses to plain continuous batching. Run speculation on your low-concurrency latency tier, use n-gram speculation on copy-heavy workloads where it's nearly free, and instrument tokens_out / target_forward_FLOPs so the regression shows up before your throughput graph does.