Speculative Decoding: Why Batching Kills Your 3x Speedup

You benchmark speculative decoding on one request against a 70B target with a 1B draft, see 3.1x, ship it, and watch p50 latency get worse the moment real traffic arrives. Nothing is misconfigured. The acceptance rate is unchanged. The speedup evaporated because speculative decoding is not a compute optimization — it is a trade that spends FLOPs to buy back memory bandwidth, and batching already spent the same currency.

TL;DR

• Speculative decoding wins at batch 1 because a decode forward pass is memory-bandwidth-bound: streaming 140 GB of weights costs the same whether you push 1 token or 200 through it. Verifying k+1 tokens is nearly free.
• The verification pass stops being free once batch_size × (k+1) exceeds your accelerator's FLOPs-per-byte ratio. For bf16 on an H100 (≈989 TFLOP/s ÷ ≈3.35 TB/s) that's about 295 tokens per forward pass — and the threshold is independent of model size for dense models.
• Past that point you pay full compute for every speculated token but keep only the accepted ones. With α=0.8 and k=4 you keep ~3.36 of 5, so goodput drops toward 67% and speculation becomes a throughput tax.
• Expected accepted tokens per round is (1 - α^(k+1)) / (1 - α), where α ≈ 1 - TV(p, q) — the distributional overlap between draft and target. α is workload-specific; measure it per route, not once.
• Fix: gate speculation on running batch size / queue depth, shrink k under load, and prefer MoE targets or n-gram drafts where the economics stay favorable.

Why does speculative decoding make decoding faster at all?

Not because the draft model is smart. Because autoregressive decode wastes the GPU.

A single decode step on a 70B bf16 model must read ~140 GB of weights from HBM and does roughly 2×70e9 = 140 GFLOPs of work per token. That is ~1 FLOP per byte. An H100 wants ~295 FLOPs per byte to saturate its tensor cores. At batch 1 you are running the GPU at well under 1% of peak FLOPs; the SMs sit idle waiting on memory.

Speculative decoding exploits the slack. The draft model autoregressively proposes k tokens. The target then scores all k+1 positions in one forward pass, because it can attend over the proposed tokens in parallel — same weight streaming, more tokens through it. Modified rejection sampling then accepts a prefix and resamples at the first rejection from the residual max(0, p - q), normalized. The output distribution is provably identical to sampling from the target alone. You get free tokens out of unused arithmetic.

That last clause is the whole thing. Free tokens out of unused arithmetic. Once the arithmetic isn't unused, the tokens aren't free.

Why does the speculative decoding speedup vanish at large batch sizes?

Because continuous batching is the other way to fill the same idle FLOPs, and it's strictly more efficient at it. Running 64 sequences through one forward pass gives you 64 tokens per weight-stream with 100% goodput. Speculation gives you k+1 tokens per sequence with acceptance-rate goodput. Both compete for the same headroom, and once that headroom is gone, only speculation keeps paying for tokens it throws away.

Concretely: the verification pass processes B × (k+1) tokens. The baseline processes B. While both are under the memory-bound threshold, they cost the same wall-clock — pure profit. Above the threshold, verification costs (k+1)× the baseline in compute, and you accepted maybe 2.5 of 5.

What is the exact crossover point?

Set memory time equal to compute time for one forward pass with N total tokens:

weight_bytes / HBM_BW  =  2 × params × N / FLOPS

For bf16, weight_bytes = 2 × params, so params cancels:

N_crossover = FLOPS / HBM_BW

The crossover is your hardware's machine balance — nothing else. H100 SXM: ~989e12 / ~3.35e12 ≈ 295 tokens. It's the same number for a 7B model and a 70B model, and tensor parallelism doesn't move it (TP splits weights and FLOPs equally).

Two important adjustments:

• MoE raises it. Weight traffic scales with total params, compute with active params. Crossover becomes 295 × (P_total / P_active). A sparse model with 1/8 activation stays memory-bound to ~2400 tokens per pass, which is why speculation holds up far better on MoE serving.
• Long context lowers effective headroom. KV-cache reads add memory traffic that scales with B × context, and speculative rollback forces you to reserve k+1 slots per sequence per step — less KV headroom means a smaller max batch, which cuts into the throughput you were trying to protect.

Here is the model end to end, including draft cost and acceptance:

HBM_BW      = 3.35e12    # H100 SXM, bytes/s
BF16_FLOPS  = 989e12     # H100 SXM, dense bf16
TARGET_P    = 70e9
DRAFT_P     = 1e9

def fwd_time(params, n_tokens):
    """Roofline: stream weights once, do 2*P FLOPs per token, take the max."""
    mem     = (2 * params) / HBM_BW
    compute = 2 * params * n_tokens / BF16_FLOPS
    return max(mem, compute)

def expected_accepted(alpha, k):
    """Leviathan-style geometric model, includes the bonus token."""
    if alpha >= 1.0:
        return k + 1
    return (1 - alpha ** (k + 1)) / (1 - alpha)

def speedup(batch, k, alpha):
    baseline = fwd_time(TARGET_P, batch)                  # 1 token/seq
    draft    = k * fwd_time(DRAFT_P, batch)               # k sequential steps
    verify   = fwd_time(TARGET_P, batch * (k + 1))
    return expected_accepted(alpha, k) * baseline / (draft + verify)

for b in (1, 16, 32, 64, 128, 256):
    print(b, [round(speedup(b, k=4, alpha=a), 2) for a in (0.9, 0.8, 0.6)])

Output (α = 0.9 / 0.8 / 0.6, k=4):

  1   3.90  3.18  2.18
 16   3.90  3.18  2.18
 32   3.90  3.18  2.18
 64   3.63  2.95  2.03
128   1.86  1.51  1.04
256   0.94  0.77  0.53

Read the last row: at batch 256 with a good draft, speculative decoding makes you 23% slower, losslessly and silently.

Treat these as an upper bound. The roofline ignores the k sequential draft launches (latency-bound; without CUDA graphs their real cost is easily 5–10x the model's estimate), rejection-sampling kernels, ragged rollback when sequences in the batch accept different lengths, and the padding waste that comes with it. In practice degradation shows up well before batch 128 on dense models.

How much does the acceptance rate actually matter?

α is the single biggest lever after batch size, and it is not a property of the draft model — it's a property of the draft/target/workload triple.

Per position, the expected acceptance probability is Σ_x min(p(x), q(x)) = 1 - TV(p, q): total-variation overlap between the target and draft distributions. That yields two non-obvious consequences.

Structured, low-entropy text has high α. JSON, code with a fixed skeleton, boilerplate, and quoted spans from a RAG context all have near-degenerate p, so even a weak draft matches. Open-ended reasoning has high entropy and low overlap. The same deployment can show α=0.85 on tool-call routes and α=0.45 on chat.

Draft/target sampling-parameter mismatch silently breaks it. The correction term max(0, p - q) is only valid if q is the exact distribution you sampled the draft token from. Apply top-p to the target and forget it on the draft (or run different temperatures on each) and you don't just lose speed — you lose the losslessness guarantee and get a distribution shift with no error, no warning, and no test that catches it. Same for tokenizer or chat-template drift between draft and target: α collapses toward zero and you're paying k draft passes for nothing.

How should you configure speculative decoding in vLLM?

Start conservative on k and gate on load. Recent vLLM takes a nested dict (the older --speculative-model / --num-speculative-tokens CLI flags moved; check your version, this area churns):

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=8,
    max_num_seqs=32,
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 4,
        "draft_tensor_parallel_size": 1,   # don't shard a 1B model across 8 GPUs
    },
)

draft_tensor_parallel_size: 1 matters more than it looks. Sharding a 1B draft over TP=8 makes every draft step an all-reduce-dominated latency floor, and you run k of them sequentially per round.

For copy-heavy workloads — RAG with quoted evidence, code edits, document rewriting — skip the draft model entirely and use n-gram / prompt-lookup speculation. Draft cost is essentially zero (a string match against the prompt), so even α=0.4 is pure profit:

speculative_config={
    "method": "ngram",
    "num_speculative_tokens": 5,
    "prompt_lookup_max": 4,
    "prompt_lookup_min": 2,
}

When should you turn speculative decoding off?

Whenever you are throughput-limited rather than latency-limited. The decision rule is load, not model:

• Low QPS, latency SLO on TTFT+ITL (interactive coding assistants, voice, single-user local inference): speculate aggressively, k=4–6.
• High QPS, throughput SLO (batch enrichment, offline eval, embedding pipelines): turn it off. Continuous batching already fills the FLOPs, and speculation just burns them on rejected tokens.
• Mixed traffic: gate dynamically. vLLM has shipped a batch-size disable threshold (the flag name has moved across releases); if yours doesn't expose one, run two pools and route by queue depth. Alternatively scale k down as the running batch grows — k=6 at batch 1, k=1 at batch 64, k=0 above your measured crossover.

And instrument it. Log the accepted-token histogram, not just a global mean: a bimodal distribution means you have two workloads sharing one config and one of them is subsidizing the other.

The short answer

Speculative decoding gives a 3x speedup at batch 1 because decode is memory-bandwidth-bound — verifying k+1 tokens in one pass costs the same weight streaming as verifying one, so the extra tokens ride along on idle tensor cores. Batching consumes exactly that same idle capacity, and once batch_size × (k+1) exceeds your accelerator's FLOPs-per-byte ratio (≈295 tokens for bf16 on H100, higher for MoE), the verification pass becomes compute-bound and every speculated token costs real FLOPs. Since only (1 - α^(k+1))/(1 - α) of the k+1 tokens are accepted, you pay full price for work you discard, and speculation flips from a 3x win to a measurable throughput loss. Treat it as a latency optimization for low-load serving, gate it on running batch size, and measure acceptance rate per workload rather than trusting a single benchmark number.