Speculative Decoding Acceptance Rate: My LLM Got 47% Slower

I turned on speculative decoding on a Friday afternoon expecting a free 2x. My local 32B model went from 34 tokens/sec to 18.

Not noise. Not a warmup artifact. Consistently, reproducibly, half speed. I had added a second model to my GPU, burned VRAM for it, and made everything worse.

The thing nobody tells you: speculative decoding is not a speedup. It's a bet, and the speculative decoding acceptance rate is the odds. Get the odds wrong and you pay for every draft token you throw away.

TL;DR

• Speculative decoding runs a small draft model for k tokens, then verifies all k in one forward pass of the big model. Output distribution is mathematically identical to normal decoding.
• It wins only when the acceptance rate (fraction of drafted tokens the target model keeps) is high enough to pay for the draft model's cost.
• Speedup is E(k) / (1 + k·c) where E(k) = 1 + α + α² + ... + αᵏ, α is acceptance rate, and c is draft-step cost divided by target-step cost.
• My α was 0.71 on code and 0.32 on English prose. At draft length 16, prose ran at 0.50x. Dropping to draft length 4 gave 1.91x on code and break-even on prose.
• Fix in order: measure α per workload, shrink k, then pick a better-matched draft model. Long drafts amplify a bad α instead of rescuing it.

What is speculative decoding actually doing?

Speculative decoding exploits one fact about LLM inference: decoding a single token is memory-bandwidth bound, not compute bound.

To produce one token, your GPU streams every weight in the model from VRAM through the compute units. A 32B model at Q4 means moving ~19 GB per token. The matrix multiplies themselves barely make the GPU sweat. The tensor cores are mostly idle, waiting on memory.

So here's the trick: if you feed the target model 5 candidate tokens instead of 1, it still reads those 19 GB exactly once. Verifying 5 positions costs roughly what verifying 1 position costs. You get four extra tokens of work for free, as long as you had candidates to check.

That's where the draft model comes in. A tiny model (0.5B, same tokenizer family) runs k cheap decode steps and guesses what comes next. The target model then verifies all k guesses in a single pass and accepts the longest prefix that matches what it would have sampled itself.

The verification step uses rejection sampling, so the output distribution is identical to running the target model alone. This isn't an approximation, and you aren't trading quality for speed. (You won't get token-for-token identical output to a non-speculative run with the same seed, because the RNG gets consumed differently, but the distribution is the same.)

That's the whole mechanism. The entire question is how many of those guesses survive.

Why did speculative decoding make my model slower?

Because every rejected draft token is pure waste, and I was drafting 16 of them at a time.

Here's the arithmetic. Let α be the per-token acceptance rate and k the draft length. The expected number of tokens you commit per target forward pass is:

E(k) = 1 + α + α² + ... + αᵏ

The 1 + is the free bonus token: even if the draft model's very first guess is wrong, the target's own verification pass produces a correct token, so you never come out with zero.

The cost of that iteration is one target step plus k draft steps. Call c the ratio of draft-step time to target-step time:

speedup = E(k) / (1 + k·c)

My setup: Qwen2.5-Coder-32B-Instruct at Q4_K_M as the target, Qwen2.5-Coder-0.5B as the draft, single 24 GB card. Measured c ≈ 0.12.

That c surprised me. The draft model has 64x fewer parameters, so I expected it to be 64x cheaper. It isn't. A 0.5B model's decode step is dominated by kernel launches, sampling, and per-step Python overhead, not by memory traffic. Small models don't get cheap in proportion to their size. Eight draft tokens cost me almost as much as a full target step.

Now plug in my two workloads:

Draft length k	Prose, α=0.32	Code, α=0.71
1	1.18x	1.53x
2	1.15x	1.79x
4	0.99x	1.91x
8	0.75x	1.68x
16	0.50x	1.18x

At k=16 on prose the formula predicts 0.50x. I measured 18 tok/s against a 34 tok/s baseline. That's 0.53x. The model wasn't broken. I was spending 16 draft steps to buy, on average, 1.47 tokens.

Notice the code column too. At α=0.71, going from k=4 to k=16 also loses speed, 1.91x down to 1.18x. Long drafts have diminishing returns on the gain side (α^k collapses toward zero) and perfectly linear growth on the cost side. That asymmetry is the whole story.

How do I measure my speculative decoding acceptance rate?

Don't guess it. Every serving stack reports it, you just have to go look.

llama.cpp: run llama-speculative or the server with -md draft.gguf, and the end-of-run stats include n_drafted and n_accept. Your α is n_accept / n_drafted.

llama-server -m qwen2.5-coder-32b-q4_k_m.gguf \
  -md qwen2.5-coder-0.5b-q8_0.gguf \
  --draft-max 4 --draft-min 1

vLLM: scrape the Prometheus endpoint for vllm:spec_decode_num_accepted_tokens_total and vllm:spec_decode_num_draft_tokens_total. Divide one by the other. Config shape depends on your version, recent builds take a JSON blob:

--speculative-config '{"model": "Qwen/Qwen2.5-Coder-0.5B",
                       "num_speculative_tokens": 4}'

Then sweep k offline before you touch the server. Six lines:

def speedup(alpha, k, c):
    accepted = sum(alpha ** i for i in range(k + 1))
    return accepted / (1 + k * c)

for k in range(1, 17):
    print(k, round(speedup(0.32, k, 0.12), 2),
             round(speedup(0.71, k, 0.12), 2))

Measure α on a real sample of your traffic, measure c with two quick benchmarks, then read the optimal k off the table. This took me ten minutes and I should have done it before I started.

What moves the acceptance rate?

α is not a property of your models. It's a property of your workload, and it swings hard.

• Predictable text wins. Code, JSON, boilerplate, and anything that echoes the prompt draft well. My highest α was a structured-output endpoint at 0.83. Brace, quote, key name, closing brace: a 0.5B model nails that.
• Open-ended prose loses. Summaries and chat replies sat at 0.32. The draft model and the target genuinely disagree about what the next adjective should be, and there's no fixing that with config.
• Temperature matters. On the same code prompts, going from T=0 to T=0.8 dropped my α from 0.71 to 0.58. Higher temperature means the target samples further from the draft's mode more often.
• Tokenizer mismatch is fatal. Draft and target must share a vocabulary. A different tokenizer isn't "lower acceptance," it's broken output or a hard error.
• Batch size can invert the whole thing. At batch size 1 verification is nearly free because the GPU was idle anyway. At high concurrency your target step is already compute-saturated, so verifying k extra positions across B sequences costs real FLOPs. Speculative decoding can reduce aggregate throughput on a busy server even with a great α. Benchmark at your actual concurrency, not at batch 1.

What I'd do differently

Route by workload instead of flipping one global flag.

My code-completion path runs with --draft-max 4 and gets a real 1.9x. My summarization path runs with speculative decoding off, because the best possible config there was 1.18x at k=1 and that isn't worth a second model in VRAM.

For the one endpoint that rewrites files I switched to n-gram speculation ("method": "ngram" in vLLM) instead of a draft model. When the output is mostly copied from the input, matching prompt n-grams is a near-perfect draft with c close to zero. No second model, no extra VRAM.

And if you're serious about a draft model, a trained speculator head like EAGLE or Medusa is a different class of tool. Those are trained against your specific target's hidden states, which is exactly the α problem attacked at the root instead of by tuning k.

So why does speculative decoding make some models slower?

Speculative decoding makes your LLM slower whenever the acceptance rate is too low to pay for the draft model's compute. The speedup is E(k)/(1 + k·c), where the gain from a longer draft decays geometrically as α^k while the cost grows linearly as k·c. On unpredictable text with α around 0.3, a draft length of 16 spends 16 cheap forward passes to buy roughly 1.5 tokens and lands near half your original throughput. Measure α per workload from your server's accepted-vs-drafted counters, sweep k with the formula before deploying, and keep in mind that the right k for JSON generation is not the right k for chat.

---

Written by the developer behind Preterview, an interview prep platform.