DEV Community

jidonglab
jidonglab

Posted on

Speculative Decoding Acceptance Rate: Why 70% Caps You Near 2x

The dashboard said 44% draft acceptance. My first instinct was to rip speculative decoding out of the serving config. Less than half the draft tokens survived, so the draft model was clearly wasting GPU time, right?

Wrong. A 44% speculative decoding acceptance rate on that dashboard meant the draft model agreed with the big model about 70% of the time per token. That number was fine. The thing actually limiting my speedup was something the dashboard didn't show at all: how many requests were sharing the GPU.

This post is the math I wish I'd done before touching the config. It's short, it fits in a Python function, and it explains almost every confusing speculative decoding result you'll see in production.

TL;DR

  • Speculative decoding has a small draft model guess k tokens, then the big target model checks all k in one forward pass. Output quality is unchanged; only speed changes.
  • With per-token acceptance α, the expected tokens per target pass is (1 - α^(k+1)) / (1 - α). At α = 0.7, k = 4 that's 2.77 tokens, and after paying for the draft model (10% of target cost per step) the ceiling is about 1.98x.
  • "Acceptance rate" on dashboards is often accepted / drafted tokens, which is much lower than α. 44% accepted/drafted at k = 4 corresponds to α ≈ 0.7.
  • More draft tokens is not better. At α = 0.7, k = 4 peaks at 1.98x; k = 8 drops to 1.78x.
  • The speedup comes from idle compute in memory-bound decoding. Large batches use up that idle compute, and speculative decoding can become a net loss.

How does speculative decoding actually work?

Speculative decoding works because a big model verifying 5 tokens costs about the same as generating 1. At small batch sizes, decoding is bound by memory bandwidth. The GPU spends its time streaming weights from HBM, and the math units mostly sit around waiting.

So you let a cheap model run ahead:

  1. The draft model generates k tokens autoregressively (cheap, because it's small).
  2. The target model runs one forward pass over all k positions plus one extra.
  3. Walk left to right. Accept draft token x with probability min(1, p(x) / q(x)), where p is the target's distribution and q is the draft's.
  4. At the first rejection, resample that position from the leftover distribution max(0, p - q) (normalized), and throw away everything after it.
  5. If all k pass, you also get the target's next token for free.

That rejection-sampling step is the clever part. It guarantees the output follows exactly the target model's distribution. Speculative decoding is lossless. If your outputs change in quality after enabling it, you have a bug somewhere else, not a tradeoff.

It also guarantees every target pass produces at least one token. The worst case is roughly normal decoding plus the wasted draft work.

How many tokens does one verification pass produce?

If each draft token is accepted independently with probability α, the expected number of tokens per target forward pass is:

E[tokens] = (1 - α^(k+1)) / (1 - α)
Enter fullscreen mode Exit fullscreen mode

This comes from the original speculative decoding analysis (Leviathan et al., 2023). The independence assumption isn't perfectly true, but it's close enough to reason with.

Here's what it gives at k = 4:

per-token α tokens per target pass
0.5 1.94
0.6 2.31
0.7 2.77
0.8 3.36
0.9 4.10

Notice how brutal the curve is. Acceptance is a chain: token 3 only counts if tokens 1 and 2 also survived. At α = 0.7, the chance all four survive is 0.7^4 = 0.24. Most of the time you're throwing away the tail of the draft.

Why does 70% acceptance cap you near 2x?

Because tokens per pass isn't speedup. You also pay for the draft model. If one draft step costs c relative to one target step, the rough speedup is:

speedup ≈ E[tokens] / (k * c + 1)
Enter fullscreen mode Exit fullscreen mode

With α = 0.7, k = 4, c = 0.1:

2.77 / (4 * 0.1 + 1) = 2.77 / 1.4 = 1.98x
Enter fullscreen mode Exit fullscreen mode

That's the ceiling under ideal memory-bound conditions. Real kernels add overhead for the extra scheduling and the verification bookkeeping, so expect to land below it.

Why c = 0.1 when a 1B draft against a 70B target is a parameter ratio closer to 0.014? Because a tiny model doesn't run 70x faster. Kernel launches, attention over the KV cache, and sampling all have fixed costs that don't shrink with parameter count. Measure c on your own hardware: time one draft decode step and one target decode step at the same batch size.

Why is my speculative decoding acceptance rate so low on the dashboard?

Because the dashboard is probably reporting accepted draft tokens divided by drafted tokens, and that number is mechanically lower than per-token α.

Expected accepted draft tokens per step is α + α² + ... + α^k. At α = 0.7, k = 4:

0.7 + 0.49 + 0.343 + 0.2401 = 1.77 accepted out of 4 drafted = 44%
Enter fullscreen mode Exit fullscreen mode

So "44% acceptance" and "70% acceptance" can describe the exact same system. Going the other way, if your dashboard shows 70% accepted/drafted at k = 4, the per-token α is about 0.86, which is excellent and puts you around 3.6 tokens per pass.

This is the metric artifact that nearly made me turn the feature off. Before you compare numbers across blog posts, papers, or your own experiments, check which definition you're looking at. Mean acceptance length (tokens emitted per target pass) is the least ambiguous one. Recent vLLM versions also log per-position acceptance rates, which show the chain decay directly: position 1 high, position 4 much lower.

Here's the converter I keep around:

def tokens_per_pass(alpha: float, k: int) -> float:
    return (1 - alpha ** (k + 1)) / (1 - alpha)

def speedup(alpha: float, k: int, c: float) -> float:
    return tokens_per_pass(alpha, k) / (k * c + 1)

def alpha_from_dashboard(accepted_over_drafted: float, k: int) -> float:
    """Invert accepted/drafted back to per-token alpha via bisection."""
    lo, hi = 0.0, 0.9999
    for _ in range(60):
        mid = (lo + hi) / 2
        accepted = mid * (1 - mid ** k) / (1 - mid)
        if accepted / k < accepted_over_drafted:
            lo = mid
        else:
            hi = mid
    return lo

print(round(alpha_from_dashboard(0.44, 4), 3))  # ~0.70
print(round(speedup(0.70, 4, 0.10), 2))          # 1.98
Enter fullscreen mode Exit fullscreen mode

How many draft tokens should I use?

Fewer than you think. Each extra draft token costs a full draft step, but only pays off if every token before it was accepted. At α = 0.7, c = 0.1:

k tokens per pass speedup ceiling
1 1.70 1.55x
2 2.19 1.83x
3 2.53 1.95x
4 2.77 1.98x
5 2.94 1.96x
6 3.06 1.91x
8 3.20 1.78x

The peak is flat and early. Going from k = 4 to k = 8 doubles your draft work for a worse result. If your α is lower, say 0.5, the optimum moves down to 2 or 3. Only push k up when α is genuinely high, like code completion or edits that copy large chunks of the prompt.

Why does speculative decoding get slower under heavy load?

Because the speedup is paid for with spare compute, and batching spends that same spare compute.

Rough numbers for an H100 SXM: about 990 TFLOPS of dense BF16 compute and about 3.35 TB/s of HBM bandwidth. That's roughly 295 FLOPs per byte before the GPU becomes compute-bound. A BF16 weight is 2 bytes and costs about 2 FLOPs per token that passes through it, so each token in the batch adds about 1 FLOP per byte of weight traffic. Ignoring attention and KV cache traffic, you need on the order of 300 tokens in flight per forward pass before the math units are actually busy.

At batch size 1, a 70B model reads 140 GB of weights to do 140 GFLOPs of work. The compute units are idle more than 99% of the time. Verifying 5 tokens instead of 1 is basically free.

Now run 60 concurrent sequences with k = 4. Every verification pass pushes 60 × 5 = 300 tokens through the target model. You're at the ridge point. Every rejected draft token is now real compute you paid for and threw away, and the draft model is competing for the same GPU. Past that point, plain continuous batching without speculation can beat you on throughput, and your tail latency gets worse too.

This is why speculative decoding shines for single-user latency (chat, local inference, coding assistants) and gets disappointing on a saturated throughput server.

What should I actually do?

A checklist, in the order I'd do it:

  1. Find out which acceptance metric you have. Convert accepted/drafted to per-token α, or use mean acceptance length directly.
  2. Measure c. Time a draft step versus a target step on your GPU at your typical batch size. Plug both into speedup().
  3. Pick k from the table, not from vibes. For α in the 0.6 to 0.75 range, start at 3 or 4.
  4. Gate speculation on load. vLLM has exposed a disable-by-batch-size option in its speculative config; check the name in your version. Turn speculation off when concurrency crosses the point where your measured speedup falls below 1.
  5. Match the drafter to the workload. Acceptance drops at higher temperature and on open-ended creative text. For RAG answers and code edits that copy from the prompt, n-gram or prompt-lookup drafting has a draft cost near zero, which changes the math in your favor.
  6. Don't evaluate quality. It's lossless. Evaluate latency at p50 and p99 under realistic concurrency.

So why does 70% acceptance only get you about 2x?

Because a 70% per-token speculative decoding acceptance rate decays along the draft chain: with 4 draft tokens, you get about 2.77 tokens per target pass, and once you pay roughly 10% of a target step for each draft step, the ceiling is about 1.98x. If your dashboard shows 70% accepted/drafted, your real per-token rate is closer to 0.86 and you have more headroom. Either way, the speedup only exists while decoding is memory-bound, so at high batch sizes the spare compute disappears and speculative decoding can make your server slower. Measure the right acceptance metric, keep k small, and turn speculation off under load.


Written by the developer behind Preterview, an interview prep platform.

Top comments (1)

Collapse
 
ahmetozel profile image
Ahmet Özel •

The 44% versus 70% distinction is worth carrying into the monitoring schema itself. I would record drafted positions, accepted prefix length, and emitted tokens per verification pass separately. Early EOS and requests finishing inside a draft block can otherwise distort comparisons across workloads with different answer lengths.

For the load gate, I would also add hysteresis around the measured crossover. Switching speculation on and off at one concurrency threshold can oscillate under bursty traffic, which makes the p99 comparison harder to interpret even when the average throughput looks stable.