DEV Community

Duncan
Duncan

Posted on

Speculative Decoding: How "Guess-and-Check" Reduces LLM Latency

So, maybe you're a developer or a machine learning engineer who has successfully stood up an open-weight Llama model in production. You've got beefy GPUs, you query your endpoint, and you sit there watching it type out... one... single... word... at... a... time.

You've spent hours tuning parameters and configuring containers, so what's the hold up? Why does generating text still feel noticeably slow?

Ok, so it is a tricky one to solve when you are first starting out, but the root of the problem is that modern LLM inference is not actually compute-bound, it is memory-bandwidth bound.

Why Standard LLM Generation Can Feel Sluggish

Before we talk about the solution, we need to look under the hood at why standard LLM generation takes so long.

Standard Transformer-based models generate text autoregressively. That means the model generates exactly one token (a word or a piece of a word) at a time. To generate the next token, the model needs the context of all previous tokens. Under the hood, this requires maintaining a Key-Value (KV) cache. Every single time you ask the GPU to generate a single new token, it must read two massive datasets from its High Bandwidth Memory (VRAM) into its compute cores (Streaming Multiprocessors):

  1. The Model Weights: The billions of parameters that make up the LLM itself (for example, a 7-billion parameter model requires roughly 14 Gigabytes of float16 data, based on standard 16-bit precision parameter calculations).
  2. The KV Cache: The saved key-value tensors representing the history of the conversation so far, which grows larger with every new token generated.

And here lies the physical catch of modern computer architecture: GPUs are insanely fast at doing floating-point math, but moving data from VRAM into the processor cores is relatively slow.

Let's look at the actual numbers to see what is happening here. According to NVIDIA's L4 GPU Specifications, an NVIDIA L4 GPU has a peak compute speed of about 242 TFLOPs of FP16 tensor operations, but its High Bandwidth Memory (VRAM) speed is capped at 300 GB/s.

  • Memory Load per Token: To generate one single token, the GPU must load all 14GB of model weights from VRAM into its compute cores.
  • Theoretical Maximum Throughput: Dividing the GPU's 300 GB/s memory bandwidth by the 14GB model weight payload yields a theoretical upper limit of about 21.4 tokens per second (300 GB/s / 14 GB = 21.4 tokens/sec), regardless of how fast the math engine operates.

This means your super powerful GPU compute cores are sitting idle for 99% of the time, simply waiting for the model weights to be loaded from VRAM. Generating a single token requires very little actual math but massive amounts of memory reads. It is a quite noticeable performance hit, and it is a pretty familiar feeling to you if you have ever sat there waiting for an AI assistant to finish its reply.

How Speculative Decoding Solves the Memory Bottleneck

So, how do we get around this memory bandwidth wall without buying double the GPUs?

Enter Speculative Decoding. Instead of forcing our large, expensive target model to generate every single token one-by-one, we introduce a tiny helper model (the draft model) to guess a sequence of future tokens rapidly, and then use the large target model to verify all those guesses in a single parallel pass.

How Does the Draft Model Predict Candidate Tokens?

The draft model is a tiny, lightweight neural network (often 10x to 100x smaller than the target model) that shares the same vocabulary tokenizer. Because its parameter footprint is tiny, its weights fit almost entirely within high-speed GPU L2/SRAM caches.

Because the draft model takes almost zero time to transfer weights from VRAM, it can generate K speculative tokens in a fraction of the time it would take the target model to generate just one.

Detailed diagram illustrating the four stages of speculative decoding: Draft Generation (Guessing), Target Parallel Evaluation (Checking), Rejection Sampling, and Final Sequence Assembly.

The Fast Typist and the Slow Editor

To really get this under-the-hood process, let’s use a simple metaphor. Imagine you have a Fast Typist and a Slow Editor working together in an office:

  • The Fast Typist (The Draft Model): This is a tiny, super lightweight model (maybe a 70-million parameter model). It is a little bit sloppy and not very smart, but because it is so small, its model weights fit easily in high-speed caches. It can type out a sequence of words incredibly quickly because loading its weights takes almost zero time.
  • The Slow Editor (The Target Model): This is the massive, 7-billion or 70-billion parameter model. It is brilliant and hyper-accurate, but it takes a long time to think because it has to read its huge library of knowledge (all those gigabytes of model weights) every time it makes a move.

In a standard world, the Slow Editor has to write every single word from scratch. The Editor slowly thinks, writes a single word, and then thinks again. This takes ages.

In a speculative world, we let the Fast Typist take a guess and write out a draft of four words incredibly quickly. The Slow Editor then looks at those four words all at once in a single glance (a single parallel forward pass).

Why is evaluating four words at once so much faster than generating them one-by-one? When the target model evaluates four tokens in parallel, it only has to load its massive model weights from memory once for that entire batch. Instead of loading 14GB four times (56GB of memory transfers), it loads 14GB exactly once. The math workload increases slightly, but because our compute cores were previously idling, they process this extra math in parallel for free.

Parallel Verification and Engineering Trade-offs

Now let's examine how the target model validates these drafted tokens, handles rejection sampling, and what practical trade-offs you need to keep in mind when running this in production.

Step-by-Step Rejection Sampling

When the target model receives the K candidate tokens from the draft model, it evaluates all K positions concurrently in a single parallel pass.

The Editor checks the work step-by-step:

  • Accepted Tokens: If the draft token's probability matches or is accepted under the target model's sampling distribution, we keep it (e.g., "the" ✓, "cat" ✓).
  • Rejected Tokens: The moment a draft token is rejected (for instance, the third token "sat" ✗), the target model discards it and all subsequent tokens in that draft batch ("on").
  • Corrected Token Insertion: Crucially, the target model then inserts its own mathematically correct token ("jumped") for that position as part of the same forward pass, meaning we still make progress even on a rejection.

Is Speculative Decoding Lossless?

Yes, speculative decoding is mathematically lossless. The final output of a speculatively decoded model is guaranteed to be 100% identical to what the large target model would have produced on its own. Because the target model enforces strict statistical rejection sampling at every step, the final text output maintains exact mathematical distribution parity.

Engineering Trade-offs to Consider

While speculative decoding offers significant speedups, there are a few practical trade-offs to consider before deploying it:

  • VRAM Overhead: Both the draft and target models must be loaded into GPU memory simultaneously. If your GPU is already packed to capacity with the primary target model, you might not have enough VRAM for the draft model's weights and KV cache.
  • The Size vs. Speed Trade-off: It is tempting to assume a larger, more accurate draft model is always better because it guesses correctly more often. However, increasing the size or layer count of the draft model increases drafting compute overhead per step. If the target model is relatively small, adding extra draft layers can cause drafting compute overhead to outweigh acceptance gains, leading to a net latency penalty compared to non-speculative baseline inference.
  • Workload Predictability: Speculative decoding excels in predictable workloads like code completion, structured JSON formatting, or repetitive data extraction where draft acceptance rates are high. It offers lower gains in highly creative, open-ended tasks where draft guesses are frequently rejected.

Where to Next?

So, now you know the theory, the physical hardware limitations, and the systems physics behind speculative decoding. But how do we actually deploy this in a real-world, production-grade cluster?

To put this into practice, you can try running speculative decoding with vLLM on your cluster. Simply configure a lightweight draft model alongside your target model, run a benchmark script, and measure the Time Per Output Token (TPOT) speedups for your specific workload.

Here are a few key resources to check out while planning your next optimization:

Top comments (0)