Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Language models generate text one token at a time. Each new token depends on the tokens generated before it, so the model usually needs another forward pass before it can produce the next token.
For a small model, this may be fast enough. For a large model, however, generating text can feel slow. The reason is not always what you might expect.
1. Why Generating Tokens Can Be Slow
You might assume the GPU spends all its time performing heavy computations. In many inference workloads, that isn't the case.
To generate a token, a large language model must access its weights, which can occupy many gigabytes of memory. For models limited by memory bandwidth, moving these weights from memory to the GPU's compute units can take longer than performing the calculations themselves.
As a result, the GPU's compute units may sit underutilized while waiting for data.
Think of a chef who spends 10 minutes fetching ingredients from a distant pantry and only 10 seconds cooking. Faster cooking wouldn't solve the main problem. Fewer trips to the pantry would.
This is similar to memory-bound inference: the bottleneck is moving data, not necessarily performing calculations.
2. The Opportunity: Process Multiple Tokens at Once
During autoregressive generation, the model typically processes one new token at a time. But when processing multiple known tokens together, the model can reuse the weights it has loaded instead of repeatedly loading them for each token.
Processing five tokens together can therefore cost much less than running five separate single-token steps, especially when inference is memory-bound.
This does not mean five tokens always cost the same as one. The exact cost depends on the model, hardware, sequence length, and workload. But processing several tokens in parallel can make better use of the available computing resources.
Now imagine you already had a good guess for the next five tokens. The large model could check those guesses together rather than generating every token sequentially.
But where would those guesses come from?
3. The Idea: A Small Draft Model
Speculative decoding uses a second, smaller model called a draft model to propose upcoming tokens.
The larger model, called the target model, then verifies those proposals.
Think of it this way:
- Draft model: A quick intern who prepares a rough draft.
- Target model: An expert editor who checks the draft.
The editor can review an existing draft more efficiently than writing every word from scratch. If most of the draft is correct, the editor can approve several words at once and correct the first mistake.
That is the basic idea behind speculative decoding.
4. How Speculative Decoding Works
Let's use this prompt as an example:
“The capital of France is”
Suppose the draft model proposes four tokens:
“ Paris”, “,”, “ a”, “ city”
Here's what happens next.
- Guess: The small model quickly guesses the next four words: “ Paris”, “,”, “ a”, “ city”.
- Check: The big model looks at all four guesses at the same time. For each spot, it figures out which word it would have picked itself.
- Keep the good ones: Going from left to right, we keep each guess that matches the big model’s choice. Here, the big model agrees with “ Paris”, “,”, and “ a”. But for the fourth spot, it would have picked “ major” instead of “ city”.
- Fix the first mistake: We swap “ city” for the big model’s word, “ major”. If there were any guesses after the wrong one, we throw them away, because they were built on a mistake.
- Start again: The small model takes the corrected sentence and guesses the next few words.
In this example, the target model produces three accepted tokens plus its correction in one verification round. That gives four new tokens instead of the single token it would ordinarily produce in a standard autoregressive step.
If all four draft tokens are accepted, the target can also contribute a bonus token, potentially producing five new tokens in that round.
The important point is that the target model verifies several proposed tokens together rather than generating each one separately.
5. What Happens When a Draft Is Wrong?
Sometimes the very first guess is wrong. When that happens, the big model rejects it and picks the word itself.
So you still get one correct word, just like you would without this trick. The only cost is a little wasted time: the small model made a guess, and the big model checked it, but the guess was thrown away.
If the small model is wrong more often, each check produces fewer words, so you save less time.
But here’s the important part: if it’s set up correctly, the final text is exactly what the big model would have written on its own.
In other words, this trick makes writing faster without changing the quality or the style of the answer. Only the speed changes.
Greedy decoding
When a model writes, it gives a score to every possible next word. There are two common ways to pick one:
- Greedy: always pick the word with the highest score. No randomness.
- Sampling: pick randomly, with higher-scoring words more likely to be chosen.
This section is about the first way, where the big model always picks its top choice.
In that case, checking the small model’s guess is simple:
- If the guess is the word the big model would have picked, we keep it.
- If the guess is a different word, we throw it away and use the big model’s word instead.
For example, say the sentence so far is “The capital of France is”. The small model guesses “ Paris”, and the big model’s top choice is also “ Paris”. Match, so we keep it. If the small model had guessed “ Lyon”, we would replace it with “ Paris”.
Result: the final text is exactly the same as what the big model would have written alone, just faster. (This holds as long as you use the same big model and the same settings.)
Sampling
Sometimes the big model adds a bit of randomness when it picks a word. If we kept every guess from the small model just because it sounds reasonable, the final text would start to follow the small model’s habits instead of the big model’s.
So we don’t keep guesses automatically. Each guess gets a chance of being kept, based on how much the big model likes it. If a guess is rejected, the big model picks a replacement.
This way, the final text still matches what the big model would have written on its own.
In short: the small model suggests, but the big model decides.
6. How Much Faster Can It Be?
The speedup depends heavily on how often the target model accepts the draft model's proposals.
Let:
- k be the number of drafted tokens per round.
- alpha be the probability that an individual draft token is accepted, assuming a simplified constant acceptance rate.
Under this simplified model, the expected number of tokens produced per target-model verification pass is:
(1 − α^(k+1)) / (1 − α)
For k=4, the expected number of tokens per verification pass is approximately:
| Acceptance rate | Expected tokens per pass |
|---|---|
| 50% | 1.9 |
| 70% | 2.8 |
| 80% | 3.4 |
| 90% | 4.1 |
These numbers show how many words we get each time the big model checks the small model’s guesses. They do not show how much faster the whole thing runs.
The real speed also depends on other things: how fast the small model is, how long the checking takes, and the extra work of running two models together.
So getting more words per check doesn’t mean you get the same amount of speedup overall. To know the real gain, you have to test it on your own model, computer, and task.
7. When Does Speculative Decoding Work Best?
Speculative decoding works best when the draft model can predict the target model's next tokens accurately.
It can be especially useful for:
- Code and boilerplate: Common patterns and repeated structures can be predictable.
- Summaries and rewrites: Some continuations closely follow the input text.
- Structured or formulaic text: Repetitive formats can make the next tokens easier to predict.
- Predictable factual answers: Familiar phrases and common sequences can lead to higher acceptance rates.
It may offer less benefit for open-ended creative writing, where many continuations are plausible and the draft model may disagree with the target more often.
The task alone doesn't determine the outcome. The acceptance rate between the particular draft and target models is what matters.
8. Trying Speculative Decoding
Several inference frameworks support speculative decoding. In Hugging Face Transformers, one approach is called assisted generation, which allows a smaller assistant model to help a target model generate tokens.
A simplified example looks like this:
output = target_model.generate(
**inputs,
assistant_model=draft_model,
max_new_tokens=200,
)
Here, target_model is the model whose output you want, and draft_model proposes candidate tokens for it to verify.
This is an illustrative example. The models and inputs must be loaded and configured appropriately, and the selected model combination must be supported by the Transformers version you use.
Conclusion
Speculative decoding addresses a key bottleneck in language model inference: generating tokens sequentially can repeatedly incur the cost of accessing a large model's weights.
A smaller model proposes several tokens, and the larger model verifies them together. When the proposals are accepted frequently enough, the target can produce multiple tokens per verification pass while preserving its original output distribution.
The central trade-off is simple: the more accurately the draft model predicts the target, the more opportunity there is to accelerate generation.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Risk-Aware AI Code Review for Business-Critical Systems
LiveReview: Risk-Aware AI Code Review for Business-Critical Systems
LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:
- Understand risky changes faster.
- Fix problems in those changes, with human attention directed to the work that needs it.
- Enforce your engineering standards.
- Learn from production outcomes, such as incidents, rollbacks, and queue wait times.
- Improve the review process over time and make it more efficient.
- Connect to the major engineering tools that your team already uses.
Auto-Approve: Ship the Safe 50% Without a Human
Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…
Click below to try LiveReview with your codebase:
Top comments (0)