MTP Explained: How Qwen and Gemma Predict Several Tokens Ahead

oooocean66 — Wed, 09 Sep 2026 07:53:18 +0000

MTP (Multi-Token Prediction) is a technique that lets a language model predict several tokens ahead while it generates text, then confirm multiple tokens at once when those predictions turn out correct.

Have you ever used an AI chat and thought, "I wish this answered a little faster"? That feeling gets stronger with "Thinking" models that reason carefully before answering, or with coding agents that iterate through trial and error. For AI systems that burn through thousands, even tens of thousands, of tokens to reach a good answer, generation speed directly shapes the user experience.

So why does AI text generation take so long in the first place? The reason is simple: LLMs can only produce one token at a time. To decide the next word, the model has to re-run its massive computations from scratch, taking into account everything generated so far. That cost adds up, and the longer the response, the longer you wait.

This raises a natural question: instead of recomputing everything one word at a time, why not compute a few words ahead in the same pass? That's exactly what MTP does. Every time the model generates a word, it also computes, in that same forward pass, a prediction for what the next few words are likely to be. If the prediction turns out correct, those predicted tokens are accepted all at once, confirming several words in a single step. If it's wrong, the model simply falls back to generating one token at a time as usual — so a wrong prediction costs little, while a correct one yields a meaningful speedup.

This technique is often called speculative decoding, a term borrowed from an older CPU optimization technique called "speculative execution." CPUs have long predicted which branch of code they're about to take, computed ahead of time based on that prediction, and either kept the result if it was right or discarded it and redid the work if it was wrong. MTP applies the same underlying idea to language model inference.

This article (the concept edition) walks through how this mechanism is actually implemented inside real models (Qwen and Gemma), and how the two approaches differ. In the next installment — the implementation/benchmark edition — we'll actually run this technique and measure exactly how much faster it gets. Later in the series, we'll also cover "DFlash," which pushes this idea even further.

What Is MTP, Exactly?

MTP (Multi-Token Prediction) is a technique where an AI predicts several tokens ahead simultaneously while generating text. By predicting tokens in advance, the model can confirm multiple tokens at once when the prediction turns out correct, speeding up output.

Similar efforts exist elsewhere: instead of generating text sequentially, some "diffusion" models generate text more like image generation, producing output in a more randomized order to speed things up (as of 2026, Google DeepMind and ELYZA, among others, appear to be researching diffusion language models). However, output quality for these remains unstable and hasn't reached a practical level yet.

At least as of 2026, MTP is arguably the most reliable LLM speedup technique available today.

Which Models Support MTP?

The models currently supporting MTP fall mainly into two categories. Each uses a different implementation approach, so support in inference engines needs to be checked individually.

Qwen-3.5 / 3.6: MTP is natively baked into the model itself — the model and its MTP drafter are a single package. It's supported out of the box in engines like vLLM. llama.cpp added experimental support relatively early on, and Unsloth has released an MTP-enabled model for Qwen-3.5-9B in GGUF format.

Gemma: MTP is achieved by pairing the model with a separate model called "Gemma-4-Assistant" — think of it as an add-on bolted onto the main model rather than something built in. llama.cpp support for this arrived somewhat later than for Qwen.

The Basic Mechanics of MTP

Let's look at how behavior differs with MTP disabled versus enabled.

[Conventional Inference (Token-by-Token)]

One forward pass predicts one token; the next step runs another forward pass. Because output comes one token at a time, this is relatively slow.

Step N (full forward pass) → token generated → Step N+1 (full forward pass) → token generated → ...

[Inference with MTP]

A single forward pass computes a certain number of tokens ahead.

Step N (forward pass, run once) ──┬─→ Token N     [confirmed]
                                   ├─→ Token N+1   [speculative]
                                   └─→ Token N+2   [speculative]

Token 1: output as confirmed.
Token 2 onward: probability values are calculated in parallel as speculative predictions.
Because output comes in multi-token units, this is relatively faster than the conventional approach.

Handling Predicted Tokens (Speculative Decoding)

Predicted tokens go through a follow-up step where they're checked against the correct answer; once they clear the acceptance criteria, they're confirmed all at once.

Draft A lightweight method quickly predicts upcoming tokens.
Verification The production model computes the probability distribution for the predicted tokens in parallel.
Acceptance The system checks how closely the probability distribution from verification matches the distribution the model would have produced without MTP, and determines whether it meets the acceptance threshold. If it matches, the tokens up to that point are accepted and confirmed all at once. If it doesn't match, drafting is cut off at the point of mismatch, and the result falls back to the production model's own inference.

Draft → Verification → Acceptance ─┬─ match ────→ accept all at once
                                    └─ mismatch ─→ cut off, fall back to the
                                                    production model's answer,
                                                    then Draft again

Because tokens appear to be generated "simultaneously," this is often mistaken for a diffusion model, but the underlying process is still strictly sequential.

In other words, this approach involves a real trade-off. When the prediction is correct, more tokens get confirmed at once, yielding faster output. But when it's wrong, the extra computation goes to waste, and the overhead can make the result slower than running the base model without MTP at all.

How the Two Approaches to MTP Differ

As mentioned, this article covers two implementations, and they differ considerably in their starting point, goals, and mechanics. Let's look at each in more detail.

Qwen's Approach to MTP

Qwen's approach is said to build on techniques researched for DeepSeek-V3, and has been used starting with Qwen3-Next.

The reason this approach is built directly into the model is that the technique itself originally started out as "a method for training a smarter model." MTP turned out to be useful, and the circuitry built into the model for that purpose is now also used, as a side effect, for speculative token prediction.

Input token (N) → Qwen3.5 Main Model → state h_N^0 → Token N+1 [confirmed]
                                             │
                                             ▼ (also feeds the MTP path)
                                       MTP Module 1 → h_N^1 → Sampling & accept
                                             │
                                   match? ───┴─── no match?
                                     │                │
                        predicted token accepted   main model's token adopted
                          → confirmed as N+2         → confirmed as N+2, chain stops
                             │
                             ▼ (only if accepted)
                       MTP Module 2 → h_N^2 → Sampling & accept → ...same check for N+3

Here's how it works, broken down. Even with MTP disabled, when a token is input, the model's main processing generates a state h_N^0 based on the token sequence so far, which is then detokenized to produce token N+1.

When MTP is enabled, as soon as state h_N^0 is produced, the requested number of MTP modules are created. If n tokens' worth of prediction is requested, n modules are prepared, each producing an embedding Emb_N^n that corresponds to a predicted word, based on state h_N^0.

After that, as shown in the figure above, the embedding information for token N+1 — the token that would have been output even without MTP — is combined to produce state h_N^1. This state h_N^1 is sent to the main model, where its sampling and acceptance mechanism checks whether it matches what the main model would have predicted on its own.
If it matches, that token is accepted and the process moves on to predicting the next token.
If it doesn't match, the state predicted by the main model is used instead, and verification stops there.

The defining feature is that the prediction and verification mechanisms are chained together one token at a time, forming a sequential flow throughout.

Gemma's Approach to MTP

Gemma's approach was built from the ground up for speed, and its key difference from Qwen is that it processes prediction and verification together, in a batch.

Input token (N) → Gemma 4 Main Model → state h_N^0 → Prediction: N+1
                                             │
                                             ▼ (KV-cache update)
                                         KV-Cache ←──────────────┐
                                             │                   │ (shares cache)
                                             ▼                   │
                                    Gemma 4 Assistants ──────────┘
                                             │  (generates sequentially inside the
                                             │   draft model, then sends as a batch)
                                             ▼
                            Prediction: N+2, N+3, N+4, ... (candidate list)
                                             │
                                             ▼
                     Main model: causal-attention masking, probabilities computed
                     in parallel  ──┬── include only as many as fit into the output
                                    └── if none fit, exclude from output

In Gemma, the draft model is external. Once the main model confirms the first token, the draft model predicts, all at once, however many tokens are allowed.

The main model first receives the input tokens, generates a state, detokenizes it, and outputs token N+1. At this point, the draft model shares a KV cache with the main model (in Qwen's case, the MTP component and the main component maintain independent KV caches).

Upon receiving the updated KV cache, the draft model generates however many predicted tokens are needed, and hands them off to the main model. The main model then checks, in a batch, whether these predicted tokens are correct using probability distributions, and determines how many of them are acceptable.

As a result, however many predicted tokens the main model accepts get output together with token N+1, all at once.

Which Approach to MTP Is Better?

Because the two approaches to MTP (Multi-Token Prediction) start from different premises, it's hard to say one is unconditionally superior. That said, evaluating primarily on maturity and compatibility with inference engines, as of July 2026, Gemma's approach appears more mature.

First, there's a difference in the scope of sequential processing. Because token prediction fundamentally assumes "predicting the next token based on the state of the previous one," the process is inherently sequential. Comparing the flow of each approach, with time running left to right:

Qwen:   Input(N) → Main model → Predict N+2 → Judge N+2 → Predict N+3 → Judge N+3 → ...
                        │             confirmed:N+2 ↑          confirmed:N+3 ↑
                        └→ confirmed: N+1

Gemma:  Input(N) → Main model ──┬→ Predict N+2 ─┐
                        │       ├→ Predict N+3 ─┼→ Judge (batch) → up to n OK → confirmed: N+2, N+3, N+4, ...
                        │       └→ Predict N+4 ─┘
                        └→ confirmed: N+1

With Qwen's approach, the main model predicts token N+1, uses that state to predict token N+2 and passes it to the main model, then predicts token N+3 only after seeing the verification result — repeating this process step by step.

By contrast, with Gemma, after predicting token N+1, the draft model predicts however many tokens are needed all at once (starting from N+2), and hands them to the main model as a single batch. Processing on the main model's side is also parallelized, making this approach far more efficient than Qwen's. Because of this structural difference, Gemma's approach comes out ahead.

Second, there's a difference in structural flexibility. Because Qwen's MTP functionality is integrated into the main model, improving the drafting logic requires additional post-training. Gemma, on the other hand, keeps the draft model separate, so it can simply be swapped out whenever better logic becomes available. This separation is also advantageous operationally.

Finally, there's the ease of implementation in inference engines. Looking at llama.cpp's implementation, Gemma's approach can run even in multimodal configurations, while Qwen's approach doesn't support multimodal use. This difference stems from the fact that Qwen embeds MTP internally, requiring the branching logic to be handled inside the model itself. Gemma's approach, by contrast, simply toggles the feature on or off depending on whether input passes through the draft model, which is a more favorable structure for engine-side implementation.

Summary (And a Preview of What's Next)

MTP, at its core, works by confirming one token while, in that same pass, speculatively predicting a few tokens ahead at minimal extra cost¹, then confirming them all together if the prediction turns out correct. Qwen builds this mechanism directly into the model itself; Gemma delegates it to a separate, dedicated draft model.

In the next installment — the implementation/benchmark edition — we'll actually run both approaches on llama.cpp and measure, with real benchmarks, exactly how much faster they get and how often the predictions turn out correct. Later in the series, we'll also cover "DFlash," which takes this idea even further.

References

Qwen3-Next: Towards Ultimate Training & Inference Efficiency
https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd

DeepSeek-V3 Technical Report
https://arxiv.org/pdf/2412.19437

Multi-token prediction with Gemma using Hugging Face Transformers
https://ai.google.dev/gemma/docs/mtp/mtp

This article is an English adaptation of the original Japanese post published on Zenn: "小さく賭けて、大きく当てる：LLMを高速化する「投機的デコード（MTP）」の正体【概念編】", by Yuichi Tominaga.

Speculating too many tokens ahead raises the cost of a wrong prediction, which can outweigh the benefit and slow things down. "Minimal extra cost" here means relative to the cost of recomputing everything from scratch for every single token in the conventional approach. ↩

DEV Community: oooocean66