In the concept edition, we saw that MTP (Multi-Token Prediction) lets a model predict several tokens ahead to speed up generation, and that Qwen and Gemma implement this in completely different ways.
The theory makes sense, but how much faster does this actually make things in practice? That's what we set out to measure directly.
In this installment — the implementation/benchmark edition — we'll actually run Gemma's MTP on llama.cpp and measure, with real numbers on an ordinary consumer PC, just how much of an effect it has.
Running It in llama.cpp
For Qwen Models
Run with the following arguments added. Specifying spec-type activates the model's internal MTP drafter. A dedicated model is provided for MTP.
/opt/llama/bin/llama-server --model /opt/llama/models/Qwen3.5-9B-MTP-UD-Q3_K_XL.gguf \
-t 4 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -ctv q8_0 -ctk q8_0 --fit off --no-warmup --no-cache-prompt -c 32768 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 4 \
--reasoning on
Argument summary:
-
--spec-type draft-mtp: specifies the speculative decoding type (
draft-mtpfor Qwen). -
--spec-draft-n-max 4: how many tokens ahead to speculate at most.
- Setting this too high increases the penalty when a prediction fails.
- Also note: concurrent processing isn't supported, and multimodal input isn't supported either.
For Gemma Models
Run with the following arguments. For Gemma, the drafter-MTP model is bolted on externally, so you enable MTP by pointing the --model-draft argument at that file.
/opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--model-draft /opt/llama/models/mtp-gemma-4-12B-it.gguf \
-t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8_0 -ctk q8_0 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-device CUDA0 \
--chat-template-kwargs '{"enable_thinking":true}'
Argument summary:
- --model-draft <gguf file>: specifies the draft model.
-
-sm layer: how the model is dispatched across multiple CUDA devices.
- At the time of writing, only layer-level distribution is supported.
- --spec-type draft-mtp: specifies the speculative decoding type.
-
--spec-draft-n-max 4: how many tokens ahead to speculate at most.
- Setting this too high increases the penalty when a prediction fails.
-
--spec-draft-device CUDA0: the device that runs drafting.
- Required when multiple CUDA devices are present.
Benchmark: Using MTP on Gemma-4-12B-it
In this section, to investigate how effectively MTP actually works and what kind of benefit it brings to users, we compared performance with and without MTP.
To also demonstrate that this kind of model verification is achievable even on ordinary consumer hardware, the benchmark was run on the author's own personal machine.
Test Environment
The experiment used the following simple test setup.
- Machine: modified HP Z440 Workstation
- CPU: Intel® Xeon® E5-2690 v4 x1
- RAM: 48GB DDR4 RDIMM
- SSD: Intel® DC S3700 Datacenter SSD (400GB SATA SSD)
- GPU: NVIDIA GeForce RTX 3060 (Ampere, 12GB GDDR6 VRAM)
- CUDA: 13.1
- Engine: llama.cpp Build 9574
- Model: gemma-4-12B-it-qat-UD-Q4_K_XL.gguf https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
- As the base model, we use Gemma-4-12b-it-QAT, a Google DeepMind model trained with quantization-aware training in mind.
- On top of that, we use the UD-Q4_K_XL model1, dynamically quantized by Unsloth2, which provides open-source fine-tuning tools.
- Drafter model: mtp-gemma-4-12B-it.gguf Downloaded from the same repository as the base model.
- We use the Assistant model prepared by Unsloth for Gemma-12B, simply quantized to 4-bit, as the draft model.
- The maximum number of speculated tokens is set to 4.
- KV Cache: 8-bit quantized
- The KV cache is quantized to 8 bits to fit efficiently into VRAM.
- Context size: 163,840 tokens
- Context size is capped at 160k to stay within VRAM capacity.
- Prompt cache: off
- Reasoning: on
Test Cases
- Goal: measure and compare speed with and without MTP.
- Only speed is measured. The actual content of the generated output is not evaluated at all.
- We measure elapsed time, token count, and token speed during generation.
- These values are taken directly from llama.cpp's own log output.
- Method: using the llama.cpp web frontend, we asked the following 4 turns of questions.
- "Write JavaScript code for Breakout (as an HTML page)."
- "Make it look cooler."
- "Slow the ball's movement down a bit."
- "Check for bugs and optimize it."
Resource Usage
Memory usage was as follows. The memory consumed specifically for MTP came to 759.68 MiB of VRAM on the GPU side and 324.79 MiB of RAM on the CPU side.
Since we used a 4-bit quantized model for the draft model here, choosing a 16-bit model instead would consume roughly 4x this amount for the weight data. Anyone planning to use a 16-bit model should keep this in mind.
| Component | CUDA0 | CPU |
|---|---|---|
| Weight data | 6,390.19 | 540.00 |
| KV cache data | 1,360.00 | |
| Sliding window KV cache | 765.00 | |
| Gated DeltaNet compute buffer | 533.80 | 180.80 |
| MTP (Assistant) model weight data | 226.90 | 144.00 |
| MTP (Assistant) model Gated DeltaNet compute buffer | 532.78 | 180.79 |
| Vision Encoder | 167.00 | |
| Audio Encoder | 167.00 | |
| Multimodal Gated DeltaNet compute buffer | 532.78 | 180.79 |
| Total | 10,675.45 | 1,226.38 |
Units: MiB. Table 1: Memory usage in this test environment.
Token Speed Comparison Results (MTP On vs. Off)
Comparing token ingestion speed turn by turn, we saw no major variation. With MTP enabled, ingestion speed dropped slightly compared to without MTP, coming in at roughly 70–73% of the baseline performance (see the "Read" rows in the table below).
Token generation speed, on the other hand, was roughly twice as fast overall with MTP enabled (see the "Generate" rows).
Below are the actual measurements. Note that time is measured in ms.
| Turn | Type | No MTP: time | No MTP: tokens | No MTP: tps | With MTP: time | With MTP: tokens | With MTP: tps |
|---|---|---|---|---|---|---|---|
| 1 | Read | 301 | 25 | 83.04 | 181 | 25 | 138.34 |
| 2 | Read | 1,836 | 1,834 | 999.10 | 3,004 | 2,135 | 710.71 |
| 3 | Read | 4,546 | 4,669 | 1,027.06 | 6,361 | 4,666 | 733.53 |
| 4 | Read | 7,017 | 7,102 | 1,012.11 | 9,559 | 7,030 | 735.43 |
| 1 | Generate | 45,326 | 1,789 | 39.47 | 33,186 | 2,084 | 62.80 |
| 2 | Generate | 80,190 | 2,802 | 34.94 | 40,281 | 2,499 | 62.04 |
| 3 | Generate | 70,164 | 2,405 | 34.28 | 35,046 | 2,337 | 66.68 |
| 4 | Generate | 84,325 | 2,851 | 33.81 | 45,532 | 2,858 | 62.77 |
Table 2: Actual data from the logs.
Without MTP, the whole run took about 293,705 ms in total — just under 5 minutes — while with MTP it completed in about 173,150 ms, just under 3 minutes. That's roughly a 40% reduction in time for this case.
This suggests MTP is a genuinely effective way to boost efficiency in tasks like today's agentic workflows, which repeat many rounds of inference.
To look more closely at the trend, we also checked how speed changed as token output progressed within each turn (again using the logs).
Below is the token output profile without MTP. Speed dips slightly but stays roughly flat overall, hovering in the mid-30s to low-40s tps across all four turns from start to finish.
Generally, with token-by-token generation, output speed tends to gradually decline as context size grows. How much it degrades depends on the model, but this can become a non-trivial problem as more turns accumulate.
By contrast, here is the token output profile with MTP enabled. Speed starts around 35–40 tps early on and climbs to 60–65 tps later, showing that tokens come out faster and faster as the conversation progresses.
A quick note before we go further: I'm a native Japanese speaker, and this model is my daily driver, so the test prompts and responses are in Japanese rather than English. Tokenizer behavior can differ quite a bit across languages, so this also happens to double as a look at how MTP performs outside the English-first benchmarks you usually see.
Next, we examined the actual content of the responses and found that each one opens with a Japanese explanation followed by code. (The original screenshot of that exchange is in Japanese, so it isn't reproduced here — the token breakdown below covers what matters for this analysis.)
Based on that, we checked the token length of the Japanese explanation portion at the start of each turn, with the following results.
| Turn | Token count of the Japanese portion |
|---|---|
| 1 | 107 |
| 2 | 147 |
| 3 | 124 |
| 4 | 256 |
Table 3: Token length of the Japanese explanation at the start of each turn.
We can't fully prove causation here, but turns where speed rose sharply (Turn 1, Turn 3) tended to have less Japanese output, while turns where speed rose more gradually (Turn 2, Turn 4) tended to have somewhat more Japanese output. This suggests the code-generation portion is where MTP's effect shows up most strongly.
A likely reason is that, regardless of programming language, code follows fairly strict grammar rules, making it relatively easy to predict what string comes next, and since the characters are essentially alphabetic, predicting the character type is also easier.
Even outside the code-generation portions, we confirmed output speeds of 40–50 tps right from the start — faster than the conventional approach — suggesting MTP was working effectively there too.
The author believes this largely comes down to characteristics of the Gemma tokenizer.
The Gemma tokenizer has a vocabulary of roughly 256k entries. Whereas many other tokenizers fall back to character-level tokenization for Japanese, Gemma's tokenizer is able to register most Japanese text as whole "words."
Because of this, the model can predict the next token at the "word" level for Japanese much like it does for English, which we believe is why Japanese output was noticeably faster with MTP than without.
Checking Token Acceptance Rates
llama.cpp is designed to log an "acceptance" value showing what fraction of tokens were accepted during the verification phase when using an MTP model. Here's an example:
prompt eval time = 9559.06 ms / 7030 tokens ( 1.36 ms per token, 735.43 tokens per second)
eval time = 45531.93 ms / 2858 tokens ( 15.93 ms per token, 62.77 tokens per second)
total time = 55090.99 ms / 9888 tokens
graphs reused = 3983
draft acceptance = 0.82931 ( 2196 accepted / 2648 generated)
Using this value, we checked acceptance for each turn, with the following results.
Table 4: Token acceptance rate during the test cases
| Turn | Acceptance | Accepted | Generated |
|---|---|---|---|
| 1 | 75.34% | 1,564 | 2,076 |
| 2 | 77.71% | 1,890 | 2,432 |
| 3 | 88.27% | 1,821 | 2,064 |
| 4 | 82.93% | 2,196 | 2,648 |
| Acceptance | Accepted Tokens | Generated Tokens | |
|---|---|---|---|
| Max | 88.27% | 2,196 | 2,648 |
| Min | 75.34% | 1,564 | 2,064 |
| Avg | 81.06% | 1,868 | 2,305 |
As mentioned above, likely because the output contained a lot of code, these tokens showed a high acceptance rate. So what does the acceptance rate look like for everyday use? Based on several days of logs, we recorded the maximum, minimum, and average, shown below.
Table 5: Token acceptance rate for everyday interactions with the Gemma-4-QAT-Assistant model
| Acceptance | Accepted Tokens | Generated Tokens | |
|---|---|---|---|
| Max | 75.55% | 2,119 | 4,800 |
| Min | 33.56% | 227 | 412 |
| Avg | 45.11% | 776 | 1,798 |
Acceptance roughly halved, and speed dropped somewhat as a result. The corresponding speeds are shown below.
Table 6: Token speeds under the conditions in Table 5
| Read | Generate | |
|---|---|---|
| Max | 1,040.82 | 84.46 |
| Min | 117.52 | 30.20 |
| Avg | 774.45 | 40.44 |
So while this isn't exactly blazing fast, it's reasonable to say it's still faster than running without MTP at all.
We confirmed that, for the Gemma-4 series, llama.cpp and Gemma-4-Assistant are a very well-matched pair, capable of delivering real benefits even in everyday use.
Summary
In this article, we looked at the new "MTP" technique — what it is, and how it actually changes generation speed in practice.
In a word, MTP is essentially the "speculative execution" of CPUs, brought into the world of LLMs: when the prediction is correct, generation gets dramatically faster; when it's wrong, the extra overhead can make it slower.
We found this feature pairs especially well with code generation. Because programming languages follow strict grammar, once you're partway through a line, it's relatively easy to predict "what string should come next." This lets MTP compensate for the parts where the tokenizer tends to slow things down, and our benchmark reflected that with solid results.
The current strength of LLMs comes precisely from their sequential nature — deciding the next word based on the words that came before. MTP, because it speeds things up without sacrificing that strength, is likely to play a genuinely important role as this technology continues to develop.
It's still early days, and there are some constraints, but we think this is a technique well worth watching as it matures and spreads.
Even as we speak, other groups are developing and releasing their own versions of this idea:
- DeepSeek (China): a token-prediction technique called DSpark
- Z-Lab at UC San Diego: a token-prediction technique called DFlash
These, too, are spreading quickly now that inference engines are starting to support them.
We hope to take a closer look at these technologies in a future article.
Supplementary Information (llama.cpp Support Status, Gemma-4 Architecture Details)
llama.cpp Support Status
- 2026/5/16: Support for Qwen's MTP feature merged (#22673)
- 2026/5/22: A further updated version merged (logit computation skip optimization: #23433)
- 2026/6/8: PR adding Gemma-4 MTP support merged (#23398).
- Enables conversion to GGUF format including the MTP drafter.
- 2026/6/8: Support for the Assistant drafter models for Gemma-4 E2B and E4B merged. (#24282)
- However, a bug caused Gemma-4-E4B specifically to malfunction.
- Apparently fixed in PR #25148 below.
- 2026/06/30: Fix for the Gemma-4 E4B bug (#25148) merged.
Gemma-4 Model Architecture
Below is the architecture of the Gemma-4-12B-it model.
Input strings ─┬─ Vision/Audio encoders (minimal, mostly folded into the model) ─┐
└─ tok → Emb ──────────────────────────────────────────────────┴─(+)─┐
│
┌── RoPE / p-RoPE ───────────────────────────────────────────────────────────────┤
│ ▼
│ [ LA → LA → LA → LA → LA → GA ] x6 groups → LNR → SoftMax → Output Probabilities
└──────────────────────────────────────────────────↑
(48 blocks total = 6 layers x 8 groups)
- Max context size: 256k tokens (limited to 160k in our test environment)
- 48 blocks overall (6 layers x 8 groups)
- Hidden size: 3,840
- FFN type: Dense (omitted from the diagram above for simplicity)
- Activation function: Gelu_Pytorch_tanh
- Activation dimension: 15,360
- The attention structure is a hybrid of Local Attention (LA) and Global Attention (GA). This structure debuted with Gemma-4.
- Local Attention (LA): uses sliding window attention.
- Handles analysis of local context via a sliding window.
- Low compute cost, fast, and low memory usage.
- Global Attention (GA): uses group query attention for accurate whole-sequence understanding.
- Captures the entire sequence.
- A high-performance coordinator handling long context and complex reasoning.
- Other notable features:
- Unified Encoder: integrates audio/vision encoders into the model.
- Normally-external audio/vision encoders are almost entirely integrated into the model itself.
- Only the CLIP portion remains an externally applied encoder.
- Per-Layer Embedding (PLE): keeps embedding information independent at each layer.
- Beyond the input-time embedding, retaining layer-specific information contributes to richer representational power during inference.
- KV Cache Sharing: a mechanism for sharing and synchronizing the KV cache.
- Normally kept independent, this mechanism shares and synchronizes the KV cache instead.
- Used to achieve more consistent inference.
- When dispatching across multiple GPUs, this shared-state synchronization causes frequent inter-GPU communication, so caution is needed in setups without NVLink.
-
DualRoPE configuration:
- Local Attention layers use RoPE.
- Embeds token position information using rotation matrices, ensuring reliable positional information for long text.
- Global Attention layers use p-RoPE3.
- A modified form of RoPE that applies rotation only to the top p% of K/V matrix pairs.
- Because it effectively suppresses the impact of noise, it's applied to the coordinating Global Attention layers, helping clarify positional information for long text.
Gemma-4-12B-Assistant Model Architecture Details
The Gemma-4-12b-it-assistant model works together with the main model to predict tokens as follows.
Main model (Gemma-4-12b-it):
Input string → tok → Emb → [LA...LA(45) → GA(46) → GA(47)] → LNR → SoftMax → N+1 output token
│ shares KV cache (layers 45-47) with the Assistant's LA/GA
▼
Assistant model (MTP-Drafter):
Query (H_N^0 + Emb of N) → LNR → LA → LA → LA → GA → Norm ─┬→ LNR → predicted state vectors
└→ Masked Embedder → predicted Token IDs (n tokens)
│
▼
Verification (inside the inference engine):
compares the Assistant's predicted state vectors against the Main model's own state vectors
→ decides how many tokens to accept → token output correction → final output
The "Assistant" model that adds MTP capability to Gemma-4-12B-it keeps Gemma-4's architecture but trims its structure down to a genuinely minimal design.
- Max context size: 256k tokens
- 4 blocks overall (a single group of 3+1)
- Hidden size: 1,024
- FFN type: Dense
- Activation function: Gelu_Pytorch_tanh
- Activation dimension: 8,192
Based on the input string, it predicts the next (N+1) token. To do this, it combines the generated state vector h_N^0 with the embedded vector of token N, and feeds the result in as a query to begin processing.
Gemma-4's attention layers normally operate as self-attention, but the structure inside the Assistant model uses cross-attention instead.
For KV data, it combines the LocalAttention block at layer 46 of the main model with the LocalAttention blocks at layers 0–2 on the Assistant side, and also integrates the GlobalAttention block at layer 47 of the main model with the GlobalAttention block at layer 3 on the Assistant side, via a shared KV cache. Through this, KV information is bound based on the query.
The output consists of two parts.
Predicted tokens: the n tokens predicted by the Assistant model.
State vector: the vector used during that prediction process.
These outputs are sent combined, just before the main model's attention blocks. They pass through all of the main model's attention blocks, generating, in parallel, the state vectors needed for verification.
Once the state vectors to be compared are ready, their contents (probability distributions) are compared to decide how many tokens should be accepted. This is handled in parallel by the inference engine (for example, the generate function in the transformers library), which outputs the resulting number of tokens to accept.
Based on this result, the candidate tokens — accounting for the accepted token count — are combined with token N+1 and passed on for output. This is how the process achieves faster output compared to conventional step-by-step inference.
This article is an English adaptation of the original Japanese post published on Zenn: "実際どれだけ速くなる？ Gemma-4のMTPを、普段使いのPCでllama.cpp検証してみた【検証編】", by Yuichi Tominaga.
Top comments (0)