DEV Community: oooocean66

Qwen3.8-27B: An Architecture and Hands-On Look at Alibaba's Compact Dense Model

oooocean66 — Mon, 28 Sep 2026 07:09:10 +0000

In August 2026, less than a month after Moonshot AI's Kimi-K3 surprised the world, Alibaba Group released "Qwen3.8-2.4T-A95B," an enormous 2.4-trillion-parameter model that once again shook the local-LLM community. Three days later, a smaller companion model quietly appeared: "Qwen3.8-27B," the model covered in this article.

It's released under the Apache-2.0 license, free for commercial use, and as a dense model it puts its full parameter count to work rather than routing through sparse experts. That combination raises a natural question: does this 27B model hold up against its much larger sibling? We look at its architecture and run hands-on benchmarks, comparing it along the way with Muse Glimmer 30B, the model we reviewed previously.

The Arrival of Qwen3.8

The Impact of Qwen3.8-2.4T-A95B

On August 12, 2026 — roughly a month after Moonshot AI's Kimi-K3 surprised the world — China's Alibaba Group published Qwen3.8-2.4T-A95B on Hugging Face. At 2.4 trillion parameters, it's a genuinely enormous model, well beyond what most people can run locally, yet it still generated considerable buzz in the local-LLM community.

Its reasoning benchmarks came close to Kimi K3's, putting it within reach of the major American frontier models.

Some users ran 1-bit or 2-bit quantized versions on their own hardware and confirmed that the numbers weren't hype¹ — reports of the model's real-world performance spread quickly.

How Does Qwen3.8-27B Measure Up?

Three days later, at midnight JST on August 15, 2026, the smaller Qwen3.8-27B was released.

One important distinction is licensing. The larger Qwen3.8-2.4T-A95B ships under Alibaba's own Qwen3.8-Max license, which imposes certain conditions on commercial use — a company large enough to offer that model as a service would likely need a separate commercial licensing agreement.

Qwen3.8-27B, by contrast, uses a plain Apache-2.0 license, so it can be used commercially at no cost. And as a dense model — as its name suggests — it puts its entire parameter count to work on every token, which is the design typically associated with higher per-parameter capability.

Qwen3.8-2.4T-A95B made a strong case for its own capability. The question is how much of that capability carries over to this smaller sibling.

We ran our own hands-on verification to dig into that question.

Qwen3.8-27B's Architecture

Architecturally, Qwen3.8 carries the Qwen3.5/Qwen3.6 design forward essentially unchanged. The structure is shown below.

Figure 1: Qwen3.8-27B's architecture

Looking at the config.json published on Hugging Face makes that continuity even clearer:

{
  "architectures": [
    "Qwen3_5ForConditionalGeneration"
  ],
  "image_token_id": 248056,
  "language_model_only": false,
  "model_type": "qwen3_5",
  "text_config": {

Beyond that, the basic specs are as follows:

Max context size: 256k tokens (limited to 64k in our test environment)
64 blocks total (GDN:GA = 3:1, 4 layers × 16 groups)
Hidden size: 5,120
FFN type: Dense
- Activation function: SiLU
- Activation dimension: 17,408

An Attention Structure That's Actually a Minority Approach

Qwen3.8's attention is a hybrid of two mechanisms — Gated Delta Network and Gated Attention (effectively GQA) — which is quite different from the "local + global" pattern that's become the popular default lately.

Gated Delta Network handles two jobs — state updates and information compression — and makes up 75% of the attention layers. The remaining 25% are Gated Attention layers, internally structured as Group Query Attention, responsible for global aggregation and precise retrieval.

This layout was first adopted in Qwen3-Next-80B and has carried through Qwen3.5 and Qwen3.6.

Rather than "local + global," it's probably more accurate to describe this as "compression + aggregation." As far as we're aware, Qwen is currently the only major model family using this approach.

The Vision Encoder Is Still SigLIP2-Based

Qwen3.8's vision encoder continues to build on the SigLIP2-based architecture (SigLIP2-SO-400M). Additional fine-tuning presumably went into it, but there's been no major structural change.

That's a notably different approach from Muse Glimmer's custom vision encoder, which we covered previously.

Running It Ourselves

For this test, we used a GPUSOROBAN High-Speed Computing instance equipped with an NVIDIA RTX A4000.

Overall Setup

The setup is as follows: we open an SSH tunnel to an access server, then reach the target instance through that tunnel.² llama.cpp's service port is relayed to localhost via port 8001 on both ends.

[ Home ]                    [ HighReso : GPUSOROBAN ]
+---------------+           +----------------+     +------------------+
|    Client     |           | Access Server  |     | Target Instance  |
|  > 8001/tcp   |--(SSH)--->|    (relay)     |---->|  8001/tcp        |
|               |           |                |     |  (llama.cpp)     |
+---------------+           +----------------+     +------------------+

Figure 2: Overview of the connection setup

Instance Used

The GPU used for this test is an NVIDIA RTX A4000. The instance specifications are as follows.

Item	Spec
Instance type	s16-1-a-standard-ubs24-v
GPU	NVIDIA RTX A4000
GPU memory	GDDR6 16GiB
GPU memory bandwidth	448.0 GB/s
FP32 compute	19.17 TFLOPS
BF16 compute	38.34 TFLOPS
INT8 compute	153.4 TOPS
INT4 compute	306.7 TOPS
vCPU	11 cores
System memory	50 GiB
Storage	Persistent 100 GiB
CUDA version	13.2
NVIDIA driver version	580
OS	Ubuntu 24.04 Server

Table 1: Instance specifications

Model Files Used

Even a 4-bit quantized version of Qwen3.8-27B exceeds 16GiB. Factoring in KV cache usage as well, we used a 2-bit quantized model optimized with Unsloth Dynamic 2.0.

We downloaded the following files from Unsloth's Qwen3.8-27B-GGUF repository:

File	Size	Description
Qwen3.8-27B-UD-Q2_K_XL.gguf	12 GB	UD2.0, 2-bit quantized model
mmproj-bf16.gguf	889 MB	Quantized vision encoder

Table 2: Files used with llama.cpp

We used llama.cpp version 0.1.0-dev (build 10450, commit ece963f41).

Since the architecture hasn't changed since Qwen3.5, being able to run it on an existing llama.cpp build without waiting for new support is arguably an advantage in its own right.

Verification: Standard Mode

First, we ran text inference using the main model on its own.

Launch Command

We used the following command to launch the web frontend for testing.

For this run, we kept the KV cache in FP16 rather than quantizing it to 8-bit, but due to memory constraints we lowered the max context size to 65,536. We also raised the log verbosity from the default of 3 to 4 to check memory usage.

build/bin/llama-server --model ./models/Qwen3.8-27B-UD-Q2_K_XL.gguf -t 12 -np 1 --prio 2 --temp 0.6 --top-p 0.95 --top-k 20 --port 8001 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 65536 --reasoning on --reasoning-effort xhigh --log-verbosity 4

Reasoning Effort Is Now Configurable

One recent improvement in llama.cpp that stood out to us is the addition of a reasoning_effort parameter.

--reasoning-effort LEVEL   reasoning effort level given to the chat template:
                           'default' to keep the template default, or a level
                           such as 'minimal', 'low', 'medium', 'high', 'xhigh'
                           or 'max' (default: default)
                           (env: LLAMA_ARG_REASONING_EFFORT)

Table 3: The reasoning_effort option, as described in llama-server's own help text

Which levels are actually usable depends on Qwen3.8-27B's own chat_template.jinja. Checking that template, the following levels appear to be supported:

xhigh (default)
medium
low

Setting these appears to inject the following text into the system prompt:

Level	Injected text
xhigh	Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.
medium	(no system prompt injected)
low	Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.

Table 4: reasoning_effort levels and their injected prompt text

So rather than setting an explicit token budget, it appears to control reasoning depth simply by appending instructions to the system prompt.

Memory Usage: KV Cache Size Is the Sticking Point

Memory usage came out as follows. As is generally true across the Qwen series, the KV cache footprint is large. We initially tried setting the max context to 131,072, but that produced an OOM error, so we halved it. With the ceiling set at 65,536, the KV cache came out to 4,096 MiB.

Device	CPU	CUDA0	Total
Model	Xeon E5-2690v4	RTX-A4000	—
Weight data (Q2_K-quant_XL)	397.85	9,567.89	9,965.74
KV cache size (f16)	—	4,096.00	4,096.00
RS buffer size (f32)	—	149.62	149.62
Gated Delta Net compute buffer	186.02	84.02	270.04
Total	583.87	13,897.53	14,481.40

Table 5: Memory usage in standard mode (main model only, units: MiB)

In other words, setting the max at 131,072 tokens in FP16 mode would require 8,192 MiB, and setting it to the model's full 262,144-token context would require 16,384 MB.

The cause is that, compared with other models, Qwen3.8-27B has more Global (GQA) layers, and each of those layers is configured with more heads and larger dimensions.

Model	Muse Glimmer 30B	Gemma-4-31B	Qwen3.8-27B
Layer structure	SWA+Global(3:1)	SWA+Global(5:1)	GDN+Global(3:1)
Total layers	52	60	64
Aggregation layers	13	10	16
Aggregation layer KV design	2 heads x 128 dim	4 heads x 512 dim	4 heads x 256 dim
KV cache size (128k, f16)	1,742.00 MiB	5,920.00 MiB	8,192.00 MiB
Design intent	Aims for lightweight inference, pushing memory reduction to the limit	Balances advanced reasoning with memory reduction	Aims squarely at advanced reasoning

Table 6: Aggregation-layer configuration and KV design across hybrid models

Gated Delta Network is, on its own, a highly efficient architecture compared with the "local + global" pattern — the layers handling information compression require essentially zero KV cache. Under otherwise identical conditions, a "local + global" design will always require a larger cache.

But Global layers appear very frequently here, the model has more layers overall than its peers, and the aggregation-layer KV design is comparatively rich — and together those factors are what push Qwen3.8-27B's KV cache size up.

That points clearly to a design goal that prioritizes reliable, deeply reasoned answers over lightweight inference.

Output

We asked the following question and examined the response.

Given the recent Kumamoto earthquake, this is a question we've been asking various models out of curiosity.³

Describe the characteristics of the ground/soil conditions in Kyushu.

The response came out as follows. Our first impression was that it looked surprisingly well-informed.

Kyushu's ground conditions are geologically, topographically, and hydrologically
complex, and their strength and character vary widely from place to place —
that variability is the single defining feature. As construction ground, the
characteristics can be summarized as follows.

1. Geological and ground diversity
Kyushu sits on a volcanic island arc, so volcanic ground, sedimentary ground,
granite ground, weathered soil, and river deposits are all mixed together over
short distances.
- Northern Kyushu (Fukuoka, Saga, Nagasaki, etc.): sedimentary basins and
  granite, with serpentinite bedrock visible in some areas.
- Central Kyushu (Kumamoto, etc.): the Aso caldera, with well-developed
  volcanic ash, pumice, lava, and river deposits.
- Southern Kyushu (Oita, Miyazaki, Kagoshima, etc.): a mix of volcanic ground,
  granite, and sedimentary plains.
As a result, ground conditions differ significantly even within Kyushu itself.

2. Plains are largely soft, recent deposits
Plains and river floodplains around Fukuoka, Saga, Nagasaki, Kumamoto, Oita,
Miyazaki, and Kagoshima often have thick layers of recent deposits — sand,
silt, and clay. Key characteristics: high water content / low shear strength /
prone to compression and settlement / prone to liquefaction during earthquakes
/ prone to differential settlement and subsidence. Soft-ground countermeasures
are especially necessary along rivers and coastal plains.

3. Volcanic ground is widespread
Kyushu has numerous volcanoes, and pumice, volcanic ash, lava, and igneous
rock all contribute to its ground composition. Pumice and volcanic ash often
have low strength, high permeability, and high compressibility, can liquefy
when wet or saturated, and are prone to settlement and changes in earth
pressure. This kind of volcanic-derived ground matters particularly around
Aso, Sakurajima, Kirishima, Kuju, Unzen, and Ibusuki. Lava and igneous rock,
on the other hand, tend to have high strength and can serve as good bearing
layers for foundations, but weathering, cracking, cavities, and erosion make
them non-uniform — the surface may be soft while deeper layers remain strong.
Because volcanic ground places "strong bedrock" right next to "soft volcanic
deposits," ground investigation is especially important here.

4. Mountain and slope areas carry high landslide/collapse risk
Kyushu's mountains are steep, and their slopes are made up of weathered rock,
spoil, volcanic soil, and river deposits. With such steep gradients,
landslides and collapses are easily triggered by rain, typhoons, or
earthquakes, and ground strength varies considerably with the degree of
weathering. Landslide and sediment-disaster risk rises sharply during heavy
rain or seismic events.

5. Active faults and earthquakes have a major effect on the ground
Parts of Kyushu lie near plate boundaries and active fault zones, so ground
motion during earthquakes can be relatively large. Earthquakes can trigger
liquefaction, landslides, slope failure, ground settlement and differential
settlement, and damage to structural foundations. This means seismic design
needs to account for the ground's seismic response, not just the structure
itself.

6. Groundwater and hydrological conditions are also complex
Ground conditions across Kyushu can involve shallow groundwater in the
plains, fissure water in volcanic rock and granite, and saltwater intrusion
along the coast. Construction needs to account for spring water, rising
groundwater levels, strength loss in soft ground, and drainage/water-sealing
during ground improvement.

7. Construction considerations
Because ground conditions vary so much by location across Kyushu,
site-specific investigation is essential rather than relying on general
assessments. Soft ground calls for pile foundations or ground improvement;
lava and bedrock require confirming the bearing layer; volcanic ground needs
liquefaction and settlement countermeasures; slopes need landslide prevention
and anchoring; and ground assessment should account for earthquakes and heavy
rain, alongside a comprehensive look at landslide, liquefaction, and
differential-settlement risk.

Summary
Kyushu's ground is characterized by a combination of soft recent deposits,
low-strength volcanic-derived soil, strong lava and bedrock, and active
faults, earthquakes, and landslides. Because of this, ground strength in
Kyushu can't be judged in general terms — it's important to check each
site's geology, topography, hydrology, and earthquake/rainfall risk
individually.

We had a DeepSearch agent built on GPT-5.6-Luna (powered by Dify) evaluate this response. The results were as follows (we've trimmed a few overly long passages).

We ran the same question and evaluation against Muse Glimmer 30B previously, and the level of the critique here was slightly higher — this response appears to be operating on a different level.

GPT-5.6-Luna judged the overall content to be sound, and its feedback moved beyond basic correctness into refinements for readability and presentation — comments aimed at polishing an already-solid answer rather than fixing a flawed one. That's a clear sign it was reading the text at a higher level.

In this case, it's fair to say this response clearly surpasses Muse Glimmer's.

Overall assessment
As a general-audience overview explaining how much Kyushu's ground varies by
region, this text's overall direction is sound. It covers volcanic ground,
alluvial plains, mountain slopes, earthquakes, and groundwater, and the
structure is easy to follow. That said, using it for construction practice or
specialized education would require correcting a few terms and overly
definitive statements.

As a rough guide: accuracy 6.5/10, readability 8/10, suitability for
construction practice 6/10. The main issues aren't outright factual errors so
much as stating conditionally variable properties as if they were universal,
and grouping geological materials, ground characteristics, and disaster risk
into the same category.

Points needing correction: the phrase "the defining feature" is too
definitive; the description of plains treats their properties too uniformly;
the earthquake section should separate ground characteristics from disaster
hazards; the groundwater section needs qualification; "strong/weak ground"
needs to be broken down further; ground materials and disaster risk should be
kept separate; and some redundancy should be trimmed.

Final assessment: as an introductory, general-audience explanation of
Kyushu's ground, this text is a usable structure. However, phrases like
"recent deposits," "spoil," grouping Oita into southern Kyushu, "volcanic ash
has high permeability," and "can liquefy when wet" need correction or
qualification. Once revised, it's suitable as a general-audience explanation.
For construction planning or foundation-type selection, this text should not
be used as the basis for decisions — it needs to be combined with geological
maps, land-condition maps, liquefaction hazard maps, active-fault data,
boring surveys, standard penetration tests, and groundwater-level surveys.

Performance

We compared this against Muse Glimmer 30B as well. Throughput came out almost identical, but token usage to reach an answer was nearly three times higher — and, correspondingly, so was the time it took.

[Qwen3.8-27B]
prompt eval time = 303.63 ms / 64 tokens ( 4.74 ms per token, 210.78 tokens per second)
       eval time = 248375.97 ms / 5771 tokens (43.05 ms per token, 23.23 tokens per second)
      total time = 248679.60 ms / 5835 tokens

[Muse Glimmer]
prompt eval time = 262.18 ms / 68 tokens ( 3.86 ms per token, 259.37 tokens per second)
       eval time = 82874.28 ms / 1866 tokens (44.41 ms per token, 22.52 tokens per second)
      total time = 83136.46 ms / 1934 tokens

Both runs used a high reasoning-effort setting, and the pattern that's generally true of the Qwen series — taking a long time to reach an answer — still held here.

Verification: Using Vision

Qwen3.8 has been released as a multimodal model since earlier versions, and this one naturally ships with a vision encoder too. In llama.cpp, attaching the vision-encoder model lets you feed it images, as shown below.

This is roughly where things start to get tight — if you want more context length, you'll need to quantize the KV cache.

Command Line

build/bin/llama-server --model ./models/Qwen3.8-27B-UD-Q2_K_XL.gguf -t 12 -np 1 --prio 2 --temp 0.6 --top-p 0.95 --top-k 20 --port 8001 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 65535 --reasoning on --reasoning_effort xhigh --log-verbosity 4 --mmproj models/mmproj-BF16.gguf

Memory Usage

Memory usage came out as follows, showing an increase of roughly 1,336 MiB.

Device	CPU	CUDA0	Total
Model	Xeon E5-2690v4	RTX-A4000	—
Weight data (Q2_K-quant_XL)	397.85	9,567.89	9,965.74
KV cache size (f16)	—	4,096.00	4,096.00
RS buffer size (f32)	—	149.62	149.62
Gated Delta Net compute buffer	186.02	84.02	270.04
Vision weight data	—	887.99	887.99
Vision metadata	—	0.12	0.12
Total	583.87	14,785.64	15,369.51

Table 7: Memory usage with the vision encoder enabled (units: MiB)

Prompt and Output

Rather than text-only inference, we tested the model's analytical ability by attaching an image and asking it questions about that image.

For this, we had the model analyze the Muse Glimmer architecture diagram from our previous article, reusing the same image⁴, and asked it to "analyze this in detail and describe everything written in it."

Note: since this article's author writes in Japanese, this test was conducted using a Japanese-language diagram as the input image. The model's response below — translated into English — conveys the diagram's content, so the image itself is kept as-is here rather than redrawn in English.

Figure 4: The image fed into the Vision Encoder

The result was as follows. The passages noted below are places where the model's interpretation was mistaken.

MUSE GLIMMER 30B Architecture Slide: Detailed Analysis

1. Overall composition of the image
Under the title "Reference: MUSE GLIMMER 30B Structure" (dark navy header
bar), the slide has three layers: top text (left = basic model specs, right
= definitions of the SA/FA attention mechanisms), a central callout
(supplementary explanation of block composition), and a bottom flow diagram
(input -> embedding -> Transformer block group -> output). Its purpose is to
visualize the architecture of a 29.6B-parameter dense multimodal LLM and the
design philosophy of its core SA/FA hybrid attention.

2-5. Analysis of parameters, attention mechanism, flow diagram, and callout
(Dense 29.6B, hidden size 6,656, SwiGLU, RoPE theta=50,000 not applied to FA,
max context 128k, DFlash-based drafter, and the respective roles of SA/FA
are laid out in detail exactly as shown in the diagram)

6. Consistency and notes (analytical observations)
Block-count inconsistency: the diagram label reads "52-block configuration"
while the callout says "50 blocks." 52 = 4x13 is consistent as a count of
groups; 50 = 12x4+2 is consistent from the text's own framing. This gap most
likely comes from counting Glimmer Vision's two layers separately — it's
natural to read the diagram's "52" as the total including vision layers, and
the callout's "50" as the text-decoder block count alone. Unifying the
definition within the slide would be preferable. Possible typo: the FA
description "no positional information" appears to be a misprint of "no
positional encoding."

7. Summary
This slide explains that MUSE GLIMMER 30B, built on a dense 29.6B-parameter
model with a 128k context, repeats SA (local, low-cost) and FA (global,
high-quality) in a 3:1 ratio, concentrates positional encoding on SA layers
while removing it from FA layers, and achieves a hybrid attention design
that balances compute efficiency with reasoning performance. Vision-input
fusion, GQA + sigmoid gating, and DFlash/MTP-based inference support are the
other key design elements, with the only consistency issue being the 50/52
block-count discrepancy between the diagram and the text.

The response came back roughly twice as long as what Muse Glimmer 30B produced for the same task previously. On top of that, the summary was nearly flawless, and it even called out an inconsistency in how we'd labeled our own diagram.

The vision encoder is still built on SigLIP2, but additional fine-tuning may well be reflected here — the way it reasons from what it extracts felt noticeably more capable.

A Look Inside the Thinking Section

Qwen models are well known for a distinctly self-doubting internal voice during their thinking phase — the model seems to talk itself into second-guessing everything right before it's about to answer, wanting to double-check just one more thing or start over from scratch, dragging out its reasoning in the process. We checked whether that pattern still holds here.

We've summarized the actual thinking-section output for the Kyushu-ground question in the appendix at the end of this article. Here, we'll first go over the key points from having Google Gemini 3.7 Flash evaluate that output.

According to Gemini's analysis, Qwen3.8-27B's thinking pattern breaks down into five main traits. First, a separation between the thinking language (English) and the output language (Japanese) — a multilingual-processing behavior where internal reasoning happens in English but the final answer comes out in Japanese. Second, multi-angle context inference and intent-filling, where it anticipates the underlying purpose behind the question (construction practice, exam prep, and so on). Third, exhaustive knowledge brainstorming and categorization, where it lists out related keywords comprehensively and then organizes them by region and geology. Fourth, real-time self-correction and fact-checking — starting to write "Miyagi" and immediately correcting it to "Miyazaki," or avoiding overgeneralizing where serpentinite is distributed, repeatedly double-checking small factual details. And fifth, iterative simulation of output structure and formatting — trying out multiple drafts of whether to use a table or bullet points, and in what order to place the headings.

The important point here is that the overtly negative emotional tone common through Qwen3.5 doesn't show up — the evaluation describes the process as proceeding "extremely logically." Since there's been no architectural change at all, this improvement appears to come down to training methodology.

So what exactly changed in how this model was trained? A clue seems to be in the following post.

Qwen3.8-Max: A New Bar for Coding and Cowork
(https://qwen.ai/blog?id=qwen3.8)

This post describes the flagship Qwen3.8-2.4T-A95B model, but given that Qwen3.8-27B is a smaller sibling, it likely went through similar training. In the "Work" section, the post explains that scaling up real-world RL systems (continuously scaling tasks, workspaces, and toolchains along independent axes), a general-purpose reward system capable of internalizing diverse forms of verification, and an online data balancer were combined to lift task-completion capability uniformly across multiple major toolchains.

That distinctive self-doubting quality, present since Qwen3, doesn't actually originate with Qwen — it traces back to a training method DeepSeek-AI calls "Long CoT Cold Start"⁵, applied to DeepSeek-R1 to boost reasoning ability in cases requiring many turns of back-and-forth. The "cold start" in the name reportedly refers to the tendency to pause mid-reasoning and start over from scratch.

Rather than relying solely on QA-style datasets, this method trains the trial-and-error process directly by drawing on data from sources like programming forums such as Reddit — and that data is reportedly full of human emotional expression, which is said to be what feeds into the model's self-doubting tone. This trait shows up not just in Qwen but broadly across DeepSeek, its originator, and other Chinese LLMs.

Qwen3.8 appears to have revisited various stages of its training pipeline, and we found a related paper, "Unified Data Selection for LLM Reasoning"⁶, which we looked into.

The paper proposes a new, training-free, and extremely lightweight data-evaluation metric called High-Entropy Sum (HES), and applies it across the SFT (supervised fine-tuning), RFT (rejection-sampling fine-tuning), and RL (reinforcement learning) stages to measure its effect.

With conventional training (based on average entropy), the important decision points in the reasoning process get diluted by a large volume of low-entropy tokens — boilerplate text, simple arithmetic logic — so the model ends up training without ever distinguishing good solutions from bad ones. Using HES instead, the paper reports, lets the optimized model learn high-quality solutions much more efficiently, with a clearly measurable training benefit.

Training a model at the scale of Qwen3.8-Max presumably involves a huge volume of answer data with substantial reasoning steps built in. Our guess is that applying the HES concept to that data made more effective solution paths — and by extension, more logical reasoning routes — easier to select, suppressing emotional noise and, as a result, producing more advanced reasoning ability.

It's also worth noting that issues with Long-CoT-style training have already been pointed out elsewhere, such as in the paper "Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubt"⁷, and it's likely that some countermeasure along those lines was applied here as well.

Some people still argue that Chinese models simply distill capability out of frontier models. Based on what we're seeing here, that framing doesn't quite fit Qwen3.8-27B — distillation alone doesn't look like the whole story behind its capability gains. If anything, the Qwen team appears to actively track and adopt a wide range of techniques, and that appetite for new ideas seems to be a real part of what makes this model work.

This shift in training methodology may also have changed how the model arrives at its answers — the reasoning path itself looks noticeably different from before. If you're doing harness engineering around this model, some of your existing prompting approaches may no longer work as expected, and revisiting them may become necessary if something that used to work stops working.

As AI models continue to evolve, the way they process information keeps changing along with them — and it may be up to us to keep updating our own approach in step.

Benchmarks: How the Wider World Sees It

Below is the benchmark comparison published by Alibaba's Qwen team. It shows results on par with — or better than — Anthropic's Claude Opus 4.6 in Max Reasoning mode. Based on what we've seen in our own hands-on testing, that claim doesn't look far-fetched.

Category	Benchmark	Qwen3.8-27B	Qwen3.6-27B	Qwen3.7-Plus	Muse Glimmer-30B	Opus4.6 Max
Coding	Agentic terminal coding (Terminal Bench 2.1)	73	63.4	64	51.7	78.2
Coding	Agentic coding (SWE-bench Pro)	61.7	53.5	57.6	51.2	53.4
Coding	Repo-level code generation (NL2Repo-Bench)	42.3	36.2	41.1	--	47.6
Coding	Agentic coding (DeepSWE 1.1)	42.2	13.3	14.2	--	--
Coding	Software engineering (QwenSWEBench)	79	49.3	59.2	--	63.8
Agent	Long-horizon office work (CoWorkBench)	70.7	61	65.1	--	68.2
Agent	Professional job tasks (JobBench)	33.4	21.8	27.6	--	--
Agent	Frontier agentic tasks Pass@1 (Agents' Last Exam)	20.4	10.6	13.2	--	--
Agent	Frontier agentic tasks Score (Agents' Last Exam)	42.9	27.3	33.6	--	--
General	Instruction following (IFBench)	79.5	69.1	79.1	77	62.5
General	Scientific reasoning (GPQA Diamond)	89.2	87.8	90.3	83.5	91.3
General	Multidisciplinary reasoning (HLE)	30.8	24	34.7	22	40
General	Competitive coding (LiveCodeBench v6)	90.3	83.9	89.6	--	88.8

What about third-party evaluations? Here's Artificial Analysis's Intelligence Index chart.

Figure 5: Artificial Analysis Intelligence Index chart

With thinking mode on, it posted a striking score of 52 — not just surpassing Claude Opus 4.6, but matching GPT-5.6 Luna, likely OpenAI's most widely used model at the moment. Seeing the score climb this far came as a genuine surprise to us.

That said, while this behavior works well for Deep Research, it also means the model is likely to be slow for general-purpose use that mixes in casual conversation, and given how many new approaches went into it, some retuning of middleware and instructions is clearly going to be needed.

In that sense, Meta's Muse Glimmer 30B, covered earlier, and Google DeepMind's longer-established Gemma-4-31B-it, still have plenty to offer. Their thinking phase is far shorter than Qwen3.8's, so they can produce answers faster.

In particular, Muse Glimmer 30B and Gemma-4 both benefit heavily from sliding-window memory savings, making it relatively easy to extend their context length. Muse Glimmer 30B's vision encoder is also appealing for its resolution. Given these different strengths, rather than defaulting to a single model for everything, what seems to matter going forward is the skill of choosing and applying the right model for the task.

Conclusion

This article covered Qwen3.8-27B, the compact model the local-LLM community had been waiting for. Its capability is no exaggeration — it delivered performance beyond what we expected. The benchmark results published on Artificial Analysis in particular likely came as a considerable surprise to a lot of people.⁸

The first use case that comes to mind is Deep Research. Paired with a search engine, it can dig into deep content without difficulty, and its resistance to falling into wasteful reasoning loops makes it a reassuring tool for search-augmented use.

Architecturally, it carries forward Qwen3.5's Gated Delta Network-based hybrid design, and being able to run it as-is without updating your inference engine is a significant point in its favor.

That said, given how firmly this design leans toward raw capability, its tendency toward a large KV cache footprint is a genuine drawback. Keeping the max context length very long requires a correspondingly capable GPU, and that memory overhead is likely to be a real headache for local-LLM users.

On the training side, the shift away from the traditional Long CoT Cold Start method has also left some users unsure what to make of it. Figuring out how to adapt to this change, and what prompt adjustments it calls for, will likely be the key going forward.

As findings from this new approach get shared through arXiv and elsewhere, it will likely spread gradually to other models. Efforts like this could act as a trigger that meaningfully lifts capability across both commercial frontier models and open-weight models alike.

New technical advances don't just show up in architecture and tokenizer specs — they also show up inside the thinking section, a part of the process that's normally hidden from view. It's worth paying attention to that usually-overlooked area and watching how model behavior changes there.

References

Qwen/Qwen3.8-27B -- Hugging Face
https://huggingface.co/Qwen/Qwen3.8-27B

unsloth/Qwen3.8-27B-GGUF - Hugging Face
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

OpenLM-AI Qwen3.8
https://openlm.ai/qwen3.8/

Qwen3.8-Max: A New Bar for Coding and Cowork
https://qwen.ai/blog?id=qwen3.8

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
https://arxiv.org/pdf/2501.12948

Unified Data Selection for LLM Reasoning
https://arxiv.org/pdf/2605.22389

Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubt
https://arxiv.org/pdf/2505.23480

Appendix

Expected KV Cache Sizes

Qwen3.8-27B requires the following KV cache sizes. Note that supporting long context requires these amounts of VRAM on top of the weight data and vision-encoder data.

Max Context Size	K:f16, V:f16	K:q8_0, V:q8_0	K:q8_0, V:turbo3⁹
65,536	4,096 MB	2,048 MB	1,488 MB
131,072	8,192 MB	4,096 MB	2,976 MB
262,144	16,384 MB	8,192 MB	5,952 MB

Table 8: Expected KV cache sizes

Thinking-Section Content for the Kyushu Ground Question (Summary)

Summarizing the actual thinking-section content for the "describe the characteristics of Kyushu's ground" question covered in the main text (the original was output in English), the trial-and-error process went roughly as follows. The full text is extremely long, so only the key points are excerpted here.

It begins by questioning itself over whether "ground" refers to geological bedrock or to civil-engineering/construction-style ground conditions, and considers the question's likely intent from multiple angles — an exam question, construction practice, or general knowledge. It then exhaustively lists out the elements worth covering: plate tectonics, volcanic activity, sedimentary layers, liquefaction risk, and regional differences (north/central/south).

Along the way, it repeatedly double-checks small factual details. For instance, it starts to write "Miyagi," catches that it's not a Kyushu prefecture, and immediately corrects it to "Miyazaki"; it also reconsiders where serpentinite is actually distributed in Kyushu and settles on the more limited phrasing "in some areas" — self-checking specific place names and geological terms at every step.

It also repeatedly adjusts its tone — not overgeneralizing, not overusing jargon, not making the answer too long — and trials multiple structural options: bullet points, paragraphs, or a table. It eventually settles on a draft that includes a table with "aspect | characteristic" columns, then tries out several versions of the concluding sentence, fine-tuning the wording right up to the end.

The sheer volume of this trial-and-error is what substantiates the self-doubting quality mentioned earlier in the article. The actual output — the Kyushu-ground answer quoted in the main text — is concise and tidy, but the internal process behind it involves repeated self-questioning and backtracking. That gap between process and output was striking.

Some users, working from the assumption that the model simply wouldn't fit on their hardware, went as far as building their own memory-reduction scheme and inference engine from scratch to verify it themselves. In 2-bit quantized mode, they reportedly recorded results well beyond what anyone expected. ↩
For more detail on this setup, see our earlier article on using GPUSOROBAN. (https://www.bluecore.net/archives/242) ↩
This article's author is based in Japan, so the model was prompted in Japanese and asked about a Japan-specific topic (the ground conditions behind the recent Kumamoto earthquake) — hence the choice of question here and elsewhere in this article. ↩
This image uses an earlier version of the model diagram than the one shown previously in this article. ↩
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (https://arxiv.org/pdf/2501.12948) ↩
https://arxiv.org/pdf/2605.22389 ↩
https://arxiv.org/pdf/2505.23480 ↩
The people running these benchmarks may well have rubbed their eyes and re-verified the results at least once, wondering if something had gone wrong — that's roughly how long it took for Qwen3.8-27B's results to come out relative to what we'd expect. ↩
Measured using a llama.cpp fork that supports TurboQuant. "turbo3" indicates TurboQuant 3-bit quantization. Note that this was tested on a 2-GPU setup, so some overhead is present. ↩

Muse Glimmer 30B: An Architecture and Hands-On Look at Meta's New Mid-Range Model

oooocean66 — Fri, 18 Sep 2026 08:35:09 +0000

In the world of generative AI, Meta's LLaMa is often considered the model that kicked off the open-weight era. Its successor, LLaMa-4, however, fell short of expectations and was widely regarded as a disappointment, leaving Meta in a difficult position over the past year.

Then, in early 2026, Meta released a new lineup called the "Muse" series. Muse Glimmer 30B, the mid-range model in that lineup covered here, appears to have drawn considerable attention in the local-LLM community.

This article walks through Muse Glimmer 30B's architecture and then verifies it hands-on on GPU hardware: standard text inference, its combination with the speculative-decoding technique DFlash, and its Vision capability. We look at what the model can actually do, based on the results below.

Meta's Muse Series

LLaMa, the Model That Started the Open-Weight Era

One of Meta AI's most significant contributions was releasing the LLaMa models as open weights.
At the time, high-performance chat models like GPT-3.5-Turbo existed, but they were closed models that couldn't be run on a typical local GPU.

To address that gap, Meta released LLaMa free of charge.
Version 1 came with restrictive licensing terms — even Alpaca, a derivative model, had commercial-use restrictions — but Version 2 relaxed those terms considerably.
That change made it much easier for a broad range of users to adopt LLMs locally, and the model drew significant attention as a result.

LLaMa 2 in particular was notable for being the first major model to adopt RoPE (Rotary Positional Embedding)¹ and Group Query Attention (GQA)², both novel techniques at the time.

Input tokens -> Embedding -+-> [ RMSNorm -> Self-Attention (GQA + RoPE) ] -+
                            |                                              |
                            +<-----------------------------(residual add)--+
                            |
                            +-> [ RMSNorm -> SwiGLU FFN ] -+
                            |                               |
                            +<------------(residual add)----+
                                   |
                            (x N decoder blocks)
                                   |
                            RMSNorm -> Linear -> Softmax -> Output tokens

Figure 1: LLaMa-2's architecture — a standard pre-norm decoder block, with GQA (fewer key/value heads shared across query heads) and RoPE-based position encoding applied at every attention layer.

These architectural choices remain standard across many open-weight models today, and models such as llm-jp-4-8b-thinking still carry that lineage forward.³

However, LLaMa-4 — the successor released after the success of LLaMa-3 — underperformed relative to expectations and was widely characterized as a disappointment. Meta itself acknowledged the setback, which was accompanied by organizational disruption within its research team and the departure of several researchers — an unusually visible decline for a lab of its standing.

From LLaMa to Muse

In April 2026, Meta announced a new model called Muse Spark.

Muse Spark launched as a closed model and initially drew little attention. Versions 1.1 and 1.2 followed in July and August respectively, and as each new version shipped, its performance gradually became better known.

Around the release of Muse Spark 1.2, Meta also introduced Muse Glimmer 30B, the mid-range model covered in this article. Its release appears to have generated notable interest within the local-LLM community.

Muse Glimmer's Architecture

Muse Glimmer uses a hybrid structure similar to several currently popular open-weight models. The architecture is shown below.

Input strings -+- Glimmer Vision (image input) -----------------+
               |                                                 |
               +- tok -> Emb -----------------------------------+-(+)-+
                                                                       |
                                RoPE (SA layers: theta=50,000)  <------+
                                RoPE (FA layers: theta=0 = NoPE) <-----+
                                                                       |
     [ SA -> SA -> SA -> FA ]  x13 groups  ------------------------->
            (52 blocks total = 4 layers x 13 groups)
                                                                       |
                                                          LNR -> SoftMax -> Output Probabilities

Figure 2: Muse Glimmer 30B's architecture. SA = Sliding Attention (local context, RoPE applied); FA = Full Attention (aggregates and consolidates, RoPE disabled / NoPE).

At a glance, the block layout resembles Qwen's more than it does Google DeepMind's Gemma-4 — it's a comparatively simple structure. It's a 29.6B-parameter, dense model that uses GeLU for its activation function.

Attention Structure

The attention structure is a hybrid built on top of Group Query Attention.

Layers labeled SA use Sliding Window Attention, which handles local context analysis. By explicitly bounding the context range each layer looks at, this reduces both memory usage and compute cost.

Layers labeled FA use Full Attention. These layers aggregate and consolidate what the preceding SA layers produced, running as standard GQA — which makes them comparatively more compute-heavy than the SA layers.

Blocks are arranged in groups of three SA layers followed by one FA layer, repeated 13 times for a total of 52 blocks.

The 3:1 SA-to-FA ratio resembles the structure used since Qwen3.5, but looking at what each layer type actually does, the closer functional match is Gemma-4's "local + global" design.⁴

A more distinctive detail is that the RoPE theta value can be set independently per layer: SA layers use 50,000, while FA layers use 0.

Looking at the config.json in the Hugging Face repository shows this setting directly.

{
  "architectures": [
    "MuseGlimmerForConditionalGeneration"
  ],
  "dtype": "bfloat16",
  "image_token_id": 200092,
  :
  :
  :
  "hidden_size": 6656,
  "initializer_range": 0.02,
  "intermediate_size": 19968,
  "layer_rope_theta": [
    500000.0,
    500000.0,
    500000.0,
    0,
    500000.0,
    500000.0,
    500000.0,
    0,
    500000.0,
    500000.0,
    500000.0,
    0,
    500000.0

In the modeling source code, a theta value of 0 causes the RoPE module to be treated as NoPE — meaning positional encoding is skipped entirely for that layer.

This behavior is confirmed in Hugging Face's transformers library:

(https://github.com/huggingface/transformers/blob/main/src/transformers/models/muse_glimmer/modeling_muse_glimmer.py)

In a separate file, modular_muse_glimmer.py, the code sets position_embeddings to None whenever the RoPE theta parameter is 0. The class that defines the model's actual forward pass then branches on that: if position_embeddings is None (i.e., NoPE), it skips the positional-encoding step entirely.

class MuseGlimmerTextAttention(nn.Module):
    :
    :
    def forward(
        self,
        hidden_states: torch.Tensor,
        position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
        attention_mask: torch.Tensor | None = None,
        past_key_values: Cache | None = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
        :
        :
        # NoPE layers receive `position_embeddings=None` from the model.
        if position_embeddings is not None:
            cos, sin = position_embeddings
            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
        :
        :

This suggests a design choice to keep FA layers focused purely on the aggregation role, stripping out anything unrelated to that job and drawing a clear division of labor between the two layer types.

Vision Encoder

The vision model, called Glimmer Vision, is a custom encoder rather than the widely used SigLIP vision transformer that most models rely on.

It has 50 blocks — 12 groups of (3 SA + 1 FA), plus one more (1 SA + 1 FA) — and its structure is nearly identical to the text decoder's, differing only at the very end. The resulting vector is treated as 1,024 tokens' worth of information and fed into the text decoder.

This design is described in Meta's own paper, "Perception Encoder: The best visual embeddings are not at the output of the network" (arXiv:2504.13181v2, published April 28, 2025), and Glimmer Vision appears to be built on the Perception Encoder architecture proposed there.

Vision encoder design has diversified across model vendors this year — some follow the approach above, while others, like Gemma-4-12B-it, fold nearly all of the weight data into the text encoder itself (aside from CLIP). It's possible this custom approach delivers stronger analytical capability than a straightforward SigLIP-based encoder, though that would need to be verified directly — which is one of the things we test below.

Running It Ourselves

For this verification, we used a GPUSOROBAN High-Speed Computing instance equipped with an NVIDIA RTX A4000.

Overall Setup

The setup is as follows: we open an SSH tunnel to an access server, then reach the target instance through that tunnel.⁵ llama.cpp's service port is relayed to localhost via port 8001 on both ends.

(The network diagram accompanying this section is omitted here — it uses several Japanese-only labels for "home," "client," "access server," and "target instance," and the setup is already fully described in the text above.)

Instance Used

The GPU used for this test is an NVIDIA RTX A4000. The instance specifications are as follows.

Item	Spec
Instance type	s16-1-a-standard-ubs24-v
GPU	NVIDIA RTX A4000
GPU memory	GDDR6 16GiB
GPU memory bandwidth	448.0 GB/s
FP32 compute	19.17 TFLOPS
BF16 compute	38.34 TFLOPS
INT8 compute	153.4 TOPS
INT4 compute	306.7 TOPS
vCPU	11 cores
System memory	50 GiB
Storage	Persistent 100 GiB
CUDA version	13.2
NVIDIA driver version	580
OS	Ubuntu 24.04 Server

Table 1: Instance specifications

Model Files Used

Even a 4-bit quantized version of Muse Glimmer exceeds 16GiB. Factoring in KV cache usage as well, we used a 2-bit quantized model optimized with Unsloth Dynamic 2.0.

We downloaded the following files from Unsloth's Muse Glimmer repository:

File	Size	Description
Muse-Glimmer-30B-UD-IQ2_M.gguf	12.3 GB	UD2.0, 2-bit quantized model
mmproj-kquant.gguf	1.40 GB	Quantized vision encoder
dflash-kquant.gguf	1.63 GB	Quantized DFlash draft model

Table 2: Files used with llama.cpp

We used llama.cpp version 10380 (0b1bad14f).

Support for Muse Glimmer had already landed via model: Muse Glimmer Support – #26841, and this build includes that change.

Verification: Standard Mode

First, we ran text inference using the main model on its own.

Launch Command

We used the following command to launch the web frontend for testing.

For this run, the KV cache was kept in FP16 rather than quantized to 8-bit. We also raised the log verbosity from the default of 3 to 4 to check memory usage.

build/bin/llama-server --model ./models/Muse-Glimmer-30B-UD-IQ2_M.gguf -t 12 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --port 8001 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 131072 --reasoning on --log-verbosity 4

Memory Usage

Memory usage came out as follows. Thanks to the sliding-window design, the footprint looks smaller than what you'd typically see from a model relying solely on standard Grouped Query Attention.

Component	CUDA0	CPU
Weight data	10,623.10	1,052.08
KV cache data	1,664.00	—
Sliding-window KV cache	97.50	—
Gated DeltaNet compute buffer	273.52	156.52
Total	12,658.12	1,208.60

Table 3: Memory usage in standard mode, main model only (units: MiB)

Output

We asked the following question and examined the response.

Given the recent Kumamoto earthquake, this is a question we've been asking various models out of curiosity.

Describe the characteristics of the ground/soil conditions in Kyushu.

The response came out as follows. Our first impression was that it looked surprisingly well-informed.

Kyushu's ground conditions are among the most affected by volcanic activity
and crustal movement anywhere in the Japanese archipelago, and the defining
characteristic is a foundation dominated by volcanic sediment and soft
alluvial deposits.

1. Geological and Topographic Features
- An island arc ringed by active volcanoes: Located at the southwestern end
  of the Japanese archipelago, on the boundary between the Philippine Sea
  Plate and the Eurasian Plate. Some of Japan's most active volcanoes are
  concentrated here — Mt. Aso, Sakurajima, the Kirishima range, and Mt.
  Unzen — and volcanically derived ground extends across a wide area.
- The Median Tectonic Line runs through the region: A roughly east-west
  fault line crosses nearly all of Kyushu, and active fault zones are
  extremely common. A chain of active faults runs through Fukuoka,
  Kumamoto, and Oita, including the Futagawa and Hinagu fault zones, which
  were the epicenters of the 2016 Kumamoto earthquakes.
- Contrast between mountainous terrain and narrow plains: Inland areas are
  ringed by the Kyushu mountains, interspersed with alluvial lowlands such
  as the Ariake Sea coast, the Chikugo Plain, Kitakyushu, and the
  Miyazaki Plain. The plains have thick sedimentary layers, while the
  mountainous areas are dominated by hard igneous rock.

2. Ground Material Characteristics
- Extensive volcanic ash and ejecta layers: Ash from the Aso caldera's
  eruptions covers central to northern Kyushu in deposits reaching tens of
  meters, and in some places over 100 meters. Pumice, scoria, and
  tuffaceous ash are known for high water content, high compressibility and
  permeability, and a tendency toward consolidation settlement.
- Thick, soft alluvial layers: Lowlands such as the Ariake Sea coast, the
  Chikushi Plain, the Yatsushiro Plain, and the Miyakonojo Basin consist of
  thick alluvial clay, sand, and silt layers. Low N-values and poor bearing
  capacity are common across broad areas of soft ground.
- A mix of volcanic and sedimentary rock: Northern Kyushu's bedrock
  consists of Paleozoic sedimentary rock, Tertiary igneous rock, and hard
  rock formations such as the Mifune Group. Southern Kyushu is dominated by
  more recent volcanic rock, which is hard but weathers easily and is prone
  to collapse.

3. Geotechnical Characteristics
- High liquefaction risk: Reclaimed land and coastal alluvial sand layers —
  particularly around Hakata Bay in Fukuoka, Nagasaki Bay, Yatsushiro Bay
  in Kumamoto, and Kagoshima Bay, as well as sandy ground near river
  mouths — are prone to liquefaction. Widespread liquefaction damage was
  reported during the 2016 Kumamoto earthquakes.
- Ground compressibility and differential settlement: Volcanic ash layers
  combined with organic alluvial clay contribute to differential
  settlement of buildings, road subsidence, and long-term settlement
  following liquefaction. Groundwater tables are shallow and sensitive to
  fluctuation.
- Ground amplification characteristics: Thick, soft volcanic ash and
  alluvial layers amplify seismic motion in many areas. Long-period ground
  motion amplification has been specifically noted around the Aso caldera
  and the Chikugo Plain.
- Cities with extensive reclaimed and improved land: Port cities such as
  Kitakyushu, Nagasaki, Kagoshima, and Sasebo are historically built on
  reclaimed land, and ground improvement is essential across much of these
  areas.

Regional Trends
- Northern Kyushu (Fukuoka, Saga, northern Oita): Dominated by alluvial
  clay in the Chikushi Plain and reclaimed land along the Genkai Sea coast.
  Ground is soft with high liquefaction risk.
- Central Kyushu (Kumamoto, southern Oita): Thickly covered by Aso volcanic
  ash. This ash compacts poorly and is prone to settlement.
- Southern Kyushu (Miyazaki, Kagoshima): A mix of volcanic ejecta directly
  beneath active volcanoes and marine-terrace deposits. Ground consists of
  volcanic soil over shallow bedrock, giving locally high bearing capacity
  in places, but with significant heterogeneity from collapse and ejecta
  deposits.

Overall, Kyushu's ground can be characterized as "heterogeneous, multi-
layered deposits from volcanic activity plus thick, soft alluvial layers,"
making earthquake resistance, ground subsidence, and liquefaction
countermeasures major challenges for urban development in the region.

We had a DeepSearch agent built on GPT-5.6-Luna (powered by Dify) evaluate this response. The results were as follows.

The evaluation flagged a recurring issue: the response tends to generalize localized Kyushu characteristics as if they applied uniformly across the whole region. Only one point — the claim about the Median Tectonic Line — was flagged as factually incorrect outright; the rest of the issues are mostly about over-generalization.

Smaller models tend to produce noticeably off-base answers to this kind of question, so getting a response of this quality without RAG suggests fairly strong reasoning ability. That's particularly notable given that we normally test with 4-bit quantized models, and this run used a 2-bit quantized model instead.

The overall thrust of the response is reasonable, but it should be treated
with caution given that it describes all of Kyushu with a single, uniform
set of ground characteristics.

In particular, the following statements need correction:
- "The Median Tectonic Line crosses all of Kyushu": there are multiple
  views on whether and where the Median Tectonic Line runs through Kyushu,
  so this cannot be stated as fact.
- "Located on the boundary between the Philippine Sea Plate and the
  Eurasian Plate": Kyushu is affected by plate motion, but describing it as
  simply sitting on a plate boundary is inaccurate.
- "Aso volcanic ash deposits reach over 100m in central-northern Kyushu":
  this does not distinguish between airfall ash, pyroclastic-flow deposits,
  and redeposited material, and reads as if a uniform 100m+ layer covers
  the entire area.
- "Volcanic ash has high compressibility and permeability": ash, pumice,
  and pyroclastic-flow deposits each have different properties and
  shouldn't be grouped together.
- "Southern Kyushu is dominated by volcanic rock": Shirasu, sedimentary
  rock, alluvial deposits, and marine terraces are also widely distributed
  there.
- "Port cities are built on reclaimed land, making ground improvement
  essential": reclaimed land is only part of the picture, and the need for
  ground improvement varies by ground conditions and structure.

On the other hand, the following points are largely correct:
- Kyushu has many active volcanoes and active faults.
- Soft alluvial layers are distributed along the Ariake Sea coast, the
  Chikushi Plain, and the Kumamoto Plain, among other areas.
- Liquefaction should be considered for reclaimed land, former riverbeds,
  and low-lying areas near river mouths.
- During the Kumamoto earthquakes, the Futagawa and Hinagu fault zones
  were active, and liquefaction damage did occur.
- Consolidation settlement due to soft ground is a recognized issue along
  the Ariake Sea coast.

A more accurate summary, then, would be: Kyushu's ground varies regionally
across volcanic sediment, bedrock, alluvial deposits, and reclaimed land.
Rather than treating volcanic ash and alluvial layers as characteristic of
all of Kyushu, subsidence, liquefaction, slope failure, and ground-motion
amplification should be assessed separately for each region based on its
specific terrain and geology.

Performance

Speed performance came out as follows, with an output rate of about 22.52 tokens/sec. Given the model's size, that's on the lower end compared to 9B–12B models, though it's not slow enough to be a practical problem. That said, raising the reasoning effort further might noticeably increase response time.

prompt eval time =     262.18 ms /    68 tokens (    3.86 ms per token,   259.37 tokens per second)
       eval time =   82874.28 ms /  1866 tokens (   44.41 ms per token,    22.52 tokens per second)
      total time =   83136.46 ms /  1934 tokens

Verification: Using DFlash

Muse Glimmer supported DFlash from release, and Unsloth's repository includes a dedicated draft model (dflash-kquant.gguf). We loaded it via --model-draft and re-ran the same question as in standard mode.

The result was underwhelming: output speed dropped from 22.52 tps to 17.80 tps, and the token acceptance rate came in at only 15.7% (mean length 1.94).

We plan to dig further into the conditions under which this speculative-decoding technique does or doesn't pay off in a follow-up article.

Verification: Using Vision

Muse Glimmer also ships with a vision encoder, with a dedicated mmproj file available in Unsloth's repository. Loading this vision encoder together with the DFlash draft model resulted in an out-of-memory error, so we tested each capability separately.

Command Line

build/bin/llama-server --model ./models/Muse-Glimmer-30B-UD-IQ2_M.gguf -t 12 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --port 8000 -c 131072 --reasoning on --log-verbosity 4 --mmproj models/mmproj-kquant.gguf

Memory Usage

Memory usage came out as follows, showing an increase of roughly 1,336 MiB.

Component	CUDA0	CPU
Weight data	10,623.10	1,052.08
KV cache data	1,664.00	—
Sliding-window KV cache	97.50	—
Gated DeltaNet compute buffer	273.52	156.52
Vision weight data	1,335.41	—
Vision metadata	0.28	—
Total	13,993.81	1,208.60

Table 4: Memory usage with the vision encoder enabled (units: MiB)

Prompt and Output

Rather than text-only inference, we tested the model's analytical ability by attaching an image and asking it questions about that image.

For this, we had the model analyze the same Muse Glimmer architecture diagram shown earlier in this article⁶ and asked it to "analyze this in detail and describe everything written in it."

Figure 3: The image fed into the Vision Encoder

The result was as follows.

This slide, titled "Reference: MUSE GLIMMER 30B Structure," is a diagram of
the MUSE GLIMMER 30B model architecture's specifications and overall data
flow.

[Spec list, left side]
- Dense-type 29.6B model
- Hidden size: 6,656 (activation function: SwiGLU, dimension 19,968)
- LNR: Linear Layer
- RoPE: theta = 50,000. Notably not applied to Full Attention layers — for
  FA layers, theta is set to 0, which signals NoPE (no positional encoding)
- Max context: 128k tokens (131,072 tokens)
- Includes a DFlash-based drafter; supports MTP

[Terminology, right side]
- SA: Sliding Attention
  - GQA-based, with a sigmoid output gate
  - Handles local context analysis via a sliding window
  - Low compute cost, fast
  - Low memory usage
- FA: Full Attention
  - GQA-based, with a sigmoid output gate
  - Aggregates and consolidates information from Sliding Attention layers
  - Maintains global structure and handles complex reasoning
  - Uses no positional encoding — positional information is left entirely
    to Sliding Attention

[Central block diagram flow]
Input strings are tokenized (tok) and embedded (Emb). RoPE is added to the
embeddings. Separately, Glimmer Vision receives input from an image icon,
and its output is added immediately after the embedding + RoPE step.

The language model body that follows is described as having 52 blocks
total: 52 blocks = 4 layers x 13 groups.

Each group runs SA → SA → SA → FA in sequence, with SA shown in blue and FA
in red. Thirteen of these groups are chained together.

At the end of each group's output, LNR normalizes the result, which then
passes through Softmax to produce the output probabilities.

In short: taking embeddings + vision features + RoPE as input, the model
runs SA three times for local analysis, then FA once to aggregate and
consolidate — a 4-layer unit repeated across 13 groups for 52 blocks total,
forming a dense causal model. The vision encoder follows a similarly dense
50-block structure, with a 128k context window, SwiGLU activation, and the
same ability to disable RoPE for Full Attention layers.

The content is largely accurate, with only minor issues, and we judged the analytical quality to be high. The response correctly captures what each element means, how the pieces connect, and even the detail that Full Attention layers use NoPE.

External Benchmark: How the Model Is Rated Elsewhere

We checked Muse Glimmer's standing against other models using Artificial Analysis, a site that publishes third-party benchmark data. The chart below is organized around their Intelligence score.

Figure 4: Artificial Analysis Intelligence scores (models shown are a custom selection for this comparison)

Muse Glimmer 30B scores 35, which is high relative to other open-weight models in the roughly-30B parameter range. Since this chart excludes coding- and agent-specialized models (a category we don't test as heavily), the comparison is limited to that scope, but within it, Muse Glimmer scores above both Qwen3.5 and Gemma-4.

What stood out more was the score for Muse Spark, the closed model this architecture is presumably derived from — its score is close enough to Claude 5 Fable's that the gap is fairly small. Reaching this level of performance within a short update cycle, combined with reports that Meta may release this model line as open weights in the future, makes it a lineup worth continuing to watch. The Muse series took a long and difficult path to reach this point, and it will be worth seeing how the rest of the lineup performs going forward.

Conclusion

This article analyzed the architecture of Meta's Muse Glimmer 30B and evaluated its behavior through hands-on testing, including basic chat inference. Based on that evaluation, Muse Glimmer 30B looks like a strong option in the mid-range model category.

Its architecture builds solidly on the "local + global" pattern that has proven effective across recent hybrid-attention models, while going a step further by omitting RoPE from the aggregation (FA) layers — a refinement that lets those layers focus purely on their consolidation role.

The vision encoder is also a custom design based on Meta's own Perception Encoder architecture, and its analytical performance held up well in testing.

The model is released under the Apache 2.0 license, which makes it a viable option for developers considering a Tokens-as-a-Service offering — a space where Gemma-4 has had little competition until now.

Muse Spark 1.2, the closed model this architecture appears to derive from, also scores well on third-party benchmarks, suggesting Meta's more recent models have made real progress since the LLaMa-4 generation. Whether Meta continues to gain ground in the frontier-model space is something worth watching going forward.

On a practical note, this test again highlighted the value of having a GPU with more VRAM on hand. GPU prices keep climbing, but without the ability to freely run mid-range models like this one locally, there's a lot that's hard to verify firsthand — a recurring frustration in this kind of investigation.

References

Muse Glimmer 30B repository (Hugging Face)
https://huggingface.co/meta-models/Muse-Glimmer-30B

Perception Encoder: The best visual embeddings are not at the output of the network
Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Choi, et al.
https://huggingface.co/papers/2504.13181
https://github.com/facebookresearch/perception_models

Muse Glimmer model code source (GitHub)
https://github.com/huggingface/transformers/tree/main/src/transformers/models/muse_glimmer

llama.cpp PR26841
https://github.com/ggml-org/llama.cpp/pull/26841

Artificial Analysis
https://artificialanalysis.ai/

This article is an English adaptation of the original Japanese post published on Zenn: "Metaの意地を見せたか　Muse Glimmer 30Bの実力を見てみる", by Yuichi Tominaga.

RoPE was first introduced in RoFormer, a Transformer-based language model similar to BERT. (https://arxiv.org/pdf/2104.09864) ↩
GQA was first introduced by Google Research's T5-XXL, in the paper that proposed the GQA approach. (https://arxiv.org/abs/2305.13245) ↩
llm-jp-4-8b-thinking's config.json lists its architecture as "LlamaForCausalLM," indicating it descends from the LLaMa-2 line, though it uses its own custom tokenizer. Its vocabulary is substantially larger, and its performance far exceeds the original LLaMa-2. ↩
In Qwen's case, the Gated Delta Network captures global information all at once, while Gated Attention acts as a global noise filter that smooths the information afterward — a structurally different approach from Muse Glimmer's. Looking across recent architectures, despite some structural variation (including how NVIDIA's Mamba series and LFM acquire local information), the "local + global" pattern used by Muse Glimmer and Gemma-4 appears to be the more dominant trend. ↩
For more detail on this setup, see our earlier article on using GPUSOROBAN. (https://www.bluecore.net/archives/242) ↩
This image uses an earlier version of the model diagram shown previously in this article. ↩

Trying "DFlash," a Diffusion-Model Approach to Parallel Draft-Token Generation, on Gemma

oooocean66 — Mon, 14 Sep 2026 07:02:56 +0000

In the concept edition and the implementation/benchmark edition, we covered a speed-up technique for LLM generation called MTP (Multi-Token Prediction). To recap briefly: a lightweight "draft model" predicts a handful of tokens ahead of time, and the main model checks them all at once. When the guesses are right, you leap ahead several tokens in a single step, which is what makes the whole thing feel faster.

There's more than one way to build that draft model, and the one we're looking at this time, "DFlash," takes an unusual approach: it uses a diffusion model — the kind of technique you'd normally associate with image generation — to predict multiple tokens all at once instead of one at a time. It claims to support a wide range of models and to significantly outperform EAGLE-3, an existing approach. Those are the claims worth testing directly.

The catch is that benchmarks like this are usually measured in an environment the vendor sets up. It's harder to find a case where someone ran it on their own GPU, against an opponent that already has a dedicated, well-optimized MTP model of its own — Gemma-4's native Assistant model.

So this time, using the same setup as the implementation/benchmark edition (an RTX 3060 with 12GB VRAM, llama.cpp, the same JavaScript coding task), we directly compared DFlash against Gemma-4-12B-it's Assistant model. The short version: DFlash did not outperform the Assistant model. The reasons are technically clear, though, and they draw a fairly clear picture of where DFlash is strong and where it isn't — that's what we'll dig into below.

Overview

Two new token-prediction techniques appeared in quick succession:

DeepSeek: a token-prediction technique called DSpark
Z-Lab at UC San Diego: a token-prediction technique called DFlash

DSpark¹ was released to speed up inference specifically for DeepSeek's own DeepSeek-V4, and it only supports DeepSeek-V4 / DeepSeek-V4-Flash.

DFlash, on the other hand, supports a much wider range of models, each with its own dedicated DFlash model. Like Google's Assistant model, it's designed to be bolted on as an add-on to achieve token prediction.

Gemma-4 series (12B, 26B-A4B, 31B)
MiniMax-M2.7
MiniMax-M2.5
Qwen 3.6 series (35B-A3B, 27B)
Qwen3.5 series (4B, 9B, 35B-A3B, 27B, 122B-A10B, 397B-A17B)
Qwen3 series (4B, 8B, Coder-30B-A3B)
Kimi-K2.6
GLM-5.1-FP8
gpt-oss (20B, 120B)
LLaMa3.1-8B-Instruct

Z-Lab is a research group led by Zhijian Liu², an assistant professor at UC San Diego who is also a research scientist at NVIDIA. The lab works across the algorithm, systems, and application layers to make AI smaller, faster, and more efficient.

This time, we wanted to understand how much of a speed-up DFlash actually delivers, verified through hands-on testing.

How DFlash Predicts Tokens

DFlash is introduced in the paper "DFlash: Block Diffusion for Flash Speculative Decoding"³.

Speculative token prediction itself isn't new — the earliest paper on the idea was published by Google DeepMind in 2023, "Accelerating Large Language Model Decoding with Speculative Sampling"⁴.

Improvements continued quietly from there, culminating in 2025 in an MTP draft model called EAGLE-3. Even so, it apparently never escaped the autoregressive paradigm, and in the end didn't deliver a dramatic speed improvement.

Meanwhile, diffusion models — the noise-removal mechanism commonly used in image generation — have been making their way into the LLM space. It started with Meta's LLaDa, and in Japan, KDDI's ELYZA Lab team released a model called "ELYZA-Diffusion-Instruct-1.0-Dream-7B"⁵.

Z-Lab's DFlash brings that diffusion-model property into the MTP draft model. The basic mechanism is shown below.

[Input Token (N)] --> [Main Model] -- h_on ----------------------------------------> [Predicted: N+1]
                                       |                                             ^
                                       v                                             |
                                  [KVCache]                                          |
                                       |                                             |
+--------------------------------------+--------------------------------+            |
| DFlash Drafter                       v (KV data injection)            |            |
|                                 [KVCache]                             |            |
|                                      |                                |            |
|                              [Diffusion Model]                        |            |
|                                      |                                |            |
|                               [Token Decoder]                         |            |
|                                      |                                |            |
|                            +---------v--------+                       |            |
|                            | Predicted N+2    |                       |            |
|                            |       +          |                       |            |
|                            | Predicted N+3    |                       |            |
|                            |       +          |                       |            |
|                            | Predicted N+4    |                       |            |
|                            +---------+--------+                       |            |
+--------------------------------------+--------------------------------+            |
                                       |                                             |
+--------------------------------------+---------------------------------------------+-------+
| Processing inside Main Model         v                                             |       |
|                             * Uses causal attention to mask;                       |       |
|                               computes probs in parallel                           |       |
|                                      +------------------------------+              |       |
|                                      v                              v              |       |
|                         (Match found)                  (No match)                  |       |
|                         Include matching portion       Nothing included in output  |       |
|                         -> Predicted: N+2                                          |       |
|                         -> Predicted: N+3                                          |       |
+------------------------------------------------------------------------------------+-------+

Figure 1: DFlash mechanism

Looking at the mechanism, it resembles the Assistant model implemented in Gemma, but the big difference is in how the draft model itself predicts tokens. The latter half — the token-evaluation stage — closely mirrors Gemma's approach.

With DFlash, the KV cache is built independently. As before, when the main model predicts token N+1, it uses that state to update its own KV cache. Data extracted from that update is then injected into DFlash's own KV cache, which is where DFlash's processing begins.

Gemma's Assistant model runs this prediction step using a very small neural network, sequentially and at high speed, producing as many predicted tokens as needed before handing them off to the evaluation logic.

DFlash, by contrast, doesn't use an autoregressive model inside its small neural network — it uses a diffusion model. Here, much like generating an image, it produces all of the needed predicted tokens, in the correct order, in one shot. The longer the maximum token length, the longer this takes, but it's still dramatically faster than doing it sequentially. What follows is the same as before: causal-attention-based masking runs in parallel, and the result determines which tokens are allowed to be output together.

So the biggest contributor to any speed advantage comes down to the parts marked with blue and red boxes in the middle of the figure below.

[For Gemma Assistant]
[Input Token (N)] --> [Main Model] --> [Predicted: N+1]
                            |
                            +-> [Predicted: N+2] -(seq)-> [Predicted: N+3] -(seq)-> [Predicted: N+4]
                                                                    |
                                                                    v
                                                     [Verify] --> Up to n OK --> [Confirmed: N+2, N+3...]

[For DFlash]
[Input Token (N)] --> [Main Model] --> [Predicted: N+1]
                            |
                            |   +-- [Predicted: N+2]
                            +---+-- [Predicted: N+3]   <-- (Outputs all at once, order included)
                            |   +-- [Predicted: N+4]
                            |               |
                            |               v
                            +--------->  [Verify] --> Up to n OK --> [Confirmed: N+2, N+3...]

Figure 2: Difference between Gemma's Assistant model and DFlash

Running It in llama.cpp

First, get the model files.

hf download google/gemma-4-12B-it
hf download z-lab/gemma4-12B-it-DFlash

This time we're using llama.cpp build 9850. It's a good idea to grab the latest build. Use the Python conversion tool included with it to convert each model to GGUF format. Start with the main model.

$ python convert_hf_to_gguf.py \
~/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/cxxxxxx...a/ \
--outtype bf16 --outfile gemma-4-12B-it-b16.gguf

Next, convert the DFlash model to GGUF.

The important thing here is --target-model-dir. When converting a DFlash model, it needs to be given a reference to the main model — that's the parameter it uses to build the converted DFlash output.

$ python convert_hf_to_gguf.py \
~/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/cxxxxxx...a/ \
--outtype bf16 --outfile models/gemma-4-12B-it-DFlash-b16.gguf \
--target-model-dir \
~/.cache/huggingface/hub/gemma-4-12B-it-qat-q4_0-unquantized/snapshots/c202...a/

If this fails with the message below, download tokenizer.model directly from the google/gemma-3-12b-it repository on Hugging Face and place it in the cache directory.

INFO:hf-to-gguf:DFlash: Using tokenizer from target model: /home/aiuser/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/5854...7ced3

Traceback (most recent call last):
  File "/data/user_data/aiuser/llama.cpp/conversion/qwen.py", line 57, in set_vocab
    self._set_vocab_sentencepiece()
  File "/data/user_data/aiuser/llama.cpp/conversion/base.py", line 1589, in _set_vocab_sentencepiece
    tokens, scores, toktypes = self._create_vocab_sentencepiece()
  File "/data/user_data/aiuser/llama.cpp/conversion/base.py", line 1606, in _create_vocab_sentencepiece
    raise FileNotFoundError(f"File not found: {tokenizer_path}")
FileNotFoundError: File not found: /home/aiuser/.cache/huggingface/hub/models--google--gemma-4-12B-it/snapshots/585...ced3/tokenizer.model

Quantizing these produces 4-bit models.

$ /opt/llama/bin/llama-quantize \
models/gemma-4-12B-it-b16.gguf models/gemma-4-12B-it-q4_0.gguf q4_0

$ /opt/llama/bin/llama-quantize \
models/gemma-4-12B-it-DFlash-b16.gguf models/gemma-4-12B-it-DFlash-q4_0.gguf q4_0

At runtime, add the following arguments. Since the DFlash model acts as an add-on draft/MTP model, use --model-draft to point to it and enable MTP.

/opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-q4_0.gguf \
--model-draft /opt/llama/models/gemma-4-12B-it-DFlash-q4_0.gguf \
-t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8_0 -ctk q8_0 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-dflash --spec-draft-n-max 4 --spec-draft-device CUDA0 \
--chat-template-kwargs '{"enable_thinking":true}'

A quick rundown of the arguments:

--model-draft <gguf file>: specifies the draft model
-sm layer: how the model is dispatched across multiple CUDA devices
- at the time of writing, only layer-wise distribution is supported
--spec-type draft-dflash: specifies the speculative-decoding type
- declares the type as DFlash so it can be handled correctly
--spec-draft-n-max 4: the maximum number of tokens to predict ahead
- setting this too high increases the penalty on a miss
--spec-draft-device CUDA0: the device used for drafting
- required when multiple CUDA devices are present

Verification: Using DFlash on Gemma-4-12B-it

Test Environment

We ran the following simplified verification setup.

Machine: modified HP Z440 Workstation
- CPU: Intel® Xeon® E5-2690 v4 x1
- RAM: 48GB DDR4 RDIMM
- SSD: Intel® DC S3700 Datacenter SSD (400GB SATA SSD)
- GPU: NVIDIA GeForce RTX 3060 (Ampere, 12GB GDDR6 VRAM)
CUDA: 13.1
Engine: llama.cpp Build 9574
- Model: gemma-4-12B-it (https://huggingface.co/google/gemma-4-12B-it)
- As the base model, we use Gemma-4-12b-it-QAT, Google DeepMind's quantization-aware-trained model.
- Drafter model: gemma-4-12B-it-DFlash (https://huggingface.co/z-lab/gemma4-12B-it-DFlash)
- We use Z-Lab's DFlash model for Gemma-12B.
- The maximum number of predicted tokens is set to 4 (matching the previous verification's settings, for comparability).
- KV Cache: 8-bit quantized (to fit efficiently in VRAM)
- Context size: 163,840 tokens (limited to 160k to fit efficiently in VRAM)
- Prompt cache: off
- Reasoning: on

Test Cases

Purpose: measure and compare speed over time when using the Assistant model versus DFlash
- Only speed is measured; the content of the generated output is not considered.
- We record wall-clock time, token count, and token throughput at write-out.
- These figures are taken directly from llama.cpp's own log output.
Method: using the llama.cpp web frontend, we ask the following over four turns:

Write JavaScript code for Breakout (as an HTML file).
Make it look cooler.
Slow down the ball's movement a bit.
Check for bugs and optimize it.

Resource Usage

Memory usage came out as follows.

Data category	Assistant CUDA0	Assistant CPU	DFlash CUDA0	DFlash CPU
Weight data	6,390.19	540.00	6,637.69	787.50
KV cache data	1,360.00	—	1,360.00	—
KV cache for Sliding Window	765.00	—	765.00	—
Gated DeltaNet compute buffer	533.80	180.80	533.80	180.80
MTP model weight data	226.90	144.00	390.42	—
MTP token-to-piece cache size	—	—	1.94	—
MTP model KV cache size	—	—	640.00	—
MTP model Sliding Window KV cache size	—	—	136.00	—
MTP model Gated DeltaNet compute buffer	532.78	180.79	519.50	176.02
Total	9,808.67	1,045.59	10,984.35	1,226.38

Table 1: Memory usage breakdown by MTP method (units: MiB). This run used text-only mode, so multimodal-model requirements are excluded.

Of that, the memory used specifically by the MTP model itself was as follows — roughly double for DFlash.

	Assistant CUDA0	Assistant CPU	DFlash CUDA0	DFlash CPU
MTP model memory usage	759.68	324.79	1,687.86	176.02

Table 2: Total memory used by the MTP model, by MTP method (units: MiB)

The main driver here is how the two approaches use the KV cache. The Assistant model shares its KV cache with the main Gemma model. DFlash, by contrast, keeps its own independent KV cache — meaning it has to hold the same cache structure as the main Gemma model a second time. That's the biggest factor behind the difference.

Token Speed Comparison (Assistant vs DFlash)

As in the previous verification, we compared token ingestion speed turn by turn. There was no major throughput difference between the Assistant model and DFlash for reading tokens in.

Turn	Assistant (tok/s)	DFlash (tok/s)
1	138.34	53.72
2	710.71	652.47
3	733.53	739.52
4	735.43	708.22

Figure 3: Token read-in speed by turn

For output speed per turn, the Assistant model was clearly ahead — DFlash consistently trailed by about 10–20 tokens per second.

Turn	Assistant (tok/s)	DFlash (tok/s)
1	62.80	39.37
2	62.04	48.42
3	66.68	52.40
4	62.77	48.73

Figure 4: Token output speed by turn

Here is the raw measurement data behind the numbers above. Note that time is in milliseconds.

Reading

Turn	Assistant time (ms)	Assistant tokens	Assistant tps	DFlash time (ms)	DFlash tokens	DFlash tps
1	181	25	138.34	577	31	53.72
2	3,004	2,135	710.71	3,374	2,202	652.47
3	6,361	4,666	733.53	6,391	4,727	739.52
4	9,559	7,030	735.43	10,173	7,205	708.22

Generation

Turn	Assistant time (ms)	Assistant tokens	Assistant tps	DFlash time (ms)	DFlash tokens	DFlash tps
1	33,186	2,084	62.80	54,413	2,142	39.37
2	40,281	2,499	62.04	51,096	2,474	48.42
3	35,046	2,337	66.68	46,701	2,447	52.40
4	45,532	2,858	62.77	55,774	2,718	48.73

Table 3: Raw measurements from the logs

Overall, inference with DFlash trailed the Assistant model by under 20 seconds. The Assistant model was already a fairly well-optimized setup going in, which may be part of why the diffusion model's theoretical advantage didn't stand out here.

As before, output speed increased somewhat over the course of each turn for both the Assistant model and DFlash, but the Assistant model was faster throughout. (We're omitting the detailed per-token scatter plots for each model here, since the turn-by-turn averages above and the raw log data in Table 3 already capture the trend.) In a few cases we also observed DFlash's output speed peaking around 1,700 tokens and then dropping off slightly after that.

Checking the Token Acceptance Rate

As in the previous verification, let's also look at the token acceptance rate.

Turn	Acceptance	Accepted	Generated
1	41.03%	1,331	3,244
2	53.36%	1,684	3,156
3	62.02%	1,744	2,812
4	57.26%	1,892	3,304

Table 4: DFlash token acceptance rate

Comparing this against the Assistant model:

	Assistant Acceptance	Assistant Accepted	Assistant Generated	DFlash Acceptance	DFlash Accepted	DFlash Generated
Max	88.27%	2,196	2,648	62.02%	1,892	3,304
Min	75.34%	1,564	2,064	41.03%	1,331	2,812
Avg	81.06%	1,868	2,305	53.42%	1,663	3,129

Table 5: Acceptance rate comparison — Assistant (left) vs. DFlash (right)

This suggests that DFlash's underwhelming result comes down to its lower acceptance rate. If model tuning progresses further and a faster-processing diffusion model becomes possible, DFlash might eventually surpass the Assistant model.

Trying to Improve Things

Through this verification, we confirmed that DFlash falls a bit short of the Assistant model. But maybe tweaking the parameters would speed things up? With that in mind, we tried the following.

Raising the Number of Predicted Tokens

The predicted-token count is currently set to 4. That's the same value DFlash used in the previous Assistant-model verification, kept for comparability. Raising this significantly might help — so we bumped --spec-draft-n-max from 4 to 15.

/opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-q4_0.gguf \
--model-draft /opt/llama/models/gemma-4-12B-it-DFlash-q4_0.gguf \
-t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8_0 -ctk q8_0 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-dflash --spec-draft-n-max 15 --spec-draft-device CUDA0 \
--chat-template-kwargs '{"enable_thinking":true}'

Here's what came out of running it:

prompt eval time = 122.83 ms / 31 tokens ( 3.96 ms per token, 252.39 tokens per second)
eval time = 48833.63 ms / 1893 tokens ( 25.80 ms per token, 38.76 tokens per second)
total time = 48956.46 ms / 1924 tokens
graphs reused = 1444
draft acceptance = 0.10901 ( 1174 accepted / 10770 generated), mean len = 2.64

The complete opposite of what we hoped for: acceptance dropped, and speed dropped along with it.

It generated a lot more candidate tokens, but almost all of them were rejected — simply lengthening the prediction window clearly wasn't the answer.

Here's the breakdown by turn:

Turn	Acceptance	Accepted	Generated
1	10.90%	1,174	10,770
2	13.59%	2,059	15,150
3	19.54%	2,090	10,695
4	19.30%	2,310	11,970

	Acceptance	Accepted Tokens	Generated Tokens
Max	19.54%	2,310	15,150
Min	10.90%	1,174	10,695
Avg	15.83%	1,908	12,146

The mean_len=2.64 value in the log above appears to represent roughly "how many tokens tend to get accepted." Looking at the per-turn breakdown above, the average was 15.83%⁶, suggesting that --spec-draft-n-max=4 was in fact the right setting for this model.

Rebuilding on Gemma-4-12B-it-QAT

What if we raised the model's own output precision instead? We tried rebuilding DFlash on that basis.

For this run, since we'd need to manually apply the same quantization level, we couldn't use an Unsloth Dynamic 2.0–quantized model the way we did for the Assistant model. Instead, we rebuilt using Gemma-4-12B-it-QAT (including the DFlash model), hoping it might improve performance.

The results, unfortunately, were not what we expected. Output was so slow that we stopped measuring after the second turn.

prompt eval time = 475.69 ms / 31 tokens ( 15.34 ms per token, 65.17 tokens per second)
eval time = 102682.33 ms / 1963 tokens (52.31 ms per token, 19.12 tokens per second)
total time = 103158.02 ms / 1994 tokens
graphs reused = 1953
draft acceptance = 0.00025 ( 2 accepted / 7844 generated), mean len = 1.00

This configuration is not viable: throughput dropped further, and the acceptance rate fell to 0.00025, or 0.25% — the lowest we observed in this test. With mean_len=1.00, essentially every generated token was rejected, a clear indication that DFlash should not be used in this configuration.

Why Couldn't It Beat Gemma-4-Assistant?

It looks like the DFlash model Z-Lab provided is really only usable with the vanilla Gemma-4-12B-it model. That's unfortunate — if that's the case, the Assistant model, which comes with a properly matched MTP model for each release, seems like the more practical choice.

The likely cause here is that Gemma-4-Assistant's draft model is simply fast enough that it processed tokens more quickly than DFlash's draft model.

As covered earlier, the speed advantage a diffusion-based drafter like DFlash offers over a conventional MTP draft model comes from being able to output candidate tokens "all at once." Against that, Gemma-Assistant has the following working strongly in its favor, which appears to have canceled out DFlash's advantage:

A shared KV-cache mechanism
A very small attention mechanism

First, Gemma-Assistant comes with KV-cache sharing built in from the start, while DFlash keeps an independent KV cache that has to be injected fresh every time. That's an advantage for Gemma-Assistant.

On top of that, Gemma-Assistant's hidden dimension is only 1,024, with a 4-layer structure. DFlash's hidden dimension, as noted at the end of this article, matches the main model at 3,840, with 5 layers — meaning its compute cost is far heavier than Gemma-Assistant's.

However good DFlash's diffusion model is at generating tokens in a single batch, more compute per step still means more time spent per step.

In this case, we should conclude that Gemma-Assistant simply already had a more optimized setup, and DFlash wasn't able to get ahead of it.

It's also worth remembering that DFlash's original benchmark comparison was against EAGLE-3, not Gemma-Assistant — so it's possible Gemma-Assistant is simply a stronger comparison baseline than the one DFlash was originally built to outperform.

That said, while the established option produced better results this time, it's possible a pairing with, say, Qwen3.5 would have produced a more favorable result.

Conclusion

This time we introduced DFlash, a technique from the UC San Diego (UCSD) research team Z-Lab that takes the "MTP" technology covered previously and pushes it in a new direction. Applying a diffusion model to a draft model was a genuinely ambitious idea, but in this test it wasn't able to beat Gemma's native draft model, Assistant.

Tracing the cause, the token acceptance rate was lower than the native draft model's, and that penalty appears to have been the dominant factor. Without a way to understand how to raise that acceptance rate, it's hard to say at this point whether further speed gains are realistic.

We also think the matchup itself didn't help. Gemma-4's Assistant model is built with practicality in mind and makes full use of Gemma-4's shared KV-cache mechanism, whereas DFlash, as before, has to write to an independent KV cache — and that overhead seems to have tipped the balance toward lower throughput.

On the broader question of putting diffusion models to work in LLMs, LLaDa and Dream are the well-known names, but more recently Google itself has released a model called Diffusion Gemma. We're currently in the middle of testing it ourselves, and the throughput numbers so far are notably high. We plan to cover those results in a dedicated article.

As Coding Agent usage keeps climbing, LLM throughput has become a common pain point, and rising VRAM costs have only sharpened the demand for models that can deliver solid throughput even on unified memory.

This kind of research is very much a "fail your way forward" field, and new techniques are emerging as we speak. We'll keep watching this space and try to keep up.

References

DFlash: Block Diffusion for Flash Speculative Decoding
Jian Chen, Yesheng Liang, Zhijian Liu
https://arxiv.org/pdf/2602.06036

Accelerating Large Language Model Decoding with Speculative Sampling
DeepMind: Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper
https://arxiv.org/pdf/2302.01318

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI
https://arxiv.org/pdf/2606.19348

[Speculative decoding] feat: add DFlash support
https://github.com/ggml-org/llama.cpp/pull/22105

Supplementary Information

DFlash Model Structure and Connection to the Main Model

(We've omitted the architecture diagram here to keep the English edition concise — the description below covers the key points.)

The DFlash model, which adds MTP capability to Gemma-4-12B-it, is architecturally distinct from Gemma-4-Assistant: rather than sharing a KV cache the way Assistant does, it's a fully separate model. In terms of its activation function and related choices, it's actually closer to a Qwen-style model. Its LA/GA notation follows Gemma-4's own convention — LA meaning Sliding Window Attention and GA meaning Full Attention.

Maximum context size: 256k tokens
5 blocks total (one group of 4+1)
Hidden size: 3,840
FFN type: Dense
- Activation function: SiLU
- Activation dimension: 7,680

During prediction, state vectors are pulled from layers 1, 10, 19, 27, 36, and 45, concatenated, and normalized. That combined information is then injected into the draft side's KV cache.

The input to the draft model is a query built by combining the embedding vectors of the N confirmed tokens with as many MASK tokens as the number of predictions needed.

Unlike the Assistant model, this approach uses non-causal attention, and internally relies on Masked Diffusion — that's the key structural difference.

This mechanism lets DFlash derive all of its predicted tokens at once; from there, the rest of the pipeline follows the same logic as the Assistant model, outputting whichever tokens get accepted.

What Is EAGLE3?

EAGLE3 is version 3 of the EAGLE (Extrapolation Algorithm for Greater Language Model Efficiency) series, documented in the following papers:

EAGLE ( https://arxiv.org/pdf/2401.15077.pdf )
EAGLE2 ( https://arxiv.org/pdf/2406.16858 )
EAGLE3 ( https://arxiv.org/pdf/2503.01840 )

Development has been led by Yuhui Li of Peking University, first author on these papers, together with a joint research team spanning Peking University, Microsoft Research, the University of Waterloo, and the Vector Institute. In the open-source community, they operate under the name SafeAILab.

Successor models are also in development: SafeAILab released EAGLE3.1 in May 2026, and a team at AWS AI Labs has developed P-EAGLE (Parallel-Drafting EAGLE), released in February 2026.

Both treat the sequential bottleneck of causal attention as the core problem, and both are focused on how to parallelize that part of the pipeline further.

This article is an English adaptation of the original Japanese post published on Zenn: "ドラフトトークンを並列生成する拡散モデル方式「DFlash」をGemmaで試した", by Yuichi Tominaga.

https://arxiv.org/pdf/2606.19348 ↩
https://zhijianliu.com/, https://z-lab.ai/ ↩
https://arxiv.org/pdf/2602.06036 ↩
https://arxiv.org/pdf/2302.01318 ↩
https://huggingface.co/elyza/ELYZA-Diffusion-Instruct-1.0-Dream-7B ↩
Average acceptance rate across the four turns shown in the table below. ↩

MTP in Practice: Benchmarking Gemma's Speculative Decoding on a Real GPU

oooocean66 — Fri, 11 Sep 2026 10:06:06 +0000

In the concept edition, we saw that MTP (Multi-Token Prediction) lets a model predict several tokens ahead to speed up generation, and that Qwen and Gemma implement this in completely different ways.

The theory makes sense, but how much faster does this actually make things in practice? That's what we set out to measure directly.

In this installment — the implementation/benchmark edition — we'll actually run Gemma's MTP on llama.cpp and measure, with real numbers on an ordinary consumer PC, just how much of an effect it has.

Running It in llama.cpp

For Qwen Models

Run with the following arguments added. Specifying spec-type activates the model's internal MTP drafter. A dedicated model is provided for MTP.

/opt/llama/bin/llama-server --model /opt/llama/models/Qwen3.5-9B-MTP-UD-Q3_K_XL.gguf \
-t 4 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -ctv q8_0 -ctk q8_0 --fit off --no-warmup --no-cache-prompt -c 32768 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 4 \
--reasoning on

Argument summary:

--spec-type draft-mtp: specifies the speculative decoding type (draft-mtp for Qwen).
--spec-draft-n-max 4: how many tokens ahead to speculate at most.
- Setting this too high increases the penalty when a prediction fails.
Also note: concurrent processing isn't supported, and multimodal input isn't supported either.

For Gemma Models

Run with the following arguments. For Gemma, the drafter-MTP model is bolted on externally, so you enable MTP by pointing the --model-draft argument at that file.

/opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--model-draft /opt/llama/models/mtp-gemma-4-12B-it.gguf \
-t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8_0 -ctk q8_0 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-device CUDA0 \
--chat-template-kwargs '{"enable_thinking":true}'

Argument summary:

--model-draft <gguf file>: specifies the draft model.
-sm layer: how the model is dispatched across multiple CUDA devices.
- At the time of writing, only layer-level distribution is supported.
--spec-type draft-mtp: specifies the speculative decoding type.
--spec-draft-n-max 4: how many tokens ahead to speculate at most.
- Setting this too high increases the penalty when a prediction fails.
--spec-draft-device CUDA0: the device that runs drafting.
- Required when multiple CUDA devices are present.

Benchmark: Using MTP on Gemma-4-12B-it

In this section, to investigate how effectively MTP actually works and what kind of benefit it brings to users, we compared performance with and without MTP.

To also demonstrate that this kind of model verification is achievable even on ordinary consumer hardware, the benchmark was run on the author's own personal machine.

Test Environment

The experiment used the following simple test setup.

Machine: modified HP Z440 Workstation
- CPU: Intel® Xeon® E5-2690 v4 x1
- RAM: 48GB DDR4 RDIMM
- SSD: Intel® DC S3700 Datacenter SSD (400GB SATA SSD)
- GPU: NVIDIA GeForce RTX 3060 (Ampere, 12GB GDDR6 VRAM)
CUDA: 13.1
Engine: llama.cpp Build 9574
- Model: gemma-4-12B-it-qat-UD-Q4_K_XL.gguf https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
- As the base model, we use Gemma-4-12b-it-QAT, a Google DeepMind model trained with quantization-aware training in mind.
- On top of that, we use the UD-Q4_K_XL model¹, dynamically quantized by Unsloth², which provides open-source fine-tuning tools.
- Drafter model: mtp-gemma-4-12B-it.gguf Downloaded from the same repository as the base model.
- We use the Assistant model prepared by Unsloth for Gemma-12B, simply quantized to 4-bit, as the draft model.
- The maximum number of speculated tokens is set to 4.
- KV Cache: 8-bit quantized
- The KV cache is quantized to 8 bits to fit efficiently into VRAM.
- Context size: 163,840 tokens
- Context size is capped at 160k to stay within VRAM capacity.
- Prompt cache: off
- Reasoning: on

Test Cases

Goal: measure and compare speed with and without MTP.
- Only speed is measured. The actual content of the generated output is not evaluated at all.
- We measure elapsed time, token count, and token speed during generation.
- These values are taken directly from llama.cpp's own log output.
Method: using the llama.cpp web frontend, we asked the following 4 turns of questions.

"Write JavaScript code for Breakout (as an HTML page)."
"Make it look cooler."
"Slow the ball's movement down a bit."
"Check for bugs and optimize it."

Resource Usage

Memory usage was as follows. The memory consumed specifically for MTP came to 759.68 MiB of VRAM on the GPU side and 324.79 MiB of RAM on the CPU side.

Since we used a 4-bit quantized model for the draft model here, choosing a 16-bit model instead would consume roughly 4x this amount for the weight data. Anyone planning to use a 16-bit model should keep this in mind.

Component	CUDA0	CPU
Weight data	6,390.19	540.00
KV cache data	1,360.00
Sliding window KV cache	765.00
Gated DeltaNet compute buffer	533.80	180.80
MTP (Assistant) model weight data	226.90	144.00
MTP (Assistant) model Gated DeltaNet compute buffer	532.78	180.79
Vision Encoder	167.00
Audio Encoder	167.00
Multimodal Gated DeltaNet compute buffer	532.78	180.79
Total	10,675.45	1,226.38

Units: MiB. Table 1: Memory usage in this test environment.

Token Speed Comparison Results (MTP On vs. Off)

Comparing token ingestion speed turn by turn, we saw no major variation. With MTP enabled, ingestion speed dropped slightly compared to without MTP, coming in at roughly 70–73% of the baseline performance (see the "Read" rows in the table below).

Token generation speed, on the other hand, was roughly twice as fast overall with MTP enabled (see the "Generate" rows).

Below are the actual measurements. Note that time is measured in ms.

Turn	Type	No MTP: time	No MTP: tokens	No MTP: tps	With MTP: time	With MTP: tokens	With MTP: tps
1	Read	301	25	83.04	181	25	138.34
2	Read	1,836	1,834	999.10	3,004	2,135	710.71
3	Read	4,546	4,669	1,027.06	6,361	4,666	733.53
4	Read	7,017	7,102	1,012.11	9,559	7,030	735.43
1	Generate	45,326	1,789	39.47	33,186	2,084	62.80
2	Generate	80,190	2,802	34.94	40,281	2,499	62.04
3	Generate	70,164	2,405	34.28	35,046	2,337	66.68
4	Generate	84,325	2,851	33.81	45,532	2,858	62.77

Table 2: Actual data from the logs.

Without MTP, the whole run took about 293,705 ms in total — just under 5 minutes — while with MTP it completed in about 173,150 ms, just under 3 minutes. That's roughly a 40% reduction in time for this case.

This suggests MTP is a genuinely effective way to boost efficiency in tasks like today's agentic workflows, which repeat many rounds of inference.

To look more closely at the trend, we also checked how speed changed as token output progressed within each turn (again using the logs).

Below is the token output profile without MTP. Speed dips slightly but stays roughly flat overall, hovering in the mid-30s to low-40s tps across all four turns from start to finish.

Generally, with token-by-token generation, output speed tends to gradually decline as context size grows. How much it degrades depends on the model, but this can become a non-trivial problem as more turns accumulate.

By contrast, here is the token output profile with MTP enabled. Speed starts around 35–40 tps early on and climbs to 60–65 tps later, showing that tokens come out faster and faster as the conversation progresses.

A quick note before we go further: I'm a native Japanese speaker, and this model is my daily driver, so the test prompts and responses are in Japanese rather than English. Tokenizer behavior can differ quite a bit across languages, so this also happens to double as a look at how MTP performs outside the English-first benchmarks you usually see.

Next, we examined the actual content of the responses and found that each one opens with a Japanese explanation followed by code. (The original screenshot of that exchange is in Japanese, so it isn't reproduced here — the token breakdown below covers what matters for this analysis.)

Based on that, we checked the token length of the Japanese explanation portion at the start of each turn, with the following results.

Turn	Token count of the Japanese portion
1	107
2	147
3	124
4	256

Table 3: Token length of the Japanese explanation at the start of each turn.

We can't fully prove causation here, but turns where speed rose sharply (Turn 1, Turn 3) tended to have less Japanese output, while turns where speed rose more gradually (Turn 2, Turn 4) tended to have somewhat more Japanese output. This suggests the code-generation portion is where MTP's effect shows up most strongly.

A likely reason is that, regardless of programming language, code follows fairly strict grammar rules, making it relatively easy to predict what string comes next, and since the characters are essentially alphabetic, predicting the character type is also easier.

Even outside the code-generation portions, we confirmed output speeds of 40–50 tps right from the start — faster than the conventional approach — suggesting MTP was working effectively there too.

The author believes this largely comes down to characteristics of the Gemma tokenizer.

The Gemma tokenizer has a vocabulary of roughly 256k entries. Whereas many other tokenizers fall back to character-level tokenization for Japanese, Gemma's tokenizer is able to register most Japanese text as whole "words."

Because of this, the model can predict the next token at the "word" level for Japanese much like it does for English, which we believe is why Japanese output was noticeably faster with MTP than without.

Checking Token Acceptance Rates

llama.cpp is designed to log an "acceptance" value showing what fraction of tokens were accepted during the verification phase when using an MTP model. Here's an example:

prompt eval time = 9559.06 ms / 7030 tokens ( 1.36 ms per token, 735.43 tokens per second)
eval time = 45531.93 ms / 2858 tokens ( 15.93 ms per token, 62.77 tokens per second)
total time = 55090.99 ms / 9888 tokens
graphs reused = 3983
draft acceptance = 0.82931 ( 2196 accepted / 2648 generated)

Using this value, we checked acceptance for each turn, with the following results.

Table 4: Token acceptance rate during the test cases

Turn	Acceptance	Accepted	Generated
1	75.34%	1,564	2,076
2	77.71%	1,890	2,432
3	88.27%	1,821	2,064
4	82.93%	2,196	2,648

	Acceptance	Accepted Tokens	Generated Tokens
Max	88.27%	2,196	2,648
Min	75.34%	1,564	2,064
Avg	81.06%	1,868	2,305

As mentioned above, likely because the output contained a lot of code, these tokens showed a high acceptance rate. So what does the acceptance rate look like for everyday use? Based on several days of logs, we recorded the maximum, minimum, and average, shown below.

Table 5: Token acceptance rate for everyday interactions with the Gemma-4-QAT-Assistant model

	Acceptance	Accepted Tokens	Generated Tokens
Max	75.55%	2,119	4,800
Min	33.56%	227	412
Avg	45.11%	776	1,798

Acceptance roughly halved, and speed dropped somewhat as a result. The corresponding speeds are shown below.

Table 6: Token speeds under the conditions in Table 5

	Read	Generate
Max	1,040.82	84.46
Min	117.52	30.20
Avg	774.45	40.44

So while this isn't exactly blazing fast, it's reasonable to say it's still faster than running without MTP at all.

We confirmed that, for the Gemma-4 series, llama.cpp and Gemma-4-Assistant are a very well-matched pair, capable of delivering real benefits even in everyday use.

Summary

In this article, we looked at the new "MTP" technique — what it is, and how it actually changes generation speed in practice.

In a word, MTP is essentially the "speculative execution" of CPUs, brought into the world of LLMs: when the prediction is correct, generation gets dramatically faster; when it's wrong, the extra overhead can make it slower.

We found this feature pairs especially well with code generation. Because programming languages follow strict grammar, once you're partway through a line, it's relatively easy to predict "what string should come next." This lets MTP compensate for the parts where the tokenizer tends to slow things down, and our benchmark reflected that with solid results.

The current strength of LLMs comes precisely from their sequential nature — deciding the next word based on the words that came before. MTP, because it speeds things up without sacrificing that strength, is likely to play a genuinely important role as this technology continues to develop.

It's still early days, and there are some constraints, but we think this is a technique well worth watching as it matures and spreads.

Even as we speak, other groups are developing and releasing their own versions of this idea:

DeepSeek (China): a token-prediction technique called DSpark
Z-Lab at UC San Diego: a token-prediction technique called DFlash

These, too, are spreading quickly now that inference engines are starting to support them.

We hope to take a closer look at these technologies in a future article.

Supplementary Information (llama.cpp Support Status, Gemma-4 Architecture Details)

llama.cpp Support Status

2026/5/16: Support for Qwen's MTP feature merged (#22673)
2026/5/22: A further updated version merged (logit computation skip optimization: #23433)
2026/6/8: PR adding Gemma-4 MTP support merged (#23398).
- Enables conversion to GGUF format including the MTP drafter.
2026/6/8: Support for the Assistant drafter models for Gemma-4 E2B and E4B merged. (#24282)
- However, a bug caused Gemma-4-E4B specifically to malfunction.
- Apparently fixed in PR #25148 below.
2026/06/30: Fix for the Gemma-4 E4B bug (#25148) merged.

Gemma-4 Model Architecture

Below is the architecture of the Gemma-4-12B-it model.

Input strings ─┬─ Vision/Audio encoders (minimal, mostly folded into the model) ─┐
                └─ tok → Emb ──────────────────────────────────────────────────┴─(+)─┐
                                                                                       │
     ┌── RoPE / p-RoPE ───────────────────────────────────────────────────────────────┤
     │                                                                                 ▼
     │      [ LA → LA → LA → LA → LA → GA ]  x6 groups   →  LNR → SoftMax → Output Probabilities
     └──────────────────────────────────────────────────↑
            (48 blocks total = 6 layers x 8 groups)

Max context size: 256k tokens (limited to 160k in our test environment)
48 blocks overall (6 layers x 8 groups)
Hidden size: 3,840
FFN type: Dense (omitted from the diagram above for simplicity)
- Activation function: Gelu_Pytorch_tanh
- Activation dimension: 15,360
The attention structure is a hybrid of Local Attention (LA) and Global Attention (GA). This structure debuted with Gemma-4.
- Local Attention (LA): uses sliding window attention.
- Handles analysis of local context via a sliding window.
- Low compute cost, fast, and low memory usage.
- Global Attention (GA): uses group query attention for accurate whole-sequence understanding.
- Captures the entire sequence.
- A high-performance coordinator handling long context and complex reasoning.
Other notable features:
- Unified Encoder: integrates audio/vision encoders into the model.
- Normally-external audio/vision encoders are almost entirely integrated into the model itself.
- Only the CLIP portion remains an externally applied encoder.
- Per-Layer Embedding (PLE): keeps embedding information independent at each layer.
- Beyond the input-time embedding, retaining layer-specific information contributes to richer representational power during inference.
- KV Cache Sharing: a mechanism for sharing and synchronizing the KV cache.
- Normally kept independent, this mechanism shares and synchronizes the KV cache instead.
- Used to achieve more consistent inference.
- When dispatching across multiple GPUs, this shared-state synchronization causes frequent inter-GPU communication, so caution is needed in setups without NVLink.
DualRoPE configuration:
- Local Attention layers use RoPE.
- Embeds token position information using rotation matrices, ensuring reliable positional information for long text.
- Global Attention layers use p-RoPE³.
- A modified form of RoPE that applies rotation only to the top p% of K/V matrix pairs.
- Because it effectively suppresses the impact of noise, it's applied to the coordinating Global Attention layers, helping clarify positional information for long text.

Gemma-4-12B-Assistant Model Architecture Details

The Gemma-4-12b-it-assistant model works together with the main model to predict tokens as follows.

Main model (Gemma-4-12b-it):
  Input string → tok → Emb → [LA...LA(45) → GA(46) → GA(47)] → LNR → SoftMax → N+1 output token
                                     │ shares KV cache (layers 45-47) with the Assistant's LA/GA
                                     ▼
Assistant model (MTP-Drafter):
  Query (H_N^0 + Emb of N) → LNR → LA → LA → LA → GA → Norm ─┬→ LNR → predicted state vectors
                                                              └→ Masked Embedder → predicted Token IDs (n tokens)
                                     │
                                     ▼
Verification (inside the inference engine):
  compares the Assistant's predicted state vectors against the Main model's own state vectors
  → decides how many tokens to accept → token output correction → final output

The "Assistant" model that adds MTP capability to Gemma-4-12B-it keeps Gemma-4's architecture but trims its structure down to a genuinely minimal design.

Max context size: 256k tokens
4 blocks overall (a single group of 3+1)
Hidden size: 1,024
FFN type: Dense
- Activation function: Gelu_Pytorch_tanh
- Activation dimension: 8,192

Based on the input string, it predicts the next (N+1) token. To do this, it combines the generated state vector h_N^0 with the embedded vector of token N, and feeds the result in as a query to begin processing.

Gemma-4's attention layers normally operate as self-attention, but the structure inside the Assistant model uses cross-attention instead.

For KV data, it combines the LocalAttention block at layer 46 of the main model with the LocalAttention blocks at layers 0–2 on the Assistant side, and also integrates the GlobalAttention block at layer 47 of the main model with the GlobalAttention block at layer 3 on the Assistant side, via a shared KV cache. Through this, KV information is bound based on the query.

The output consists of two parts.

Predicted tokens: the n tokens predicted by the Assistant model.

State vector: the vector used during that prediction process.

These outputs are sent combined, just before the main model's attention blocks. They pass through all of the main model's attention blocks, generating, in parallel, the state vectors needed for verification.

Once the state vectors to be compared are ready, their contents (probability distributions) are compared to decide how many tokens should be accepted. This is handled in parallel by the inference engine (for example, the generate function in the transformers library), which outputs the resulting number of tokens to accept.

Based on this result, the candidate tokens — accounting for the accepted token count — are combined with token N+1 and passed on for output. This is how the process achieves faster output compared to conventional step-by-step inference.

This article is an English adaptation of the original Japanese post published on Zenn: "実際どれだけ速くなる？ Gemma-4のMTPを、普段使いのPCでllama.cpp検証してみた【検証編】", by Yuichi Tominaga.

MTP Explained: How Qwen and Gemma Predict Several Tokens Ahead

oooocean66 — Wed, 09 Sep 2026 07:53:18 +0000

MTP (Multi-Token Prediction) is a technique that lets a language model predict several tokens ahead while it generates text, then confirm multiple tokens at once when those predictions turn out correct.

Have you ever used an AI chat and thought, "I wish this answered a little faster"? That feeling gets stronger with "Thinking" models that reason carefully before answering, or with coding agents that iterate through trial and error. For AI systems that burn through thousands, even tens of thousands, of tokens to reach a good answer, generation speed directly shapes the user experience.

So why does AI text generation take so long in the first place? The reason is simple: LLMs can only produce one token at a time. To decide the next word, the model has to re-run its massive computations from scratch, taking into account everything generated so far. That cost adds up, and the longer the response, the longer you wait.

This raises a natural question: instead of recomputing everything one word at a time, why not compute a few words ahead in the same pass? That's exactly what MTP does. Every time the model generates a word, it also computes, in that same forward pass, a prediction for what the next few words are likely to be. If the prediction turns out correct, those predicted tokens are accepted all at once, confirming several words in a single step. If it's wrong, the model simply falls back to generating one token at a time as usual — so a wrong prediction costs little, while a correct one yields a meaningful speedup.

This technique is often called speculative decoding, a term borrowed from an older CPU optimization technique called "speculative execution." CPUs have long predicted which branch of code they're about to take, computed ahead of time based on that prediction, and either kept the result if it was right or discarded it and redid the work if it was wrong. MTP applies the same underlying idea to language model inference.

This article (the concept edition) walks through how this mechanism is actually implemented inside real models (Qwen and Gemma), and how the two approaches differ. In the next installment — the implementation/benchmark edition — we'll actually run this technique and measure exactly how much faster it gets. Later in the series, we'll also cover "DFlash," which pushes this idea even further.

What Is MTP, Exactly?

MTP (Multi-Token Prediction) is a technique where an AI predicts several tokens ahead simultaneously while generating text. By predicting tokens in advance, the model can confirm multiple tokens at once when the prediction turns out correct, speeding up output.

Similar efforts exist elsewhere: instead of generating text sequentially, some "diffusion" models generate text more like image generation, producing output in a more randomized order to speed things up (as of 2026, Google DeepMind and ELYZA, among others, appear to be researching diffusion language models). However, output quality for these remains unstable and hasn't reached a practical level yet.

At least as of 2026, MTP is arguably the most reliable LLM speedup technique available today.

Which Models Support MTP?

The models currently supporting MTP fall mainly into two categories. Each uses a different implementation approach, so support in inference engines needs to be checked individually.

Qwen-3.5 / 3.6: MTP is natively baked into the model itself — the model and its MTP drafter are a single package. It's supported out of the box in engines like vLLM. llama.cpp added experimental support relatively early on, and Unsloth has released an MTP-enabled model for Qwen-3.5-9B in GGUF format.

Gemma: MTP is achieved by pairing the model with a separate model called "Gemma-4-Assistant" — think of it as an add-on bolted onto the main model rather than something built in. llama.cpp support for this arrived somewhat later than for Qwen.

The Basic Mechanics of MTP

Let's look at how behavior differs with MTP disabled versus enabled.

[Conventional Inference (Token-by-Token)]

One forward pass predicts one token; the next step runs another forward pass. Because output comes one token at a time, this is relatively slow.

Step N (full forward pass) → token generated → Step N+1 (full forward pass) → token generated → ...

[Inference with MTP]

A single forward pass computes a certain number of tokens ahead.

Step N (forward pass, run once) ──┬─→ Token N     [confirmed]
                                   ├─→ Token N+1   [speculative]
                                   └─→ Token N+2   [speculative]

Token 1: output as confirmed.
Token 2 onward: probability values are calculated in parallel as speculative predictions.
Because output comes in multi-token units, this is relatively faster than the conventional approach.

Handling Predicted Tokens (Speculative Decoding)

Predicted tokens go through a follow-up step where they're checked against the correct answer; once they clear the acceptance criteria, they're confirmed all at once.

Draft A lightweight method quickly predicts upcoming tokens.
Verification The production model computes the probability distribution for the predicted tokens in parallel.
Acceptance The system checks how closely the probability distribution from verification matches the distribution the model would have produced without MTP, and determines whether it meets the acceptance threshold. If it matches, the tokens up to that point are accepted and confirmed all at once. If it doesn't match, drafting is cut off at the point of mismatch, and the result falls back to the production model's own inference.

Draft → Verification → Acceptance ─┬─ match ────→ accept all at once
                                    └─ mismatch ─→ cut off, fall back to the
                                                    production model's answer,
                                                    then Draft again

Because tokens appear to be generated "simultaneously," this is often mistaken for a diffusion model, but the underlying process is still strictly sequential.

In other words, this approach involves a real trade-off. When the prediction is correct, more tokens get confirmed at once, yielding faster output. But when it's wrong, the extra computation goes to waste, and the overhead can make the result slower than running the base model without MTP at all.

How the Two Approaches to MTP Differ

As mentioned, this article covers two implementations, and they differ considerably in their starting point, goals, and mechanics. Let's look at each in more detail.

Qwen's Approach to MTP

Qwen's approach is said to build on techniques researched for DeepSeek-V3, and has been used starting with Qwen3-Next.

The reason this approach is built directly into the model is that the technique itself originally started out as "a method for training a smarter model." MTP turned out to be useful, and the circuitry built into the model for that purpose is now also used, as a side effect, for speculative token prediction.

Input token (N) → Qwen3.5 Main Model → state h_N^0 → Token N+1 [confirmed]
                                             │
                                             ▼ (also feeds the MTP path)
                                       MTP Module 1 → h_N^1 → Sampling & accept
                                             │
                                   match? ───┴─── no match?
                                     │                │
                        predicted token accepted   main model's token adopted
                          → confirmed as N+2         → confirmed as N+2, chain stops
                             │
                             ▼ (only if accepted)
                       MTP Module 2 → h_N^2 → Sampling & accept → ...same check for N+3

Here's how it works, broken down. Even with MTP disabled, when a token is input, the model's main processing generates a state h_N^0 based on the token sequence so far, which is then detokenized to produce token N+1.

When MTP is enabled, as soon as state h_N^0 is produced, the requested number of MTP modules are created. If n tokens' worth of prediction is requested, n modules are prepared, each producing an embedding Emb_N^n that corresponds to a predicted word, based on state h_N^0.

After that, as shown in the figure above, the embedding information for token N+1 — the token that would have been output even without MTP — is combined to produce state h_N^1. This state h_N^1 is sent to the main model, where its sampling and acceptance mechanism checks whether it matches what the main model would have predicted on its own.
If it matches, that token is accepted and the process moves on to predicting the next token.
If it doesn't match, the state predicted by the main model is used instead, and verification stops there.

The defining feature is that the prediction and verification mechanisms are chained together one token at a time, forming a sequential flow throughout.

Gemma's Approach to MTP

Gemma's approach was built from the ground up for speed, and its key difference from Qwen is that it processes prediction and verification together, in a batch.

Input token (N) → Gemma 4 Main Model → state h_N^0 → Prediction: N+1
                                             │
                                             ▼ (KV-cache update)
                                         KV-Cache ←──────────────┐
                                             │                   │ (shares cache)
                                             ▼                   │
                                    Gemma 4 Assistants ──────────┘
                                             │  (generates sequentially inside the
                                             │   draft model, then sends as a batch)
                                             ▼
                            Prediction: N+2, N+3, N+4, ... (candidate list)
                                             │
                                             ▼
                     Main model: causal-attention masking, probabilities computed
                     in parallel  ──┬── include only as many as fit into the output
                                    └── if none fit, exclude from output

In Gemma, the draft model is external. Once the main model confirms the first token, the draft model predicts, all at once, however many tokens are allowed.

The main model first receives the input tokens, generates a state, detokenizes it, and outputs token N+1. At this point, the draft model shares a KV cache with the main model (in Qwen's case, the MTP component and the main component maintain independent KV caches).

Upon receiving the updated KV cache, the draft model generates however many predicted tokens are needed, and hands them off to the main model. The main model then checks, in a batch, whether these predicted tokens are correct using probability distributions, and determines how many of them are acceptable.

As a result, however many predicted tokens the main model accepts get output together with token N+1, all at once.

Which Approach to MTP Is Better?

Because the two approaches to MTP (Multi-Token Prediction) start from different premises, it's hard to say one is unconditionally superior. That said, evaluating primarily on maturity and compatibility with inference engines, as of July 2026, Gemma's approach appears more mature.

First, there's a difference in the scope of sequential processing. Because token prediction fundamentally assumes "predicting the next token based on the state of the previous one," the process is inherently sequential. Comparing the flow of each approach, with time running left to right:

Qwen:   Input(N) → Main model → Predict N+2 → Judge N+2 → Predict N+3 → Judge N+3 → ...
                        │             confirmed:N+2 ↑          confirmed:N+3 ↑
                        └→ confirmed: N+1

Gemma:  Input(N) → Main model ──┬→ Predict N+2 ─┐
                        │       ├→ Predict N+3 ─┼→ Judge (batch) → up to n OK → confirmed: N+2, N+3, N+4, ...
                        │       └→ Predict N+4 ─┘
                        └→ confirmed: N+1

With Qwen's approach, the main model predicts token N+1, uses that state to predict token N+2 and passes it to the main model, then predicts token N+3 only after seeing the verification result — repeating this process step by step.

By contrast, with Gemma, after predicting token N+1, the draft model predicts however many tokens are needed all at once (starting from N+2), and hands them to the main model as a single batch. Processing on the main model's side is also parallelized, making this approach far more efficient than Qwen's. Because of this structural difference, Gemma's approach comes out ahead.

Second, there's a difference in structural flexibility. Because Qwen's MTP functionality is integrated into the main model, improving the drafting logic requires additional post-training. Gemma, on the other hand, keeps the draft model separate, so it can simply be swapped out whenever better logic becomes available. This separation is also advantageous operationally.

Finally, there's the ease of implementation in inference engines. Looking at llama.cpp's implementation, Gemma's approach can run even in multimodal configurations, while Qwen's approach doesn't support multimodal use. This difference stems from the fact that Qwen embeds MTP internally, requiring the branching logic to be handled inside the model itself. Gemma's approach, by contrast, simply toggles the feature on or off depending on whether input passes through the draft model, which is a more favorable structure for engine-side implementation.

Summary (And a Preview of What's Next)

MTP, at its core, works by confirming one token while, in that same pass, speculatively predicting a few tokens ahead at minimal extra cost¹, then confirming them all together if the prediction turns out correct. Qwen builds this mechanism directly into the model itself; Gemma delegates it to a separate, dedicated draft model.

In the next installment — the implementation/benchmark edition — we'll actually run both approaches on llama.cpp and measure, with real benchmarks, exactly how much faster they get and how often the predictions turn out correct. Later in the series, we'll also cover "DFlash," which takes this idea even further.

References

Qwen3-Next: Towards Ultimate Training & Inference Efficiency
https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd

DeepSeek-V3 Technical Report
https://arxiv.org/pdf/2412.19437

Multi-token prediction with Gemma using Hugging Face Transformers
https://ai.google.dev/gemma/docs/mtp/mtp

This article is an English adaptation of the original Japanese post published on Zenn: "小さく賭けて、大きく当てる：LLMを高速化する「投機的デコード（MTP）」の正体【概念編】", by Yuichi Tominaga.

Speculating too many tokens ahead raises the cost of a wrong prediction, which can outweigh the benefit and slow things down. "Minimal extra cost" here means relative to the cost of recomputing everything from scratch for every single token in the conventional approach. ↩