Qwen4-Exp: How Per-Layer N-gram Embeddings and Sparse Attention Are Reshaping Hybrid LLM Architecture

Alibaba's Qwen team released Qwen3.8-Flash-Next on August 26, 2026 — an experimental model that serves as the architectural preview for the upcoming Qwen4 series. The model's configuration type, qwen4_exp, is now supported in Hugging Face Transformers v5.16.0, and it introduces four interlocking design choices that are worth understanding on their own terms: Per-Layer Embedding (PLE), Qwen Sparse Attention (QSA), Gated Residual (GR) connections, and a dedicated Multi-Token Prediction (MTP) head.

This isn't just another incremental model release. The architecture makes some genuinely unusual bets — particularly the decision to store 51 billion parameters in a deterministic n-gram lookup table that runs entirely in host RAM, contributing zero GPU FLOPs. That's a design philosophy worth unpacking.

The Core Problem: Scaling Capacity Without Scaling Compute

Most large language models face a familiar tension: adding more parameters improves capability, but also increases the compute cost of every forward pass. Mixture-of-Experts (MoE) architectures partially solve this by activating only a subset of parameters per token, but the routing overhead and memory fragmentation of large expert pools create their own problems.

Qwen4-Exp takes a different angle. The model has 176 billion total parameters but activates only 6 billion per token — a 29:1 ratio. The key to making this work is that the bulk of those "inactive" parameters aren't in MoE experts at all. They live in the n-gram embedding table, which is accessed via a deterministic hash lookup rather than a learned routing decision.

Per-Layer Embedding: 51 Billion Parameters at Zero GPU Cost

The most distinctive component of Qwen4-Exp is its Per-Layer Embedding (PLE) system. At layer 2 of the 48-layer stack, the model injects lexical features derived from hashed token bigrams and trigrams. The embedding table contains 20 million entries and totals 51 billion parameters — but because it's a lookup table rather than a matrix multiplication, it lives in host RAM and is prefetched asynchronously.

The practical implication: you get the representational benefit of a massive embedding space without paying for it in GPU memory bandwidth or FLOPs. The features are combined with the residual stream using a dilated depthwise convolution, which adds a small amount of local context sensitivity without the cost of full attention.

For deployment, frameworks like llama.cpp support offloading the n-gram table to CPU via ple_ngram_embd=CPU, which makes the model viable on workstations with limited GPU memory. The table can also be sharded across devices using the split_ngram_parts configuration parameter.

Qwen Sparse Attention: Block-Level Selection Instead of Token-Level Indexing

Standard sparse attention mechanisms typically work at the token level: they score individual key-value pairs and select the top-k most relevant ones. This creates two problems at long context lengths — the indexer itself becomes expensive, and the selected tokens are often scattered across memory in ways that hurt cache locality.

Qwen Sparse Attention (QSA) addresses both issues by operating at the micro-block level. Rather than scoring individual tokens, QSA compresses the key sequence into blocks, scores those blocks, and selects the most relevant contiguous regions for full attention. The trailing (incomplete) block is always kept uncompressed to preserve precision at the boundary.

The result is that QSA's compute cost scales with a fixed budget — 512 blocks per layer — rather than with sequence length. At one million tokens, this translates to up to 7.6× faster prefill and 4.9× faster decode compared to full attention, according to NVIDIA benchmarks on the GB300 NVL72 platform.

QSA is designed to work alongside Gated DeltaNet (GDN), a linear attention mechanism that handles long-tail history compression via recurrent states. The macro-block structure alternates between GDN layers (for efficient sequence compression) and QSA layers (for precise retrieval), with each followed by an MoE block. In the 48-layer model, this produces 12 macro-blocks, each containing three GDN→MoE layers and one QSA→MoE layer.

Gated Residual: Four-Branch Information Flow

The Gated Residual (GR) architecture replaces the standard single-path residual connection with a four-branch structure. Before each attention and MoE block, the model uses Hyper-Connections combined with GatedNorm to mix these parallel residual streams. A learned, element-wise gating mechanism then controls how block outputs are injected back into the streams.

The residual state itself is stored in FP8 to reduce memory pressure, while the gating weights remain in higher precision. According to the Enera Labs analysis, this design improves training convergence and stability at scale — the model uses the Muon optimizer with refined scaling laws and doesn't require a warmup phase.

The practical benefit for practitioners is that GR connections make the model more robust to the kind of gradient instability that often appears when training hybrid architectures that mix linear and sparse attention mechanisms.

Multi-Token Prediction Head: Speculative Decoding Built In

The fourth component is a 4-billion-parameter MTP head — a dense-attention layer with its own 512-expert MoE — that predicts the next-next token during inference. This is specifically designed to improve acceptance rates in speculative decoding workflows.

Rather than requiring a separate draft model (as in standard speculative decoding setups), the MTP head is trained jointly with the main model and shares its representations. This means the draft predictions are better calibrated to the main model's distribution, which typically translates to higher acceptance rates and therefore higher effective throughput.

What This Means for Practitioners

Qwen3.8-Flash-Next is explicitly labeled as an architectural preview rather than a production model — the production version will be Qwen3.8-Flash. But the qwen4_exp architecture is already supported in Hugging Face Transformers v5.16.0 and has been ported to llama.cpp, so practitioners can evaluate it today.

A few things worth noting for teams considering this architecture:

Context window: The native context is 262,144 tokens, with YaRN scaling extending it to one million tokens. At one million tokens on GB300 NVL72 hardware, the model achieves 16,000+ tokens per second per GPU and 200+ tokens per user — numbers that make long-document workflows genuinely practical.

Licensing: Qwen3.8-Flash-Next is released under Apache 2.0, unlike the flagship Qwen3.8-Max which carries a bespoke commercial license. This makes it a viable option for teams that need open-weight deployment without licensing constraints.

Hardware path: For prototyping, the model runs on four RTX PRO 6000 Blackwell Max-Q GPUs or a DGX Spark cluster. For production serving, the GB300 NVL72's 130 TB/s NVLink bandwidth eliminates the cross-network bottlenecks that typically plague MoE models at scale.

The Broader Design Philosophy

What makes Qwen4-Exp interesting as an architectural statement is the way it separates different kinds of capacity. The n-gram table provides lexical breadth at zero GPU cost. The GDN layers provide efficient long-range compression. The QSA layers provide precise retrieval at fixed compute cost. The MoE blocks provide task-specific depth. And the MTP head provides throughput without a separate draft model.

Each component is doing a specific job, and the jobs don't overlap much. That's a different approach from architectures that try to solve everything with a single attention mechanism and a large MoE pool — and it's worth watching to see whether the Qwen4 production release validates the tradeoffs.

The full model documentation and llama.cpp integration are available for teams that want to experiment before the production release.