DiffusionGemma 26B: How Google's Text Diffusion Model Generates Tokens in Parallel

Google DeepMind released DiffusionGemma 26B-A4B on June 10, 2026 — an open-weight language model that generates text using discrete diffusion rather than the autoregressive approach used by virtually every major LLM today. The result is a model that can exceed 1,000 tokens per second on an H100 GPU at small batch sizes, compared to roughly 70 tokens per second for the standard autoregressive Gemma 4 26B on the same hardware. This post explains how that works, what the tradeoffs are, and when you'd actually want to use it.

The Core Problem with Autoregressive Generation

Standard language models — GPT, Llama, Gemma, Claude — generate text one token at a time. Each new token is conditioned on all previous tokens, which means the model must run a full forward pass for every single output token. At low batch sizes (one or a few concurrent requests), this is memory-bandwidth-bound: the GPU spends most of its time moving model weights from VRAM into compute units, not actually doing matrix multiplications. The tensor cores sit underutilized.

This is fine for high-throughput cloud deployments where you can batch hundreds of requests together. But for interactive, local, or latency-sensitive workloads — a coding assistant responding in real time, an in-browser tool, a local deployment on a single GPU — autoregressive generation is slow by design.

How Diffusion Changes the Equation

DiffusionGemma borrows the core idea from image diffusion models (like Stable Diffusion) and applies it to discrete text tokens. Instead of generating tokens left-to-right, the model starts with a "canvas" of 256 masked or random tokens and iteratively denoises them over multiple passes.

At each denoising step, the model uses bidirectional attention — every position in the 256-token block attends to every other position simultaneously. This is architecturally closer to BERT than to GPT. The model refines the entire block in parallel, locking in high-confidence tokens first and continuing to refine uncertain ones. Generation stops when the average canvas entropy drops below a threshold or tokens stabilize across two consecutive steps (up to a maximum of 48 denoising steps).

For sequences longer than 256 tokens, the model uses a block-autoregressive approach: it processes and commits each 256-token block to the KV cache sequentially, then moves to the next block. This preserves the speed advantage within each block while handling longer outputs.

The key speed gain: generating 256 tokens requires T forward passes on a 256-position sequence. An autoregressive model doing the same output requires 256 forward passes on sequences growing from 1 to 256 positions. At T=10 denoising steps, DiffusionGemma does 10 forward passes instead of 256 — and each forward pass is compute-bound (dense matrix multiplications across all 256 positions), which saturates GPU tensor cores rather than memory bandwidth.

Architecture Details

DiffusionGemma is built on the same Mixture-of-Experts backbone as Gemma 4 26B. The full specs:

• Total parameters: 25.2B
• Active parameters per step: 3.8B (8 active experts out of 128 total, plus 1 shared)
• Context length: Up to 256K tokens
• Canvas length: 256 tokens per diffusion block
• Vocabulary size: 262K tokens
• Modalities: Text, image, and video inputs; text output

The encoder-decoder structure is worth noting: an autoregressive encoder processes and caches the prompt context (the input), while the decoder applies bidirectional attention over the generation canvas (the output). This means the prompt processing is still sequential, but output generation is parallel.

The recommended sampler uses entropy-bounded denoising with adaptive stopping: temperature decays linearly from 0.8 to 0.4 across denoising steps, and the lowest-entropy tokens are locked in first. Generation terminates early if average canvas entropy drops below 0.005 and top tokens are stable across two steps.

Benchmark Numbers

The speed gains are real, but they come with quality tradeoffs. Comparing DiffusionGemma to standard Gemma 4 26B on the same benchmarks:

Benchmark	DiffusionGemma	Gemma 4 26B
MMLU Pro	77.6%	82.6%
AIME 2026 (no tools)	69.1%	88.3%
LiveCodeBench v6	69.1%	77.1%
GPQA Diamond	73.2%	82.3%
Codeforces ELO	1429	1718

The gap is consistent: DiffusionGemma scores roughly 5–20 percentage points lower than its autoregressive counterpart across reasoning and coding benchmarks. The one exception is HLE (Humanity's Last Exam) without tools, where DiffusionGemma scores 11.0% vs. Gemma 4's 8.7% — though both numbers are low enough that this may not be meaningful.

On throughput, deployment benchmarks show:

• H100, FP8, batch 1: ~1,100 tokens/sec
• L40S, INT8, batch 4: ~1,400 tokens/sec
• L40S, INT8, batch 1: ~480 tokens/sec (at T=10)
• Autoregressive Gemma 4, L40S, batch 1: ~70 tokens/sec

The speedup is most pronounced at small batch sizes. At batch sizes of 32 or more, autoregressive models with continuous batching close the gap significantly.

Hardware Requirements

The 26B MoE architecture loads all expert weights into VRAM at startup, even though only 3.8B activate per forward pass. This means VRAM requirements are determined by total parameters, not active ones:

• BF16: ~52 GB VRAM (requires H100 80GB or RTX PRO 6000)
• INT8: ~28 GB (fits on L40S 48GB)
• NVFP4 (4-bit): ~18 GB (fits on RTX 4090 24GB)

The model is available under Apache 2.0 on Hugging Face and works with vLLM, Hugging Face Transformers, SGLang, MLX, Unsloth, and NVIDIA NeMo.

Where This Makes Sense (and Where It Doesn't)

The bidirectional attention mechanism makes DiffusionGemma particularly well-suited for tasks where the output has global structure — code infilling, structured document completion, or constrained generation like filling in a JSON schema. The model can "see" the full output block at once and resolve dependencies that autoregressive models handle awkwardly (e.g., knowing a closing bracket is needed before generating the content inside it).

It's less suited for:

• High-concurrency APIs where autoregressive models with continuous batching are more efficient at batch sizes of 32+
• Long-context tasks where the 256-token block structure adds overhead
• Streaming output where users expect to see tokens appear one by one — DiffusionGemma generates a full block before committing it
• Production quality-critical applications where the benchmark gap matters

The model is explicitly labeled experimental by Google DeepMind, and the recommendation in the official documentation is to use standard Gemma 4 for quality-critical outputs.

What This Signals

Text diffusion has been a research direction for several years — models like MDLM, PLAID, and earlier masked diffusion approaches explored this space — but DiffusionGemma is the first time a major lab has shipped a production-scale open-weight text diffusion model built on a frontier MoE backbone. The fact that it's Apache 2.0 and compatible with standard inference frameworks (vLLM, SGLang) makes it immediately usable rather than a research artifact.

The quality gap relative to autoregressive Gemma 4 is real and worth taking seriously. But for the specific use case of interactive, low-latency local inference — where 1,000 tokens per second on a single GPU is the goal — DiffusionGemma offers a meaningfully different point in the design space. Whether the diffusion approach can close the quality gap with further training and architectural refinement is an open question, and one that's now much easier to explore with open weights available.

---

Primary source: Google DeepMind blog post on DiffusionGemma

Model card: Hugging Face — google/diffusiongemma-26B-A4B-it

Deployment guide: Spheron — Deploy DiffusionGemma on GPU Cloud