DiffusionGemma: How Google DeepMind's Text Diffusion Model Achieves 1,000 Tokens Per Second

Most language models generate text the same way: one token at a time, left to right, each word depending on everything that came before it. It's a well-understood approach, but it has a fundamental speed ceiling tied to memory bandwidth. Google DeepMind's DiffusionGemma, released on June 10, 2026, takes a different path — borrowing the denoising process from image generation to produce entire blocks of text in parallel. The result is throughput that reaches over 1,000 tokens per second on a single NVIDIA H100, roughly four times faster than comparable autoregressive models.

This post explains how that works, what the trade-offs are, and where this kind of model actually makes sense to use.

The Core Idea: Text as a Denoising Problem

Diffusion models for images work by starting with random noise and iteratively refining it toward a coherent picture. DiffusionGemma applies the same principle to text, but with a twist suited to discrete tokens.

Instead of generating tokens sequentially, the model starts with a "canvas" of masked placeholder tokens — up to 256 at a time — and runs multiple forward passes to refine them. In each pass, every token can attend to every other token in the block (bidirectional attention), which means the model can resolve dependencies in any direction. By the end of the denoising process, the entire block is finalized at once.

This is fundamentally different from autoregressive generation, where token N can only see tokens 1 through N-1. The bidirectional view lets DiffusionGemma handle tasks where the correct answer at position 5 depends on what ends up at position 20 — something autoregressive models have to approximate through chain-of-thought or other workarounds.

Architecture: Built on Gemma 4's MoE Backbone

DiffusionGemma is built on the Gemma 4 26B-A4B architecture, a Mixture-of-Experts design with 25.2 billion total parameters but only 3.8 billion active during any given inference step. The MoE structure means the model routes each token through a subset of specialized "expert" sub-networks rather than activating the full parameter set — keeping compute manageable while maintaining a large effective capacity.

On top of this backbone, Google added a diffusion head that handles the masked token refinement process. The model accepts multimodal inputs (text, images, and video) but produces text output only.

Key hardware numbers from NVIDIA's deployment guide:

• NVIDIA H100: 1,000+ tokens/second
• NVIDIA DGX Station: up to 2,000 tokens/second
• GeForce RTX 5090: ~700 tokens/second
• VRAM requirement: ~18GB when quantized (fits on an RTX 4090 or 5090)

The model is available under the Apache 2.0 license on Hugging Face (google/diffusiongemma-26B-A4B-it) and has day-zero support in Hugging Face Transformers, vLLM, and Unsloth.

Why Local Inference Benefits More Than Cloud

One detail worth understanding: the speed advantage of diffusion is more pronounced in local, single-user settings than in large-scale cloud deployments.

Cloud inference systems run autoregressive models efficiently by batching requests from many users simultaneously, keeping GPUs saturated and making good use of high-bandwidth memory (HBM). In that environment, the sequential nature of autoregressive generation is less of a bottleneck because the hardware is always busy.

Local inference is different. A single user's GPU sits idle between tokens, and the memory bandwidth required to load model weights for each token becomes the limiting factor. Diffusion sidesteps this by shifting the bottleneck from memory bandwidth to raw compute — generating 256 tokens per forward pass means the weight-loading cost is amortized across a much larger output block.

This is also why Google's cloud-based Gemini models remain autoregressive: the batching advantages of cloud serving reduce the relative benefit of diffusion, while the quality trade-offs (more on those below) remain.

Where Diffusion Has a Genuine Advantage

The bidirectional attention in DiffusionGemma opens up use cases that are awkward for autoregressive models:

Code infilling: When you need to insert code in the middle of an existing function, the correct insertion depends on both what comes before and after it. Autoregressive models have to guess at the future context; DiffusionGemma can see both sides simultaneously.

In-line text editing: Revising a sentence in the middle of a paragraph while keeping surrounding text coherent is a similar problem. Diffusion handles this naturally.

Constraint-heavy generation: Tasks like Sudoku solving, molecular sequence generation, and mathematical formatting benefit from the model's ability to self-correct across the entire output block rather than committing to each token before seeing the rest.

High-throughput local applications: Any application that needs to generate large amounts of text quickly on local hardware — interactive coding assistants, real-time document drafting tools — benefits from the raw speed advantage.

The Trade-Offs Are Real

DiffusionGemma is explicitly described as experimental, and the benchmark numbers from DataNorth's analysis show a meaningful quality gap compared to standard Gemma 4:

Benchmark	DiffusionGemma	Gemma 4 26B
MMLU Pro	77.6%	82.6%
GPQA Diamond	73.2%	82.3%
LiveCodeBench v6	69.1%	77.1%
Codeforces (Elo)	1429	1718

The accuracy gap is consistent across reasoning, science, and coding benchmarks. This reflects a known challenge with text diffusion: language is discrete, so a single badly predicted token can make an entire block incoherent in a way that a misplaced pixel in an image does not. The model has to do more work to recover from errors, and the iterative refinement process doesn't always converge to the same quality as careful left-to-right generation.

There's also a practical inefficiency for short outputs. Diffusion models must run the full denoising process regardless of output length — generating five tokens requires nearly as much compute as generating 256. Autoregressive models scale linearly with output length, making them more efficient for brief responses.

What This Signals About the Direction of Open Models

DiffusionGemma is notable not just as a standalone release but as a signal that Google is actively exploring non-autoregressive architectures in its open model line. The concurrent existence of Gemini Diffusion (a closed, waitlist-only infrastructure experiment) and DiffusionGemma (open weights, Apache 2.0) suggests Google is running parallel tracks: testing diffusion at scale internally while releasing open models for the community to experiment with.

For developers, the practical takeaway is straightforward: DiffusionGemma is a useful tool for latency-sensitive, local inference workloads where throughput matters more than peak accuracy. For tasks requiring careful reasoning, long-context retrieval, or high-stakes outputs, the standard Gemma 4 models remain the better choice.

The Apache 2.0 license and broad framework support (Transformers, vLLM, Unsloth) make it easy to experiment with. If you're building applications where generating text fast on local hardware is the primary constraint, it's worth testing whether the quality trade-off is acceptable for your specific use case.

Getting Started

The model weights are available at google/diffusiongemma-26B-A4B-it on Hugging Face. For local deployment on consumer hardware, quantization to INT8 or INT4 brings VRAM requirements down to 28GB and ~18GB respectively. NVIDIA's deployment guide covers setup for RTX GPUs and DGX systems, and vLLM's day-zero support means you can serve it with a standard inference stack without additional configuration.

---

Sources: Google DeepMind DiffusionGemma page, Ars Technica coverage, NVIDIA deployment blog, DataNorth benchmark analysis