Autoregressive vs Diffusion LLMs: How the Next Generation of Language Models Actually Writes Text

If you have watched an AI write, you know the ritual. Tokens appear left to right, one after another, like someone typing very fast. It feels like proof of intelligence. It is actually a constraint. Every mainstream language model, from GPT to Claude to the small model running on your laptop, is locked into a strictly sequential process: emit a token, condition on it, emit the next one. Never look ahead. Never go back.

That constraint is now being attacked from an unexpected direction. This week, two deep explanatory posts are circulating on Hacker News at the same time: a guide from the Kuleshov group at Cornell titled "How to Build a Diffusion Language Model," and Sander Dieleman's post on continuous diffusion language models. They land on top of a real product wave. Inception Labs' Mercury generates over 1,000 tokens per second per user on standard GPUs. NVIDIA's open-weight Nemotron Diffusion models report 2 to 8 times the throughput of comparable autoregressive models while retaining up to 99 percent of their quality. Google shipped Gemma Diffusion as an open-weights release.

One disclosure before we go further. I am a backend engineer who runs his own AI agent infrastructure, not an ML researcher. I have never trained a diffusion model. Everything below comes from reading the primary sources this week, and I will link them so you can check me. But I found that the core idea is surprisingly buildable once you see it, and the "which one should I care about" question has a concrete answer now. That is what this article is for.

Autoregressive generation: the incumbent's superpower and its three defects

An autoregressive model generates text the way a very strict typist would. It predicts the next token given all previous tokens, appends it, and repeats. This simple recipe won because it is perfectly suited to GPUs during training and produces a clean probability for every token, which makes reinforcement learning post-training straightforward.

But the recipe carries three defects that are structural, not incidental:

• No error correction. Once a token is emitted, it is permanent. Early mistakes compound, because every later token is conditioned on the flawed ones.
• Speed is capped by sequence length. Generating N tokens takes N sequential forward passes. You cannot parallelize your way out of a process where each step depends on the last one.
• Causal attention only. The model looks backward, never at future context, even when the "future" is text it is about to write anyway.

For years the field accepted these defects as the price of doing business. Speculative decoding and KV caching shave the cost, but the fundamental loop stayed sequential. Diffusion language models change the loop itself.

Diffusion for text: start with the whole page wrong, then fix it

Diffusion models already generate almost every image you have seen from an AI. The idea there is beautifully dumb: take a clean image, add a little noise, repeat until you have pure static. Then train a network to run the tape backward, removing a bit of noise at each step. Generation means starting from static and denoising your way to a picture.

Text is discrete. Words are not blurry, and you cannot add a fraction of a word. So researchers replaced Gaussian noise with something text-shaped: masking. The most influential formulation, popularized by the Kuleshov group and known as masked diffusion, is best understood as a generative BERT. You take clean text, hide a random fraction of tokens, and train a bidirectional transformer to fill in the blanks. Two differences from BERT matter. The masking rate is randomized across training, which turns out to make the model genuinely generative rather than just a fill-in-the-blanks classifier, and it comes with a principled training objective that closed most of the quality gap with autoregressive models.

Generation then works like this: start from a sequence that is entirely blanks, ask the model to fill in every blank, deliberately re-mask most of the sequence while keeping slightly more tokens fixed than last round, and repeat. The text assembles itself out of order, wherever the model is most confident.

Here is the sampling loop in pseudocode, so you can hold the whole algorithm in your head:

# Masked diffusion sampling, the entire idea in 10 lines
sequence = [MASK] * length            # start: all blanks
for step in range(num_steps):
    predictions = model(sequence)     # fill in EVERY blank (a guess)
    sequence = predictions            # accept the full guess
    keep = num_kept[step]             # grows each round
    sequence = remask_random(         # re-noise, but keep the best
        sequence, keep_count=keep)
# each round leaves fewer blanks, until none remain

If you know BERT, you already know 80 percent of this. The remaining 20 percent, the randomized masking schedule and the re-noising loop, is what turns a fill-in-the-blanks model into a generator that can write an entire passage in parallel and revise it mid-stream.

The four upgrades that made it production-ready

Plain masked diffusion had real problems: fixed-length output, no way to fix a token after unmasking it, and slow sampling relative to its potential. The current generation of models stacks four fixes, and the Kuleshov post traces each one. These are worth knowing by name, because they are the vocabulary the next year of model releases will use.

• Block diffusion solves length. Instead of diffusing one fixed-length canvas, the model generates blocks of tokens conditioned on everything before them, then KV-caches each finished block exactly like an autoregressive model would. Block size becomes a tuning knob: pick it to match your domain, or to maximize GPU utilization.
• Encoder-decoder architectures solve speed. Researchers noticed diffusion does two jobs, representing finished tokens and denoising broken ones, so modern models split those jobs between a full encoder and a lighter decoder. Gemma Diffusion and Nemotron Diffusion both use this shape.
• Remasking and uniform noise solve error correction. In remasking samplers, a small subset of already-revealed tokens gets re-masked each step and regenerated, so a grammatical error introduced early can literally be un-written once context arrives. Uniform state diffusion takes a different route: it replaces tokens with random vocabulary words instead of masks, meaning any token is revisable at any step. This is what makes parallel generation coherent instead of self-contradictory.
• Distillation solves step count. Progressive distillation, borrowed from image diffusion, trains the model on its own generations to skip steps, halving sampling cost each round. Combined with the fixes above, this is where the 5 to 10x speedups come from.

There is a fifth layer worth a sentence: post-training. Diffusion models complicate standard RL because estimating the likelihood of a full sampled sequence is expensive, so techniques like diffu-GRPO approximate it, and newer estimators have already pushed diffusion models to state-of-the-art results on logical and math reasoning benchmarks. The training recipe that made autoregressive models smart is being ported over.

The model landscape right now

So who is actually shipping this? Four names cover the field today.

• LLaDA proved it scales. An 8B-parameter open-weights masked diffusion model built roughly along the LLaMA recipe, it anchors most academic research in the area.
• Mercury from Inception Labs was the first commercial diffusion LLM, and its whole pitch is speed: over 1,000 tokens per second per user on standard GPUs, no exotic hardware. Its successor claims to rival speed-optimized frontier models at 5 to 10 times their speed.
• Gemma Diffusion is Google's open-weights entry, combining the uniform-noise backbone with block diffusion and the encoder-decoder design, and it is already supported in mainstream tooling.
• Nemotron Diffusion is NVIDIA's family, scaled up to 35B parameters with a pragmatic twist: a single checkpoint can fall back to plain autoregressive decoding when you want it.

That last detail deserves a pause. A model that speaks both languages, writing in parallel when you need throughput and sequentially when you want maximum reliability, tells you the industry does not see this as a religious war. It sees it as a per-request routing decision.

Autoregressive vs diffusion: which one wins, and when

Here is the honest comparison, with the limits included, because 99 percent of autoregressive quality is not 100 percent, and diffusion models have not yet been trained at frontier scale.

• Choose autoregressive when you need streaming UX where users read as the text appears, when your stack depends on the mature ecosystem of tooling and fine-tuning recipes built around next-token models, or when you need the highest raw quality available, because frontier investment still flows overwhelmingly to autoregressive systems.
• Choose diffusion when throughput is your bottleneck: bulk summarization, batch classification, large-scale code generation, any pipeline where you pay for tokens by the million. Speed on standard GPUs without specialized hardware is the current killer feature.
• Watch diffusion for controllable generation. Because the model refines globally instead of committing to edits one token at a time, it is naturally better at hitting target properties, a constraint you steer during generation rather than hope for afterward. Early demonstrations span code and generated DNA sequences validated in wet labs.
• Do not bet your product on either monopoly. The Nemotron fallback design is the tell. The likely future is hybrid checkpoints, and your abstraction layer should assume a single model may generate both ways.

My own takeaway from a week of reading: nothing here changes what I build this quarter, but it changes what I assume. I had quietly filed "LLMs generate left to right" next to physics constants. It is not a law. It is one algorithm, and another one now matches it on quality while beating it on speed.

Why this might matter more than it looks

The Kuleshov post ends with an argument I have not stopped thinking about. The transformer did not win because it was smarter than RNNs. It won because it was parallel, and parallelism is what let training scale. Since around 2024, gains from scaling pre-training have been flattening, and most new intelligence comes from post-training and inference-time compute, both of which are bottlenecked by how fast a model can generate, which today means a sequential algorithm. If diffusion makes inference fully parallel the way transformers made training parallel, the authors argue it could unlock a comparable jump. Their phrasing: diffusion may be to inference-time scaling what the transformer was to RNNs for pre-training scaling.

I would not bet the farm on any single research thesis. But I have read enough of these arcs to respect the shape of one: an incumbent approach with a structural speed limit, a challenger that removes the limit rather than optimizing around it, and open-weights releases making the challenger downloadable today. That is exactly what this looks like.

Here is what I would actually do with this:

• If you run inference pipelines, benchmark one diffusion model on your real workload this month. Throughput claims like 2 to 8x deserve a test on your data, not a retweet.
• If you are learning how LLMs work, learn masked diffusion next, not more transformer trivia. The 10-line sampling loop above plus the Kuleshov post is a weekend of reading that will not be wasted.
• If you build products on top of models, keep your model layer swappable. The fallback-to-autoregressive design in Nemotron is what your architecture should look like too.

I write about AI, backend engineering, and the tools I actually run, every week. Subscribe, it is free.

Have you tried a diffusion language model yet: Mercury, Gemma Diffusion, LLaDA, anything? Did the speed difference show up in your real workload, or was it a benchmark-only win? I am genuinely curious which way this breaks in practice.

Sources and further reading:

• How to Build a Diffusion Language Model, Kuleshov group at Cornell
• Continuous Diffusion Language Models, Sander Dieleman