Today Inception Labs dropped Mercury 3: a diffusion-based LLM with a 1M-token context window doing 1,068 tokens/sec on NVIDIA GPUs, claiming to beat GPT-6 Luna and Claude Haiku 5.5 on benchmarks — at $0.20/$0.70 per million tokens, half price during launch. It's live on OpenRouter, Baseten, Vercel, and behind an OpenAI-compatible API, so you can try it by changing a base URL.
If you're a builder, here's the part that matters: this is not "a faster autoregressive model." It's a different generation paradigm, and it changes what your inference code can assume.
The 30-second architecture primer
Autoregressive decoding: for i in range(n): token[i] = model(tokens[:i]). Serial. Unfixable — once token 5 is emitted, it's baked in.
Diffusion decoding: start with a fully masked/noisy sequence of length n, then iteratively denoise all positions in parallel over K refinement steps. Position 5 can be revised in step 12 based on what position 500 looks like. The model gets K parallel passes over the whole sequence instead of n serial commits.
That parallelism is where 1,068 tok/s comes from. On a GPU, parallel refinement across the sequence exploits the hardware's actual strength (wide vector math) instead of its weakness (serial dependency chains).
Why this should change your evals
Three workload classes where diffusion's architecture is structurally advantaged:
- Infilling and editing. Code completion mid-file, document revision, structured-output repair. Autoregressive models fake this with fill-in-the-middle hacks; diffusion models do it natively because every position is refinable.
- Constrained generation. JSON schemas, formats with global consistency requirements, anything where a late constraint should rewrite an early choice. Serial models can't go back; diffusion can.
- Self-correction loops. Exposure bias — the "one wrong token poisons everything downstream" problem — is an autoregressive artifact. Parallel refinement doesn't have it, at least not in the same form.
The flip side, from the research literature (Diffusion-LM, SEDD, LLaDA): open-ended long-form coherence and precise instruction-following have historically lagged autoregressive models. Mercury 3 claims to close the gap. Claims. Test it yourself.
Trying it: the 5-minute eval
from openai import OpenAI
client = OpenAI(
base_url="https://api.inception.ai/v1", # OpenAI-compatible endpoint
api_key="YOUR_INCEPTION_KEY",
)
resp = client.chat.completions.create(
model="mercury-3",
messages=[{"role": "user", "content": "Refactor this function..."}],
)
print(resp.choices[0].message.content)
What to actually measure — don't just vibe-check:
- Time to first usable token vs. full completion time. Diffusion models refine the whole sequence; "first token" semantics differ. Measure what your UX actually needs.
- Throughput at YOUR context length, not the headline 1,068 tok/s (measured at favorable conditions). Plot tok/s against 4K / 32K / 128K / 1M context. The 1M-context + max-speed pairing is almost certainly not simultaneous.
- Edit/infill accuracy on your real tasks: give it broken JSON, half-written functions, documents with a paragraph to rewrite. This is where diffusion should win — verify it does.
- Instruction-following precision on multi-constraint prompts. This is where diffusion historically loses — verify it doesn't.
The honest caveats (day-one edition)
- Vendor benchmarks, vendor hardware, vendor conditions. Independent evals will land in days; don't migrate production on launch-day numbers.
- The entire inference ecosystem (KV-cache tricks, speculative decoding, every serving framework's optimizations) was built for autoregressive decoding. Diffusion uses different kernel patterns — expect immature tooling and surprising failure modes.
- Prompt engineering behaves differently. Techniques tuned for left-to-right generation (chain-of-thought scaffolding, "let's think step by step") may not transfer. Budget time to re-learn what works.
Why I'm paying attention anyway
October has been the month the LLM monoculture cracked: open weights counter-attacking (816B-param models last week), inference prices collapsing (DeepSeek's $0.50 agent yesterday), and now a non-autoregressive architecture shipping at frontier-adjacent quality. Even if Mercury 3's numbers turn out optimistic, the direction is set — and the builders who understand diffusion decoding now will have a head start when (not if) every lab ships one.
Try it this week. Report back what breaks. That's how new architectures earn trust.
Source: Inception announcement via PANews, Oct 20, 2026. Benchmark figures are vendor-reported. Written by Shivam Kumar (VisionQuantech).
Top comments (2)
The line that matters here is "it changes what your inference code can assume," and I don't think most builders have internalized how much tooling is quietly autoregressive-shaped. Token-by-token streaming UIs, speculative decoding, per-token usage metering, KV-cache sizing heuristics — all of that assumes serial left-to-right emission. A model that denoises all positions in parallel over K steps doesn't stream the same way, so a lot of "just swap the base URL" migrations will hit weird edges in the layer above the API.
The infilling/editing advantage is the real story for me, more than the headline tok/s. Code-repair and structured-output fixups where a late constraint should rewrite an early choice are genuinely awkward with FIM hacks, and diffusion does it natively. That said, I'd want to see the latency distribution, not the peak throughput — K refinement steps means time-to-first-useful-output behaves very differently from TTFT on an AR model, and for interactive agents that tail is what users feel. Has anyone benchmarked it on a real agent loop yet rather than single-shot generation?
The headline metric I'd add here is cost per token, not peak tok/s: diffusion decoding needs K full-sequence forward passes, and each pass touches the whole context window — that's K times the read cost an autoregressive decoder with a KV cache avoids at steady state. So the real comparison is throughput per GPU-dollar at long contexts, which is exactly where KV-cache quantization is currently shrinking the autoregressive cost side. Has anyone seen tok/s or latency numbers for Mercury 3 at 128K+ input context with tight output budgets? That's where this architecture bet actually has to pay off.