How DFlash Uses Block Diffusion to Break the Speculative Decoding Bottleneck

Autoregressive LLM inference has a fundamental problem: every token depends on the one before it. Even with speculative decoding — where a small draft model proposes tokens and the target model verifies them in parallel — the drafting step itself has remained sequential. DFlash, a framework from researchers at UC San Diego's Z Lab, changes that by replacing the autoregressive drafter with a block diffusion model that generates an entire candidate block in a single forward pass.

The results are notable: 6× lossless acceleration on Qwen3-8B, 2.5× improvement over the previous state-of-the-art EAGLE-3, and up to 15× throughput gains on NVIDIA Blackwell hardware at production concurrency levels. The framework is now integrated into SGLang and vLLM, making it accessible without application-level changes.

Why Speculative Decoding Still Had a Bottleneck

Speculative decoding works by having a lightweight draft model generate a sequence of candidate tokens, which the target model then verifies in a single parallel forward pass. If the target model accepts most of the draft tokens, you get significant speedups — the expensive target model runs less often.

The catch is that existing draft models like EAGLE-3 are themselves autoregressive. They generate tokens one at a time, so drafting γ tokens takes γ sequential steps. This creates a ceiling: the faster you want to draft, the more you're constrained by sequential computation. EAGLE-3 achieves roughly 2–3× speedups in practice, which is useful but leaves substantial GPU capacity underutilized.

Diffusion language models offer an alternative — they can generate tokens in parallel — but standalone diffusion LLMs have historically underperformed autoregressive models on quality, making them poor candidates for the verification step.

What DFlash Does Differently

DFlash's core insight is to use a diffusion model only for drafting, not for final generation. The target model remains a standard autoregressive LLM that handles verification. This lets DFlash capture the parallelism of diffusion generation while preserving the quality guarantees of autoregressive verification.

The drafting process works as follows:

1. Context extraction: The target model processes the input prompt and produces hidden states at multiple layers.
2. KV injection: These hidden states are projected and injected into the Key-Value cache of every layer in the draft model. This is the critical difference from earlier diffusion-based speculative decoding approaches, which only conditioned the drafter on the first layer's features. By injecting target context throughout the draft model's depth, DFlash maintains strong alignment between draft and target even as the draft model grows deeper and more expressive.
3. Parallel block drafting: The draft model fills in an entire block of masked token positions in a single forward pass, treating the problem as a joint denoising task rather than a sequential prediction.
4. Verification: The target model checks the proposed block. Accepted tokens are kept; the first rejected token triggers a new draft cycle.

Because the drafting cost is roughly constant regardless of block size, DFlash can use deeper draft models and larger block sizes without the linear latency penalty that constrains autoregressive drafters. A 5-layer DFlash model drafting 16 tokens runs faster than a single-layer EAGLE-3 model drafting 8 tokens.

Training the Draft Model

Training DFlash draft models involves a few design choices that matter for acceptance rates. The draft model shares token embeddings and the language model head with the target model, which keeps the output distribution aligned. During training, random block positions are sampled from the training data rather than always starting from the beginning of a sequence — this improves generalization to arbitrary context lengths.

Loss weighting uses exponential decay across positions within a block, prioritizing accuracy at earlier positions where errors compound. The intuition is that a wrong token early in a block will cause the entire remaining block to be rejected, so it's worth spending more training signal there.

Benchmark Results

On Qwen3-8B with greedy decoding, DFlash achieves:

• 6.08× speedup on code generation (HumanEval)
• 5.15× speedup on math (MATH-500)
• 5.62× speedup on chat (MT-Bench)

Compared to EAGLE-3 on the same tasks, DFlash is 1.4–1.8× faster. For reasoning models at temperature 1, the gains are even larger: 4.5× acceleration on AIME benchmarks.

At production scale on NVIDIA Blackwell (DGX B300), the NVIDIA engineering team reports up to 15× throughput improvement over standard autoregressive decoding for gpt-oss-120B at 500–600 tokens/sec per user interactivity targets. Even against EAGLE-3, DFlash delivers 1.5–2.6× higher throughput depending on task type, with coding and multilingual tasks showing the largest gains.

Integration with SGLang and vLLM

The LMSYS team's Spec V2 blog post describes how DFlash is now the default speculative decoding engine in SGLang. The integration adds an overlap scheduler that reduces host-device synchronization overhead by overlapping draft processing with KV cache allocation for the next batch. This alone adds roughly 33% throughput on top of DFlash's base gains — on Qwen3-8B, throughput goes from 11,400 to 15,300 tokens/second.

For vLLM users, DFlash integrates through the Speculators library. Switching from EAGLE-3 requires updating the checkpoint path and specifying the algorithm; no application code changes are needed. TensorRT-LLM support is also available for Blackwell and Hopper deployments.

Z Lab has released over 20 DFlash draft model checkpoints on Hugging Face covering Qwen, Llama, Gemma, and Kimi K2.6 model families. The original paper and project page include training code and quick-start examples for both SGLang and the Transformers library.

What This Means for Inference Infrastructure

Speculative decoding has been a useful but niche optimization — effective mainly when you have a good draft model and the right hardware setup. DFlash makes the case that the drafting step itself was the limiting factor, not the verification step.

The practical implication is that inference serving costs for large models can drop substantially without any change to model quality. For teams running LLMs at scale, the combination of DFlash with modern inference frameworks like SGLang or vLLM represents a meaningful reduction in GPU hours per token — particularly for coding and reasoning workloads where token acceptance rates are high.

The framework also points toward a broader pattern: diffusion models may be most useful not as standalone generators but as components within hybrid systems where their parallelism can be exploited without sacrificing the quality guarantees of autoregressive verification.