Inco AI released DFlash 2, a drafter model for speculative decoding that reaches 3.43 times the throughput of ordinary autoregressive decoding on a single request, while producing byte-identical output. More unusually, the company published the full benchmark grid, including the rows where its own advantage collapses: at 32 concurrent requests on conversational prompts, the speedup falls to 1.01 times.
Key facts
- DFlash 2 reaches 3.43x throughput over autoregressive decoding on grade-school math problems at concurrency 1, falling to 1.45x at concurrency 32.
- Released August 2026 by Inco AI; the drafter is public as incoai/Qwen3.8-27B-DFlash2.
- Benchmarked on one NVIDIA H200 with FlashAttention 3, block size 8, seven draft tokens per verification step.
- Primary sources: the model card and Inco AI's DFlash 2 blog post.
Speculative decoding is one of the few free lunches in machine learning. A small, fast model guesses the next several tokens; the large, slow model checks all of them in a single pass; correct guesses are kept and wrong ones discarded. Because verifying a batch is far cheaper than generating one token at a time, the result comes out faster and, critically, identical. DFlash 2's model card puts it directly: "Decoding is lossless: greedy output matches the target model exactly, and sampling preserves its distribution."
What DFlash 2 changes is the guesser. Rather than predicting tokens one at a time, it is a block-diffusion drafter that "predicts a whole block of tokens in a single pass and keeps the top candidates at every position. A lightweight selector then traces one coherent path through them." Two-tap dynamic convolutions in the backbone stop the draft quality from decaying toward the end of the block, which is the characteristic failure of block drafting: the first two guesses are good, the last two are noise.
The analogy is a court stenographer who anticipates the end of a sentence. A good one gets ahead of the speaker and is usually right; the speaker only has to glance at the transcript to confirm. DFlash 2's improvement is that instead of committing to one guess per word, the stenographer writes several plausible continuations at each position and then picks the single most coherent line through them before handing it over.
The numbers, from the model card's own tables, are worth reading as a grid rather than a headline:
| Concurrency | Grade-school math | Code generation | Conversation |
|---|---|---|---|
| 1 | 3.43x | 3.11x | 2.67x |
| 8 | 2.84x | 2.67x | 2.27x |
| 32 | 1.45x | 1.16x | 1.01x |
Two things stand out. First, DFlash 2 beats both Qwen3.8's own built-in multi-token prediction and a community drafter at every concurrency level tested, on every task. Second, and this is the part vendors normally omit, the competing methods go negative under load. At 32 concurrent requests the built-in multi-token prediction runs at 0.94 times baseline on hard math, 0.84 on code, and 0.77 on conversation. Speculating actively makes the server slower than not speculating at all.
Why that happens is worth understanding, because it generalizes. Speculative decoding buys latency with spare compute, and it works so well on a single request because generating one token at a time leaves a graphics card overwhelmingly idle, a problem our lesson on why LLM inference is memory bound explains in detail. Batching many users together fills that idle capacity with real work. Once the card is saturated, the draft-and-verify passes are no longer free, and every wrong guess is wasted compute that a paying request could have used.
Why it matters practically: a production serving stack lives at high concurrency, not at concurrency 1. The honest reading of the table is that speculative decoding pays most on structured, constrained output for single users, and least on prose-like chat under load. Anyone quoting 3.4x without naming the concurrency is quoting the best cell in the grid.
Inco AI's blog additionally claims DFlash 2 yields more output per verification pass at only about 1 percent extra cycle latency, with gains spanning 16 to 25 percent across benchmarks over the original DFlash, and says DFlash now runs in SGLang, vLLM, TensorRT-LLM, llama.cpp and oMLX with DFlash models past 3.5 million downloads. Those are company-asserted ecosystem numbers; the benchmark tables above come from the model card and are reproducible from the documented setup. The original DFlash paper was published at ICML 2026 by Jian Chen, Yesheng Liang and Zhijian Liu.
The honest caveat: both the SGLang and vLLM launch commands in the model card point at specific pull-request branches rather than released versions, so this is bleeding-edge integration rather than a stable dependency. And the whole result is measured on one H200 with one target model. Speculative decoding gains are famously sensitive to hardware, batch policy and prompt distribution, and there is no reason to expect this exact grid to reproduce on different silicon. We have written before about speculative decoding taking the front page; the difference here is a vendor publishing the row where its own technique stops paying.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)