Sparse block‑prefill kernels now make 128 K token prompts tractable for dense LLMs, shattering the long‑standing quadratic bottleneck that forces most services to truncate inputs. FlashPrefill V2 flips the script by delivering an order‑of‑magnitude speedup while keeping output quality essentially intact.
Before this work, inference on very long contexts relied on dense attention implementations such as FlashAttention‑2, whose quadratic scaling still dominated latency even after low‑level optimizations. The community accepted that serving 100 K‑plus tokens would require either massive hardware or severe approximation, leaving high‑quality long‑context use cases out of reach.
FlashPrefill V2 achieves a 47.26× speedup over FlashAttention‑2 at 128 K context length in FP8 precision, and still delivers 27.19× faster inference in BF16 while achieving substantially lower latency than dense baselines, delivering up to 30.49× speedup over an FA3/4‑aligned dense baseline in FP8 precision [1]. The authors attribute this gain to a mean‑corrected block‑sparse operator coupled with warp‑specialized kernels, ping‑pong pipelining, and native support for paged KV caches that align with the latest FlashAttention‑3/4 stack.
The method “confining the accuracy loss to within about one point on RULER and LongBench averages, and to within points at 128K on RULER where fewer than of the blocks are computed” demonstrates that the aggressive sparsity does not materially degrade generation quality even at extreme context lengths [1]. This bound holds across both FP8 and BF16 runs, suggesting the mean‑correction term successfully mitigates approximation error.
The paper’s evaluation stops at 128 K tokens and focuses on homogeneous workloads; it does not report performance on mixed‑length batches, multi‑GPU scaling, or models larger than those tested on Hopper GPUs. Moreover, integration is demonstrated only with SGLang, leaving open whether the same gains translate to other serving stacks or to inference under heavy request concurrency.
If these results hold in production, any service that currently falls back to truncation for inputs beyond 64 K tokens should swap its dense attention backend for FlashPrefill V2, cutting end‑to‑end latency by roughly an order of magnitude and unlocking real‑time summarisation or code completion on massive documents.
Top comments (0)