Discarding attention scores speeds up LLM generation by as much as 43 % while barely denting accuracy[1]. The gain comes from eliminating the quadratic scoring pass that dominates KV‑cache eviction, letting the model focus on moving data rather than ranking it. In practice the method plugs into existing inference stacks without any model fine‑tuning.
Prior KV‑cache compression schemes typically score each token to decide what to keep, using the score as a signal of future relevance; this scoring introduces additional latency and memory bandwidth overhead.
Random Attention removes the scoring pass entirely and achieves up to a 43 % boost in token‑per‑second throughput in vLLM deployments[1]. By “evicts uniformly at random within each attention head, computing no score at all,” the eviction round costs only the compaction itself, making it the fastest evictor in both Hugging Face and vLLM stacks.
Across four models (Qwen3‑4B, 14B and 32B, and Phi‑4‑reasoning) and six reasoning benchmarks it matches the strongest existing evictor, beating it outright in 31 of 60 head‑to‑head comparisons[1]. The result shows that precise scoring contributes almost nothing once the prompt is protected; random draws retain enough redundant copies of the reasoning trace to preserve performance.
The study is limited to chain‑of‑thought style tasks and to models up to 32 B parameters; it does not evaluate open‑ended dialogue or retrieval‑augmented generation where prompt length dominates the cache. This suggests that Random Attention’s benefits may shrink when prompts themselves exhaust the KV budget, a regime the paper does not explore.
For any service that serves reasoning‑heavy requests, swapping the default evictor for Random Attention’s random_pp module should shave roughly a third off inference latency without retraining models. Re‑running existing throughput benchmarks (e.g., vLLM’s token‑per‑second test) with this plug‑and‑play optimizer will likely confirm the advertised speedup while keeping task accuracy within a few points of the baseline.
Top comments (0)