Sparse attention accesses fewer KV entries per decode step. The full KV cache still requires fast, high-capacity memory.
Our new XM-Sparse testbed results with SGLang HiSparse and Netpreme X-Mem™ show how high-bandwidth access to offloaded KV and greater memory capacity can reduce time per output token (TPOT) and increase peak decode throughput compared with host DRAM.
These results were measured in the X-Mem™ Pilot Virtual Machine. X-Mem™ silicon will be available later.
Kudos to the Netpreme engineering team.
Read the architecture and full benchmark results on Netpreme’s blog
Top comments (0)