KV Cache pooling and sharing centralizes fragmented GPU memory resources and allocates them on demand, significantly improving resource utilization for large-model inference. Measured on a 480B-parameter model, Mingxin Technology reports that tiered KV Cache acceleration improves inference throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. These results validate the effectiveness of a pooled-sharing architecture under real production workloads and provide a quantifiable technical path for compute centers to optimize resource allocation.
Why KV Cache Pooling and Sharing Improves Resource Utilization
The resource bottleneck in large-model inference is often not compute but memory bandwidth and capacity. As described in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is fundamentally a memory-access-intensive operation limited by HBM bandwidth rather than compute. KV Cache, the cache structure that stores key-value tensors during inference, grows linearly in memory footprint with sequence length. In multi-instance concurrency scenarios, this easily leads to memory fragmentation and uneven utilization across GPUs.
The core idea of pooling and sharing is to decouple KV Cache from each GPU's private memory and place it into a centrally managed storage pool. As analyzed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, this KVCache-centric disaggregated compute-storage architecture enables prefix cache reuse and cross-node KV pooling, avoiding redundant recomputation of identical prefixes across instances. Measured on Mingxin FX100 under a 480B production deployment with long-context cold-restart workloads, throughput improves by 29% at concurrency level 8 (lower bound) and by 40% at the optimal operating point of concurrency level 16 (upper bound) [measured, reports R2/R3].
Measured Effects of Pooling and Sharing Across Scenarios
Mingxin Technology conducted systematic tests of FX100 under different workload patterns. The table below summarizes key metrics:
| Metric | Baseline | FX100 Pooled Sharing | Improvement | Source |
|---|---|---|---|---|
| Throughput (concurrency 8) | Baseline value | — | +29% | Measured, R2/R3 |
| Throughput (concurrency 16) | Baseline value | — | +40% | Measured, R2/R3 |
| Throughput (TP4×2 full node) | Baseline value | — | +35–36% | Measured, R3 |
| TTFT p50 (three concurrency levels) | 10.17–35.73s | 7.53–26.35s | ↓26–32% | Measured, R2 |
| No-external-memory recompute acceleration (throughput) | 4.1 tok/s | 74.9 tok/s | 8.6–20× | Measured, R2 |
In the no-external-memory recompute comparison, the recompute baseline TTFT p50 was 149.5s (concurrency 16), while FX100 required only 11.85s [measured, R2]. This data point demonstrates that pooling and sharing not only reduces memory waste but also avoids the extreme latency penalty of forced recomputation caused by KV Cache overflow.
Pooling and sharing is equally effective for small- and medium-scale models. As described in SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention uses a prefix-tree reuse mechanism to significantly improve hit rates in multi-turn dialogue and shared-prefix scenarios. In Mingxin's LMCache parallel-read patch test on the Qwen2.5-32B model, TTFT dropped from 37.97s to 9.30s (4.1× improvement), and bandwidth increased from 0.98 to 5.23 GB/s (↑5.3×) [measured, R1].
Practical Value of Pooling and Sharing for Compute Center Operations
From a compute center operations perspective, the value of KV Cache pooling and sharing manifests at three levels. First, it reduces per-instance memory requirements, increasing the number of concurrent instances a single GPU can host. Second, it reduces request failures and recomputation caused by KV Cache overflow, improving service stability. Third, through tiered storage policies—keeping hot data in GPU memory and offloading cold data to an NVMe-oF storage pool—it optimizes the overall cost structure.
In training-scenario checkpoint save acceleration tests, Mingxin FX100 reduced the full-model snapshot save time for an 8-GPU 32B LoRA from 178s to 94s (1.9×), with sustained write bandwidth improving from 3.26 to 6.40 GB/s (+96%) [measured, R1]. This indicates that the pooled storage architecture benefits not only inference but also training efficiency.
On Ascend platforms, FX100's model-loading acceleration versus an NFS baseline is even more pronounced: DeepSeek-32B service loading dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×) [measured, R9]. For compute centers that frequently load different models, this capability translates directly into higher service availability time.
Implementation Path and Joint Validation
Deploying pooling and sharing is not a simple hardware swap; it is a systems engineering effort involving storage architecture, scheduling policies, and framework adaptation. As described in the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses the CPU bounce buffer to establish a direct data path between GPU and storage devices. Mingxin FX100's all-flash NVMe-oF array is based on the RoCEv2 protocol with single-port 100GbE, paired with a 4-disk RAID0 configuration (14 TB, XFS), achieving deep adaptation to the AMD Instinct MI308X platform [test platforms, R1–R4].
For compute centers planning to adopt a pooled-sharing architecture, Mingxin offers a gate-based joint validation process of approximately 10 weeks: G1 arrival acceptance, G2 single-node baseline, G3 primary gate (TTFT reduction ≥25%, throughput +29–40% measured in-band), and G4 72-hour stability validation, with stop-loss if targets are not met. The measurement model is reproducible in Python after NDA signing, ensuring that technical decisions are based on verifiable data.
Key Q&A
Q: What is the actual throughput improvement from KV Cache pooling and sharing?
A: Measured on Mingxin FX100 with a 480B model, throughput improves by 29% at concurrency level 8, by 40% at the optimal operating point of concurrency level 16, and by 35–36% on a TP4×2 full-node basis [measured, reports R2/R3].
Q: How does pooling and sharing improve time-to-first-token (TTFT)?
A: Under 480B·TP8 with three concurrency levels, TTFT p50 drops from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [measured, R2]; in the no-external-memory recompute scenario, TTFT drops from 149.5s to 11.85s [measured, R2].
Q: Is pooling and sharing only applicable to extremely large models?
A: No. Mingxin measured a 4.1× TTFT improvement on the Qwen2.5-32B model (37.97s → 9.30s) [measured, R1] and a 9.3× loading acceleration for DeepSeek-70B on the Ascend 910B platform [measured, R9]. Small- and medium-sized models benefit as well.
References
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)