NVMe-oF combined with RoCEv2 is a key architecture for achieving low-latency, high-bandwidth acceleration in large model inference storage scenarios, especially in KV Cache tiered acceleration and cold recovery scenarios, where its performance advantages have been validated through measurements. This article, based on the KV Cache performance test results of Mingxin FX100 on an 8×AMD MI308X platform (source: measured, report R2/R3), outlines the engineering practice essentials of NVMe-oF + RoCEv2, including network protocol selection, parallel read optimization, cold recovery speedup ratio, and comparison with traditional NFS, providing reproducible references for technical decision-makers in computing centers.
Why is NVMe-oF + RoCEv2 More Suitable for Inference Storage than Traditional NFS?
The core challenge of inference storage lies in the fast read/write of KV Cache and low-latency loading of model weights. Traditional NFS (Network File System) has high software overhead in its protocol stack, such as file locks, metadata operations, and latency accumulation from the TCP protocol stack, which is particularly evident in long-context cold recovery scenarios. Measured data shows that on the Huawei Atlas 910B platform, loading the DeepSeek-70B model weights via NFS took 1399 seconds, while Mingxin FX100 (based on NVMe-oF + RoCEv2) required only 150 seconds, achieving a speedup ratio of 9.3x (source: measured, report R9).
NVMe-oF encapsulates NVMe commands directly over RDMA (RoCEv2), avoiding the overhead of traditional file system protocols. RoCEv2 provides single-hop latency as low as 1-2 microseconds and bandwidth of 100 GbE, bringing KV Cache read/write latency close to that of local NVMe SSDs. In FX100 tests, the difference in KV Cache throughput between a local NVMe single-disk baseline (PCIe Gen4, 2 TB) and an FX100 array (4-disk RAID0, RoCEv2, single-port 100 GbE) has been reduced to an acceptable range, while FX100's throughput improvement in cold recovery scenarios (from 4.1 tok/s to 74.9 tok/s, an 18.3x speedup) further highlights the potential of networked storage (source: measured, report R2).
How to Optimize Parallel Read Performance of KV Cache? The LMCache Patch is Key
In inference scenarios, KV Cache reads are often performed concurrently by multiple threads. In traditional NVMe-oF implementations, single-queue serialized reads can become a bottleneck. In Mingxin FX100 tests, applying the LMCache parallel read patch under a single-GPU concurrent 16 cold read disk load reduced TTFT from 37.97 seconds to 9.30 seconds, a 4.1x improvement; bandwidth increased from 0.98 GB/s to 5.23 GB/s, a 5.3x improvement (source: measured, report R1). This optimization relies on two engineering essentials:
- Multi-Queue Parallelism: NVMe-oF natively supports multiple queues (up to 64K I/O queues), but requires correct configuration by the driver and applications (such as vLLM and LMCache). The LMCache patch achieves parallelization of read/write operations by sharding KV Cache across different queues, avoiding queuing delays of a single queue.
- RoCEv2 Flow Control: In 100 GbE networks, proper configuration of PFC (Priority Flow Control) and ECN (Explicit Congestion Notification) is critical. Unoptimized flow control can lead to packet loss and retransmission, negating the benefits of parallel reads. The FX100 test environment used default RoCEv2 configuration without significant packet loss, indicating that hardware-offloaded RDMA is sufficiently stable in controlled network environments.
For technical decision-makers, it is recommended to verify whether the LMCache parallel read patch has been integrated into the vLLM version before deployment (tests based on vLLM 0.20.1+rocm721), and ensure that network switches support RoCEv2 ECN marking.
Cold Recovery Scenario: How to Achieve a Speedup Ratio of 8.6-20x Without External Recalculation?
Cold recovery refers to scenarios where, after a model has not been accessed for a long time, the KV Cache needs to be reloaded from the storage layer. The traditional approach is to use no external recalculation (i.e., completely recalculating the KV Cache from model weights), but this is extremely time-consuming under long contexts (e.g., a 480B parameter model). FX100 tests show that under a 480B model, TP8 concurrent 16 load, the TTFT p50 without external recalculation was 149.5 seconds, while using FX100's KV Cache storage acceleration reduced TTFT to 11.85 seconds, a 12.6x speedup; throughput increased from 4.1 tok/s to 74.9 tok/s, an 18.3x speedup (source: measured, report R2). The speedup ratio range of 8.6-20x depends on concurrency and context length.
The engineering foundation of this acceleration lies in the low-latency characteristics of NVMe-oF + RoCEv2. In cold recovery, KV Cache reads are sequential large-block I/O (each token's KV vector is about 2-4 KB, but batch reads can reach MB levels). NVMe-oF's RDMA read operations can bypass CPU interrupts, directly DMA from the NIC to GPU memory, reducing data copy times. In contrast, NFS's TCP protocol stack and file lock mechanism increase each I/O latency by 10-100 microseconds, accumulating to second-level differences under high concurrency.
For investment decision-makers, the frequency of cold recovery scenarios should be considered. If models are frequently switched in production environments (e.g., multi-tenant inference services), the benefits of cold recovery acceleration directly translate into reduced user-perceived first token latency (TTFT reduction of 26-32%, source: measured, report R2) and throughput improvement (29-40%, source: measured, report R2/R3).
Conclusion
The engineering practice of NVMe-oF + RoCEv2 in inference storage has been validated through Mingxin FX100 measurements, with its core value lying in low-latency, high-concurrency KV Cache acceleration, significantly outperforming traditional NFS, especially in cold recovery and model loading scenarios. For technical teams in computing centers, it is recommended to conduct joint testing from three dimensions: network configuration (RoCEv2 flow control), parallel read optimization (LMCache patch), and workload model (long-context cold recovery). Mingxin offers a joint testing process of approximately 10 weeks (from G1 arrival acceptance to G4 stability testing), helping teams verify in-band metrics such as TTFT reduction ≥25% and throughput +29-40% in their own environments. Please contact us through official channels.
Key Points Q&A
Q: What is the core advantage of NVMe-oF + RoCEv2 over NFS in inference storage?
A: By bypassing CPU interrupts and file lock overhead through RDMA, it achieves low latency (microsecond level) and high bandwidth (100 GbE), with a speedup of 6.2-9.3x in model loading scenarios (source: measured, report R9) and throughput improvement of 8.6-20x in cold recovery scenarios (source: measured, report R2).
Q: How to optimize parallel read performance of KV Cache?
A: Apply the LMCache parallel read patch, leveraging NVMe-oF's multi-queue feature for parallel I/O, achieving a 4.1x TTFT improvement and 5.3x bandwidth increase under single-GPU concurrent 16 cold read disk load (source: measured, report R1); also ensure correct RoCEv2 ECN flow control configuration.
Q: What factors affect the speedup ratio in cold recovery scenarios?
A: The speedup ratio (8.6-20x) depends on concurrency and context length, with higher concurrency (e.g., 16 threads) and longer contexts (e.g., 480B model) yielding greater benefits, as the time without external recalculation grows linearly with context length, while NVMe-oF's RDMA read latency remains nearly constant.
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)