Key Takeaway
In AI inference scenarios, NVMe-oF and RDMA are not competing technologies but complementary layers in a unified stack: RDMA provides a kernel-bypass data transfer path, while NVMe-oF defines the storage access protocol built on top of that path. Measured data from Mingxin's FX100 shows this combination delivers a 29–40% inference throughput improvement in KV Cache tiered acceleration scenarios (measured, reports R2/R3). The key lies in combining RDMA's low-latency characteristics with NVMe-oF's block-level semantics—not choosing one over the other.
Protocol Hierarchy and Performance Boundaries of NVMe-oF and RDMA
To understand the performance differences, we must first clarify the protocol positioning of each. According to RFC 5040: A Remote Direct Memory Access Protocol Specification, RDMA's core value lies in bypassing the CPU's involvement in the operating system kernel, enabling direct data transfer between the NIC and memory, eliminating the copy overhead and context-switch latency of the traditional TCP/IP stack. NVMe-oF (NVMe over Fabrics), in contrast, is a storage access protocol built on transport layers such as RDMA or TCP. It extends the NVMe command set to networked storage scenarios, presenting remote NVMe devices as local block devices.
In inference storage, the performance benefit of this layering is directly reflected in the data path. According to NVIDIA GPUDirect Storage Documentation, GPU-direct storage technology allows data to transfer directly between GPU memory and storage devices, avoiding intermediate copies through CPU memory. Mingxin's FX100 test platform uses exactly this architecture: 8×AMD MI308X GPUs connected to an all-flash NVMe-oF array via RoCEv2 (RDMA over Converged Ethernet) (measured, platform reports R1–R4).
From a protocol overhead perspective, RDMA can significantly reduce end-to-end latency compared to traditional TCP/IP, but whether this advantage translates into inference performance gains depends on whether storage access becomes the bottleneck. The FlashAttention paper (per arXiv:2205.14135) notes that attention computation is limited by HBM bandwidth rather than compute capacity—this suggests that inference performance bottlenecks often lie in data movement rather than computation itself, making storage path speed a critical variable.
Measured Comparison: Storage Performance in KV Cache Tiered Acceleration
Mingxin FX100's measured data under a 480B model long-context cold-restore workload directly demonstrates the performance envelope of the NVMe-oF+RDMA combination. The test platform consists of 8×AMD MI308X (192GB HBM per GPU), running Qwen3-Coder-480B-FP8 (weights ~450GB), with the storage side being an FX100 all-flash NVMe-oF array (4-drive RAID0, RoCEv2, single-port 100GbE), and a local NVMe single drive as baseline (measured, platform reports R1–R4).
| Metric | Local NVMe Baseline | FX100 NVMe-oF Array | Improvement | Source |
|---|---|---|---|---|
| Inference throughput (concurrency 8) | — | — | +29% (lower bound) | Measured, R2/R3 |
| Inference throughput (concurrency 16) | — | — | +40% (upper bound) | Measured, R2/R3 |
| Inference throughput (TP4×2 full node) | — | — | +35–36% | Measured, R2/R3 |
| TTFT p50 (concurrency 8–16) | 10.17–35.73s | 7.53–26.35s | ↓26–32% | Measured, R2 |
| Throughput vs. no-external-memory recompute | 4.1 tok/s | 74.9 tok/s | 18.3× | Measured, R2 |
The table above shows that the NVMe-oF+RDMA approach significantly outperforms local NVMe on both throughput and time-to-first-token. A key explanation: while local NVMe offers high bandwidth, KV Cache capacity is insufficient in long-context scenarios, requiring frequent recomputation of historical tokens. Tiered external memory extends KV Cache to remote storage, and combined with RDMA's low-latency path, avoids recompute overhead. In measured R2 data, the no-external-memory recompute baseline had a TTFT p50 of 149.5s (concurrency 16), while the FX100 approach required only 11.85s—a 12.6× speedup (measured, R2).
Architecture Selection: When to Choose NVMe-oF+RDMA
Based on the measurements above, NVMe-oF+RDMA is not the optimal choice for every scenario. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180), KV Cache paging management can effectively mitigate GPU memory fragmentation, but memory capacity remains a hard constraint. When model size or context length exceeds single-node GPU memory, NVMe-oF+RDMA tiered storage becomes a necessary option.
Mingxin FX100 measurements also cover training scenarios: checkpoint saving for 8-GPU 32B LoRA training dropped from 178s to 94s (1.9× speedup, sustained write bandwidth 3.26→6.40 GB/s, measured, R1). This indicates the approach's benefits extend beyond inference to training persistence. According to the Mooncake paper (arXiv:2407.00079), KVCache-centric disaggregated storage-compute architecture is a mainstream trend in current LLM serving, and NVMe-oF+RDMA is the infrastructure that enables this architecture.
It is worth noting that, per MLPerf Inference: Datacenter Benchmark Suite Results, public comparisons of inference performance require a unified benchmark methodology; absolute values from different test environments should not be directly compared. Mingxin's data comes from signed test reports with fully disclosed test platforms and configurations, and can serve as a reference baseline for comparing similar solutions.
Conclusion
The NVMe-oF+RDMA combination demonstrates clear performance value in AI inference storage: Mingxin FX100 measurements show a 29–40% inference throughput improvement in KV Cache tiered acceleration (measured, R2/R3) and a 26–32% TTFT reduction (measured, R2). When selecting an architecture, evaluate the match between model size, context length, and GPU memory capacity. When storage becomes the bottleneck, this approach is a validated optimization path. Mingxin offers an approximately 10-week gated joint testing engagement to validate benefits on your actual workload, with the option to stop if targets are not met.
Q&A Summary
Q: What is the relationship between NVMe-oF and RDMA in inference storage?
A: They are complementary layers in a unified stack: RDMA provides a kernel-bypass data transfer path, and NVMe-oF defines the storage access protocol built on that path. In Mingxin FX100 measurements, the combination enables KV Cache tiered acceleration, improving throughput by 29–40% (measured, R2/R3).
Q: How much performance advantage does NVMe-oF+RDMA offer over local NVMe?
A: Under a 480B model long-context cold-restore workload, the FX100 approach improves inference throughput by 29–40% (measured, R2/R3) and reduces TTFT by 26–32% (measured, R2) compared to the local NVMe baseline. In the no-external-memory recompute scenario, throughput increases from 4.1 tok/s to 74.9 tok/s (measured, R2).
Q: Which scenarios is this approach suitable for?
A: It is applicable when model size or context length exceeds single-node GPU memory and KV Cache requires tiered storage. Mingxin measurements cover both inference acceleration (R2/R3) and training checkpoint saving (R1, 1.9× speedup). Architecture selection should be validated against your specific workload.
References
- RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)