NVMe-oF vs. Local NVMe: A Comparative Analysis for Inference Performance

Key Takeaways

In long-context inference and KV Cache tiered acceleration scenarios, the throughput and latency advantages of an NVMe-oF all-flash array over a local NVMe single drive are supported by reproducible measurements: in a 480B production deployment, tiered KV acceleration improves inference throughput by +29–40% (measured, reports R2/R3), and reduces time-to-first-token (TTFT) by 26–32% (measured, report R2). However, NVMe-oF is not a universal replacement—its benefits are concentrated in specific workload patterns such as multi-GPU sharing, cold read workloads, and checkpoint persistence. Selection decisions must be based on concurrency levels, context length, and SLA constraints.

Mechanism Differences: NVMe-oF vs. Local NVMe

NVMe-oF (NVMe over Fabrics) maps remote NVMe devices as local block devices via the RDMA protocol, with the data path bypassing the CPU bounce buffer—the NIC exchanges data directly with the GPU or memory. According to NVIDIA GPUDirect Storage documentation, the core value of this mechanism lies in reducing CPU involvement during data movement, thereby lowering latency and CPU utilization. In contrast, the advantage of a local NVMe single drive is the shortest path—no network hops, no protocol encapsulation overhead—making it the more economical choice in single-node, single-GPU, or low-concurrency scenarios.

Per RFC 5040 (RDMA protocol specification), the semantic boundary of RDMA is "remote direct memory access," and its design goal is not to replace local storage but to address data-sharing efficiency across multiple nodes. This implies that NVMe-oF's applicability is inherently tied to "multi-node sharing" and "centralized storage," rather than single-node performance limits.

The trade-off between the two paths is essentially a balance between data path length and sharing capability: local NVMe offers lower latency but cannot be shared across nodes, while NVMe-oF adds network overhead but unlocks centralized management and concurrent multi-GPU reads.

Measured Data: Quantified Benefits of NVMe-oF in Inference Scenarios

Measured data from the Mingxin FX100 on an 8×AMD MI308X platform (reports R1–R3) provides a comparable quantitative basis:

Metric	Local NVMe Single Drive (Baseline)	FX100 NVMe-oF Array	Improvement	Source
480B·TP8·Concurrency 16·TTFT p50	Upper bound of 10.17–35.73s range	Upper bound of 7.53–26.35s range	↓26–32%	Measured, report R2
480B·KV tiered·Throughput (optimal operating point, concurrency 16)	—	+40% (lower bound at concurrency 8: +29%)	+29–40%	Measured, reports R2/R3
Throughput without external memory recomputation (concurrency 16)	4.1 tok/s	74.9 tok/s	8.6–20× range	Measured, report R2
TTFT p50 without external memory recomputation (concurrency 16)	149.5s	11.85s	12.6×	Measured, report R2
LMCache cold read·TTFT (Qwen2.5-32B·single GPU·concurrency 16)	37.97s	9.30s	4.1×	Measured, report R1
Model loading (DeepSeek-70B·Ascend 910B)	1399s (NFS baseline)	150s	9.3×	Measured, report R9

It should be clarified that "local NVMe single drive" and "without external memory recomputation" represent two different baselines in the comparisons above: the former represents the conventional local storage path, while the latter represents the worst case of not using external memory at all and relying solely on GPU VRAM. The benefits of NVMe-oF over local NVMe are primarily evident in cold read scenarios (4.1× TTFT improvement with LMCache patching) and multi-GPU concurrent sharing (480B throughput +29–40%), rather than across all workload types.

Applicability Boundaries and Selection Criteria

The benefits of NVMe-oF are not universal; its applicability boundaries can be defined along three dimensions:

Concurrency pattern: Measured data from R2/R3 shows that throughput improvement varies with concurrency level—+29% at concurrency 8 (lower bound) and +40% at concurrency 16 (upper bound). In low-concurrency scenarios, the network overhead share increases, narrowing NVMe-oF's relative advantage.

Context length: The core premise of KV Cache tiered acceleration is that in long-context scenarios, KV data exceeds VRAM capacity and must spill to the storage layer. In short-context scenarios, the KV Cache can reside entirely in VRAM, rendering NVMe-oF unnecessary. According to the PagedAttention paper (SOSP '23), the motivation for paged KV Cache management lies in VRAM fragmentation and capacity limits—precisely the precondition for external memory tiering.

Workload type: Training checkpoint saving (1.9× acceleration, measured, report R1) and model loading (6.2–9.3×, measured, report R9) are strong use cases for NVMe-oF, as these workloads inherently require high-bandwidth sequential writes/reads and involve multiple GPUs accessing the same dataset concurrently. Conversely, for single-GPU, small-scale, one-off inference, the latency advantage of local NVMe may be more pronounced.

Selection should follow this decision sequence: first define the SLA (TTFT ceiling) and concurrency pattern, then estimate KV Cache spill volume, and finally determine whether centralized storage sharing capability is needed. If concurrency ≥8 and context length ≥32K, the measured benefit range of NVMe-oF can serve as an expected reference; for single-GPU, low-concurrency, short-context scenarios, local NVMe remains the reasonable default.

Conclusion

NVMe-oF and local NVMe are not substitutes but complementary solutions for different workload patterns. Measured data from the Mingxin FX100 demonstrates that in multi-GPU, long-context inference scenarios, the throughput and latency benefits of NVMe-oF have clear quantitative support; however, its applicability is constrained by three factors: concurrency level, context length, and workload type. Decision-makers should base their selection on their own SLA and concurrency model, rather than on a single benchmark result. Mingxin supports an approximately 10-week gated joint testing cycle (from G1 arrival acceptance to G4 stability verification), allowing validation of the above metrics under real workloads to determine fit for your scenario.

Key Questions and Answers

Q: What are the measured benefits of NVMe-oF over local NVMe in inference scenarios?
A: Under 480B long-context workloads, tiered KV acceleration improves throughput by +29–40% (measured, reports R2/R3) and reduces TTFT by 26–32% (measured, report R2). Benefits are concentrated in multi-GPU concurrency and cold read scenarios; advantages narrow under low concurrency and short context.

Q: What are the applicability boundaries of NVMe-oF?
A: Three constraint dimensions: concurrency level (network overhead share increases at low concurrency), context length (KV spill is a prerequisite), and workload type (checkpoint and model loading show the most significant benefits). For single-GPU, low-concurrency, short-context scenarios, local NVMe remains the reasonable default.

Q: How can one verify whether NVMe-oF fits their specific scenario?
A: First define the SLA and concurrency pattern, then estimate KV Cache spill volume. Mingxin supports an approximately 10-week gated joint testing cycle (from G1 arrival acceptance to G4 stability verification), allowing metric suitability to be validated under real workloads.

References

1. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
2. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
3. RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
4. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

---

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).