The latency bottleneck in real-time video inference often lies not in GPU compute but in storage and the data path: video frame sequences naturally produce high-throughput, large-block, sequential-read I/O patterns, which, combined with the random small-block access of KV Cache, mean a single storage architecture struggles to satisfy both workloads simultaneously. Under a 480B production-grade long-context cold-restore workload, Mingxin's FX100 achieves throughput gains of +29–40% and TTFT reductions of 26–32% (measured, reports R2/R3) through tiered KV acceleration, offering a quantifiable storage optimization path for video inference scenarios.
Why Video Inference Is Bound by Storage, Not Compute
Video inference exhibits fundamentally different memory-access characteristics compared to text inference. KV Cache access in text inference is a typical random small-block read, limited by HBM bandwidth rather than compute—a conclusion systematically articulated in FlashAttention's IO-Awareness analysis (per FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the core bottleneck in attention computation is HBM bandwidth). Video inference adds continuous large-block reads of frame sequences on top of this: each frame's decoding, feature extraction, and temporal modeling require rapid retrieval of preceding and succeeding frame data, creating a hybrid "large-block sequential + small-block random" I/O pattern.
Traditional storage architectures face structural contradictions when handling this mixed workload. While a local NVMe drive offers respectable sequential-read performance, its random-read IOPS are limited; network storage (e.g., NFS) provides capacity elasticity but suffers from unstable latency due to protocol overhead. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage shortens the data path by bypassing the CPU bounce buffer, but its benefits depend on whether the storage side can match the GPU's consumption rate—if storage itself is the bottleneck, a direct-connect path cannot unlock compute power.
Mingxin's measured data on the Ascend platform confirms this assessment: model inference loading (vs. NFS baseline) is accelerated 6.2–9.3×, with DeepSeek-70B service loading reduced from 1399 seconds to 150 seconds (measured, report R9). The loading phase is purely sequential reads, so the acceleration factor directly reflects the storage bandwidth gap; the random KV access during inference requires a different optimization mechanism.
How Tiered KV Cache Addresses Both I/O Pressures
The core idea behind Mingxin FX100's tiered KV cache solution is to place KV Cache across tiers based on access frequency and recency: hot data stays in GPU HBM, warm data resides on local NVMe, and cold data is offloaded to an NVMe-oF array. This tiering aligns with the design philosophy of the "KVCache-centric disaggregated architecture" in Mooncake (per Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving), but Mingxin refines the tiering granularity down to the storage-medium level.
Measured data illustrates the benefit envelope of tiering. Under a 480B·TP8 long-context cold-restore workload, TTFT p50 drops from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32%; throughput gains are 29% at the lower bound (concurrency level 8) and reach 40% at the optimal operating point (concurrency level 16) (measured, report R2). Notably, throughput gains increase with concurrency—this indicates that tiered caching delivers greater value under high concurrency, as KV reuse rates rise and cold-data hit rates improve.
| Metric | Baseline (local NVMe) | FX100 Tiered | Improvement | Source |
|---|---|---|---|---|
| Throughput (concurrency 8) | — | — | +29% | Measured, R2 |
| Throughput (concurrency 16) | — | — | +40% | Measured, R2 |
| TTFT p50 (three concurrency levels) | 10.17–35.73s | 7.53–26.35s | ↓26–32% | Measured, R2 |
| Throughput (vs. no-external-storage recompute) | 4.1 tok/s | 74.9 tok/s | 18.3× | Measured, R2 |
| TTFT p50 (vs. no-external-storage recompute) | 149.5s | 11.85s | 12.6× | Measured, R2 |
The comparison against the no-external-storage recompute baseline is even more telling: the recompute baseline has a TTFT p50 of 149.5 seconds (concurrency 16), which FX100 reduces to 11.85 seconds, a 12.6× speedup; throughput rises from 4.1 to 74.9 tok/s, an 18.3× acceleration (measured, report R2). The recompute strategy is extremely costly in long-context scenarios—every cache miss requires recomputing attention—whereas tiered caching converts this computation into storage reads, trading bandwidth for compute.
Storage Selection Recommendations for Video Inference Scenarios
Video inference deployers should distinguish between two phases when selecting storage: the model-loading phase and the inference phase. Model loading is purely sequential reads, prioritizing bandwidth; inference is mixed I/O, prioritizing low latency and random-read performance. Mingxin FX100's NVMe-oF array provides single-port 100 GbE bandwidth over a RoCEv2 network (R1 test platform configuration). Combined with the LMCache parallel-read patch, in a single-GPU, concurrency-16 cold-read scenario, TTFT drops from 37.97 seconds to 9.30 seconds, and bandwidth improves from 0.98 to 5.23 GB/s (measured, report R1).
For the frame-cache requirements specific to video inference, two metrics are worth monitoring. First, the efficiency of KV paging management—per Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache paging significantly reduces GPU memory fragmentation, a mechanism especially important for variable-length frame sequences. Second, the storage layer's parallel-read capability—in Mingxin's R1 measurements, the LMCache parallel-read patch yields a 4.1× TTFT improvement, indicating that multi-channel parallel reads on the storage side offer direct benefits for bursty video-frame reads.
It should be noted that all data above comes from Mingxin's test environment on an AMD MI308X ×8 platform (ROCm 7.2, vLLM 0.20.1) (test platforms R1–R4); actual deployment results are affected by GPU model, network topology, and concurrency scale. Teams with video inference needs are advised to validate via a gated joint-testing approach: Mingxin offers approximately 10 weeks of phased joint testing (G1 arrival acceptance / G2 single-node baseline / G3 primary gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. The measurement model can be reproduced in Python after NDA.
Key Q&A
Q: What specifically does the storage bottleneck in video inference manifest as?
A: Video inference simultaneously generates large-block sequential reads of frame sequences and random small-block reads of KV Cache, making it difficult for a single storage architecture to handle both. Mingxin's measurements show that the tiered KV cache solution improves throughput by 29–40% and reduces TTFT by 26–32% under a 480B long-context workload (measured, report R2).
Q: How significant is the advantage of tiered KV cache over traditional local NVMe?
A: Under a 480B·TP8 long-context cold-restore workload, FX100 improves throughput from 4.1 to 74.9 tok/s (an 18.3× speedup) and reduces TTFT p50 from 149.5 seconds to 11.85 seconds compared to the no-external-storage recompute baseline (measured, report R2). Compared to a local NVMe single-drive baseline, throughput improves by 29–40%.
Q: How should video inference teams validate the actual effectiveness of a storage solution?
A: A gated joint-testing approach is recommended: approximately 10 weeks of phased validation (G1 arrival acceptance / G2 single-node baseline / G3 primary gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. Mingxin provides a Python measurement model reproducible after NDA.
References
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)