The core tension in GPU compute leasing is straightforward: compute is billed by the hour, yet the speed at which data moves between storage and GPUs often becomes the bottleneck constraining effective compute output. Improving data transfer efficiency directly impacts the actual utilization of leased compute and the latency performance of inference services. Based on measured data from Mingxin's proprietary test platform, combined with public protocol specifications and cloud vendor billing practices, this article outlines three verifiable optimization paths: storage architecture, network protocols, and cache tiering.
Why Data Transfer Becomes a Hidden Cost in Compute Leasing
Public cloud GPU instances are typically billed by the hour. According to Amazon Web Services' EC2 On-Demand pricing page, billing is based on instance family and hourly usage. This means every second a GPU spends waiting for data incurs cost without generating compute value. Microsoft Azure's Linux virtual machine pricing page shows billing models that differentiate between on-demand, reserved, and spot modes, with significant regional and instance-spec variations. Regardless of the billing model chosen, data transfer efficiency directly affects the actual output per unit of compute cost.
In model inference and training scenarios, the data transfer bottleneck is usually not network bandwidth itself, but rather the random read performance of the storage system and the overhead of the protocol stack. When model weights, KV Cache, or training checkpoints need to be loaded from remote storage, high storage path latency causes GPUs to frequently enter waiting states. According to public research from Epoch AI, AI compute scale and costs continue to grow steadily, meaning the optimization headroom in data transfer efficiency will only carry more weight in total cost of ownership.
Storage Architecture Optimization: From Network File Systems to NVMe-oF All-Flash Arrays
Traditional compute clusters often use Network File Systems (NFS) as shared storage, where protocol stack overhead is particularly pronounced in high-concurrency random read scenarios. Mingxin's measurements on the Huawei Atlas 910B platform show that after switching the model serving load path from NFS to an FX100 all-flash NVMe-oF array, DeepSeek-32B service load time dropped from 691 seconds to 112 seconds, and DeepSeek-70B from 1399 seconds to 150 seconds—speedups of 6.2x and 9.3x respectively [source: measured, report R9]. This comparison demonstrates that storage medium random read performance and protocol path simplicity have significant impact on read-dominated workloads like model loading.
The essence of NVMe-oF (NVMe over Fabrics) is extending the NVMe command set to the network transport layer. As defined by RFC 5040, the RDMA (Remote Direct Memory Access) protocol allows data to transfer directly between network interface cards and memory, bypassing the operating system's network protocol stack, thereby reducing CPU involvement and transfer latency. NVMe-oF is a storage access protocol built on RDMA semantics, designed to preserve NVMe device semantics while enabling low-latency cross-node access. In contrast, traditional TCP transmission requires multiple copies and interrupt handling through the kernel protocol stack. Per RFC 9293's specification of TCP transmission semantics, TCP is designed for reliable byte-stream transport and is not specifically optimized for low-latency storage access scenarios.
Cache Tiering: Bringing Hot Data Closer to the GPU
Even when the storage system itself has high throughput capacity, latency-sensitive model inference workloads still need a cache layer close to the GPU to absorb access hotspots. KV Cache is critical data that grows dynamically with requests during LLM inference, and its access pattern exhibits clear locality—subsequent token generation for the same sequence requires repeated reads of previous tokens' key-value states.
Mingxin conducted long-context inference measurements on an 8-GPU AMD Instinct MI308X platform using the Qwen3-Coder-480B-FP8 model (MoE architecture, approximately 450GB of weights). Results showed that with KV tiered acceleration, inference throughput improvement fell within the 29% to 40% range: 29% improvement at concurrency level 8 (lower bound), 40% at concurrency level 16 (upper bound), and 35% to 36% at the full-machine TP4×2 level [source: measured, reports R2/R3]. Time-to-first-token (TTFT) at 480B·TP8 with three concurrency levels dropped from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26% to 32% [source: measured, report R2].
The core logic of cache tiering is placing the most frequently accessed KV data in the storage tier closest to GPU compute units (such as local high-speed disks), while cold data falls back to the remote storage pool. The effectiveness of this tiering strategy depends on accurately identifying hot data and dynamically adjusting placement policies. In joint testing with LMCache, Mingxin observed that a parallel read patch improved TTFT by 4.1x in cold-read disk scenarios: with single-GPU concurrency of 16 and cold reads from disk, TTFT dropped from 37.97 seconds to 9.30 seconds, and bandwidth increased from 0.98 GB/s to 5.23 GB/s [source: measured, report R1]. This data indicates that cache layer read efficiency optimization still has order-of-magnitude improvement headroom.
Applicability Boundaries and Selection Criteria for Optimization Methods
These three paths are not mutually exclusive; real deployments require combining them based on workload characteristics. Storage architecture replacement (e.g., migrating from NFS to NVMe-oF) suits large-file sequential read/write scenarios such as model loading and checkpoint saving; cache tiering suits KV-access-intensive scenarios like long-context inference and multi-turn dialogue; network protocol selection determines the performance ceiling of the other two.
Selection requires clarifying constraints. According to Alibaba Cloud's official GPU instance family classification page, different instance families are explicitly categorized for AI training, HPC, and graphics rendering scenarios. When evaluating optimization options, compute leasing providers should first determine their workload's SLA requirements (such as TTFT upper bounds), context length, and concurrency patterns, then select storage and cache solutions accordingly. Mingxin employs an approximately 10-week gate-based joint testing process—from arrival acceptance and single-node baseline to the main gate (requiring TTFT reduction of no less than 25% and throughput improvement within the measured 29%–40% band)—using reproducible test data as the basis for decisions, avoiding bias from experience-based estimation.
Key Questions and Answers
Q: What is the most direct lever for optimizing data transfer efficiency in compute leasing?
A: Three paths: migrating storage architecture from NFS to NVMe-oF all-flash arrays, cache tiering to bring hot data closer to the GPU, and selecting RDMA paths for network protocols. Mingxin's measurements show model loading accelerates by 6.2x to 9.3x when switching from NFS to the FX100 array [measured, report R9].
Q: What are the measured results of KV Cache tiered acceleration?
A: In 480B model long-context inference, throughput improved by 29% to 40% (concurrency levels 8 to 16), and TTFT decreased by 26% to 32% [measured, reports R2/R3]. Against a baseline without external memory recomputation, speedup reached 8.6x to 20x [measured, report R2].
Q: How can the effectiveness of optimization solutions be verified?
A: Gate-based joint testing is recommended: set quantitative targets for TTFT reduction and throughput improvement, and conduct phased acceptance under real workloads. Mingxin's main gate requires TTFT reduction of no less than 25% and throughput improvement within the measured 29%–40% band; failure to meet these thresholds triggers immediate stop-loss.
References
- Epoch AI — https://epoch.ai/
- RFC 9293: Transmission Control Protocol (TCP) — https://datatracker.ietf.org/doc/html/rfc9293
- RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
- EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
- Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
- Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)