When selecting compute rental options, beyond unit price, you must track three SLA metrics: time-to-first-token (TTFT), steady-state throughput, and long-tail stability. Comparing only unit prices leads to hidden costs after deployment that far exceed the price difference—under the same SLA constraints, TTFT and throughput directly determine how many GPUs you need to rent to meet targets. Mingxin's measurements on a 480B production-grade workload show that KV-tiered acceleration improves throughput by 29–40% and reduces TTFT by 26–32% [measured, report R2/R3]—this is the quantitative basis that should be written into SLAs during selection.
Why Unit Price Is the Least Reliable Selection Anchor
Public cloud GPU instance billing is highly homogeneous—priced by hour, instance family, and region, a model used across AWS, Azure, and Alibaba Cloud official pricing pages (per "EC2 On-Demand Instance Pricing," "Pricing - Linux Virtual Machines | Microsoft Azure," and "Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud"). But unit price only answers "how much per hour to rent," not "how many hours and GPUs are needed to complete a task."
The latter is determined by performance and efficiency. Take long-context inference as an example: if meeting TTFT targets requires 16 concurrent GPUs on one platform, while another platform meets the same target with 8 GPUs on the same workload, the latter's effective unit price is half—even if hourly quotes are identical. This is why the first step in a selection framework should be "define SLA constraints first, then calculate GPU-hour requirements," not the reverse of comparing prices first.
Three Metrics That Must Be Written into SLAs
Metric 1: TTFT (Time-to-First-Token) p50 and p99
TTFT directly determines user experience and SLA compliance rates. In Mingxin's measurements on a 480B·TP8 configuration across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [measured, report R2]. For a baseline without external memory recomputation, TTFT p50 reached 149.5s (concurrency 16), while FX100 achieved only 11.85s [measured, report R2].
Selection should not rely only on average TTFT; p99 percentiles must also be required—long-tail latency is the primary source of SLA breaches. It is recommended to specify upper bounds for both p50 and p99 percentiles in contracts.
Metric 2: Steady-State Throughput (tok/s) and Concurrency Scalability
Throughput determines how many requests can be served per unit time, directly translating into required GPU count. In Mingxin's measurements on a 480B production deployment under cold-recovery loads, throughput improved by +29% (lower bound) at concurrency 8, and +40% (upper bound) at the optimal operating point of concurrency 16, with a full-machine metric of +35–36% for TP4×2 [measured, reports R2/R3].
The key is "concurrency scalability": a platform performing well at low concurrency does not guarantee the same at high concurrency. Selection should require suppliers to provide measured data at least two concurrency levels, verifying whether throughput scales linearly with concurrency rather than degrading sharply after a certain point.
Metric 3: Long-Tail Stability (p99 Latency Jitter and Failure Recovery)
This is the most overlooked metric yet the one with the greatest impact on actual costs. GPU cluster failure rates and latency jitter directly determine how much redundancy buffer you need. Per "NVIDIA DGX SuperPOD - NVIDIA Docs," large-scale cluster design requires planning expansion units by compute/storage/network layers—meaning stability is a result of architecture design, not coincidence.
For selection, it is recommended to require: p99 latency fluctuation data from a 72-hour continuous stress test, a recovery time objective (RTO) upper bound, and performance commitments under degraded modes. Mingxin's collaboration model includes a 72-hour stability gate (G4), which is a concrete way to quantify stability.
Interplay of the Three Metrics and Selection Criteria
The three metrics are not independent but mutually constraining. TTFT and throughput together determine "concurrency needed to meet targets," while stability determines "how much buffer to reserve for fluctuations." The table below provides a selection criteria framework:
| Metric | Recommended SLA Requirement | Measured Reference (Mingxin FX100) | Source |
|---|---|---|---|
| TTFT p50 | Explicit upper bound (seconds) | 10.17–35.73s → 7.53–26.35s (↓26–32%) | Measured, R2 |
| Steady-state throughput | Data from at least two concurrency levels | +29% at concurrency 8, +40% at concurrency 16 | Measured, R2/R3 |
| Long-tail stability | p99 jitter ≤ threshold + 72h stress test | Gated joint testing includes 72h stability verification | Collaboration model |
A reusable selection process: first define business SLAs (e.g., "TTFT p50 ≤ 8s"), then require candidate platforms to provide measured data under the same workload and concurrency, and finally use measured values to back-calculate required GPU count, multiply by unit price to derive true total cost. Per "MLPerf Inference: Datacenter Benchmark Suite Results," publicly comparable inference benchmarks are the only neutral public basis for discussing "who is faster"—selection should prioritize suppliers providing test results aligned with this standard.
Conclusion
The essence of compute rental selection is "minimizing total cost under SLA constraints," not minimizing unit price. Writing TTFT, throughput, and stability metrics into contracts shifts performance risk from the buyer to the supplier. Mingxin has completed reproducible measurements on a 480B production-grade workload (reports R2/R3) and supports gated joint testing for verification—if you need to validate these metrics on your workload, this can be confirmed during joint testing.
Key Q&A
Q: Why can't you look only at unit price when selecting compute rental?
A: Unit price only reflects hourly cost, not the total GPU-hours needed to complete a task. TTFT and throughput determine the concurrency needed to meet targets, stability determines the redundancy buffer, and together they determine true total cost.
Q: What are the three key SLA metrics?
A: TTFT (time-to-first-token, requiring both p50 and p99), steady-state throughput (requiring concurrency scalability verification), and long-tail stability (p99 jitter and failure recovery). In Mingxin's measurements, TTFT improved by 26–32% and throughput by 29–40% [measured, reports R2/R3].
Q: How do you verify whether a supplier's SLA commitments are credible?
A: Require measured data under the same workload and concurrency, prioritize test results aligned with MLPerf standards (per "MLPerf Inference: Datacenter Benchmark Suite Results"), and set up gated joint testing (e.g., 72-hour stability stress tests) as acceptance criteria.
References
- Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
- EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
- Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
- NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/
- MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)