Checkpointing is the least glamorous part of a training cluster and the one that most reliably decides its effective utilisation. Everyone budgets for the write bandwidth of a clean checkpoint. Almost nobody budgets for the correlated burst: hundreds of ranks deciding to flush state in the same second, onto a shared pool that was sized from an average, while the fabric is still carrying gradient traffic.
Local SSD gives each node a private, predictable write path. The failure domain is small, the latency is stable, and a restart needs a reshuffle or a recovery-from-peer protocol. The cost is that drives sit idle most of the time, replicated across every node, and a node failure can take its slice of the checkpoint with it unless the framework writes across neighbours.
Checkpoint storage is not a capacity purchase; it is an insurance contract whose premium is paid in stolen training time.
Part One: What is actually being compared
To compare disaggregated NVMe-oF checkpoint pool with node-local NVMe checkpointing honestly you first have to name the quantity being optimised. In large training cluster operations it is tempting to use the headline specification, but the binding quantity is the one that appears in the bill after a full year of operation: the sum of the committed cost, the per-unit cost and the cost of the occasions when the system behaves badly. Once that is written down, most of the specification-sheet argument simply disappears.
The cost structure behind disaggregated NVMe-oF checkpoint pool is dominated by a fixed commitment that must be paid before the first unit of value appears, while node-local NVMe checkpointing converts that commitment into a variable charge on every unit. That difference is invisible in a per-unit comparison and decisive in a budget, because a fixed cost rewards scale and a variable cost punishes it. Whichever side your volume forecast lands on determines the answer long before the specification sheet does.
Part Two: The arithmetic that decides it
A disaggregated NVMe-oF pool inverts that trade. Capacity is pooled and utilisation is high, so the per-terabyte bill falls, and a failed node leaves its state safely in the pool. But the write path now crosses the fabric, and the fabric is shared with the collective operations that keep the training step synchronised. Writing a checkpoint into a congested network does not just take longer; it slows the step that is running concurrently.
The second-order consequence is organisational. Choosing disaggregated NVMe-oF checkpoint pool creates a dependency on a capability that must be staffed, monitored and kept current; choosing node-local NVMe checkpointing keeps the capability outside the team and turns it into a vendor relationship. Neither is free, and the one that looks cheaper in a spreadsheet is frequently the one that adds a standing meeting to the calendar.
Part Three: Where the simple model breaks
The arithmetic that matters is expected lost work. If the mean time between interruptions is T and the checkpoint interval is C, the average compute thrown away per failure is roughly C over 2, plus the write overhead of every checkpoint. Shortening C reduces lost work but increases steady-state overhead, and a shared pool makes each of those writes more expensive and less predictable. The optimal interval on a shared pool is therefore longer than on local drives, and the cluster loses more work per failure.
A 0.4 TB/s shared pool serving 512 ranks at once turns a nominal 22-second checkpoint into a tail stall that can outlast the job's own timeout.
Part Four: The cost that arrives later
There is a subtle failure mode in the other direction. When many ranks checkpoint simultaneously, the pool's internal garbage collection and metadata layer become the bottleneck, and tail latency stretches far beyond the mean. A checkpoint system measured on the 50th percentile will look healthy right up to the day a 99th-percentile stall coincides with a fabric pause and the job's timeout fires.
Expect a split by scale rather than by industry. Large operators can amortise the fixed side of disaggregated NVMe-oF checkpoint pool and will move first; smaller ones keep node-local NVMe checkpointing because their utilisation is spiky and their tolerance for operational complexity is low. The interesting competition happens in the middle, where a managed service can make either option look like the other.
Part Five: What this changes outside the engineering team
The best deployments split the two. Incremental, frequent snapshots go to local drives where latency is stable; a less frequent consolidated snapshot goes to the shared pool where capacity is cheap. That is a tiering decision, not a vendor decision, and it is worth more than any single drive specification.
The everyday version is the autosave indicator on a document. When it flickers at the wrong moment the whole machine hesitates, and you learn to save manually before anything risky. Training clusters have the same habit: engineers schedule checkpoints away from evaluation steps because they know the write will steal time from the computation.
A market sizing sanity check
AI cluster storage is a 20 to 30 billion dollar annual market by the end of the decade, and the checkpoint tier is perhaps a tenth of that. If fabric-attached flash takes a third of new large-cluster builds, the controller, tiering and burst-buffering software around it is a 0.8 to 1.5 billion dollar annual business by 2029, sold mostly to customers who already bought the drives.
Treat the numbers as a scale model. They are built to show where the curves cross and which term dominates, and they deliberately round the inputs that vary most between deployments. The ranking is robust to those rounds; the exact threshold is not.
The comparison that matters
Local SSD optimises for predictable write latency and a small failure domain. A disaggregated pool optimises for capacity utilisation and clean recovery, and charges for it in correlated tail latency. Choose by computing expected lost work per failure, not by comparing dollars per terabyte.
The honest caveat is that this comparison is sensitive to one assumption: that the workload keeps its current shape. Shift the mix toward the case where node-local NVMe checkpointing is strong and the result changes without any technology changing at all. That is why the durable output of an evaluation is not a verdict but the threshold at which the verdict flips.
The cheapest terabyte is the one that is unavailable exactly when every rank wants it in the same second.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)