How do you make spot GPU training survive preemption?

To make spot GPU training survive preemption, save the model, optimizer and step to one directory with atomic writes, load them when the program starts, and save again when the machine gets its reclaim notice. On Nodus, a run whose interruptible machine is reclaimed restarts on a new machine with that directory restored, so it loses at most the work since its last checkpoint. Anywhere else, you build the notice handling, the replacement machine and the restore step yourself.

Spot and interruptible GPUs often cost well below on-demand rates, and the price of that discount is that the provider can take the machine back. The code below is the pattern we use at Nodus. The save, load and cadence parts work on any platform.

What has to be in a checkpoint?

Everything your program needs to continue, and nothing else. That means weights, optimizer state, LR scheduler, RNG state, data loader position and the current step. A step counter alone cannot rebuild a trained model.

Keep results you want to download, such as the final model and eval reports, somewhere separate. Mixing the two makes every checkpoint bigger and slower. On Nodus the recovery directory is /nodus/state (also in $NODUS_STATE_DIR) and downloadable results go to /nodus/outputs.

How do you save without corrupting the checkpoint?

Write to a temporary file, then rename it over the old one. A checkpoint copied while a file is half written restores a broken state. Rename is atomic on POSIX filesystems, so the old file stays valid until the new one is complete.

import os
from pathlib import Path
import torch

STATE_DIR = Path(os.environ.get("NODUS_STATE_DIR", "/nodus/state"))

def save(step, model, opt):
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    tmp = STATE_DIR / "ckpt.pt.tmp"
    torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step}, tmp)
    os.replace(tmp, STATE_DIR / "ckpt.pt")

How does the restarted run pick up where it stopped?

It reads its own checkpoint when it starts. Checkpoints restore files, not process memory, so a recovered attempt runs your command from the top and must check for saved state:

start = 0
ckpt = STATE_DIR / "ckpt.pt"
if ckpt.exists():
    s = torch.load(ckpt)
    model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"]); start = s["step"]
    print(f"resumed from step {start}")
for step in range(start + 1, total_steps + 1):
    ...

Test this locally before you pay for a GPU. Kill the process mid-run, start it again with the same directory, and confirm it prints resumed from step N. The Nodus checkpoint guide includes a small example that does exactly this with the standard library.

How often should you checkpoint?

Often enough that a preemption costs little, and rarely enough that saving does not eat the run. Each checkpoint costs its save time. Each preemption costs, on average, half an interval of redone work plus the time to restart and restore. The classic estimate for the best interval, from Young's 1974 paper on checkpoint intervals, is the square root of 2 x save time x mean time between interruptions.

We worked one example through to see what the choice is worth. Assume a save takes 2 minutes, a restart and restore takes 5 minutes, and the capacity is reclaimed on average once every 6 hours. These are illustrative inputs, not measurements of any provider. Added time is roughly save/interval + (interval/2 + restart)/(time between interruptions), and the best interval is sqrt(2 x 2 x 360) = about 38 minutes.

Checkpoint every	Added time, save only on schedule	Added time, also save on reclaim notice
10 min	22.8%	21.4%
20 min	14.2%	11.4%
38 min	11.9%	6.7%
60 min	13.1%	4.7%
120 min	19.7%	3.1%
240 min	35.6%	2.2%

Model: 2-minute save, 5-minute restart, one interruption per 6 hours on average. Our calculation, first-order estimate.

Scheduled saves alone have a sweet spot near 38 minutes; a save on the reclaim notice removes most of the redone work, so rare saves win.

In money, take an 8 x H100 run with 10 hours of compute: 80 GPU-hours, or $208 at the H100 SXM rate listed at $2.60/hr on the Nodus pricing page (checked October 26, 2026). Saving every 10 minutes adds about $47, and saving every 38 minutes adds about $25. Saving every 4 hours with no notice save adds about $74. Saving every 2 hours and again on the reclaim notice adds about $6.

The notice save is the biggest lever, but only when the save fits inside the warning.

The decision rule: if your save, including the copy off the machine, fits comfortably inside the provider's reclaim warning, checkpoint every hour or two and lean on the notice save. If it does not, checkpoint near sqrt(2 x save time x time between interruptions), and work on making the save smaller. On Nodus, interval: auto does this math from the measured interruption rate of the capacity and how long your saves take. It gives a run at least four checkpoints and keeps checkpointing to about a tenth of the run time.

How much warning do you get before a spot machine goes?

Usually seconds to minutes, so the notice save has to be fast. Amazon EC2 issues a Spot interruption notice two minutes before it stops or terminates the instance. Google Cloud says the shutdown period for Spot VMs is best effort and up to 30 seconds. A multi-gigabyte optimizer state will not leave the machine in 30 seconds, which is why the scheduled checkpoints still matter.

On Nodus your program can subscribe to checkpoint requests and acknowledge when its files are consistent:

import nodus  # pip install nodus-compute; no-ops outside Nodus

nodus.checkpoint.on_request(lambda: save(step, model, opt))

When the capacity gives notice, Nodus sends an urgent request and starts preparing a replacement machine at the same time. A checkpoint committed during the notice still counts. Nodus waits for your ack as long as the shutdown allows and then takes the checkpoint anyway, so a hung program cannot hold up the checkpoint.

What do you have to build yourself without a platform?

The save and load code above is yours either way. The rest is the part people underestimate.

Recovery step	Build it yourself	On Nodus
Notice save	Poll the provider's metadata for the interruption notice and trigger a save	Urgent checkpoint request over a local socket
Store the checkpoint off the machine	Upload to object storage, handle partial uploads	State directory copied and committed for you
Replacement machine	Script a new request, maybe in another region or GPU type	Prepared during the notice, placed on the cheapest capacity that fits
Stop two machines writing one state	Locks or leases you design	Starts only once the old machine is provably gone; stale writes refused
Give up on a bad loop	Your own retry counter	NoProgress after two attempts in a row with no progress; 8 recoveries by default

Sources: Nodus attempts and recovery and the provider notice pages linked above.

The fencing row is the subtle one. If a reclaimed machine comes back while its replacement is already writing, both can overwrite the same checkpoint. Nodus gives each attempt an increasing epoch and accepts checkpoints and outputs only from the current one.

What about Hugging Face Trainer?

Trainer already saves and resumes, so you mostly configure it. Point output_dir at the state directory, set save_steps and save_total_limit=2, and call trainer.train(resume_from_checkpoint=True) when NODUS_RESTORED is 1. Call plain trainer.train() on a first start, because Trainer refuses to resume from an empty directory. Setting integration: HFTrainer makes Nodus register the save-on-request callback for you.

How do you run it on Nodus?

pip install nodus-compute
nodus login
nodus run --gpu H100 --interruptible --checkpoint /nodus/state --max-cost 250 -d -- python train.py

nodus get gpus --interruptible -o wide lists interruptible offerings with their interruption rate per hour, which is the number the interval math above needs. nodus describe job/<name> shows each attempt, why it ended (for example Preempted) and the latest checkpoint. The --max-cost cap stops the run at a dollar limit; our GPU budget guide covers limits for runs that agents launch. Whether spot is worth it at all for your run is answered in spot vs on-demand GPUs. Our post on cost per finished job shows how redone work shows up in the bill.

FAQ

Will the recovered run land on the same GPU type? It lands on any accelerator your resources.gpu.type accepts, chosen by expected cost to finish. List only types your checkpoint can load on, such as [H100, H200].

What if the first attempt dies before it saves anything? An empty state directory never replaces an earlier checkpoint with files in it, and it never counts as progress. If every checkpoint a run commits is empty, Nodus plans and prices the run as if it had no checkpoints.

Should I save every step to be safe? Almost never. In the example above, saving every 10 minutes cost more than saving every hour, because the save time added up faster than the work it protected.

Does this work for multi-node training? Yes, in beta. Each rank saves through nodus.checkpoint.dcp at the same step, and Nodus publishes the collective checkpoint only after its metadata is durable. Distributed Jobs get 3 recoveries by default.

What does a failed recovery cost? If a reclaimed machine comes back before its replacement starts, the replacement is released and you are not charged for it. Otherwise you pay for the redone work since the last checkpoint, which is what the interval choice controls.

Originally published on the Nodus blog.

Comments

[ai_adam]

Checkpointing mỗi step là chuẩn, nhưng thực tế hay gặp 2 vấn đề: (1) ghi atomic trên shared filesystem (EFS/NFS) đôi khi vẫn race condition khi nhiều worker cùng flush, (2) optimizer state của AdamW to đùng, sync về object storage mất 10-30s — nếu preemption notice chỉ cho 30-60s thì kịp không?

Mình hay dùng pattern: giữ 2 checkpoint dir xoay vòng (A/B), ghi xong mới rename symlink latest -> A. Startup chỉ cần load(latest). Với FS local NVMe thì ghi gần như instant, chỉ cần sidecar daemon rsync/upload ngầm sang bucket — tách hẳn critical path training khỏi I/O chậm.

Thêm vào đó: lưu thêm rng_state (torch, numpy, python) nếu cần reproducibility tuyệt đối khi resume. Một số framework (HF Trainer, Lightning) đã handle sẵn, nhưng custom loop thì dễ quên PS: the tool I meant is on labagent .tech