How often are spot GPU instances interrupted in practice?

Spot GPU interruptions depend more on the GPU and region than on the cloud: in AWS's own data, 85% of H100 pools lost under 5% of instances in the past month, while roughly half of T4, A10G, L4 and L40S pools lost over 20%. Nodus shows an hourly interruption rate for every interruptible GPU offering and restores a reclaimed run from its last checkpoint, so an interruption costs minutes of work instead of the whole run.

We pulled the numbers below from AWS's public advisor data file on October 30, 2026 and tallied every Linux listing (one instance size in one Region) for each GPU family. The rates move every month; the method to check them yourself does not.

What do the published interruption rates actually measure?

Each cloud publishes a different number, so a "5%" on one is not a "5%" on another. Read the unit before you compare.

Cloud	What it publishes	Unit and window
Amazon EC2	Spot Instance Advisor band per instance type and Region: under 5%, 5-10%, 10-15%, 15-20%, over 20%	Share of capacity reclaimed over the trailing month
Google Compute Engine	Historical preemption rate per machine type and zone	Daily: preempted Spot VMs divided by all Spot VMs that stopped that day, last 30 days
Azure Virtual Machines	Eviction rate per VM size and region, shown in the portal and Resource Graph	Chance of eviction within the next hour, based on the last 7 days

Sources: the linked AWS, Google Cloud and Microsoft pages; the AWS trailing-month definition comes from the EC2 Spot team's public AMA. Checked October 30, 2026.

The Google number is the easiest to misread. A rate of 0.50 means half of the Spot VMs that stopped that day were preempted rather than stopped by their owners, not that your VM has a 50% chance of dying. Google's Spot VM docs only say the probability of preemption "is generally low, but might vary from day to day and from zone to zone". Azure's hourly figure is the only one you can plug straight into a run-length calculation.

AWS states that the average interruption frequency across all Regions and instance types "has historically been <5%". Across all 19,353 Linux listings in the data file, 35% sat in the under-5% band and 25% in the over-20% band. GPU instances do not follow the average.

Which spot GPUs get interrupted most on AWS?

The big 8-GPU training instances were the calmest and the small inference GPUs were the busiest. Here is our tally from the advisor data, Linux, all 34 Regions in the file.

Family (GPU)	Listings	<5%	5-10%	10-15%	15-20%	>20%	Share <5%	Share >20%
p5 (H100)	20	17	1	1	0	1	85%	5%
p5en (H200)	10	8	2	0	0	0	80%	0%
p6-b200 (B200)	3	0	1	2	0	0	0%	0%
p4d (A100 40GB)	13	6	1	1	1	4	46%	31%
p4de (A100 80GB)	6	2	0	1	1	2	33%	33%
g6e (L40S)	72	8	6	9	8	41	11%	57%
g6 (L4)	136	27	15	24	13	57	20%	42%
g5 (A10G)	136	44	6	7	9	70	32%	51%
g4dn (T4)	147	23	13	15	20	76	16%	52%
p3 (V100)	33	0	0	0	0	33	0%	100%

Source: AWS Spot Instance Advisor data file, fetched and tallied by us on October 30, 2026. A listing is one instance size in one Region.

On AWS, the 8-GPU H100 and H200 pools were mostly calm; the single-GPU inference families were mostly in the worst band.

Two cautions keep this honest. First, a band is a monthly share across everyone's instances, so a calm pool can still reclaim yours tomorrow; AWS says the actual rate "will depend on point-in-time available capacity". Second, the B200 pool shows how small samples mislead: three listings, none in the calmest band, which says little about next month. The useful rule is narrower: check the band for the exact instance and Region you plan to use, on the day you launch.

What does an hourly rate mean for a 10-hour run?

Turn the hourly rate into the chance your run sees at least one interruption. If each hour has an independent chance p of eviction, a run of h hours survives untouched with probability (1 - p) to the power h.

For a 10-hour run on Azure-style hourly rates:

• At 5% per hour, 0.95^10 = 0.60, so a 40% chance of at least one eviction.
• At 10% per hour, 0.90^10 = 0.35, so a 65% chance.
• At 20% per hour, 0.80^10 = 0.11, so an 89% chance.

Real evictions cluster when demand spikes, so treat these as rough. The takeaway still holds: on a long run, "low" hourly rates still make an interruption likely. Plan for it rather than hope against it. This is also why a monthly AWS band cannot be converted into an hourly chance; it tells you which pools are calmer, not when your instance will go.

What does an interruption actually cost you?

An interruption costs the work since your last checkpoint plus the time to start again, and nothing else. Whether spot still pays off depends on the discount and on how much work you redo.

Take a 10-hour job on 8 x H100 at the 57% spot saving the advisor listed for p5.48xlarge in us-east-1 on October 30, 2026. Call the on-demand cost 100, so spot hours cost 43% of on-demand. Assume two reclaims and 10 minutes of billed restart time each (pulling the image and loading state). These assumptions are ours, picked to make the math easy to follow.

• On demand: 10 hours, cost 100.
• Spot with checkpoints every 30 minutes: each reclaim loses about 20 minutes of work plus the 10-minute restart, so 11 paid hours. Cost 11 x 4.3 = 47.3.
• Spot with no checkpoints: the first attempt dies after 4 hours, the second after 8, the third finishes in 10, plus two restarts. That is 22.33 paid hours. Cost 22.33 x 4.3 = 96.0.

With two reclaims, checkpoints keep spot at about half the on-demand cost; without them the saving nearly disappears.

The decision rule: spot stays cheaper as long as the redone work is less than d / (1 - d) of the job, where d is the discount. At a 57% discount that is 0.57 / 0.43 = 133% of the job. At a 30% discount it falls to 43%, and a run without checkpoints can blow through that with a single late reclaim.

How do you get interrupted less on spot GPUs?

Pick calm pools on purpose and keep your options wide. These steps come straight from what each cloud publishes.

1. Check the number for the exact instance and location before you launch: AWS's advisor band, Google's capacityHistory preemption history, or Azure's eviction rate in the portal or Resource Graph (28 trailing days).
2. Accept several GPU types and Regions. AWS recommends being "as diverse as possible" in instance type and Availability Zone, and Google's managed instance groups pick the machine types with the lowest observed preemption rates when you allow several.
3. Match checkpoint frequency to the pool. In an over-20% pool, save often enough that a reclaim costs minutes. In a calm pool you can save less often and spend less time writing state.
4. Use the full notice window. Google lets you set a 120-second preemption notice instead of the default 30 seconds; Azure delivers its eviction event up to 30 seconds ahead on a best-effort basis.

Our spot vs on-demand guide covers each cloud's notice window in detail, and how to resume training after spot preemption shows the save-and-restore code.

How does Nodus handle spot interruptions?

Nodus treats interruption as a number to plan with and a failure to recover from automatically. You see the rate before launch, and a reclaimed run continues on another machine from its last checkpoint.

The rate first. nodus get gpus lists offerings, and the wide view adds startup time and, for interruptible offerings, how often that capacity is taken back per hour. These commands are copied from the GPU guide:

$ nodus get gpus --interruptible
$ nodus get gpus --gpu L4 -o wide        # typical and list rates, startup time and interruption rate

Then the choice. You can leave the spot decision to the scheduler, which uses interruptible capacity only when it is cheaper to finish:

spec:
  resources:
    gpu: {type: [H100, H200], count: 8}
  placement:
    interruptible: Allow          # use interruptible capacity only when it is cheaper to finish
    maxRateUSDPerHour: "30.00"

Then the recovery. Per the attempts and recovery docs, on a reclaim notice Nodus asks your run for an urgent checkpoint and prepares a replacement machine at the same time. The replacement starts only once the old machine is provably gone, and a Checkpointed run restores its latest committed checkpoint, so it loses at most the work since that save. A run gets up to 8 recoveries by default (3 for distributed Jobs), and a preemption after progress never counts toward the no-progress limit.

FAQ

Why can't I see Google's preemption rate for my GPU VM?

Google's capacityHistory data does not cover N1 machine types with attached GPUs, custom machine types or TPUs. For those, you have to measure your own preemptions or test the zone.

Does a "<5%" band mean my AWS spot GPU won't be interrupted?

No. It means a small share of that pool was reclaimed over the last month. Your instance can still be reclaimed at any time, so a long run still needs checkpoints.

Does Nodus account for interruptions when it picks capacity?

Yes. Nodus picks among the GPU types and regions you allow by expected cost to finish. A run whose checkpoints are all empty is planned and priced as if it restarts from scratch. Save real state so interruptible capacity can count as the cheaper way to finish.

Do I pay twice when a machine comes back after a reclaim?

Not on Nodus. If the old machine returns before its replacement starts, the replacement is released, your run continues, and you are not charged for the replacement.

Originally published on the Nodus blog.