GPU capacity is the tightest it has been in a while. Amazon just crossed three trillion dollars largely on cloud AI demand and the reporting says even AWS can't add capacity fast enough. When supply is that tight, the price gap between on-demand and spot GPUs gets wide and interesting, and every team running inference is staring at the same question: how much of this can we safely move to spot?
I'm a cloud associate and this is a decision I've had to make with real money attached, so instead of the usual "spot is 70% cheaper, go use it" take, here are the actual rules we use to decide what goes where. Spoiler: the headline discount is the least important number.
First, the number everyone quotes and why it's misleading
Yes, spot GPU instances often run 60-70% below on-demand. That number is real and it's also a trap, because it quietly assumes your workload doesn't care about being interrupted. GPU spot capacity is the first thing reclaimed when demand spikes, which right now is often. So the real comparison isn't "cheap vs expensive," it's "cheap-but-can-vanish vs expensive-but-guaranteed," and the right answer depends entirely on what the workload does when it gets a two-minute eviction notice.
The rule: sort workloads by interruption cost, not by size
We put every GPU workload into one of three buckets based on a single question, what does an interruption actually cost us.
Bucket 1: interruption is free-ish, go spot. Batch inference, offline embedding jobs, eval runs, anything that can checkpoint and resume. If a job can be killed and restarted with no user impact and minimal lost work, spot is close to a no-brainer. We run these on spot with checkpointing every few minutes so an eviction costs us seconds, not the whole job.
Bucket 2: interruption is survivable if you engineered for it. Real-time inference behind a load balancer, where losing one node degrades but doesn't break service. Spot works here only if you've done the work: spread across multiple instance types and AZs so one capacity pool draining doesn't take you down, keep a small on-demand baseline for the floor, and let spot handle the burst. This is where most of the savings actually live, and also where most of the outages come from when people skip the engineering.
Bucket 3: interruption is unacceptable, stay on-demand (or reserved). Anything user-facing with a hard latency SLA and no graceful degradation, or a single-node workload that can't tolerate a restart. Paying full price here isn't a failure, it's buying reliability you actually need. Trying to force these onto spot is how you end up explaining an incident.
The costs the discount hides
Three line items that don't show up in the "spot is 70% off" pitch and that we learned to budget for:
- Cold starts and model load time. A fresh GPU node pulling a large model into memory can take minutes. If you're churning through spot nodes, you pay that startup cost repeatedly in both time and instance-hours. A frequently-interrupted spot node can be more expensive in effective throughput than a stable on-demand one.
- The on-demand baseline you still need. Almost nobody runs pure spot for serving. You keep a baseline on-demand fleet for the floor, so your blended cost is never the spot headline number, it's a weighted average. Model the blend, not the best case.
- Engineering and operational time. Multi-pool spread, checkpointing, drain handling, fallback logic. That's real work, and it only pays off above a certain spend. Below it, the engineering costs more than the savings.
Where the biggest win actually is (and it isn't spot)
The most honest thing I can tell you: before you optimize the price of the GPU, check whether the GPU should be running at all. Our biggest inference savings didn't come from spot, they came from scheduling. Eval and dev GPU pools have no business running overnight or on weekends, and utilization on "always-on" inference fleets is usually far below what people assume. We schedule non-production GPU capacity to scale down off-hours the same way we schedule any other non-prod resource (that scheduling is a core part of what ZopNight does for us, but you can do the crude version with a cron job and an autoscaler). A 70% spot discount on a node that shouldn't be on is still 100% waste.
The decision in one line
Bucket the workload by interruption cost first, model the blended price including cold starts and your on-demand baseline second, and before any of that, make sure the capacity is even supposed to be running. The spot discount is real, but it's the last lever, not the first.
If you're running inference on spot, which bucket gave you the most trouble? For us it was bucket 2, the "survivable if engineered" tier, where a bad week of evictions taught us to spread across instance types the hard way.
Top comments (0)