This article was originally published at sivaro.in

Spot Instances vs On Demand for Training Cost: A Practitioner's Guide

I burned $47,000 in a single weekend last March. Not on a product launch. Not on a marketing campaign. On GPU hours for a fine-tuning run that could have cost me $9,400 if I'd used spot instances instead of on-demand.

That was the moment I stopped treating spot instances vs on demand for training cost as an academic comparison. It's a P&L decision. It's a hiring decision. It's whether you make payroll or you don't.

Here's what most people get wrong: they assume spot is always cheaper and therefore always better. That's false. Spot is usually cheaper, but it comes with interruption risk that can triple your effective wall-clock time if you architect around it badly. On-demand is expensive but predictable. The right answer depends on your training workload, your checkpointing maturity, and how much engineering time you can afford to spend on infrastructure.

This guide breaks down both options with real numbers, real trade-offs, and the decision framework I use at SIVARO when clients ask whether to run their training jobs on spot or on-demand capacity.

The Real Difference Between Spot and On-Demand Pricing

On-demand is simple. You request an instance, you get it, you pay the listed hourly rate. A p5.48xlarge (8x H100) on AWS runs about $98.32/hour on-demand as of September 2026. A single A100 80GB instance sits around $4.10/hour. You reserve it, you use it, you release it.

Spot instances are spare capacity that cloud providers sell at a discount. That discount ranges from 60% to 90% depending on instance type, region, and current demand. The catch: the provider can reclaim your instance with as little as a 30-second warning when they need the capacity back.

I've seen a p5.48xlarge spot instance go for $28.50/hour in us-east-1 on a quiet Tuesday. Same instance, same region, three days later during a competitor's launch week? $71/hour. Spot pricing floats.

The keyword here is interruption. It's not "shutdown." The instance stops, your job dies mid-step unless you've checkpointed. That's the whole game.

Why Spot Interruption Rates Matter More Than Spot Discounts

Everyone focuses on the discount. Amateur move.

What matters is the interruption rate for your specific instance type in your specific region at your specific time. AWS publishes a spot instance advisor that shows interruption frequency buckets (less than 5%, 5-10%, 10-15%, 15-20%, and greater than 20%). If you're running a 72-hour training job on an instance type with a 20% interruption rate, you're not saving 70%. You're losing time.

Here's the math that changed how I think about this.

Let's say you have a job that takes 100 GPU-hours on-demand at $4/hour = $400. Now run it on spot at $1.20/hour. Looks like $120, right? Wrong.

If your job gets interrupted every 4 hours on average and you lose 20 minutes of progress each time (no checkpointing), your effective cost per GPU-hour balloons. You're paying for compute you throw away. In a bad case, you spend 140 GPU-hours to complete 100 GPU-hours of work, and that's if you consistently find spot capacity. If you don't, you're waiting.

The only way spot wins is with frequent checkpointing. I'm talking every 5-10 minutes for large training runs, not every hour.

import boto3
import time
from datetime import timedelta

def should_checkpoint(last_checkpoint_time, interval_minutes=5):
    """Return True if enough time has elapsed since last checkpoint."""
    elapsed = time.time() - last_checkpoint_time
    return elapsed >= (interval_minutes * 60)

def training_loop_with_checkpoints(model, optimizer, dataloader, ckpt_interval=5):
    last_ckpt = time.time()
    step = 0
    for batch in dataloader:
        loss = model.train_step(batch)
        optimizer.step()
        step += 1

        if should_checkpoint(last_ckpt, ckpt_interval):
            save_checkpoint(model, optimizer, step)
            last_ckpt = time.time()
            print(f"Checkpointed at step {step}")

Every checkpoint has a cost. If you're writing a 70B parameter model state to S3 every 5 minutes, that's real money and real time. From my testing, checkpointing a 70B model takes 40-90 seconds depending on network throughput. Do that every 5 minutes and you've added 8-15% overhead to your training run.

When On-Demand Actually Makes Financial Sense

Contrarian take: on-demand isn't a fallback. For some workloads, it's the correct primary choice.

Short jobs. Anything under about 4 hours. Interruption risk compounds over time, and short jobs rarely hit an interruption window.

Latency-sensitive training. If you're doing online fine-tuning or RLHF where you need to respond to data quickly, on-demand's predictability wins.

Small clusters that can't tolerate any loss. If you're running 8 GPUs and one gets interrupted, your entire distributed job collapses. The overhead of recovery doesn't justify the discount.

When you're competing on time-to-market and not on cost. I've worked with startups where burning an extra $30K on on-demand compute to ship a model three weeks earlier was obviously correct because the funding round closed on that timeline.

And here's a real one: when spot capacity isn't available. In Q1 2026, H100 spot capacity in us-west-2 was essentially zero for six weeks. Everyone wanted H100s. Nobody was giving up capacity. You couldn't find spot at any price. Teams that had architected exclusively for spot got stuck.

When Spot Is the Obvious Choice

Long pre-training runs. Anything over 12 hours where you can checkpoint properly. The savings are dramatic.

Hyperparameter sweeps. Hundreds of independent short jobs that each run 30-90 minutes. If one dies, you just relaunch it. Perfect spot workload.

Batch inference and embedding generation. Not exactly training, but same cost model. Interruption just means you redo a batch.

Fault-tolerant fine-tuning. Any fine-tuning job with robust checkpointing and automatic relaunch. This is the sweet spot for most teams.

import boto3
import json
from botocore.exceptions import ClientError

def handle_spot_interruption(event, context):
    """Lambda handler triggered by EC2 spot interruption notice."""
    detail = event['detail']
    instance_id = detail['instance-id']
    action = detail['instance-action']  # usually 'terminate'

    print(f"Spot interruption for {instance_id}, action: {action}")

    # Trigger graceful shutdown: save final checkpoint, drain dataloader
    trigger_graceful_shutdown(instance_id)

    # Log the interruption for capacity planning
    log_interruption(instance_id, detail['instance-action'])

    return {'statusCode': 200, 'body': 'handled'}

You get a two-minute warning via the instance metadata service (or EventBridge, as above). Two minutes is enough for a graceful checkpoint if your save is fast. It's not enough if you're saving a 400GB optimizer state.

The Architecture That Makes Spot Work

Spot doesn't work because spot is cheap. Spot works because you built a system that survives interruption.

Here's what production-grade spot training looks like at SIVARO:

Checkpoint to object storage, not local disk. Local NVMe dies with the instance. S3 or GCS survives. Yes, writing to S3 is slower than local disk. Do it anyway.

Use a job queue. Don't launch instances manually. Use a queue (SQS, Ray, Slurm, or a homegrown one) that tracks pending work. When an instance dies, the work returns to the queue.

Mix spot and on-demand in the same cluster. This is the move most teams miss. Run your critical path on on-demand, your throughput-loaded work on spot. If spot dries up, the critical path keeps moving.

Set a maximum price, not a fixed price. Spot instances let you bid a max price. If the current spot price exceeds it, you don't get the instance. If you set your max too low, you get nothing. Set it to roughly 60-70% of on-demand. That's where capacity is usually available.

Persist the random seed and the optimizer state. Restart determinism is real. Without it, your loss curves look like noise after every restart.

# Kubernetes Job spec with spot node selector and checkpoint sidecar
apiVersion: batch/v1
kind: Job
metadata:
  name: training-run-001
spec:
  backoffLimit: 10
  template:
    spec:
      nodeSelector:
        node.kubernetes.io/instance-type: p5.48xlarge
        karpenter.sh/capacity-type: spot
      tolerations:
        - key: "sku"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"
      containers:
        - name: trainer
          image: sivaro/trainer:0.4.2
          command: ["python", "train.py", "--resume-from-latest-checkpoint"]
          volumeMounts:
            - name: checkpoint-store
              mountPath: /checkpoints
      volumes:
        - name: checkpoint-store
          persistentVolumeClaim:
            claimName: s3-checkpoint-pvc
      restartPolicy: OnFailure

Karpenter (for EKS) and similar autoscalers make this much easier than it used to be. They'll provision spot capacity, replace interrupted nodes, and keep the cluster stable.

Actual Numbers From Real Workloads

Numbers matter more than theory. Here's what I've observed running production training workloads on AWS and GCP over the past two years.

70B parameter fine-tuning, 8x H100, 72-hour run:

• On-demand: ~$7,080 (at $98.32/hr)
• Spot with proper checkpointing: ~$2,180 (at ~$30/hr average)
• Spot without checkpointing (interrupted 4 times, restarted from scratch twice): $4,700 in wasted compute plus $10,000 in lost engineer time

7B parameter fine-tuning, 4x A100, 18-hour run:

• On-demand: ~$295
• Spot with checkpointing: ~$95
• Spot interruption rate in us-east-1: 3 out of 10 runs got interrupted at least once

1B parameter sweep, 240 short runs, 45 min each:

• On-demand: ~$1,476
• Spot: ~$410

The savings on the sweep are dramatic because each job is short and independent. Doesn't matter if one dies; the queue just reruns it.

For reference on pricing changes, AWS cut spot prices for H100 instances by about 12% in July 2025, then they floated up again through early 2026. Spot is not a fixed discount; it's a market.

The Hidden Costs Nobody Talks About

Everyone quotes the hourly rate. Nobody quotes the engineering cost.

Building interruption-tolerant training took my team about six weeks of engineering time across three quarters. That's roughly $80,000 in fully-loaded salary cost. It pays back fast if you're spending $50K/month on compute. It never pays back if you're spending $2K/month.

Ongoing operational load matters too. Spot interruption dashboards, alerting, capacity planning, region failover logic. All of that is real work.

I'd put the break-even at roughly $8,000-$12,000 per month in training compute. Below that, on-demand is cheaper once you price in your engineers' time. Above that, spot's savings dominate.

The other hidden cost: opportunity cost of interrupted experiments. When a researcher is iterating on an architecture, three interruptions in an afternoon kills their flow. If that researcher costs $250/hour, a $40 spot saving can easily be a net loss.

A Decision Framework You Can Actually Use

Here's the loose heuristic I use. It's not perfect but it beats analysis paralysis.

Run on spot if:

• Your job runs longer than 4 hours
• You can checkpoint every 10 minutes or less
• You're spending over $10K/month on training compute
• You have a job queue with automatic retry
• Your team can tolerate some operational complexity

Run on-demand if:

• Jobs are short (under 4 hours)
• Interruption cascades break distributed training
• You have tight time-to-market constraints
• Your training volume is under $10K/month
• Your team is small and can't afford infra work

Run a mix if: (this is most teams)

• You have a critical-path training run on on-demand
• Parallel sweeps and experiments on spot
• You route based on capacity availability at submission time

The mix is what I actually recommend. You don't have to choose sides. You can architect so that spot is your default for parallel work and on-demand is the safety net.

Cloud Provider Differences That Actually Matter

AWS spot: mature, has capacity rebalancing and instance hibernation. Best tooling. Also the most expensive on-demand pricing, so the spot discount is genuinely meaningful.

GCP preemptible/spot: simpler model, 24-hour max lifetime on preemptible (spot has no time limit as of 2024). Generally available in most regions. Slightly less flexible than AWS.

Azure spot: got much better in 2025 with the spot eviction policy API. Still catching up on tooling.

Lambda and Cloud Run GPU: worth a mention for inference, less so for training. Not really part of the spot vs on-demand conversation for training workloads, but the pricing model sits in between.

If you're running large-scale training and cost matters, AWS or GCP are the real choices.

FAQ: Spot Instances vs On-Demand for Training Cost

How much cheaper are spot instances than on-demand for training?
Typically 60-90% cheaper on the hourly rate. In practice, once you account for interruption and restart overhead, expect effective savings of 40-70% for workloads with good checkpointing.

Will my spot instance definitely get interrupted?
No. Some instance types in some regions have less than 5% interruption frequency per month. Others regularly hit 20%+. Check AWS's Spot Instance Advisor before choosing.

Can I use spot for multi-node distributed training?
Yes, but it's harder. One interrupted node collapses the whole job in most frameworks. You need elastic training frameworks (like TorchElastic) or fault-tolerant approaches (like Gemini or specialized checkpointing). Adds engineering overhead.

How often should I checkpoint?
Every 5-10 minutes for spot. Every 30-60 minutes for on-demand. The right number balances checkpoint I/O overhead against lost progress on interruption.

Is on-demand ever cheaper than spot for training?
Only if your effective spot cost (including wasted compute and engineering overhead) exceeds on-demand pricing. This happens for short jobs, latency-sensitive work, and workloads without checkpointing.

What's the two-minute warning on spot interruption?
AWS gives you a two-minute notice via instance metadata and EventBridge before terminating a spot instance. It's enough to save a checkpoint if the save is fast, not enough for a full model dump.

Do spot prices change often?
Yes, sometimes multiple times per hour. They track supply and demand closely. Popular instance types during busy periods can spike 3-5x.

Should I use spot for inference?
Inference is a different beast. If you have SLA requirements, on-demand or reserved capacity usually wins. Spot can work for batch inference and asynchronous workloads.

Where This Is Heading

Spot capacity for GPUs has become much more contested since mid-2025, when demand from foundation labs spiked. Some regions have almost no savable capacity. Others still have plenty.

Multi-cloud spot arbitrage is emerging as a real practice. Tools that submit the same job to whichever cloud has cheap spot at that moment. I think this becomes standard in the next 18 months. Too much money on the table not to.

For most teams I work with, the answer to spot instances vs on demand for training cost comes down to one question: have you built the infrastructure to survive interruption? If yes, spot is nearly always cheaper. If no, on-demand is what you actually pay, and the "spot discount" is a fantasy that only survives on a spreadsheet.

Build the checkpointing. Build the queue. Then use spot for everything you can.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.