How do you train on AWS spot instances without losing work?

To train on AWS spot instances without losing work, save checkpoints to storage that outlives the instance, reload them on start, and restart the job after each interruption. Amazon SageMaker managed spot training handles the restart and S3 sync, while plain Amazon EC2 Spot leaves all of it to you. A Nodus job checkpoints and restores itself, including when it runs in your connected AWS account.

The part most guides skip is the two-minute warning. It is only useful if your checkpoint can leave the machine inside it, and for a full fine-tune of a large model it usually can't. Below we show the sizes, the setup for each path, and the rule we use to pick one.

What actually happens when AWS reclaims a spot instance?

You get about two minutes, and then the instance is terminated unless you asked for something else. AWS issues an interruption notice two minutes before it stops or terminates a Spot Instance, as an Amazon EventBridge event and as the spot/instance-action item in instance metadata. AWS recommends checking for it every 5 seconds and says notices are best effort.

Termination is the default interruption behavior. Anything on the instance, including a checkpoint on local disk, goes with it. A rebalance recommendation can arrive earlier as a sign of elevated risk, but AWS says it may also arrive together with the two-minute notice. Treat it as a bonus, not a plan.

AWS is also candid that training is a hard fit. Its Spot best practices say Spot is not suited to stateful or tightly coupled workloads, and suggest being flexible across at least 10 instance types. A training run is stateful, and a GPU job usually fits only a handful of instance types. Checkpointing is what turns it into something Spot can handle.

Can your checkpoint leave the machine before the instance goes?

Only if it is small. We worked out what a full training checkpoint weighs and how fast it would have to upload to make it out in time. We budget 90 of the 120 seconds for the upload, leaving 30 seconds to finish the current step and shut down. That split is our assumption, not an AWS figure.

A full fine-tune with mixed precision and the Adam optimizer saves about 14 bytes per parameter. That is 2 for bf16 weights, 4 for fp32 master weights and 8 for Adam's two moment estimates. A LoRA run saves only the adapter, at 12 bytes per adapter parameter (fp32 weights plus Adam). Adapter counts use rank 16 on every linear layer of the Llama 3 8B and 70B shapes.

What you train	Checkpoint size	Upload rate to finish in 90 s
8B model, full fine-tune	112 GB	1.24 GB/s
70B model, full fine-tune	980 GB	10.89 GB/s
8B model, LoRA r16 (41,943,040 adapter params)	0.50 GB	0.0056 GB/s
70B model, LoRA r16 (207,093,760 adapter params)	2.49 GB	0.028 GB/s

Our arithmetic: parameters x bytes per parameter, then size / 90 seconds. Model shapes from the published Llama 3.1 model configs.

A LoRA checkpoint is a rounding error next to a full fine-tune of the same model.

Upload speed decides the rest. At 1 GB/s, the 112 GB checkpoint takes 112 seconds, which misses the 90-second budget. You need 1.25 GB/s to get it down to 89.6 seconds. At 0.5 GB/s it takes 224 seconds, longer than the whole notice.

Faster links leave room to spare: 56 seconds at 2 GB/s and 28 seconds at 4 GB/s. We have not measured instance-to-S3 rates here, so time an aws s3 cp of a file your checkpoint's size from your own instance before you rely on the notice.

Below about 1.25 GB/s, a full 8B checkpoint cannot leave inside the notice.

Our rule: if your measured upload time is under 90 seconds, save on the notice and keep scheduled saves infrequent. If it is not, save on a schedule and write the checkpoint to durable storage as you go. The spot checkpointing walkthrough has the interval math for that case.

How does SageMaker managed spot training keep your progress?

It syncs a local folder to S3 and puts it back after every restart. Your script writes checkpoints to /opt/ml/checkpoints. SageMaker copies that folder to the S3 URI you give it during training and copies it back into the container when the job restarts on a new spot instance. Your script still has to load the latest checkpoint itself.

The request needs three fields beyond a normal CreateTrainingJob call:

import boto3

sm = boto3.client("sagemaker")
sm.create_training_job(
    TrainingJobName="llama-8b-lora-spot",
    AlgorithmSpecification={"TrainingImage": IMAGE_URI, "TrainingInputMode": "File"},
    RoleArn=ROLE_ARN,
    OutputDataConfig={"S3OutputPath": "s3://my-bucket/output/"},
    ResourceConfig={"InstanceType": "ml.p5.48xlarge", "InstanceCount": 1, "VolumeSizeInGB": 500},
    EnableManagedSpotTraining=True,
    StoppingCondition={"MaxRuntimeInSeconds": 36000, "MaxWaitTimeInSeconds": 86400},
    CheckpointConfig={"S3Uri": "s3://my-bucket/ckpt/llama-8b-lora/", "LocalPath": "/opt/ml/checkpoints"},
)

In the training script, point the checkpoint folder at that path and resume when something is there. With Hugging Face Transformers:

import glob
from transformers import Trainer, TrainingArguments

args = TrainingArguments(output_dir="/opt/ml/checkpoints", save_steps=200, save_total_limit=2)
trainer = Trainer(model=model, args=args, train_dataset=train_ds)
has_ckpt = bool(glob.glob("/opt/ml/checkpoints/checkpoint-*"))
trainer.train(resume_from_checkpoint=True if has_ckpt else None)

Three details from the AWS docs matter. MaxWaitTimeInSeconds covers waiting for spot capacity plus running, and it must be at least MaxRuntimeInSeconds, per the StoppingCondition reference. Set it to the runtime plus the delay you can accept: above, 10 hours of runtime with a 24-hour deadline.

Deleting a checkpoint in the container also deletes it in S3, so save_total_limit=2 keeps two copies in both places. With several instances, give each one its own file names, because SageMaker writes everything to one S3 prefix.

Billing follows productive time. AWS reports savings as (1 - BillableTimeInSeconds / TrainingTimeInSeconds) x 100. Its managed spot page says training time excludes intervals lost to an interruption within their first hour, plus time spent waiting for capacity. The same page says managed spot never falls back to on-demand: if capacity does not come back, the job stops at MaxWaitTimeInSeconds.

What do you build yourself on plain EC2 Spot?

You build the restart, the sync and the notice handling. On raw EC2 you need a launcher that replaces a lost instance and a checkpoint copy to Amazon S3 or an EBS volume. You also need a watcher for the notice and a script that resumes. AWS suggests an EventBridge rule on interruption and rebalance events that triggers a checkpoint, and an Auto Scaling group or EC2 Fleet using the price-capacity-optimized allocation strategy.

One AWS option fits training better than most guides admit: stop instead of terminate. With a persistent request and stop behavior, EC2 keeps the EBS volumes and restarts the instance when capacity returns in the same Availability Zone and instance type. While it is stopped you pay only for the volumes.

# the root device name (/dev/sda1 here) depends on your AMI
aws ec2 run-instances --image-id "$AMI" --instance-type g6e.xlarge \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=500,VolumeType=gp3}' \
  --instance-market-options 'MarketType=spot,SpotOptions={SpotInstanceType=persistent,InstanceInterruptionBehavior=stop}'

The catch is in the same sentence: it waits for that exact type in that exact zone, with no deadline. Checkpoints must also live on EBS, because instance store data does not survive a stop. Use stop behavior when the run can wait and the checkpoint is too big to upload in the notice. Use terminate plus S3 when you need the run to move to whatever capacity exists.

How does Nodus run the same training job?

A Nodus Job saves a state directory and restores it into the next attempt, wherever that attempt runs. Your script writes model, optimizer and step to /nodus/state and loads them on start. Nodus copies that directory on a cadence it sets from how often the capacity is interrupted, and again when the capacity gives notice, as the checkpoints guide describes. A replacement attempt starts with the files restored.

nodus run --name llama-8b-lora --gpu L40S --image nodus/pytorch \
  --checkpoint /nodus/state --max-cost 20 -- python train.py

With the Hugging Face example above, set output_dir="/nodus/state" and keep the same resume check. The --max-cost cap stops the job at a dollar limit you choose; it is not a quote.

If you have AWS credits, Nodus can spend them first. Connect the account, then create the execute stack it opens and run nodus cloud connect aws aws-main --credit-limit 500, per the cloud accounts guide. That feature is in Beta. Single-node Jobs then go to your account first, in us-east-1, us-east-2 or us-west-2.

AWS bills you for the machine and disk, and Nodus adds a route fee of $0.02 per device-hour. If AWS refuses a machine for quota, Nodus skips that family in that region for an hour and runs the work elsewhere. When the account cannot take the work, the default sends it to Nodus capacity at Nodus rates; --burst Deny makes it wait instead.

To be precise about what we claim: the docs describe Nodus launching its own tagged machines in your account. They do not promise AWS Spot pricing there, so plan on spending your credits, not on a spot discount. Multi-node runs use Nodus capacity.

Which path should you choose?

Pick by how much you want to build and what happens when capacity runs out.

	SageMaker managed spot	Plain EC2 Spot	Nodus Job
Restarts after an interruption	SageMaker	Your launcher or fleet	Nodus
Checkpoint copy	/opt/ml/checkpoints synced to S3	Your code (S3 or EBS)	/nodus/state, copied by Nodus
When spot capacity is gone	Waits, then stops at MaxWait	Waits, or your fallback code	Other capacity, or waits if you choose
Multi-node	Supported; you name files per instance	Your orchestration	Runs on Nodus capacity
Hard limit	MaxRuntime up to 28 days	None built in	Dollar cap per job

Sources: the AWS pages linked in each section and the Nodus checkpoints and cloud accounts guides, checked November 4, 2026.

Our decision rule. Choose SageMaker managed spot if you already use SageMaker and the run can slip by a day. Choose plain EC2 Spot with stop behavior if the run can wait and you want only the raw instance bill. Choose Nodus if the run has a deadline, since the job moves to other capacity instead of waiting.

The SageMaker alternatives comparison prices the same run across platforms, and our interruption-rate breakdown shows which AWS GPU types get reclaimed most.

FAQ

Can you use managed spot with a SageMaker built-in algorithm that doesn't checkpoint?

Yes, but only for short jobs. AWS limits built-in and marketplace algorithms that do not checkpoint to a MaxWaitTimeInSeconds of 3,600 seconds, so the whole job, waits included, has to fit in an hour.

Is the SageMaker spot savings figure the same as the EC2 spot discount?

No. AWS says the managed spot savings compare the SageMaker spot price with the SageMaker on-demand price. They are not comparable to the EC2 Spot price history in the EC2 console, which has no SageMaker service rate in it.

Do you need the optimizer state in a spot checkpoint?

Yes, for a clean resume. Reloading weights without Adam's moment estimates restarts the optimizer from zero, so the resumed run is not the run you started. For a LoRA run the optimizer state is small (the whole 8B checkpoint above is 0.50 GB), so there is no reason to drop it.

What does Nodus list for an H100 if you don't use your AWS account?

The NVIDIA H100 SXM is listed at $2.60/hr on the Nodus pricing page (checked November 4, 2026). That is the rate overflow work pays when your account cannot take it.

Originally published on the Nodus blog.