Can you use serverless GPUs for long training runs?

You can, up to a point: Modal caps a single function execution at 24 hours, RunPod Serverless caps a job at 7 days, and on both a preempted or failed run starts over unless your own code saved a checkpoint. Nodus runs long training as a Job or TrainingJob with an optional time limit, saves your state directory automatically, and resumes from the latest checkpoint when capacity is reclaimed. Our rule: serverless is fine for training that finishes well inside one execution and is cheap to repeat; past that, use a platform that owns the checkpoint.

How long can a serverless GPU function run?

Each serverless platform puts a hard ceiling on one execution, and the defaults are much lower than the maximums.

Platform	Default limit per execution	Maximum	What happens at the limit
Modal	5 minutes	24 hours	FunctionTimeoutError after any retries
RunPod Serverless	10 minutes	7 days	The job fails and the worker stops
Nodus Job or TrainingJob	No limit unless you set timeout	You can raise or remove it on a running Job	DeadlineExceeded; time spent suspended does not count

Sources: Modal timeouts, RunPod endpoint settings, Nodus Jobs guide. Checked October 29, 2026.

So a 30-hour fine-tune does not fit in one Modal call at all. It fits in one RunPod Serverless job only if you raise the execution timeout from its 10-minute default, which is easy to forget until the first job dies at minute ten.

What happens when a long serverless run is interrupted?

The platform restarts the work, but your progress only survives if your code wrote it somewhere first. Modal's preemption guide says all Modal Functions are preemptible by default, that a preempted Function is restarted on the same input, and that training Functions should checkpoint because the chance of interruption grows with run time. GPU Functions cannot opt out of preemption.

Here is what that costs. Take a 30-hour run on one H100 that is interrupted at hour 20:

• Without checkpoints, the restart begins at step 0, so you pay for 20 + 30 = 50 GPU-hours.
• With a checkpoint every hour, you lose at most one hour, so you pay for at most 31 GPU-hours.

At Modal's H100 rate of $0.001097 per second (about $3.95 per hour), those 20 lost hours are $78.98 of repeated work. The fix is not exotic, but it is code you have to write, test and keep correct: save model, optimizer and step state, then detect a restart and load it.

One interruption without checkpoints makes a 30-hour run bill 50 GPU-hours.

How does Nodus keep a multi-day training run going?

Nodus treats checkpointing as the platform's job. Your program writes what it needs to continue under /nodus/state, and Nodus copies that directory while the run is going. With the default interval: auto, the checkpoint guide says the cadence follows how often the capacity is interrupted and how long a save takes, with at least four checkpoints over the expected run. Nodus also checkpoints when the capacity gives notice that it is about to be reclaimed, then starts the next attempt with the directory restored and NODUS_RESTORED=1 set.

The managed TrainingJob runtimes do the save and restore for you, including optimizer, scheduler and RNG state. This is the SFT example from the docs:

apiVersion: nodus.dev/v1beta1
kind: TrainingJob
metadata:
  name: support-sft
spec:
  runtime: nodus/sft
  model:
    uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca
    secret: hf-token            # only for gated repositories
  data:
    volume: support-chats       # a Volume holding chats.jsonl
    path: chats.jsonl
    format: JSONL
    columns: {prompt: question, completion: answer}
  parameters:
    steps: 2000
    lora: {r: 16, alpha: 32}
  maxCostUSD: "20.00"

The maxCostUSD cap is the other half of running long jobs safely. When the cap is reached, the run is suspended rather than failed, and raising the cap resumes the same run from its checkpoint. For a broader look at which platforms do this for you, see Which GPU platforms checkpoint and resume training for you?

What does a 30-hour H100 fine-tune cost on each option?

At list prices, Nodus is the cheapest of the four for the same 30 GPU-hours, and it is the only one where the restart logic is not your code.

Option	H100 list rate	30 GPU-hours	Fits in one run?
Nodus Job or TrainingJob	$2.60/hr	$78.00	Yes
Modal Function	$3.95/hr ($0.001097/s)	$118.48	No, 24-hour cap
RunPod Pod (H100 SXM)	$3.99/hr	$119.70	Yes, not serverless
RunPod Serverless (H100 PRO)	$5.39/hr	$161.70	Yes, if the timeout is raised

Prices checked October 29, 2026: Nodus H100 SXM listed at $2.60/hr on the Nodus pricing page, Modal pricing (GPU only; Modal bills CPU and memory separately), RunPod pricing for Pods and Serverless.

The math is 30 times each hourly rate. On those numbers the Nodus run costs 34% less than the Modal run, before counting any repeated hours from an interruption.

Same 30 GPU-hours, list prices: the gap is the hourly rate before any lost work.

When is serverless the right place to train?

Serverless training makes sense when the run is short, cheap to repeat, or already part of a serverless app. A LoRA fine-tune of a small model that takes two hours fits inside Modal's 24-hour cap with room to spare. An interruption there costs at most two hours, so writing restart logic may not be worth it.

Use this decision rule. If the run would take less than a few hours and a full restart is cheap, train where your app already lives. If it takes longer than one execution allows, or one lost run would cost more than you would spend writing and testing checkpoint code, use a platform that checkpoints for you.

Where is Modal or RunPod the better choice?

Modal is the better choice when your training code is already a Modal app and runs fit in 24 hours. You keep one deployment model for training and inference, and its Python-first workflow avoids writing manifests. Modal also publishes a long training example that shows the checkpoint pattern if you want to stay.

RunPod Serverless is the better choice when one job can take days but you want queue-based workers that scale down when idle. Its 7-day execution limit is far longer than Modal's. RunPod Pods suit teams that want a plain GPU machine and are happy to handle restarts themselves.

Nodus is the better choice when runs are long, you want them on the cheapest capacity that will finish them, and you would rather the platform own checkpoints, recovery and the spending cap. We cover the platform-level comparison in What are the best serverless GPU platforms in 2026?.

How do you move a long training run to Nodus?

If you use one of the managed runtimes, save the manifest above as sft.yaml, check the estimate, then apply it:

$ pip install nodus-compute
$ nodus login
$ nodus apply -f sft.yaml --dry-run -o yaml     # status.estimate: cost to completion, first hold, start time
$ nodus apply -f sft.yaml
$ nodus logs job/support-sft -f

For your own training script, point its save and load paths at /nodus/state and run it as a Job with nodus run --checkpoint /nodus/state. The checkpoint guide has a complete Python example.

FAQ

Can I chain several Modal calls to train past 24 hours?

Yes. Each call loads the last checkpoint from a Volume, trains until close to the limit, saves, and the next call continues. You own the orchestration, the save format and the edge cases, such as a preemption that lands mid-save.

Does the 24-hour Modal limit include time waiting for a GPU?

No. Modal's timeout measures execution time only, not scheduling, and it applies per execution attempt when retries are configured.

How long does Nodus keep checkpoints after a run finishes?

Checkpoint settings take a retainAfterFinish value, shown as 168h in the docs example, after which only the final checkpoint is kept. nodus describe job/NAME shows the latest one.

Does a Nodus TrainingJob need my code to handle checkpoints?

Not with the managed runtimes such as nodus/sft, nodus/dpo and nodus/grpo-lora; they save full recovery state themselves. Custom code only needs to write and read its files under /nodus/state.

Originally published on the Nodus blog.