Wrap your model in DistributedDataParallel (DDP) if it fits on one GPU, or shard it with FSDP if it doesn't, and start one process per GPU with torchrun. On Nodus, nodus run --gpu H100:8 -- torchrun train.py runs that script on all eight GPUs of one machine with no launcher flags, and --nodes 2 --launcher torchrun spreads it across two machines. Fill one machine before you add a second: the second machine buys wall-clock time, not a cheaper run.
This post walks through the three decisions in order: which wrapper, how to launch it, and when more than one machine is worth paying for. The code follows the PyTorch tutorials and the Nodus docs; the memory and cost figures are our own arithmetic, shown in full so you can redo it for your model.
Should you use DataParallel, DDP or FSDP?
Use DDP by default and skip nn.DataParallel. The PyTorch DDP tutorial explains why: DataParallel is a single process with many threads, so it fights the Python GIL and only works on one machine, while DDP runs one process per GPU and works across machines too.
DDP gives every GPU a full copy of the model. Each one trains on a different slice of the batch, then an all-reduce averages the gradients so every copy stays identical. That is simple and fast, but it means the whole training state has to fit on each GPU.
FSDP removes that limit. The PyTorch FSDP tutorial describes it as sharding model parameters, gradients and optimizer states across GPUs, which "makes it feasible to train models that cannot fit on a single GPU." Each GPU holds only its shard and gathers a layer's full weights just before it needs them.
The decision rule we use: if the model, its gradients and the optimizer state fit on one GPU with room for activations, use DDP. If they don't, use FSDP. If even FSDP is tight, look at LoRA before you add hardware.
How much memory does each GPU need with DDP and FSDP?
Full training with Adam in mixed precision costs about 16 bytes per parameter before activations. That figure comes from the ZeRO paper: 2 bytes for 16-bit weights, 2 for gradients and 12 for the FP32 master weights plus Adam's two moments. DDP keeps all 16 bytes on every GPU. FSDP with full sharding divides them by the number of GPUs.
| Model | Model state per GPU with DDP | FSDP on 8 GPUs | FSDP on 16 GPUs |
|---|---|---|---|
| 1B parameters | 16 GB | 2 GB | 1 GB |
| 8B parameters | 128 GB | 16 GB | 8 GB |
| 70B parameters | 1,120 GB | 140 GB | 70 GB |
The math is parameters x 16 bytes, divided by the GPU count for FSDP. Activations, temporary all-gather buffers and the CUDA context come on top, so treat these as floors.
On 80 GB GPUs, an 8B model needs FSDP and a 70B model needs more than one 8-GPU machine of them.
Read it against an 80 GB NVIDIA H100. A 1B model fits comfortably with DDP. An 8B model does not fit at 128 GB, but drops to 16 GB per GPU with FSDP on one 8-GPU machine. A full 70B run needs 140 GB per GPU even when sharded over 8 GPUs, so on H100s it needs two machines.
Even then, 70 GB on each of 16 GPUs leaves little room for activations. That is why our VRAM guide for 70B models points full 70B fine-tuning at one machine with 8 NVIDIA B200s instead: a bigger single machine beats a second small one.
What does a DDP training script look like?
Four changes turn a single-GPU script into a DDP script: start a process group, pin each process to its GPU, wrap the model, and give each process its own slice of the data. This sketch follows the DDP tutorial's pattern:
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler
dist.init_process_group("nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
model = DDP(build_model().cuda(local_rank), device_ids=[local_rank])
sampler = DistributedSampler(dataset)
loader = DataLoader(dataset, batch_size=32, sampler=sampler)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
for epoch in range(epochs):
sampler.set_epoch(epoch)
for x, y in loader:
loss = loss_fn(model(x.cuda(local_rank)), y.cuda(local_rank))
opt.zero_grad()
loss.backward()
opt.step()
dist.destroy_process_group()
build_model, dataset, epochs and loss_fn are yours. Two lines are easy to miss. DistributedSampler stops every GPU from training on the same examples, and sampler.set_epoch(epoch) reshuffles each epoch; without it, PyTorch's own docstring warns, the same ordering is used every time.
The batch size above is per GPU. On 8 GPUs the effective batch is 8 x 32 = 256, so revisit your learning rate when you scale out.
How do you switch the same script to FSDP?
Replace the DDP wrapper with fully_shard from FSDP2, applied to each transformer block first and then to the whole model. The FSDP tutorial uses exactly this order:
from torch.distributed.fsdp import fully_shard
model = build_model()
for layer in model.layers:
fully_shard(layer)
fully_shard(model)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
Sharding per block matters. It lets FSDP gather one block's weights, run it and free them again, so only one block is ever whole on a GPU. Create the optimizer after sharding so it sees the sharded parameters. The data loading and training loop stay the same as the DDP version.
How do you launch it on one machine with several GPUs?
Run the script under torchrun, which starts one process per GPU and sets RANK, LOCAL_RANK and WORLD_SIZE for each. On your own machine you pass --nproc-per-node. On Nodus you ask for the GPUs and leave the flags off:
nodus run --gpu H100:8 --max-cost 250 -d -- torchrun train.py
The Nodus GPU guide explains why no flags are needed: Nodus sets PET_NPROC_PER_NODE to the GPU count, so torchrun starts one process per GPU, and /dev/shm is already sized for NCCL. --max-cost caps the spend in dollars, here just above the $208.00 estimate worked out below, and the same guide shows a --dry-run that prints the estimate before anything starts.
When is a second machine worth it?
Add a machine when one machine cannot hold the job or cannot finish it in time. More GPUs rarely make a run cheaper, because the gradient traffic between machines is slower than inside one, so each GPU does a little less useful work.
Here is a worked example. Say a run takes 10 hours on one machine with 8 H100s. H100s are listed at $2.60/hr on the Nodus pricing page, so that is 80 GPU-hours, or $208.00.
Now run it on two machines. The cost depends on scaling efficiency, the share of perfect speed-up you actually get, which you have to measure for your own model and network.
| Setup | Wall-clock | GPU-hours | Cost at $2.60/GPU-hour |
|---|---|---|---|
| 1 machine, 8 GPUs | 10.0 h | 80.0 | $208.00 |
| 2 machines, 16 GPUs, 90% efficiency | 5.6 h | 88.9 | $231.11 |
| 2 machines, 16 GPUs, 75% efficiency | 6.7 h | 106.7 | $277.33 |
Wall-clock is 10 hours divided by (2 x efficiency); GPU-hours is 16 times that. Prices checked October 27, 2026. The efficiencies are assumptions for the example, not measurements.
Two machines finish sooner but cost 11% to 33% more in this example.
So the second machine is a time purchase. At 90% efficiency you pay $23.11 more to get the result about 4.4 hours sooner. If nobody is waiting on it, one machine is the cheaper answer.
How do you run one job across several machines on Nodus?
Add --nodes and --launcher torchrun to the same command. The Nodus multi-node guide uses this as its two-node smoke test:
nodus run --name torchrun-2-node --gpu H100 --nodes 2 --launcher torchrun --max-cost 2.00 -- torchrun train.py
Nodus acquires every machine, checks that they can reach each other, and only then starts your command on all of them together. It sets MASTER_ADDR, NODE_RANK and the PET_* variables that torchrun reads, so the script itself does not change. --gpus-per-node 8 gives each machine eight GPUs, and --total-gpus lets Nodus pick the machine shape for you.
Multi-node training is in Beta, and the docs are specific about its limits. Gangs top out at 8 machines.
By default all machines are placed on one private network in one region. If you allow wider placement, traffic between machines on public addresses is not encrypted in the Beta. The guide also says a two-machine DDP result does not yet qualify cross-machine FSDP or several GPUs per machine across machines, so check the exercised combinations before a large FSDP run.
What happens when one GPU in a multi-GPU run fails?
The whole run stops, because every process waits on the others in each all-reduce. That makes checkpoints more important as you add GPUs: one lost machine out of two throws away everything since the last save.
On Nodus, the multi-node guide describes the recovery. If a machine is lost or reclaimed, Nodus stops every rank, keeps the surviving machines, replaces only the lost one, and restarts all ranks with the latest checkpoint restored into the state directory. Your script has to save there and load from it on start; the checkpoint guide in the Nodus docs shows how, and our post on automatic checkpointing and resume compares how other platforms handle it.
FAQ
How many GPUs can torchrun use on one machine?
As many as the machine has, one process per GPU. On Nodus, the GPU count you ask for is 1, 2, 4 or 8 per machine, and anything larger needs a multi-node job.
Do I need NVLink for multi-GPU training?
DDP and FSDP run over NCCL on whatever links the GPUs share, so NVLink is not required. It does speed up the gradient and all-gather traffic, which matters most for FSDP because it moves weights as well as gradients.
Can I use Hugging Face Accelerate or DeepSpeed instead of torchrun on Nodus?
Nodus accepts Accelerate and Deepspeed as launchers for multi-node jobs, setting their variables or a hostfile, but the multi-node guide marks them accepted in the Beta and qualified later. torchrun, Ray and verl are documented without that caveat.
Should I use LoRA instead of FSDP for a large model?
Often, yes. LoRA freezes the base weights and trains small adapters, so gradients and optimizer state shrink to a fraction of a percent of the model. If LoRA quality is good enough for your task, it usually fits a model on one GPU that full training would need a cluster for.
Originally published on the Nodus blog.
Top comments (0)