Every tensor in PyTorch lives on a device: the CPU, an NVIDIA GPU (cuda), or an Apple Silicon GPU (mps). An operation runs on the device where its inputs are, and all its inputs must be on the same device. Running a model on a GPU therefore comes down to two things: putting the model's tensors on the GPU, and putting each batch of data on the same GPU before it is passed to the model. This article explains how .to() does both, what goes wrong when one of them is missed, and how to write code that runs unchanged on any machine.
Detecting the device
import torch
device = (
"cuda" if torch.cuda.is_available()
else "mps" if torch.backends.mps.is_available()
else "cpu"
)
print("Using:", device)
Output on a machine without a GPU:
Using: cpu
On Google Colab with a GPU runtime this prints cuda, and on a Mac with Apple Silicon it prints mps. The variable is defined once at the top of the script, and every later line uses device instead of a hard-coded name, so the same script runs on all three.
A device can also be written as a torch.device object. "cuda" means the current GPU, and "cuda:0" means GPU number 0. They are not equal as objects:
print(torch.device("cuda"), torch.device("cuda").index)
print(torch.device("cuda:0"), torch.device("cuda:0").index)
print(torch.device("cuda") == torch.device("cuda:0"))
cuda None
cuda:0 0
False
A tensor that was moved with .to("cuda") reports its device as cuda:0, so tensor.device == torch.device("cuda") is False for it. To check which kind of device a tensor is on, compare the type instead: tensor.device.type == "cuda".
Moving the model
import torch.nn as nn
model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 3))
model = model.to(device)
print(next(model.parameters()).device)
.to(device) on a module moves every parameter and every registered buffer, in every sub-module. On a CPU machine the print shows cpu, and on a GPU machine it shows cuda:0.
For a module, .to() works in place. It changes the module's own tensors and returns the same module object:
w_before = model[0].weight
m2 = model.to(torch.float64)
print(m2 is model)
print(model[0].weight is w_before)
True
True
This check uses a dtype conversion so that it runs without a GPU, and a device move goes through the same .to() code. The module is the same object and its Parameter objects are the same objects, only their data was replaced. Because the parameters keep their identity, an optimizer that was created before .to() still points at the model's parameters. The usual order is still to move the model first and create the optimizer after, which avoids depending on this detail.
Only registered tensors move
.to() moves the tensors the module knows about: parameters and buffers registered with register_buffer. A tensor stored as a plain attribute is not moved:
class Scale(nn.Module):
def __init__(self):
super().__init__()
self.lin = nn.Linear(3, 3)
self.plain = torch.ones(3) # plain attribute
self.register_buffer("buf", torch.ones(3)) # registered buffer
m = Scale().to("meta")
print(m.lin.weight.device, m.buf.device, m.plain.device)
meta meta cpu
(meta is a device with no data that PyTorch provides for testing. It is used here because it behaves like a second device on a machine that has only a CPU.) The weight and the buffer moved, the plain tensor stayed on the CPU. On a GPU, self.plain would stay on the CPU in the same way, and the first operation that combines it with a GPU tensor fails. A registered buffer is also saved in the state_dict, and a plain attribute is not.
Moving the data
The model is on the GPU, but the data still starts on the CPU. Each batch is moved inside the training loop:
for bx, by in loader:
bx = bx.to(device)
by = by.to(device)
pred = model(bx)
loss = loss_fn(pred, by)
...
Both the inputs and the targets are moved, because the loss function combines pred (on the GPU) with by.
For a tensor, .to() is not in place. It returns a tensor on the new device and leaves the original where it was:
x = torch.randn(4, 10)
y = x.to("meta")
print(x.device, y.device)
cpu meta
A line like bx.to(device) without assignment creates a GPU copy and throws it away, and bx stays on the CPU. The result must be assigned: bx = bx.to(device).
If the tensor is already on the target device (and has the target dtype), .to() returns the tensor itself without copying:
t = torch.randn(3)
print(t.to("cpu") is t)
True
So model.to(device) and bx.to(device) cost nothing on a CPU-only machine, and the device-agnostic code has no overhead there.
The same-device error
When an operation receives tensors on different devices, it raises a RuntimeError. The message always begins with Expected all tensors to be on the same device. The rest depends on the operation. These are two of the forms in the torch 2.15 binaries, the first a general one and the second from the matrix multiply used by nn.Linear:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
RuntimeError: Expected all tensors to be on the same device. addmm: expected 'mat1' to be CUDA, but got CPU
The fix is to find which tensor is on the wrong device and move it. The usual causes are:
- a batch that was not moved, or was moved without assignment (
bx.to(device)instead ofbx = bx.to(device)). - the targets left on the CPU while the inputs were moved, which makes the loss function fail.
- a tensor stored as a plain attribute of the module instead of a registered buffer, as shown above.
- a new tensor created inside
forward()without a device.
The last case looks like this:
class Bad(nn.Module):
def __init__(self):
super().__init__()
self.lin = nn.Linear(3, 3)
def forward(self, x):
return self.lin(x) + torch.zeros(x.shape[0], 3) # created on the CPU
class Good(Bad):
def forward(self, x):
return self.lin(x) + torch.zeros(x.shape[0], 3, device=x.device)
With the model and the input on the meta device:
Bad -> RuntimeError: Tensor on device cpu is not on the expected device meta!
Good -> meta
torch.zeros, torch.ones, torch.randn and torch.arange create tensors on the CPU unless a device is given. Inside forward(), device=x.device creates them on the same device as the input, whatever that device is.
Moving results back to the CPU
Some operations need CPU tensors, mainly converting to NumPy (and so plotting with matplotlib). .numpy() raises an error in two cases. For a tensor that requires gradients:
a = torch.randn(3, requires_grad=True)
a.numpy()
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
And for a tensor that is not on the CPU. On the meta device the message is:
TypeError: can't convert meta device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
For a GPU tensor the same message names the GPU device instead of meta. The pattern that works for any tensor is:
pred_np = pred.detach().cpu().numpy()
.detach() removes the tensor from the autograd graph, and .cpu() copies it to the CPU (or returns it unchanged if it is already there). For a single number, such as a loss, .item() returns a Python float from a tensor on any device.
Loading checkpoints across devices
torch.save records, for every tensor, the device it was on. A checkpoint saved from a GPU model has its tensors tagged cuda:0. Loading that file on a machine without a GPU fails:
sd = torch.load("model_gpu.pt", weights_only=True)
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
map_location tells torch.load where to put the tensors instead:
sd = torch.load("model_gpu.pt", map_location="cpu", weights_only=True)
sd = torch.load("model_gpu.pt", map_location=device, weights_only=True)
sd = torch.load("model_gpu.pt", map_location={"cuda:0": "cpu"}, weights_only=True)
All three loaded the CUDA-tagged file on the CPU-only machine, and the loaded weights were equal to the originals. The first always loads on the CPU. The second loads on whatever device the script is using. The third maps specific devices, for example a checkpoint saved on GPU 1 loaded on GPU 0 with {"cuda:1": "cuda:0"}.
load_state_dict copies the loaded values into the model's existing tensors, so they keep the model's device and dtype. Loading float32 values into a float64 model gave float64 weights. In the same way, a CPU state_dict loaded into a model that is already on the GPU puts the weights on the GPU.
Should the whole dataset go on the GPU?
Moving data batch by batch keeps only one batch on the GPU at a time. Moving the whole dataset at once avoids the copy on every step, but only works if it fits in GPU memory, which ranges from about 8 GB on consumer cards to 80 GB or more on data-centre GPUs. The size of a dataset stored as float32 is the number of values times 4 bytes:
MNIST train 60,000 x 1 x 28 x 28 = 0.19 GB
CIFAR-10 train 50,000 x 3 x 32 x 32 = 0.61 GB
ImageNet train 1,281,167 x 3 x 224 x 224 = 771.41 GB
MNIST and CIFAR-10 fit on any GPU next to a small model. ImageNet does not fit on any single GPU, and has to be loaded in batches. The memory also has to hold the model, its gradients, the optimizer state and the activations of each batch, so the dataset can use only part of it.
For batch-by-batch loading, two options make the copy faster. DataLoader(..., pin_memory=True) puts each batch in page-locked CPU memory, from which the GPU can copy directly. bx.to(device, non_blocking=True) then lets the copy run while the GPU is still working on the previous step.
The device-agnostic training loop
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
device = (
"cuda" if torch.cuda.is_available()
else "mps" if torch.backends.mps.is_available()
else "cpu"
)
x = torch.randn(500, 10)
y = torch.randint(0, 3, (500,))
loader = DataLoader(TensorDataset(x, y), batch_size=32, shuffle=True)
model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 3)).to(device)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for epoch in range(20):
model.train()
for bx, by in loader:
bx, by = bx.to(device), by.to(device)
loss = loss_fn(model(bx), by)
opt.zero_grad()
loss.backward()
opt.step()
model.eval()
with torch.no_grad():
preds = model(x.to(device)).argmax(dim=1).cpu()
The device appears in four places: the detection at the top, the model's .to(device), the batch's .to(device), and the .cpu() when results leave the model. Nothing else in the loop changes compared with a CPU-only loop.
| Step | Code |
|---|---|
| Detect the device | "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" |
| Model to device |
model = model.to(device) (in place, returns the same module) |
| Batch to device |
bx, by = bx.to(device), by.to(device) (returns new tensors, must be assigned) |
New tensors in forward()
|
torch.zeros(..., device=x.device) |
| Result to CPU |
t.detach().cpu().numpy(), or t.item() for one number |
| Load a checkpoint | torch.load(path, map_location=device, weights_only=True) |
Summary
- Every tensor has a device, and every operation needs all its inputs on the same device.
- Detect the device once and use the
devicevariable everywhere. Compare devices with.typewhen the index does not matter. -
model.to(device)moves all parameters and registered buffers in place. Tensors stored as plain attributes are not moved, so useregister_buffer. -
tensor.to(device)returns a new tensor. Assign the result. If the tensor is already on the device, the same tensor is returned. -
Expected all tensors to be on the same devicemeans one tensor is on the wrong device: usually an unmoved batch or target, a plain attribute, or a tensor created inforward()withoutdevice=x.device. - Use
.detach().cpu().numpy()to get results into NumPy. - Load checkpoints with
map_location. A checkpoint saved on a GPU does not load on a CPU-only machine without it.
This is one chapter's worth of an idea from my book, PyTorch From Ground Up, which builds everything from tensors upward so nothing stays vague. If it helped: 8 chapters are free, no email required, there's a free one-page tensor cheat-sheet here, every example runs in the companion notebooks on GitHub, and the full book is on Leanpub or in paperback and Kindle on Amazon.
More in this series
How Training Actually Works:
- PyTorch Autograd Explained: What
.backward()Actually Does - Backpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees
- What
optimizer.step()Actually Does: SGD, Momentum and Adam by Hand - What Your Loss Function Actually Tells the Model
- nn.Module Explained: The Same Model Built with Raw Tensors and with nn.Module
- model.train() vs model.eval() vs torch.no_grad() in PyTorch
- Saving and Loading Models in PyTorch: state_dict, Checkpoints and Resuming Training
Shape mechanics:
- Reshape vs View in PyTorch
- PyTorch Broadcasting Explained
- What Does
unsqueezeDo in PyTorch? - What Does
keepdimDo in PyTorch? - PyTorch
permutevstranspose
Next in this series: overfitting and regularization.
Top comments (0)