You ran a training job. It completed. You stopped the pod to save money. You came back and the checkpoints were gone.
No error. No warning. Just an empty directory where the weights should be.
This is one of the most common failures in rented GPU compute and it is entirely preventable.
Why the checkpoints disappeared
RunPod pods have two kinds of storage:
Container storage: The root filesystem (/, /root, /tmp, /home). Fast. Exists only while the pod is running. When you stop or terminate the pod, everything on container storage is wiped. This is ephemeral storage.
Network volume: Mounted at /workspace when you attach one. Persists across pod stops and restarts. Costs a small amount per GB per month but survives when the pod does not.
Most training frameworks default to writing checkpoints to container storage. Hugging Face Trainer defaults to ./output. Axolotl defaults to a path under the working directory. Stop the pod and those files are gone.
One important distinction: stopping a pod and terminating a pod behave differently depending on your RunPod configuration. Spot instances can be terminated abruptly. For safety, treat container storage as temporary regardless of pod lifecycle.
The paths that disappear
If your checkpoint output directory resolves to any of these, the weights will not survive pod termination:
/root/...
/home/...
/tmp/...
./output — resolves to container storage
~/.cache/...
If the path is under /workspace, it survives provided /workspace is backed by a network volume you have attached.
Two separate settings. Both must be fixed.
This is the detail that causes most repeat failures. There are two independent configuration settings that both default to ephemeral storage. Fixing one does not fix the other.
- Checkpoint output directory
python
-# Hugging Face Trainer
training_args = TrainingArguments(
output_dir="/workspace/outputs",
...
)
yaml
-# Axolotl
output_dir: /workspace/outputs
Not ./output. Not /root/outputs. Explicitly /workspace/outputs.
- HuggingFace cache
Base model weights download to HF_HOME. If it is unset, they go to ~/.cache/huggingface — container storage. When the pod stops, every model shard you downloaded is gone and must be downloaded again.
bash
export HF_HOME=/workspace/.cache/huggingface
Add this to your pod startup script so it runs before Python starts. If your training launches via SSH or an automated script, set it inside the launch script itself rather than relying on .bashrc:
bash
-#!/usr/bin/env bash
export HF_HOME=/workspace/.cache/huggingface
python train.py
Why this keeps happening
Training frameworks assume a permanent local filesystem. Cloud GPU providers assume the developer knows how storage works. Between them sits an unmonitored gap.
During eight structured user research sessions with ML engineers, checkpoint loss to ephemeral storage was the most consistently reported failure. The majority had experienced it. In most cases there was no error the job completed successfully and the weights simply were not there afterwards.
Catch it before the job starts
bash
uvx computefence doctor
Real output from a RunPod A40 pod running v0.3.2 — this session had HF_HOME correctly set, so storage passed. The disk warning on /workspace is real and would have caused a checkpoint failure mid-run:
ComputeFence v0.3.2 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
3 WARNINGS · 0 BLOCKERS · 5 PASSED
Environment
✓ Python 3.11.10
✓ PyTorch 2.4.1+cu124 detected (CUDA available — NVIDIA A40)
✓ NVIDIA A40 compute capability sm_86 is supported by this PyTorch build
⚠ No Accelerate config found — skipping GPU count check
Fix: Run 'accelerate config' to create a configuration file
Storage
✓ HF_HOME is set to /workspace/.cache/huggingface
✓ Persistent volume(s) detected: /workspace
⚠ Volume /workspace — 9.8 GB free of 10.0 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
3 warning(s) found. Review before launching.
Exit codes: 0 = all passed, 1 = warnings, 2 = blockers
9.8 GB free on a 10 GB volume. A 7B model checkpoint alone would have filled it. Caught before billing started.
Before your next training job checklist
Network volume is attached and mounted at /workspace
HF_HOME is set to /workspace/.cache/huggingface
output_dir is set to /workspace/outputs or equivalent
Run uvx computefence doctor to verify before billing starts
GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence
Top comments (0)