DEV Community: Francisco Booth

“No space left on device” on RunPod: What Causes It and How to Fix It Before Billing Starts

Francisco Booth — Tue, 10 Nov 2026 16:21:32 +0000

You rent a GPU on RunPod, download your model, start fine-tuning. Everything looks fine until the job crashes:

OSError: [Errno 28] No space left on device

Or your training completes but GGUF export fails:

RuntimeError: [Errno 28] No space left on device

You have spent money on GPU compute, potentially hours of training, and now you cannot finish the job.

Having a large storage volume attached does not mean every process is using it. Here is why it happens, how to diagnose it, and how to catch it before the next run starts billing.

Why RunPod pods run out of disk

RunPod pods have two storage layers:

Container disk (/): Fast, small (typically 10–20GB on standard templates), wiped when the pod stops.

Network volume (/workspace): Persistent, sized by you when you create it.

The problem is that several things write to container disk by default even when a network volume is attached:

HuggingFace model cache (~/.cache/huggingface unless HF_HOME is set)
Training checkpoints (most frameworks default to ./output)
pip cache (~/.cache/pip)
GGUF export artifacts

Any one of these is manageable. Combined on a 10–20GB container disk, they fill it fast.

Find out which filesystem is actually full first

Before changing any configuration, identify where space is being consumed:

df -h

Look at / and /workspace separately. If / is almost full but /workspace has space, you have a configuration problem not a capacity problem.

To find what is consuming space on the root filesystem:

du -xhd1 / 2>/dev/null | sort -h

The GGUF export case

GGUF export is the peak disk moment in a typical fine-tuning pipeline. For a 12B model you briefly hold simultaneously:

The 16-bit merged model (~24GB)
The f16 GGUF (~24GB)
The quantised output (~6GB)
Additional temporary working files

Combined peak can exceed 50GB. The exact amount depends on your conversion workflow, quantisation method, and whether intermediate files coexist. If your working directory only has 20GB free, the export fails partway through. Hours of training wasted.

One operator running a Gemma 12B fine-tune on Vast.ai hit exactly this training ran clean for hours, the GGUF export step failed because the working directory had ~20GB against a ~45–47GB peak requirement.

Three things to set before every job

Point HF_HOME at the network volume

export HF_HOME=/workspace/.cache/huggingface
mkdir -p $HF_HOME

Add this to your pod startup script, not just your shell profile if training launches via SSH or an automated script it will not load .bashrc.

Point your output directory at the network volume

Hugging Face Trainer

training_args = TrainingArguments(
output_dir="/workspace/outputs",
save_total_limit=2, # prevents old checkpoints filling the volume
)

Axolotl

output_dir: /workspace/outputs

Size your volume for the full pipeline, not just training

Account for base model weights, checkpoints, and export artifacts simultaneously:

7B model: at least 60GB
12B model: at least 100GB
70B model: at least 350GB

Catch it before billing starts

uvx computefence doctor

Real output from a RunPod A40 pod that caught a disk warning before launch:

ComputeFence v0.3.2 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

3 WARNINGS · 0 BLOCKERS · 5 PASSED

Storage
✓ HF_HOME is set to /workspace/.cache/huggingface
✓ Persistent volume(s) detected: /workspace
⚠ Volume /workspace — 9.8 GB free of 10.0 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume

9.8GB free on a 10GB volume. The HF_HOME was correctly set. The volume was mounted. But the available capacity was too low for a full training run. Caught before the job started.

ComputeFence warns below 20GB and blocks below 5GB.

Current limitation: ComputeFence does not yet calculate peak storage requirements from your model size. It checks available capacity and cache configuration. That is a narrower claim than “your pipeline will fit” but it catches the cases where you are obviously going to run out before you start paying.

Make the check automatic:

alias train="uvx computefence doctor && python train.py"

If disk is below 5GB, ComputeFence exits with a non-zero code and the training command does not run.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

“no kernel image is available for execution on the device” What This PyTorch Error Means On RunPod Blackwell GPUs

Francisco Booth — Mon, 09 Nov 2026 13:12:16 +0000

You launch a training job on RunPod. PyTorch detects your GPU, torch.cuda.is_available() returns True, and your environment checks appear to pass.

Then your script reaches its first real CUDA operation and crashes:

RuntimeError: CUDA error: no kernel image is available for execution on the device

If you are running on a Blackwell GPU with an older PyTorch build, you have hit a GPU architecture compatibility problem. The frustrating part is that your environment looks perfectly healthy right up until the first CUDA kernel executes.

What causes this error

CUDA kernels are compiled for specific GPU architectures, identified by compute capability for example sm_86 for an A40, sm_90 for an H100.

When you install PyTorch, the build you download contains kernels compiled for a fixed range of architectures. PyTorch 2.4.1+cu124 supports sm_50 through sm_90.

Blackwell GPUs: the RTX PRO 4000, RTX PRO 4500, RTX PRO 6000, and RTX 5090 have compute capability sm_120. They are outside that range.

With an incompatible PyTorch build, PyTorch does not fail immediately. torch.cuda.is_available() returns True. The GPU name appears in the output. Everything looks fine. Then the training script reaches the first real CUDA kernel call the first matrix multiplication, the first forward pass and it throws the error above.

By that point the pod is already allocated, your dependencies are installed, and billing has started.

Note on the B200: The B200 belongs to the sm_100 architecture family, not sm_120. Always check the actual compute capability reported by your GPU rather than assuming every Blackwell card uses the same architecture.

You can check this in two lines:

import torch
cap = torch.cuda.get_device_capability(0)
print(f"Device capability: sm_{cap[0]}{cap[1]}")
print(f"Supported architectures: {torch.cuda.get_arch_list()}")

If your device capability is not in the supported list, the run will fail when the first kernel executes.

Why torch.cuda.is_available() returning True is not enough

This is where the problem becomes deceptive.

torch.cuda.is_available() confirms that PyTorch can access CUDA and identify a device. It does not confirm that the installed build contains compatible kernels for every operation you intend to run. Architecture compatibility is a separate check.

GPU detection and GPU compatibility are two different things.

First-hand reproduction on RunPod

I reproduced this on a RunPod RTX PRO 4000 Blackwell pod using the default PyTorch 2.4.1+cu124 template.

The terminal output looked clean:

✓ PyTorch 2.4.1+cu124 detected (CUDA available — NVIDIA RTX PRO 4000 Blackwell)

Meanwhile PyTorch was printing a warning to stderr that sm_120 was not supported by this build. The pre-flight check passed. Training would have failed on the first real kernel call.

The warning that appeared in stderr:

UserWarning: CUDA device capability sm_120 is not supported by this PyTorch build.
Supported capabilities: sm_50 through sm_90.

If your training script runs in a tmux session, pipes output to a log file, or uses a framework like Axolotl or Unsloth, you will miss this warning entirely until the forward pass crashes.

The pod also reported /workspace - 466126.9 GB free of 2190892.5 GB unusually large volume figures that appear to be a virtualisation reporting artefact on early Blackwell deployments rather than real capacity.

This reproduction exposed a gap in the original pre-flight checks and led directly to the GPU architecture check added in ComputeFence v0.3.2.

Which GPUs are affected with standard PyTorch images

As of PyTorch 2.4.x, these cards are not supported by the builds shipped in standard RunPod PyTorch templates:

NVIDIA RTX PRO 4000 Blackwell: sm_120
NVIDIA RTX PRO 4500 Blackwell: sm_120
NVIDIA RTX PRO 6000 Blackwell: sm_120
NVIDIA RTX 5090: sm_120
NVIDIA B200: sm_100 (different architecture, same class of problem)

If RunPod assigns one of these GPUs and you are on the default template, you will hit this error on the first kernel call.

How ComputeFence catches it

Since v0.3.2, ComputeFence compares each GPU’s compute capability against the architectures supported by the installed PyTorch build using get_device_capability() and get_arch_list(). A mismatch is flagged as a blocker not a warning:

✗ BLOCKER: GPU architecture mismatch
NVIDIA RTX PRO 4000 Blackwell has compute capability sm_120
This PyTorch build supports sm_50 through sm_90
Training will fail on the first CUDA kernel call
Fix: Use a PyTorch build compiled for sm_120, or select a different GPU

This runs before billing starts.

uvx computefence doctor

Note: get_arch_list() reports architectures compiled into the PyTorch installation. It is a useful compatibility signal but does not guarantee that every third-party CUDA extension will work. Run a small representative CUDA operation before committing to a long training run on new hardware.

How to fix it

Option 1: Install a PyTorch build that supports your GPU

Use a release compiled for sm_120. Check pytorch.org/get-started/locally for current compatibility. After installing, verify with the two-line check above before launching your job.

Option 2: Select a different GPU

If you need to use an existing PyTorch image with pinned dependencies, select a GPU whose architecture is in the supported range. An A40 (sm_86) or H100 (sm_90) will work with PyTorch 2.4.x.

The broader lesson

GPU detection is not GPU compatibility. As providers add newer hardware generations, the gap between the latest cards and the PyTorch builds in standard templates will keep appearing. Passing torch.cuda.is_available() is not proof that your training workload can execute.

Check the architecture before billing starts.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

Your RunPod Checkpoints Disappeared When The Pod Stopped. Here Is Why.

Francisco Booth — Sun, 08 Nov 2026 18:56:29 +0000

You ran a training job. It completed. You stopped the pod to save money. You came back and the checkpoints were gone.

No error. No warning. Just an empty directory where the weights should be.

This is one of the most common failures in rented GPU compute and it is entirely preventable.

Why the checkpoints disappeared

RunPod pods have two kinds of storage:

Container storage: The root filesystem (/, /root, /tmp, /home). Fast. Exists only while the pod is running. When you stop or terminate the pod, everything on container storage is wiped. This is ephemeral storage.

Network volume: Mounted at /workspace when you attach one. Persists across pod stops and restarts. Costs a small amount per GB per month but survives when the pod does not.

Most training frameworks default to writing checkpoints to container storage. Hugging Face Trainer defaults to ./output. Axolotl defaults to a path under the working directory. Stop the pod and those files are gone.

One important distinction: stopping a pod and terminating a pod behave differently depending on your RunPod configuration. Spot instances can be terminated abruptly. For safety, treat container storage as temporary regardless of pod lifecycle.

The paths that disappear

If your checkpoint output directory resolves to any of these, the weights will not survive pod termination:

/root/...
/home/...
/tmp/...
./output — resolves to container storage
~/.cache/...

If the path is under /workspace, it survives provided /workspace is backed by a network volume you have attached.

Two separate settings. Both must be fixed.

This is the detail that causes most repeat failures. There are two independent configuration settings that both default to ephemeral storage. Fixing one does not fix the other.

Checkpoint output directory

python
-# Hugging Face Trainer
training_args = TrainingArguments(
output_dir="/workspace/outputs",
...
)

yaml
-# Axolotl
output_dir: /workspace/outputs

Not ./output. Not /root/outputs. Explicitly /workspace/outputs.

HuggingFace cache

Base model weights download to HF_HOME. If it is unset, they go to ~/.cache/huggingface — container storage. When the pod stops, every model shard you downloaded is gone and must be downloaded again.

bash
export HF_HOME=/workspace/.cache/huggingface

Add this to your pod startup script so it runs before Python starts. If your training launches via SSH or an automated script, set it inside the launch script itself rather than relying on .bashrc:

bash
-#!/usr/bin/env bash
export HF_HOME=/workspace/.cache/huggingface
python train.py

Why this keeps happening

Training frameworks assume a permanent local filesystem. Cloud GPU providers assume the developer knows how storage works. Between them sits an unmonitored gap.

During eight structured user research sessions with ML engineers, checkpoint loss to ephemeral storage was the most consistently reported failure. The majority had experienced it. In most cases there was no error the job completed successfully and the weights simply were not there afterwards.

Catch it before the job starts

bash
uvx computefence doctor

Real output from a RunPod A40 pod running v0.3.2 — this session had HF_HOME correctly set, so storage passed. The disk warning on /workspace is real and would have caused a checkpoint failure mid-run:

ComputeFence v0.3.2 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

3 WARNINGS · 0 BLOCKERS · 5 PASSED

Environment
✓ Python 3.11.10
✓ PyTorch 2.4.1+cu124 detected (CUDA available — NVIDIA A40)
✓ NVIDIA A40 compute capability sm_86 is supported by this PyTorch build
⚠ No Accelerate config found — skipping GPU count check
Fix: Run 'accelerate config' to create a configuration file

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
3 warning(s) found. Review before launching.
Exit codes: 0 = all passed, 1 = warnings, 2 = blockers

9.8 GB free on a 10 GB volume. A 7B model checkpoint alone would have filled it. Caught before billing started.

Before your next training job checklist

Network volume is attached and mounted at /workspace
HF_HOME is set to /workspace/.cache/huggingface
output_dir is set to /workspace/outputs or equivalent
Run uvx computefence doctor to verify before billing starts

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

I Ran Eight User Research Sessions With ML Engineers About GPU Training Failures. Here Is What They Had In Common.

Francisco Booth — Sat, 07 Nov 2026 21:08:25 +0000

The most expensive failure in machine learning infrastructure is not a crash. A crash throws an exception, halts the container, and stops the billing meter.

The most expensive failure is a job that succeeds at doing the wrong thing.

Before building ComputeFence I ran eight structured research sessions with ML engineers running fine-tuning workloads on rented GPU compute RunPod, Vast.ai, Lambda Labs. I asked what had gone wrong, what it cost, and whether they saw an error message.

The pattern was consistent enough that I stopped being surprised after the third session.

1. Checkpoints lost to ephemeral storage

The majority of engineers I spoke to had lost model weights because their checkpoint output directory was writing to container storage instead of a persistent volume.

On providers like RunPod and Vast.ai, storage is partitioned:

Ephemeral container root (/root, /tmp) is wiped the moment a pod terminates
Persistent network storage lives on dedicated paths like /workspace

Default training scripts routinely dump checkpoints to ./output or ~/.cache. The job runs for eight hours, logs a clean exit, terminates the pod. The weights are gone.

The fix is one line set --output_dir /workspace/checkpoints but training harnesses assume the filesystem is permanent, and nothing tells you otherwise.

2. Silent garbage and false convergence

Most had also run a job that finished without errors but produced a model that scored at random chance or far below expectations.

The causes varied:

Duplicate rows poisoning low-rank adapter updates.
Conflicting labels introducing contradictory supervision.
Class weight misconfiguration causing binary classification loss to collapse to 0.693 immediately the exact value of ln(2), which is what you get when a model predicts the majority class for every example.

From the outside, the terminal logged steady iterations, GPU utilisation sat at 98%, and loss ticked down slightly. The failure only revealed itself during post-training evaluation after the compute was spent.

3. No error message for any of it

This was the consistent thread across every session. The expensive failures were not loud crashes. They were silent completions that only revealed themselves when the model was evaluated.

One engineer described it as: “The job succeeded at doing the wrong thing.”

Training frameworks assume the local environment has been provisioned correctly. The cloud GPU host assumes the developer knows what they are doing. Between them sits an unmonitored gap and that gap is where the money goes.

The enterprise version of the same problem

One session stood out. An engineer at a major industrial company described how their ML team maintains a five to six part manual bash script that they run before every GPU job. Environment checks, storage checks, dependency checks all hand-rolled and maintained internally.

The reason: no standardised tool exists for this. Every team either builds its own pre-flight script or skips it and pays the cost when something fails silently.

What the sessions changed about ComputeFence

Before the research I was focused on CUDA CPU fallback the failure I had hit personally. 24 seconds per iteration instead of 0.4. Training on CPU for hours at full GPU rates. No warning.

After eight sessions I understood that the most common failure was not the GPU check. It was storage.

HF_HOME pointing to ephemeral disk. Checkpoint directory writing to container storage. Disk filling up during export. These failures are quieter than a CUDA mismatch and more expensive because they surface at the end of a long run, not the beginning.

The storage checks became the first thing ComputeFence runs. GPU visibility is second.

The 30-second pre-flight check

uvx computefence doctor

In order of how commonly the failures appeared across the eight sessions:

HuggingFace cache path: Flags if HF_HOME is unset or pointing to ephemeral storage.
Checkpoint output directory: Flags output paths under known ephemeral prefixes.
Disk headroom: Warns below 20GB, blocks below 5GB.
GPU and CUDA visibility: Fails if torch.cuda.is_available() returns False.
Accelerate configuration: Flags mismatches between configured process count and visible GPUs.
Dataset integrity: Optional scan for duplicates and conflicting labels with --dataset.

Every warning prints the exact fix command. The check takes under 30 seconds.

The rule of cloud compute

A job that crashes on line one is cheap fix the error and rerun.

A job that crashes after six hours because the disk filled up on checkpoint four is expensive.

A job that runs for twelve hours, reports exit code 0, and saves its weights to ephemeral storage is the most expensive of all.

Pre-flight checks cannot verify model quality or guarantee convergence. They catch the entirely preventable configuration errors that are visible before python train.py. That is a narrower claim than “prevent all failures” but it is an honest one.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

Your GGUF Export Will Fail At The End. The Disk Was Never Going To Be Enough.

Francisco Booth — Fri, 06 Nov 2026 15:41:53 +0000

Training finishes clean. The LoRA adapter saved. You kick off the GGUF export.

Then it dies.

No space left on device.

The frustrating part is that nothing warned you. The training job ran for hours without a single disk error. The problem only appeared at the worst possible moment the final step, after all the compute was spent.

Why export kills jobs that training survived

During fine-tuning the disk footprint stays small. You are writing a LoRA adapter and periodic checkpoints a few hundred megabytes at most. The base model weights stay in memory, not on disk.

GGUF export is different. To convert a fine-tuned model you briefly hold multiple large files simultaneously:

The 16-bit merged model
The f16 GGUF
The quantised outputs

For a 12B model that peak sits at approximately 45–47GB. If your working directory only has 20GB free, the export fails partway through. The merge is incomplete. The GGUF is corrupted or missing.

A Vast.ai operator described this exactly: training ran clean, disk footprint stayed small, then GGUF export failed because the working directory had ~20GB against a ~45–47GB peak requirement. Silent end-of-pipeline failure.

Why this is hard to see before it happens

The disk check that matters for export is different from the disk check that matters for training. During training you need headroom for checkpoints a few GB. During export you need headroom for the full model merge plus conversion artifacts simultaneously.

Most pre-flight thinking is focused on training. Export is an afterthought.

There is no error before the job starts. The disk looks fine. Training looks fine. The failure only surfaces at the conversion step.

The check before you start

Before any run that ends in GGUF export, check how much space the export will actually need.

Rough estimate for a model of size N billion parameters:

16-bit merge: approximately 2 × N GB
f16 GGUF: approximately 2 × N GB
Quantised output (Q4): approximately 0.5 × N GB
Peak simultaneous: approximately 4–4.5 × N GB

For a 12B model that is 48–54GB. For a 7B model that is 28–32GB.

If your working directory does not have that headroom before the job starts, the export will fail.

-# Check available disk on your working volume
df -h /workspace

-# Check total size of base model weights
du -sh /workspace/.cache/huggingface/

Or run a full environment check in 30 seconds:

uvx computefence doctor

Real output from a RunPod A40 pod (v0.2.11 — current version is v0.3.1):

ComputeFence v0.2.11 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

4 WARNINGS · 0 BLOCKERS · 4 PASSED

Environment
✓ Python 3.12.3
✓ PyTorch 2.8.0+cu128 detected (CUDA available — NVIDIA A40)
⚠ No Accelerate config found — skipping GPU count check
Fix: Run 'accelerate config' to create a configuration file

Storage
✓ HF_HOME is set to /workspace/.cache/huggingface/
✓ Persistent volume(s) detected: /workspace
⚠ Volume /workspace — 6.0 GB free of 9.3 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume
⚠ Root disk (/) — 16.2 GB free of 20.0 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4 warning(s) found. Review before launching.

That disk warning 6GB free on a 9.3GB volume would catch you before you start a 12B export that needs 45–47GB. The job would not even begin with that headroom. That is the point.

The fix

Three options before any export job:

Option 1 — Attach a larger volume
Size it to at least 4–5× your model’s parameter count in GB. For a 12B model that means at least 60GB free before you start.

Option 2 — Export in stages
Merge first, check disk, then convert. Do not run the full pipeline in a single script if you are not certain about headroom.

Option 3 — Export locally or on cheaper compute
GGUF conversion does not need a GPU. Run it on a CPU instance with large cheap disk rather than holding a GPU pod open for the conversion step.

What ComputeFence catches today

ComputeFence checks disk headroom on both the workspace volume and the root disk before launch. It warns below 20GB and blocks below 5GB.

It does not yet estimate the full GGUF export peak requirement from your model size. That check is on the roadmap. For now the disk warning is the signal if you are below 20GB free before training starts, you will almost certainly be below what export needs at the end.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

Why Your Training Script Will Fail on the Next GPU Pod Even If It Worked Last Time

Francisco Booth — Mon, 26 Oct 2026 17:45:11 +0000

You ran a training job last week. It completed cleanly. Today you rent a fresh pod, run the same script, and something breaks sometimes silently.

Same script.

Same model.

Different pod.

Different result.

This is environment drift and it is one of the most expensive, least documented failure modes in rented GPU compute.

The myth of the standard pod

When you rent a GPU on RunPod or Vast.ai you are not spinning up a standardised environment. You are renting whatever machine a host made available, layered with a base image that varies by provider, instance type, and when it was last updated.

Between two ostensibly identical pods, the following vary without warning:

CUDA version: One pod runs CUDA 12.1, the next runs 12.6, the next runs 12.8
PyTorch build: A cu118 build behaves differently from cu121 and cu128, particularly around custom kernels and FlashAttention
System libraries: gcc version, glibc, NVIDIA driver revision
Pre-installed ML packages: One image ships transformers 4.41, another ships 4.48

None of these differences crash your SSH session. The pod boots fine, nvidia-smi shows the GPU, your /workspace mounts cleanly. The divergence is invisible until your script hits the specific dependency that behaves differently on this image.

How drift breaks training

Environment drift rarely produces a clean early traceback. It fails in three ways:

Silent CPU fallback

If the PyTorch build on the image does not align with the underlying CUDA driver, torch.cuda.is_available() can return False without throwing an exception. Your script resolves device to CPU and keeps running.

I hit this on a real Vast.ai run iteration time jumped from 0.4 seconds to 24 seconds. GPU utilisation: 0%. The script ran to completion at CPU speed while billing at full GPU rates.

Adding an explicit guard prevents this:

python
import torch
if not torch.cuda.is_available():
raise RuntimeError("CUDA unavailable refusing to start training")

Dependency version mismatch

Running ModernBERT requires transformers>=4.48.0. A pod with an older base image ships 4.41. The model may load via remote code but behave incorrectly, or fail partway through the first epoch with an obscure attribute error. The failure is not at import it is 40 minutes into your run.

Accelerate configuration mismatch

Your Accelerate config was created on a 2-GPU node with num_processes: 2. Your new pod has one GPU. The launcher may hang, produce garbage output, or corrupt checkpoints. No error at launch.

Why this is expensive to debug

Environment drift failures share one trait: they are invisible at pod start. The GPU is visible, the script launches, everything looks normal. The failure surfaces at the first forward pass, the first checkpoint save, or not at all if the job completes but produces wrong output.

By the time you notice, you have paid for the compute.

What to check before every new pod

-# Verify PyTorch can see the GPU

python3 -c "import torch; print(torch.version, torch.version.cuda, torch.cuda.is_available())"

-# Check key package versions

pip show transformers accelerate | grep Version

Or run all of this plus HF_HOME path, persistent volume detection, and disk headroom in one command before billing starts:

uvx computefence doctor

The check takes under 30 seconds.

The honest framing

The CPU fallback failure is confirmed from my own Vast.ai runs. The CUDA version variance is confirmed from my A40 session PyTorch 2.8.0+cu128 on that specific pod, not universal. The dependency and Accelerate examples are well-understood engineering realities, not personal war stories.

The point is not that every new pod will break your script. It is that a new pod is a new environment and treating it as identical to the last one is an assumption worth checking before the meter starts running.

Your last pod working is evidence that your setup worked once. It is not proof that your next pod is ready.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

Why Axolotl Users On RunPod Lose Checkpoints Silently And How To Catch It Before Your Job Starts

Francisco Booth — Thu, 15 Oct 2026 21:25:39 +0000

Axolotl is one of the cleanest fine-tuning frameworks available. The problem is not Axolotl. The problem is what happens to your checkpoints when you run it on a rented GPU pod.

What I found running Axolotl on a real RunPod A40

Last night I spun up a RunPod A40 pod at $0.59 per hour with a 10GB network volume mounted at /workspace. I installed ComputeFence and then installed Axolotl 0.20.0.

Before launching any training job I ran:

computefence doctor

Here is the exact output:

ComputeFence v0.2.11 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

4 WARNINGS · 0 BLOCKERS · 4 PASSED

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4 warning(s) found. Review before launching.

The GPU passed. HF_HOME passed correctly pointing to the persistent volume. But the disk warning was real and would have caused a problem.

Why the disk warning matters for Axolotl specifically

Axolotl is not a small install. Its dependency tree pulled in over 200 packages torch, transformers, accelerate, flash-attention, deepspeed, gradio, and dozens more. By the time the install finished, my 10GB network volume had 6GB free and my root disk had 16.2GB free. Both below the 20GB safe threshold.

Axolotl saves checkpoints during training. If you are fine-tuning a 7B model with save_steps: 100, checkpoints can be several gigabytes each. With 6GB free on the workspace volume, a checkpoint save mid-run would fail. The job would die without completing. The weights you had trained up to that point would be gone.

No error before the job starts. No warning from Axolotl. Just a failed save at step 100 or 200 or wherever the disk fills.

The fix

Two options before launching:

Option 1 - Attach a larger volume
If you are planning a long fine-tuning run, attach at least a 50GB network volume. Axolotl’s dependencies alone consume 3–4GB. Your base model weights consume more. Leave headroom for checkpoints.

Option 2 — Configure checkpoint saving to the persistent volume explicitly

In your Axolotl config:

output_dir: /workspace/outputs

Not /root/outputs. Not a relative path. Explicitly /workspace/outputs so checkpoints land on the persistent volume, not the ephemeral root disk.

Run this before every Axolotl job

pip install computefence && computefence doctor

Or without installing:

uvx computefence doctor

It checks GPU visibility, HF_HOME path, persistent volume detection, and disk headroom before billing starts. The whole check takes under 30 seconds.

The disk warning it caught on my A40 pod was real. If I had launched a training job immediately after installing Axolotl without checking, the job would have run until the first checkpoint save and died.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

Why HF_HOME Points To Ephemeral Storage On RunPod By Default And How To Fix It

Francisco Booth — Sun, 11 Oct 2026 14:23:40 +0000

If you have ever run a HuggingFace model download on RunPod and come back to find your cache gone, this is why.

The default is wrong for rented GPU pods

HuggingFace sets HF_HOME to ~/.cache/huggingface when no environment variable overrides it. On a standard Linux machine that is fine your home directory persists.

On RunPod, your home directory is on the container volume, which is ephemeral. When the pod stops, it is gone.

RunPod has two storage layers: the container volume, which is ephemeral, and the network volume mounted at /workspace, which persists across pod restarts. HF_HOME defaults to the wrong one.

This means every model download, every tokeniser, every dataset you pulled via datasets.load_dataset() gone on pod restart. You pay to download them again on the next run.

On large models that is not a minor inconvenience. A 7B model download at pod startup adds minutes and real cost every single time.

The fix is one line in your startup script:

export HF_HOME=/workspace/.cache/huggingface

/workspace is RunPod’s persistent volume mount. One line and your cache survives every restart.

Why this is easy to miss

There is no error message. HuggingFace downloads the files successfully to the ephemeral path. Your training run completes. The problem only surfaces when you restart the pod and the download happens again and even then it looks like a slow startup, not a misconfiguration.

Silent failures are the hardest to debug because you are not looking for them.

Automating the check

If you want this caught automatically before every job rather than relying on remembering to set the variable, ComputeFence checks it as part of its pre-flight scan:

uvx computefence doctor

It flags HF_HOME pointing to ephemeral storage and prints the exact export command for your platform. It also checks GPU visibility, Accelerate config, checkpoint directory persistence, and disk headroom the other silent failures that cost compute budget without a log entry.

The two-line pre-flight for every RunPod job

export HF_HOME=/workspace/.cache/huggingface
uvx computefence doctor

The first line fixes the cache. The second confirms everything else is configured correctly before the GPU bill starts.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

My GPU Training Job Ran for 20 Hours and Produced Nothing. There Was No Error Message.

Francisco Booth — Tue, 06 Oct 2026 20:12:49 +0000

I was fine-tuning the Exolio AI classifier that detects machine-generated texts. The classifier can be seen at huggingface.co/FranciscoBooth1/exolio-ai-detector.

Between January and April 2026, I ran dozens of training jobs using Vast.ai. Cost of runs: approximately £1,000.

All of the eventually identified failed runs had one thing in common: none of them gave me an obvious sign that they had failed.

The run that ran all night and gave no output

For Exolio v15, I was switching from DeBERTa-v3-large to ModernBERT. I rented a machine, uploaded the training script, installed dependencies, and initiated the run.

Everything in the terminal looked exactly as expected:

Uploading training script...
Installing Python dependencies (this may take several minutes)...
[Dependencies installed successfully]
Starting training in tmux session 'ssh_tmux'...
Training started successfully! Instance will auto-destroy when complete.

The script returned to my local terminal and training started running in tmux on the remote machine. There was no way to monitor what happened inside that tmux session. Everything looked good. I went to sleep.

The next day I tried to SSH back in:

ssh: connect to host [instance IP] port 55157: Connection refused

The instance was gone. The log files disappeared along with it. I checked my HuggingFace repository to see if there were any results:

Last modified: 2026-04-12 01:20:33+00:00
.gitattributes
config.json
deberta_3class_latest.zip
model.safetensors
tokenizer.json
training_args.bin

There was no modernbert_latest.zip. The last modification time was April 12 which was the previous DeBERTa run. No new files had been added.

That was the only sign that something had gone wrong. Not an error message. Not a bad benchmark score. Absence of new files.

I could not recover the logs the instance auto-destroyed and took everything with it so I cannot identify the root cause. Twenty hours had passed, the instance was gone, and no new model had appeared in my HuggingFace repository.

I ran a training job, nothing obviously failed, and after twenty hours the infrastructure disappeared and there was no artifact left. I was not able to tell exactly why.

Another failure: training on CPU for hours without knowing

Earlier in the same period I rented what seemed to be a good machine and started fine-tuning DeBERTa. The loss function was printing. The iteration counter was counting. Everything was going as expected.

Except the speed was wrong. I saw approximately 24 seconds per iteration. On working instances the same workload had been around 0.4 seconds per iteration.

I ran nvidia-smi. GPU utilisation: 0%.

My script resolved the device to CPU because torch.cuda.is_available() returned False no exception thrown, just CPU. The machine had CUDA installed but my PyTorch environment could not see the GPU. Training continued on CPU with no errors while I was paying full GPU rates.

I stopped the run. Rented a different machine. Restarted.

The pattern

In both cases there was no error message that I could act on. In the first case there was no way to know anything had gone wrong until I tried to SSH back in. In the second case I noticed a strange iteration speed and stopped the job.

In the second case the problem was visible before python train.py. torch.cuda.is_available() returning False is visible before the job starts. GPU utilisation at 0% is visible within seconds of launch.

Additionally and not as an explanation of the overnight run, since I had no logs there are other silent failure modes that are also detectable before the training job starts. At least some of the conditions surrounding these failures were detectable before python train.py if I had known to check them:

Whether PyTorch could see the GPU: torch.cuda.is_available() returns False before the job starts
Whether the HuggingFace cache was configured to use a potentially ephemeral path: an environment variable check
Whether the checkpoint output directory was under a known instance-local prefix: a path check

None of these checks were complicated. I just did not perform them systematically on each new machine.

What I built

After enough of these I decided to build a pre-flight checker. One command before each training job:

pip install computefence && computefence doctor

Or without installing the package:

uvx computefence doctor

Real output from uvx computefence doctor on my local Mac — on a real GPU pod the PyTorch and volume checks run against actual hardware:

ComputeFence v0.2.6 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

5 WARNINGS · 0 BLOCKERS · 1 PASSED

Environment
✓ Python 3.14.3
⚠ PyTorch not found — skipping GPU checks
Fix: pip install torch torchvision torchaudio

Storage
⚠ HF_HOME is not set. HuggingFace may cache to instance-local storage
that is not persistent across pod termination.
Fix: Set HF_HOME to a persistent volume path before training
⚠ No mounted persistent volumes detected at /workspace, /runpod-volume, or /vast
Fix: Mount a network volume at /workspace before launching your job
⚠ Root disk (/) — 12.1 GB free of 460.4 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume

Dataset
⚠ No dataset path provided — skipping dataset checks

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
5 warning(s) found. Review before launching.

Checks:

1) GPU and CUDA visibility - fails if torch.cuda.is_available() returns False; reports relevant CUDA and PyTorch version information and warns on detected version skew

2) HuggingFace cache path — warns if HF_HOME is not set or points to a path known to be instance-local on supported providers.

3) Accelerate configuration — warns if there is an inconsistency between the configured number of processes and the GPUs visible in the environment.

4) Disk headroom — warns when less than 20 GB is available; blocks if less than 5 GB is available.

5) Checkpoint output directory — warns if the path is under known instance-local prefixes that may not survive pod termination.

6) Dataset — optionally checks for duplicates and conflicting labels with --dataset.

Checks that it does not do: training script correctness, model architecture, learning rate safety, runtime behaviour during training, or GPU utilisation during training. ComputeFence is a pre-flight check. It does not monitor the running job.

A passing check does not guarantee that the training job will succeed.

The pre-flight check took under 30 seconds on my test pod.

The ask

ComputeFence is free, open source, and MIT licensed.

If you run training jobs on Vast.ai, RunPod, or any other rented GPU, try uvx computefence doctor before your next run. If it finds something interesting, paste the output in the comments below.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

24 seconds per iteration instead of 0.4. I paid for six hours of GPU compute and trained on CPU the entire time.

Francisco Booth — Sun, 06 Sep 2026 10:22:30 +0000

Failure 1 — CUDA silently fell back to CPU

My training job launched on Vast.ai and ran to completion. Iteration time was 24 seconds instead of 0.4 seconds. CUDA had fallen back to CPU silently. PyTorch logged nothing. I had been billed for six hours of GPU compute while training on an unaccelerated CPU thread the entire time.

Failure 2 — HF_HOME on ephemeral disk

Every fresh pod re-downloaded base model weights to /root/.cache — the ephemeral container disk wiped on pod shutdown. Same download, same cost, every run. The fix is one line. I did not know it for weeks.

export HF_HOME=/workspace/.cache/huggingface

Failure 3 — Accelerate config mismatch

My accelerate config had num_processes: 2. The pod had one GPU. Training launched, appeared to run, and produced garbage output. No error thrown. The configuration simply did not match the hardware.

Failure 4 — Dirty dataset

28,432 duplicate rows. 312 conflicting labels. Loss collapsed to 0.693 on step one — the exact cross-entropy value for random guessing on a binary classification problem. I spent three days debugging model architecture and learning rates before I scanned the dataset.

The pattern

None of these printed an exception. All of them were visible before python train.py if I had known what to check.

I spoke to 13 ML engineers on RunPod, Vast.ai, and AWS. Nine had lost checkpoints to ephemeral disk. Eight had shipped silent garbage with no log error.

An ML engineer at a major German industrial company told me his team maintains a five or six part manual bash script that they run before every GPU job because no standardised tool exists.

When enterprise teams are hand-rolling bash scripts, the problem is real.

So I built ComputeFence.

Standard install:

pip install computefence
computefence doctor

Zero install on a fresh pod:

uvx computefence doctor

30 seconds. Every warning prints the exact fix command.

ComputeFence v0.2.5 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 WARNINGS · 0 BLOCKERS · 3 PASSED

Storage
⚠ HF_HOME is not set — model weights will cache to ephemeral disk
Fix: export HF_HOME=/workspace/.cache/huggingface
⚠ Root disk (/) — 14.3 GB free (below 20 GB)
Fix: Free up disk or move checkpoints: df -h to check usage

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 warning(s) found. Review before launching.

It checks GPU and CUDA visibility, HuggingFace cache path persistence, Accelerate GPU count versus what is actually on the instance, disk headroom for checkpoints, and checkpoint output directory persistence via --output-dir.

It does not check training script correctness, learning rate safety, or anything that only fails during the run. HuggingFace cache and your checkpoint output directory are separate paths — fixing one does not fix the other. ComputeFence catches configuration mistakes before the GPU starts billing.

Add it to your pod startup script. Exit code is 0 on warnings and 1 only on hard blockers so training will still launch on warnings:

pip install computefence && computefence doctor && python train.py

Does it catch anything real?

Three operators have run it before paid training jobs on RunPod.

One caught HF_HOME writing to ephemeral disk and an Accelerate config mismatch on a RunPod A100. He fixed both before launch.

One confirmed the storage warning matched real pod behaviour and said he would not have caught it without the tool.

Aaron, an ML engineer who ran ComputeFence on a RunPod A40, said the Accelerate warning would have made him stop and investigate before launching.

Still very early.

If you run it before your next paid job on RunPod, Vast.ai, Lambda Labs, or any bare metal instance — paste your computefence doctor output in the comments or open an issue on GitHub. I want to know which checks fire on real setups and which are noise.

Free. MIT licensed. Works on any bare metal GPU provider.

GitHub: github.com/Francisco-Booth/ComputeFence