Axolotl is one of the cleanest fine-tuning frameworks available. The problem is not Axolotl. The problem is what happens to your checkpoints when you run it on a rented GPU pod.
What I found running Axolotl on a real RunPod A40
Last night I spun up a RunPod A40 pod at $0.59 per hour with a 10GB network volume mounted at /workspace. I installed ComputeFence and then installed Axolotl 0.20.0.
Before launching any training job I ran:
computefence doctor
Here is the exact output:
ComputeFence v0.2.11 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4 WARNINGS · 0 BLOCKERS · 4 PASSED
Environment
✓ Python 3.12.3
✓ PyTorch 2.8.0+cu128 detected (CUDA available — NVIDIA A40)
⚠ No Accelerate config found — skipping GPU count check
Fix: Run 'accelerate config' to create a configuration file
Storage
✓ HF_HOME is set to /workspace/.cache/huggingface/
✓ Persistent volume(s) detected: /workspace
⚠ Volume /workspace — 6.0 GB free of 9.3 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume
⚠ Root disk (/) — 16.2 GB free of 20.0 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4 warning(s) found. Review before launching.
The GPU passed. HF_HOME passed correctly pointing to the persistent volume. But the disk warning was real and would have caused a problem.
Why the disk warning matters for Axolotl specifically
Axolotl is not a small install. Its dependency tree pulled in over 200 packages torch, transformers, accelerate, flash-attention, deepspeed, gradio, and dozens more. By the time the install finished, my 10GB network volume had 6GB free and my root disk had 16.2GB free. Both below the 20GB safe threshold.
Axolotl saves checkpoints during training. If you are fine-tuning a 7B model with save_steps: 100, checkpoints can be several gigabytes each. With 6GB free on the workspace volume, a checkpoint save mid-run would fail. The job would die without completing. The weights you had trained up to that point would be gone.
No error before the job starts. No warning from Axolotl. Just a failed save at step 100 or 200 or wherever the disk fills.
The fix
Two options before launching:
Option 1 - Attach a larger volume
If you are planning a long fine-tuning run, attach at least a 50GB network volume. Axolotl’s dependencies alone consume 3–4GB. Your base model weights consume more. Leave headroom for checkpoints.
Option 2 — Configure checkpoint saving to the persistent volume explicitly
In your Axolotl config:
output_dir: /workspace/outputs
Not /root/outputs. Not a relative path. Explicitly /workspace/outputs so checkpoints land on the persistent volume, not the ephemeral root disk.
Run this before every Axolotl job
pip install computefence && computefence doctor
Or without installing:
uvx computefence doctor
It checks GPU visibility, HF_HOME path, persistent volume detection, and disk headroom before billing starts. The whole check takes under 30 seconds.
The disk warning it caught on my A40 pod was real. If I had launched a training job immediately after installing Axolotl without checking, the job would have run until the first checkpoint save and died.
GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence
Top comments (0)