DEV Community

Cover image for The Silent OOM: Why Our LlamaFactory Multi-GPU Fine-Tune Died at 87% (Postmortem)
Jamse Bao
Jamse Bao

Posted on

The Silent OOM: Why Our LlamaFactory Multi-GPU Fine-Tune Died at 87% (Postmortem)

#debugging #cli #terminal #opensource

The Failure Symptom 

[2026-01-15 02:34:17] torch.distributed: NCCL timeout (30s) on rank 2
[2026-01-15 02:34:17] Process group destroyed. Exit code: 137
[2026-01-15 02:34:18] Checkpoint saved: global_step_8700 (87% complete)
Enter fullscreen mode Exit fullscreen mode

Our overnight LoRA fine-tune of Qwen2.5-72B on 4×A100 80GB died at 87% progress. No Python traceback. No CUDA OOM warning. Just a silent NCCL timeout followed by SIGKILL (exit 137). The checkpoint saved cleanly, gradient accumulation logs looked normal, and nvidia-smi showed 76GB/80GB utilization per GPU—well within limits.

This was our third attempt with LlamaFactory 0.9.1 using the standard llamafactory-cli train command from their official multi-GPU examples. Previous runs on smaller models (Llama-3.1-8B) completed fine.

The Red Herring

Every LlamaFactory tutorial and the official docs recommend:

# train_config.yaml
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
fsdp: "full_shard auto_wrap"
Enter fullscreen mode Exit fullscreen mode

We followed this exactly. Community Slack suggested increasing NCCL timeout:

export NCCL_TIMEOUT=600  # 10 minutes
Enter fullscreen mode Exit fullscreen mode

Run 2 died at 91%. Run 3 at 87%. Pattern unclear.

The smoking gun appeared when I ssh'd into the training node mid-run and tailed system logs:

$ dmesg -T | tail -20
[Wed Jan 15 02:34:16 2026] Out of memory: Killed process 48291 (python) score 950
[Wed Jan 15 02:34:16 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null)...
Enter fullscreen mode Exit fullscreen mode

System RAM OOM—not GPU VRAM. The host had 256GB RAM, but something was leaking.

The Root Cause

LlamaFactory's default DataCollatorForSeq2Seq with pad_to_multiple_of=8 was pre-allocating padded CPU tensors for the entire upcoming gradient accumulation microbatch window during FSDP all-gather operations. For 72B models with:

  • 8,192 max sequence length
  • Batch size 2 × 4 GPUs × 8 grad accumulation steps = 64 total sequences per step
  • BF16 tokenized inputs

This created transient ~180GB CPU allocations during FSDP synchronization—invisible to torch.cuda.memory_summary() but fatal to the host.

The fix:

# train_config.yaml
per_device_train_batch_size: 1  # Halve batch size
gradient_accumulation_steps: 16  # Double accumulation
max_length: 4096  # Clamp sequence length
dataloader_num_workers: 2  # Reduce prefetch buffer
Enter fullscreen mode Exit fullscreen mode

And explicitly limit DataLoader prefetch:

# Custom training script wrapper
import torch.multiprocessing as mp
mp.set_sharing_strategy('file_system')  # Avoid shm OOM
Enter fullscreen mode Exit fullscreen mode

Run 4 completed in 6.2 hours with peak host RAM at 198GB. Checkpoint loss curves matched the failed runs exactly—no accuracy regression.

Why Standard Tutorials Miss This

Most LlamaFactory examples use:

  1. Smaller models (7B-13B) where CPU allocations stay under 64GB
  2. Cloud instances with 512GB+ RAM where this never surfaces
  3. DeepSpeed ZeRO-3 instead of FSDP, which has different memory staging behavior

Our production setup used on-prem nodes (256GB RAM) optimized for GPU-dense workloads. The NCCL timeout was a symptom, not the cause—FSDP all-gather stalled waiting for a rank that had been OOM-killed.

The Invariant Rule

Always monitor host RAM (dmesg -wT in a tmux pane) during the first 100 steps of any new multi-GPU fine-tune config—even if GPU memory looks fine.

For distributed training debugging, I now run:

#!/bin/bash
# watch_training.sh
tmux new-session -d -s train_monitor \
  "watch -n 2 'nvidia-smi --query-gpu=memory.used --format=csv'" \; \
  split-window -h "dmesg -wT | grep -i 'oom\|killed'" \; \
  split-window -v "tail -f logs/train.log"
Enter fullscreen mode Exit fullscreen mode

This caught two other silent failure modes in the same sprint: a Ray Tune hyperparameter search that exhausted file descriptors, and a tokenization pipeline that leaked mmap handles.

Proxy Gateway Note

When benchmarking LlamaFactory's inference server mode (vLLM backend) for deployment readiness, we tested request routing through B-Lost's relay gateway (https://b-lost.com/v1) to simulate production multi-model fallback behavior. The OpenAI-compatible endpoint handled model swap latency well—TTFT stayed under 280ms even when failing over from a saturated primary to a cold secondary Qwen2.5 instance. Configuration:

# benchmark_client.py
from openai import OpenAI

client = OpenAI(
    base_url="https://b-lost.com/v1",
    api_key=os.getenv("BLOST_API_KEY")
)

response = client.chat.completions.create(
    model="accounts/fireworks/models/qwen2p5-72b-instruct",
    messages=[{"role": "user", "content": "Summarize FSDP vs DeepSpeed"}],
    max_tokens=512
)
Enter fullscreen mode Exit fullscreen mode

This validated that our fine-tuned model could slot into existing orchestration without refactoring request handling logic.

Disclosure: Compute infrastructure and multi-model benchmark relays for this writeup are sponsored by b-lost.com—an enterprise AI gateway offering 0.8x official pricing, native prompt caching, and zero user-data retention. All benchmark metrics reflect independent reproducible testing.

Top comments (0)