The 2am Loss Spike That Broke Everything
You're 12 hours into training a transformer on A100s. Loss curve looks clean. You check again at epoch 15: NaN. All of it. The optimizer state is corrupted, your checkpoint is useless, and you just burned $47 in compute credits on nothing.
Mixed precision training (FP16 activations + FP32 master weights) cuts memory usage by ~40% and speeds up training by 2-3x on modern GPUs. But the precision reduction introduces numerical instability that doesn't show up in FP32 training. I've seen four specific patterns that cause NaN losses in production, and they all have different fixes.
This isn't about "just use gradient clipping" — that's a band-aid. The real culprits are gradient scaling overflow, unsafe loss functions, bad initialization, and accumulation precision bugs.
Why FP16 Breaks When FP32 Doesn't
Continue reading the full article on TildAlice
Top comments (0)