Every deep learning journey has the same plot twist: the loss curve goes flat, or worse, turns into NaN at epoch 3 like it just discovered nihilism. The framework error says nothing useful. Stack Overflow says "lower your learning rate." You lower it by 10x. Nothing changes—now it just reaches NaN with more dignity.
The first time this happened to me, I had written a custom classification loop where I manually applied torch.softmax() at the end of forward() and then fed the probabilities into torch.log() inside my own cross-entropy function. On clean toy inputs, it looked fine. On a real batch with one wildly confident misprediction (a logit gap over 104 in float32), the target class probability underflowed past IEEE 754 subnormals straight to 0.0, torch.log(0.0) returned -inf, and the backward pass cheerfully multiplied -inf * 0.0 into NaN across the network. I spent two hours blaming my learning rate scheduler before realizing I had reinvented numerical instability from scratch.
I am currently working through the Foundations of Deep Learning micro-degree on KI-Campus. The resource that has turned its lectures into real intuition is, surprisingly, a 2016 book: Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville. It predates Transformers as we know them, yet I use it less as a cover-to-cover textbook and more as a debugging manual.
Why a 2016 book next to a modern course?
Courses give you structure, deadlines, and graded checkpoints. But a 10-minute lecture video has to compress derivations to fit the clock. When a slide skips three algebraic steps and expects you to nod along, you need a reference that explains things from first principles—ideally one that is freely readable online.
The KI-Campus micro-degree is split into three courses (~36 hours total), and they map remarkably well to the book:
| KI-Campus Course | Topics | Book Chapters |
|---|---|---|
| I: Basics | Math prereqs, MLPs, backprop, SGD/Adam, regularization | Ch. 2–8 |
| II: Architectures & Methodology | CNNs, RNNs/LSTMs, Transformers, transfer learning, debugging | Ch. 9–11, 15 |
| III: Advanced Topics | Autoencoders, VAEs, GANs, diffusion, VI, MCMC, HPO | Ch. 11, 14, 17, 19–20 |
The cheat sheet: symptom → chapter → PyTorch fix
Instead of reading the book front to back, I look up whatever is currently on fire. Here is the quick-reference table, followed by the exact PyTorch patterns each chapter explains.
| Your symptom | Look here | The idea that helps |
|---|---|---|
Loss becomes NaN
|
Ch. 4 | Finite-precision floats: overflow, underflow, and log-sum-exp stability |
| Training is painfully slow or oscillates | Ch. 4, 8 | Ill-conditioning, momentum, adaptive optimizers, batch normalization |
| Great train accuracy, bad test accuracy | Ch. 5, 7 | Optimization ≠ learning: you are minimizing a surrogate on finite data |
| "Why cross-entropy and not MSE?" | Ch. 3, 5, 6 | Cross-entropy is maximum likelihood (minimizing KL divergence) |
| RNN forgets everything or explodes | Ch. 10 | Repeated Jacobian multiplication → vanishing/exploding gradients |
| "I don't know what to try next" | Ch. 11 | Baseline first, overfit a micro-batch, change one variable at a time |
| VAEs feel like black magic | Ch. 19–20 | ELBO and the reparameterization trick |
1. Loss becomes NaN (Chapter 4: Numerical Computation)
Chapter 4 explains why float32 is not real-number arithmetic: exp(89) overflows to inf, and exp(-104) underflows past the smallest float32 subnormal (1.4e-45) to 0.0. Even though torch.softmax shifts by max(z) internally to prevent overflow, computing torch.log(torch.softmax(z)) in two separate steps still underflows to 0.0 whenever a wrong class logit beats the target class logit by 104 or more. Fusing the log and the softmax via log-sum-exp keeps the math in log-space from start to finish:
import torch
import torch.nn.functional as F
# Class 0 logit is 105.0; true target (Class 1) logit is 0.0 -> gap of 105
logits = torch.tensor([[105.0, 0.0, 0.0]], requires_grad=True)
targets = torch.tensor([1])
# ❌ Fragile: exp(0 - 105) underflows to 0.0 in float32 -> log(0.0) = inf -> NaN grad
probs = torch.softmax(logits, dim=-1)
bad_loss = -torch.log(probs[0, targets[0]]) # inf (and logits.grad becomes NaN)
# ✅ Stable: F.cross_entropy fuses log_softmax (log-sum-exp) + NLLLoss
good_loss = F.cross_entropy(logits, targets) # 105.0, with clean finite gradients
2. "Why cross-entropy and not MSE?" (Chapters 3, 5, 6)
If you pair a sigmoid output with Mean Squared Error (MSE), the gradient picks up a factor of σ'(z) = σ(z)(1 - σ(z)). When the neuron is confidently wrong (σ(z) ≈ 0 while y = 1), σ'(z) ≈ 0—the gradient vanishes precisely when the model needs to correct itself the fastest. Negative log-likelihood cancels that saturation term out.
import torch.nn as nn
# ❌ Sigmoid + MSE saturates gradients when predictions are confidently wrong
bad_criterion = nn.MSELoss()
# ✅ BCEWithLogitsLoss applies log-sigmoid stability + maximum likelihood gradient
good_criterion = nn.BCEWithLogitsLoss()
3. Training oscillates or stalls (Chapters 4 & 8)
When the Hessian is poorly conditioned—steep in one direction, flat in another—vanilla SGD bounces across the canyon walls instead of walking down the valley. Momentum smooths out the zig-zag; BatchNorm rescales layer activations so gradients stay well-behaved across depth.
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9, nesterov=True)
# Or adaptive per-parameter scaling:
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)
(Note: AdamW decouples weight decay from the adaptive gradient denominator—a refinement published right after the 2016 book and covered in KI-Campus Course I.)
4. RNN forgets everything or explodes (Chapter 10)
Unrolling an RNN over T steps multiplies the recurrent Jacobian T times. If its largest singular value is < 1, gradients vanish exponentially; if > 1, they explode. LSTMs/GRUs fix the vanishing half with gated additive paths; gradient clipping fixes the exploding half.
loss.backward()
# ✅ Cap the global L2 norm of gradients before stepping
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
5. "I don't know what to try next" (Chapter 11: Practical Methodology)
Before touching hyperparameters, prove your architecture and training loop can memorize a tiny slice of data. If a network cannot overfit 16 examples to near-zero loss, adding 100,000 more examples will only waste electricity faster.
# ✅ Sanity check: can the model memorize a single mini-batch?
x_tiny, y_tiny = next(iter(train_loader))
x_tiny, y_tiny = x_tiny[:16], y_tiny[:16]
for step in range(200):
optimizer.zero_grad()
loss = F.cross_entropy(model(x_tiny), y_tiny)
loss.backward()
optimizer.step()
if step % 50 == 0:
print(f"step={step:3d} | loss={loss.item():.4f}")
6. VAEs feel like black magic (Chapters 19–20)
You cannot backpropagate through a random sampling node z ~ N(mu, sigma^2) directly. Chapter 20 shows how the reparameterization trick moves the randomness into an auxiliary input ε ~ N(0, I), turning sampling into a deterministic, differentiable operation with respect to mu and logvar.
def reparameterize(mu: torch.Tensor, logvar: torch.Tensor) -> torch.Tensor:
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std) # randomness lives here, outside the gradient path
return mu + eps * std
Three ideas that changed how I think
1. Optimization ≠ learning.
Your optimizer minimizes loss on the training set. What you actually care about is error on data you have never seen. Once that distinction clicks, regularization (weight decay, early stopping, data augmentation, dropout) stops looking like a random bag of tricks and becomes one coherent idea: anything meant to reduce generalization error, even if it hurts training loss.
2. Architectures are assumptions, not magic.
A CNN assumes that nearby pixels matter together and that the same pattern can appear anywhere (sparse connectivity and parameter sharing). An RNN assumes the same transition rule applies at every time step. Seeing architectures as inductive biases explains both why they work and why they break when those assumptions don't hold.
3. Chapter 11 is the most underrated chapter in the book.
Two habits from it have saved me more hours than any architecture diagram:
- Use random search instead of grid search for hyperparameters (grid search wastes trials on dimensions that barely matter).
- Check that your model can overfit a tiny batch.
That second habit saved me from a particularly embarrassing bug in an image classifier: validation accuracy was stuck at 50% (coin-flip territory), and I was already shopping for a deeper backbone. I ran the 16-sample overfit test, and the loss still refused to drop below 0.693. Turns out I had transposed (H, W, C) to (C, W, H) in a preprocessing helper, scrambling the spatial structure while keeping the tensor shape valid. Five minutes with Chapter 11's sanity check beat two days of hyperparameter tuning. 🔍
Honest review: what to skip
I would rather tell you this upfront than pretend the book is an easy weekend read:
- It is dense. The math is not decorative. Read slowly, with pen and paper, or you will slide right off the page.
- Part III is a mountain. Chapters 16–18 (structured probabilistic models, Monte Carlo, partition functions) are fascinating but heavy. If your goal is understanding modern generative models in Course III, jump to Ch. 19 (ELBO) and Ch. 20 (VAEs/GANs), and come back to 16–18 later.
- It is from 2016. Transformers and self-attention only appear in their early seq2seq form, and modern denoising diffusion models come later. That is literally where KI-Campus picks up (Course II for Transformers, Course III for diffusion).
- The web version is read-only. Great for keeping open in a browser tab, less great if you like annotating a PDF.
Even with those caveats, the gap is smaller than it sounds. Once you understand score matching (Ch. 14, 18), the ELBO (Ch. 19), and attention (Ch. 12), modern papers feel like continuations rather than alien technology.
My workflow
- Watch the KI-Campus module for the map and vocabulary.
- Read the matching section of Deep Learning for the intuition and the edge cases.
- Do the graded quizzes to find out what I only thought I understood.
- Break something on purpose: overfit a mini-batch, remove normalization, crank the learning rate up. Seeing the failure beats reading about it.
Takeaway
You do not need to read all 800 pages of Goodfellow, Bengio, and Courville from start to finish. You need it open in a tab for the day your loss curve misbehaves, paired with a structured course to give you a route.
Your turn: what is the book or reference that "saved" you when a model refused to cooperate—and what is the sneakiest NaN you have ever debugged? 👇
Top comments (0)