You can build the most elegant neural network architecture in the world, and it will learn absolutely nothing until you answer one question first: wrong compared to what, exactly?
That's what a loss function is. It's not a technical afterthought bolted onto the end of a network — it's the thing that turns "prediction" into "learning." Everything from here on, every gradient that flows backward through every layer, starts as a number produced by this one function. Get it wrong, and the rest of the machinery doesn't matter.
So let's actually derive where these loss functions come from, instead of just memorizing "use MSE for regression, cross-entropy for classification" as a rule of thumb.
Loss functions aren't arbitrary — they're probability in disguise
Here's the thing most intro explanations skip: loss functions aren't hand-picked because they "feel right." They fall directly out of maximum likelihood estimation — you assume a probability distribution over your labels, and the loss function is just the negative log of that distribution's likelihood.
Start with regression. Assume your network's prediction is the mean of a Gaussian, with some noise:
$$p(y \mid x, w, \beta) = \mathcal{N}\big(\hat{y}(x, w), \, 1/\beta\big)$$
Take the negative log-likelihood over your whole dataset, expand the Gaussian, and something clean happens: every term that doesn't depend on your weights — constants, the noise variance — drops away during optimization. What's left is:
$$\frac{\beta}{2} \sum_{m=1}^{M} \big(y_m - \hat{y}(x_m, w)\big)^2$$
That's mean squared error. Not chosen because "distance feels intuitive," but because it's exactly what falls out of assuming Gaussian noise on a continuous target. If you ever get asked "why L2 loss for regression" on an exam, this is the actual answer, not "it penalizes big errors more."
Now do the same thing for classification, except your label distribution isn't Gaussian — it's categorical. For a coin flip, your label model is a Bernoulli distribution. For $K$ classes, it's the multinoulli:
$$\mathfrak{C}(y \mid p) = \prod_{k} p_k^{\,y_k}$$
Take the negative log-likelihood of that, and the product turns into a sum of logs:
$$L(w) = -\sum_{m=1}^{M} \sum_{k} y_{k,m} \, \ln \hat{y}_{k,m}$$
This is cross-entropy. Same derivation pattern, different assumed distribution. Once you see it this way, "which loss function should I use" stops being a lookup-table question and becomes: what's the actual probabilistic story behind my output? Continuous and Gaussian-ish → L2. Discrete and categorical → cross-entropy.
Why one-hot encoding isn't just a formatting choice
If you have 5 classes and the true label is class 3, you write it as:
$$y = \begin{bmatrix}0\0\0\1\0\end{bmatrix}$$
This looks almost too simple to explain, but it's doing real mathematical work. Encode your labels as raw integers (0, 1, 2, 3, 4) instead, and you've accidentally told your model that class 4 is somehow "more" than class 0 — an ordinal relationship that doesn't exist between "cat" and "dog." One-hot encoding removes that false structure entirely, and it maps cleanly onto the multinoulli distribution above, where exactly one $y_k$ is nonzero.
That last part is what makes cross-entropy so cheap to compute. Plug a one-hot vector into $-\sum_k y_k \ln \hat{y}_k$, and every term where $y_k = 0$ vanishes. You're left with:
$$L = -\ln \hat{y}_{k^*}$$
— the negative log of whatever probability your model assigned to the correct class. That's it. Cross-entropy loss, in the one-hot case, is just "how much did the model doubt the right answer." Confident and correct → loss near zero. Confident and wrong → loss explodes toward infinity. That asymmetry is the whole point — it's what makes the model care about fixing its most confident mistakes first.
The elegant accident of softmax and cross-entropy
Softmax converts a network's raw, unbounded output scores (logits) into something that behaves like a probability distribution:
$$\hat{y}_k = \frac{e^{z_k}}{\sum_j e^{z_j}}$$
Every output lands strictly between 0 and 1, and they all sum to exactly 1. Good — but softmax by itself is expensive to differentiate. Because of that normalization term in the denominator, every output depends on every input logit, which means its gradient is a full Jacobian matrix, not a simple per-element derivative.
Here's where it gets genuinely elegant. Pair softmax with cross-entropy — the combination that shows up in essentially every classification network you'll ever train — and take the gradient of the loss with respect to the pre-softmax logits. All the exponentials cancel. All the log terms cancel. The messy Jacobian collapses into:
$$\frac{\partial L}{\partial z_k} = \hat{y}_k - y_k$$
Predicted probability, minus the true one-hot label. That's the entire gradient. No exponentials, no matrix, just a vector subtraction.
Sit with how strange that is for a second. You started with an exponential normalization and a logarithm, stacked two genuinely nonlinear, seemingly-unrelated functions on top of each other, and the derivative simplified into something a first-grader could compute. That's not a coincidence dictated by convenience — it's a direct consequence of softmax being the correct inverse link function for a categorical likelihood, and cross-entropy being the correct loss for that same likelihood. When you pick the mathematically consistent pair, the calculus rewards you.
Practically, this is also why deep learning frameworks fuse softmax and cross-entropy into a single operation internally — softmax_cross_entropy_with_logits-style implementations skip the unstable, wasteful path of computing softmax, then log, then backpropagating through both separately.
Descending the loss surface
Once you have a loss value, you need to actually reduce it. The update rule is deceptively small:
$$W_{\text{new}} = W - \eta \cdot \frac{\partial L}{\partial W}$$
The gradient points toward steepest increase; you subtract it to head downhill. The learning rate $\eta$ controls your stride length. Too large, and you overshoot the valley floor, bouncing between its walls instead of settling into it — in the worst case the loss diverges outright. Too small, and you crawl toward the minimum so slowly that you might never get there in a reasonable number of steps, or get trapped in some shallow dip along the way with no momentum to escape it.
That tension — step too far and oscillate, step too little and crawl — is the entire reason the rest of optimization research exists.
The first fork in the road is how much data you use to compute each gradient step. Use the whole dataset every time (batch gradient descent) and your gradient estimate is smooth and accurate, but each step is painfully slow. Use a single random sample (stochastic gradient descent) and each step is nearly instant, but wildly noisy. Mini-batch gradient descent — a few dozen to a few hundred samples per step — is the practical compromise almost everyone actually uses: fast enough per step, smooth enough to make real progress, and it happens to line up perfectly with how GPUs like to be fed data.
Giving the optimizer memory and self-adjustment
Plain gradient descent has no memory — every step reacts only to the current gradient and forgets everything that came before. Momentum fixes that by keeping a running, decaying average of past gradients:
$$v^{(k)} = \mu \, v^{(k-1)} + \nabla L(W^{(k)})$$
Picture a heavy ball rolling downhill instead of a point mass that stops and restarts at every step. In directions where the gradient keeps pointing the same way, the ball builds up speed. In directions where the gradient keeps flipping sign — the classic symptom of a narrow, steep-walled ravine in the loss surface — the accumulated momentum in one direction cancels the momentum in the other, and the oscillation dampens out. Nesterov's variant refines this further by computing the gradient at the look-ahead position the momentum is already carrying you toward, correcting course slightly earlier and cutting down oscillation even more in badly-conditioned landscapes.
Momentum, though, still uses one global learning rate for every parameter in the network. But not every parameter needs updating at the same rate — some features fire constantly, others rarely. AdaGrad addressed this by tracking each parameter's accumulated squared gradient and scaling its individual learning rate inversely — frequent, large-gradient parameters get gently reined in, rare ones get relatively larger steps. Its flaw: that accumulator only ever grows, so learning rates eventually decay toward zero regardless of whether training is actually done. RMSProp fixes this by decaying the accumulator over time instead of summing forever, so recent gradients matter more than ancient ones.
Adam — by far the most common optimizer you'll encounter in practice — combines both ideas: a momentum term for direction, and an RMSProp-style adaptive scale for step size, with a bias correction added for both since they start at zero. It's not magic, and it's worth knowing it isn't perfect: Adam's original convergence proof turned out to contain an actual bug, later patched by a variant called AMSGrad. And in a lot of published state-of-the-art results, plain SGD with Nesterov momentum and a carefully tuned learning rate schedule still edges out Adam on final performance, even though Adam converges more predictably out of the box. The honest recommendation, and the one worth remembering: start with mini-batch SGD plus momentum, reach for Adam once you have a feel for your data, and keep your eyes open regardless of which one you pick.
Trusting your own backpropagation
There's a quieter, less glamorous idea buried in this chapter that matters just as much as any of the optimizers: how do you know your backprop implementation is actually correct?
You check it against a version that doesn't rely on calculus at all — the definition of a derivative, approximated directly:
$$\frac{\partial f}{\partial x} \approx \frac{f(x+\epsilon) - f(x-\epsilon)}{2\epsilon}$$
This centered version is deliberately more accurate than the simpler one-sided version $\frac{f(x+\epsilon) - f(x)}{\epsilon}$ — the first-order error terms cancel out in the subtraction, leaving an error that shrinks quadratically instead of linearly as $\epsilon$ shrinks. Compute your gradient this way, compare it to what your analytical backprop implementation produces, and if they agree closely, you have real evidence your chain rule is implemented correctly. If they don't, you have a bug, and you have it before wasting hours or days training on top of it.
This isn't a minor footnote. Backprop bugs are famously silent — a broken gradient doesn't crash your program, it just quietly trains a worse network, and you often only notice something's "off" without ever finding the actual cause. Even major, widely-used software frameworks have shipped subtly incorrect gradient computations that went unnoticed for a long time. The lesson generalizes past this one chapter: gradient-based methods are forgiving enough that as long as you're roughly following the right direction, you'll still get some kind of result — which is exactly what makes them so hard to debug when something's subtly wrong.
Where this leaves us
Zoom out, and this chapter is really about one idea wearing several disguises: define what "wrong" means in a way that's grounded in probability, turn that definition into a number you can differentiate, and then descend that number's landscape as efficiently as you can — while occasionally checking, by brute force, that your descent direction is actually correct.
Next up: what happens when the inputs aren't just flat vectors of independent features, but images, where nearby pixels are correlated and position actually matters. That's where convolutions come in.
Top comments (0)