Part 11 of my "revisiting my AI/ML notes" series. Earlier in this series I covered sigmoid and plain ReLU. ReLU fixed sigmoid's vanishing gradient problem for positive inputs — but it introduced a new failure mode of its own, and there's a whole family of activation functions built to fix it.
The problem with plain ReLU: dying neurons
ReLU is max(0, z) — for any negative input, the output is exactly zero, and so is the gradient. If a neuron's weights drift into a state where its input is consistently negative, that gradient stays zero forever: the neuron stops updating completely, no matter how much more training happens. This is called the dying ReLU problem, and it can silently remove a meaningful chunk of a network's capacity.
Leaky ReLU
The fix is almost embarrassingly small: instead of outputting exactly 0 for negative inputs, output a small fraction of the input.
LeakyReLU(z) = z if z > 0, else 0.01 * z
That tiny non-zero slope for negative values is enough to keep a small gradient flowing, so a neuron that drifts negative can still recover instead of dying permanently.
ELU (Exponential Linear Unit)
ELU(z) = z if z > 0, else alpha * (e^z - 1)
ELU takes a different approach for negative inputs — an exponential curve that smoothly approaches -alpha instead of a straight line. This gives it smoother gradients around zero, which can help training stability. The tradeoff: computing e^z on every negative input is meaningfully more expensive than Leaky ReLU's simple multiplication, so it's a real speed-vs-quality tradeoff, not a strict upgrade.
PReLU (Parametric ReLU) takes Leaky ReLU one step further by making that slope (the alpha) a learnable parameter instead of a fixed 0.01 — the network figures out the best slope for itself during training.
Swish
Swish(z) = z * sigmoid(z)
Swish is a newer activation, generally recommended for very deep networks (notes mention 40+ layers). Unlike ReLU, it's smooth everywhere and slightly non-monotonic near zero, which empirically tends to help gradient flow in very deep architectures.
Softmax: for the output layer, not hidden layers
Everything above is a hidden-layer activation. Softmax is different — it's used specifically at the output layer for multi-class classification, converting a set of raw scores into a valid probability distribution that sums to 1:
softmax(z_i) = e^(z_i) / sum over j of e^(z_j)
If a model outputs raw scores across four classes and softmax turns them into something like [0.6, 0.2, 0.1, 0.1], the class with the highest probability (0.6) is the model's prediction — and unlike sigmoid, this generalizes naturally beyond two classes.
Why this matters
There's no single "best" activation function — each one trades off gradient behavior, computational cost, and training stability differently. What matters is recognizing which problem a given activation function was built to solve, so the choice for a specific architecture is deliberate rather than just "whatever the tutorial used."
Original Notes:
Top comments (0)