Activation Functions Explained: Sigmoid vs ReLU

Part 4 of my "revisiting my AI/ML notes" series. Previous post covered forward propagation — the weighted sum and the sigmoid squash at the end. This one zooms into activation functions themselves: what they're for, and the two most common ones.

Why activation functions exist at all

Without an activation function, a neuron is just a weighted sum — a linear equation. Stack as many linear layers as you want, and the whole network still collapses mathematically into one giant linear function. That's a serious limitation: most real-world patterns (images, language, anything genuinely complex) are not linearly separable.

Activation functions introduce non-linearity, which is what actually lets a deep network learn complex patterns instead of just fitting a straight line.

Sigmoid

sigmoid(y) = 1 / (1 + e^-y)

Sigmoid squashes any input into a value strictly between 0 and 1 — an S-shaped curve. This makes it naturally interpretable as a probability, which is exactly why it's still common at the output layer for binary classification.

A useful way to think about it in a classification context:

• If the output is less than 0.5, the neuron effectively doesn't activate — the network is leaning toward "no" / class 0.
• If the output is 0.5 or higher, the neuron activates — the network is leaning toward "yes" / class 1.

The catch: for very large positive or negative inputs, sigmoid's curve flattens out almost completely. That means the gradient (the signal used to update weights during training) shrinks toward zero — this is the vanishing gradient problem, and it's the main reason sigmoid fell out of favor for hidden layers in deep networks.

ReLU (Rectified Linear Unit)

ReLU(y) = max(y, 0)
ReLU is almost aggressively simple:

• If y is negative, the output is 0 — the neuron doesn't fire at all.
• If y is positive, the output is y itself, completely unchanged.

This simplicity is exactly why ReLU became the default choice for hidden layers in modern deep networks: it's computationally cheap, and it doesn't suffer from vanishing gradients for positive inputs the way sigmoid does.

Where each one is actually used

The practical convention that falls out of these tradeoffs:

• Hidden layers → ReLU (or a variant like Leaky ReLU), because it trains faster and avoids vanishing gradients across many stacked layers.
• Output layer → sigmoid for binary classification (single probability between 0 and 1), or softmax for multi-class problems.

This split — ReLU in the middle, sigmoid/softmax at the very end — is close to the default architecture choice for most classification networks today.

Why this matters

Picking the wrong activation function isn't a cosmetic choice — it directly affects whether a deep network can train at all. A network built entirely out of sigmoid hidden layers will train noticeably slower (or stall completely) on deep architectures, purely because of how quickly the gradient vanishes as it's backpropagated through many layers. That's the exact topic for the next post.

Next up: backpropagation — how the network actually uses the loss to update every weight.

---

Original Notes: