TL;DR: Why neural networks need activation functions. Sigmoid, tanh, ReLU, GELU and softmax in plain English, with a comparison table and a rule for choosing one.
Here is a strange fact: you could stack 100 layers of a neural network and it would be exactly as powerful as a single layer. Sounds like a scam, right? It would be, if it weren’t for one tiny ingredient called the activation function.
If you have ever opened a deep learning tutorial and seen words like ReLU, sigmoid or softmax fly by, this post is for you. No heavy maths, no fear. By the end you will know what activation functions do, why they exist, and which one to pick without googling in a panic.
What is an activation function, really?
Every artificial neuron does two jobs. First it takes its inputs, multiplies each by a weight and adds them up. Second, it passes that total through a small function that decides how strongly the neuron should "fire". That second step is the activation function.
Think of a bouncer at a club. The neuron’s total is the crowd noise outside. The bouncer (activation function) decides: too quiet, stay out; loud enough, come in; ridiculously loud, still only one person gets in at a time. Different bouncers, different rules.
Why can’t we skip it? The 100-layer trick
Multiplying and adding is called a linear operation. The awkward thing about linear operations is that combining them gives you… another linear operation. Two straight-line rules stacked together are still just one straight-line rule. So a network with no activation functions, however deep, can only draw straight lines through your data.
Real problems are not straight. Recognising a cat, translating Hindi to English or predicting engine wear all involve curves and twists. The activation function adds a bend at every layer, and those bends stacked together let the network trace almost any shape. That is the whole secret.
Layers give a network depth. Activation functions give it the ability to use that depth.
The main characters
1. Sigmoid: the classic that got old
Sigmoid squashes any number into a value between 0 and 1, which makes it look like a probability. Big positive input gives nearly 1, big negative gives nearly 0. It was the star of early neural networks because it loosely mimics a biological neuron that is either firing or quiet.
The catch: at the far ends the curve goes almost flat. Flat means the slope is nearly zero, and networks learn by following slopes. Even at its steepest, sigmoid’s slope is only 0.25. Multiply a number like that through many layers and the learning signal shrinks to almost nothing. This is the famous vanishing gradient problem. Early layers basically stop learning, like a game of telephone where the message fades before it reaches the first person.
Where it still shines: the final layer of a yes/no prediction (spam or not spam).
2. Tanh: sigmoid’s more balanced sibling
Tanh (say "tanch") squashes numbers between -1 and 1. Because its output is centred around zero, learning is usually smoother than with sigmoid. It still flattens at the ends, so it shares the vanishing gradient headache, just a little less. You will still meet it inside older recurrent networks such as LSTMs.
3. ReLU: the boring one that changed everything
ReLU stands for Rectified Linear Unit, which is a very grand name for a very simple rule: if the number is negative, output 0; otherwise pass it through unchanged. That’s it. In code it is literally max(0, x).
Why did such a plain rule take over deep learning? Two reasons. It is extremely cheap to compute (just a comparison, no exponentials). And for positive inputs its slope is exactly 1, so the learning signal does not fade as it travels back through layers. ReLU became the default after it powered the AlexNet image model that won the 2012 ImageNet competition and kicked off the modern deep learning boom.
The catch: a neuron that keeps receiving negative inputs outputs zero forever, and with zero slope it can never recover. People call this a "dead" neuron, or the dying ReLU problem. Poor thing.
4. Leaky ReLU: giving the dead a pulse
Leaky ReLU fixes the dead neuron issue with a small tweak: instead of outputting exactly 0 for negatives, it lets a tiny amount through (often 1% of the input). The neuron always has some slope, so it can always recover.
5. GELU and Swish: the smooth modern ones
ReLU has a sharp corner at zero. GELU and Swish (also called SiLU) round that corner off into a gentle curve, which often trains a little better in very large models. GELU is used in well-known language models such as BERT and the GPT family. A fun bit of trivia: Swish was discovered by researchers using an automated search that tried many candidate functions, rather than by someone sitting down and deriving it.
6. Softmax: the vote counter
Softmax is different. Instead of treating one neuron at a time, it looks at a whole group of outputs and turns them into percentages that add up to 100%. If a model is deciding between cat, dog and bird, softmax might output 85%, 10% and 5%. It belongs in the last layer of a multi-class classifier, never in the middle.
Quick comparison
| Function | Output range | Best used for | Watch out for |
|---|---|---|---|
| Sigmoid | 0 to 1 | Final layer, yes/no output | Vanishing gradients |
| Tanh | -1 to 1 | Older recurrent networks | Vanishing gradients |
| ReLU | 0 to infinity | Hidden layers, default choice | Dead neurons |
| Leaky ReLU | Slightly below 0 to infinity | When ReLU neurons die | One extra setting to pick |
| GELU / Swish | Smooth, mostly positive | Transformers, large models | Slightly more compute |
| Softmax | 0 to 1, sums to 1 | Final layer, many classes | Not for hidden layers |
So which one should you pick?
A beginner-safe recipe that works for a surprising number of projects:
- Hidden layers: start with ReLU. If many neurons seem dead, try Leaky ReLU.
- Output for yes/no: sigmoid.
- Output for many categories: softmax.
- Output for a number (like predicting a temperature): no activation at all, just the raw value.
- Building a transformer: follow what similar models use, usually GELU or a Swish-style function.
Try it yourself in 30 seconds
import numpy as np
x = np.array([-3, -1, 0, 1, 3])
sigmoid = 1 / (1 + np.exp(-x))
relu = np.maximum(0, x)
tanh = np.tanh(x)
print("sigmoid:", sigmoid.round(2))
print("tanh :", tanh.round(2))
print("relu :", relu)
Run it and compare. Notice how sigmoid barely changes between 3 and 100, while ReLU keeps growing. That difference is exactly why one struggles with depth and the other doesn’t.
Frequently asked questions
Can I use different activation functions in different layers?
Yes. Hidden layers commonly use one function (like ReLU) while the output layer uses another (like softmax) to match the task.
Is ReLU always the best?
No. It is the best starting point. Newer smooth functions can edge it out in very large models, but for most beginner projects ReLU is fast, simple and hard to beat.
What is a vanishing gradient in one sentence?
It is when the learning signal shrinks as it travels backwards through the layers until the early layers get almost no feedback and stop improving.
Final thought
An activation function is a tiny bend that turns a pile of straight-line maths into something that can recognise faces, translate languages and predict failures. Small switch, huge effect. Which one surprised you most: the 100-layer trick or the fact that ReLU is just max(0, x)? Tell me in the comments, and if you want a follow-up on how neural networks actually learn (backpropagation without tears), say so.
Related guides you may like
- How Logistic Regression Works: Beginner to Pro Guide
- Complete Guide to Fine-Tuning LLMs: From Prompt-Tuning to RLHF
- Linear Regression Explained: A Beginner’s Guide to Prediction
💬 Over to you
Which activation function do you reach for by default, and has switching ever fixed a model that wasn't learning? I'd love to hear your war stories.
If this helped, hit ❤️ and follow for more plain-English ML and engineering guides. New posts go out every Tuesday and Friday, and this one is part of the Machine Learning Basics series.
Originally published on ML AI Insights Hub. Find more plain-English ML guides, worked engineering examples and free Python/Excel calculators at mlaiinsightshub.blog.
Top comments (0)