I Finally Understood Why Neural Networks Need Activation Functions

karki vasagnavi — Tue, 21 Jul 2026 15:06:14 +0000

Today I Finally Understood Why We Plot the Derivative of Activation Functions

When I started learning Deep Learning, I thought activation functions were just mathematical equations we had to memorize.

Today, while implementing Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax from scratch in Python, I realized something much more interesting.

The activation function itself is only half of the story.

The derivative is what actually teaches the neural network how to learn.

My Experiment

I generated 100 equally spaced values between -10 and 10:

z = np.linspace(-10, 10, 100)

Then I implemented:

Sigmoid
Tanh
ReLU
Leaky ReLU
Softmax

For the first four, I plotted both the activation function and its derivative.

Initially, I wondered:

Why am I plotting the derivative? Isn't the activation function enough?

After learning about backpropagation, the answer became clear.

What the Activation Function Tells Us

The activation function answers:

"What output should this neuron produce?"

For example, Sigmoid compresses any input into a value between 0 and 1.

ReLU outputs 0 for negative inputs and passes positive inputs unchanged.

This is what happens during the forward pass.

Input
   ↓
Weighted Sum
   ↓
Activation Function
   ↓
Prediction

What the Derivative Tells Us

The derivative answers a completely different question:

"How much should this neuron change to reduce the error?"

During the backward pass, the neural network computes gradients.

Those gradients come directly from the derivatives of the activation functions.

Without derivatives, the model wouldn't know how to update its weights.

Prediction
     ↓
Calculate Error
     ↓
Compute Derivatives
     ↓
Update Weights

That was the moment when the plots finally made sense to me.

My Biggest Observation

Sigmoid

Beautiful S-shaped curve.

But its derivative almost becomes zero for very large positive or negative inputs.

That means learning slows down because the gradients almost disappear.

This is called the vanishing gradient problem.

Tanh

At first glance, it looked almost identical to Sigmoid.

But I noticed something important:

Its output is centered around zero.

That small difference helps optimization because the gradients are more balanced around the origin.

ReLU

The simplest graph was also the most surprising.

f(x)=max(0,x)

Its derivative is

0   (negative inputs)

1   (positive inputs)

This explains why ReLU trains deep neural networks much faster than Sigmoid.

Leaky ReLU

ReLU has one weakness.

Negative neurons completely stop learning because the derivative becomes zero.

Leaky ReLU fixes that with a tiny negative slope.

Instead of

Gradient = 0

it becomes

Gradient = 0.01

A small change mathematically—but an important one during training.

Softmax

Softmax was different from the others.

It doesn't transform a single value.

It transforms an entire vector into probabilities.

Example:

Input

[2,4,1]

↓

Output

[0.114
 0.844
 0.042]

Every probability depends on every other input.

That explains why we usually don't visualize Softmax as a simple curve.

Today I stopped thinking of activation functions as just formulas.

Now I see them as two complementary ideas:

The activation function decides what a neuron outputs.
Its derivative decides how that neuron learns.

The graph explains the prediction.

The derivative explains the learning.

My next goal is to implement:

Forward propagation from scratch
Backpropagation from scratch
Gradient Descent
A simple neural network without TensorFlow or PyTorch

I want to understand the mathematics before relying on deep learning frameworks.

When you first learned neural networks, what concept changed your understanding the most—activation functions, backpropagation, or gradient descent?

DeepLearning #MachineLearning #Python #ArtificialIntelligence #NeuralNetworks #LearningInPublic #Bioinformatics #100DaysOfCode #DevCommunity

DEV Community: karki vasagnavi