The Easiest Way to Understand Backpropagation

Fouad Elhamra — Wed, 05 Aug 2026 14:41:07 +0000

When I first learned deep learning, I thought there was only one way to train a neural network: compute the gradient and update the weights.

Then I discovered there are actually three different ways to do it.

Let's understand them with a simple example.

Imagine you have a dataset with 10,000 images.

Your goal is to minimize the loss function by updating the model's weights.

Batch Gradient Descent

Batch Gradient Descent processes the entire dataset before making a single update.

10,000 samples -> Compute total loss -> Compute gradients -> Update weights

Advantages

Stable gradient
Smooth convergence
Accurate update direction

Disadvantages

Slow for large datasets
Requires lots of memory
One update only after processing every sample

Think of it like reading an entire book before writing a summary.

2. Stochastic Gradient Descent (SGD)

Instead of waiting for all 10,000 samples, SGD updates the weights after every single sample.

Sample 1 → Update
Sample 2 → Update
Sample 3 → Update
...
Sample 10,000 → Update

Now the model learns much faster.

The downside?

Each sample may point in a slightly different direction, causing the optimization path to bounce around.

Imagine hiking toward the bottom of a valley while someone changes your direction every few seconds.

This creates the famous zigzag optimization path.

Advantages

Very fast updates
Low memory usage
Can escape some local minima

Disadvantages

Noisy gradients
Unstable convergence
Loss fluctuates a lot

3. Mini-Batch Gradient Descent

Mini-batch combines the best parts of both approaches.

Instead of using one sample or the entire dataset, we split the data into small batches.

For example:

10,000 samples

Batch 1 = 128 samples → Update
Batch 2 = 128 samples → Update
Batch 3 = 128 samples → Update
...

Now every update is based on enough data to reduce noise, but not so much that training becomes slow.

This is why frameworks like PyTorch and TensorFlow use mini-batches by default.

Advantages

Faster than Batch GD
More stable than SGD
Efficient GPU utilization
Standard choice for deep learning

Comparison

Method	Update Frequency	Speed	Stability
Batch Gradient Descent	After the entire dataset	Slow	High
Stochastic Gradient Descent	After every sample	Fast	Low
Mini-Batch Gradient Descent	After every small batch	Fast	High

Final Thoughts

If you're training modern neural networks, Mini-Batch Gradient Descent is usually the best choice.

It balances speed, memory usage, and convergence, making it the default optimization strategy in most deep learning libraries.

Understanding these three optimization strategies helped me understand why neural networks train the way they do—not just how they train.

DEV Community: Fouad Elhamra

The Easiest Way to Understand Backpropagation A