When I first learned deep learning, I thought there was only one way to train a neural network: compute the gradient and update the weights.
Then I discovered there are actually three different ways to do it.
Let's understand them with a simple example.
Imagine you have a dataset with 10,000 images.
Your goal is to minimize the loss function by updating the model's weights.
Batch Gradient Descent
Batch Gradient Descent processes the entire dataset before making a single update.
10,000 samples -> Compute total loss -> Compute gradients -> Update weights
Advantages
- Stable gradient
- Smooth convergence
- Accurate update direction
Disadvantages
- Slow for large datasets
- Requires lots of memory
- One update only after processing every sample
Think of it like reading an entire book before writing a summary.
2. Stochastic Gradient Descent (SGD)
Instead of waiting for all 10,000 samples, SGD updates the weights after every single sample.
Sample 1 → Update
Sample 2 → Update
Sample 3 → Update
...
Sample 10,000 → Update
Now the model learns much faster.
The downside?
Each sample may point in a slightly different direction, causing the optimization path to bounce around.
Imagine hiking toward the bottom of a valley while someone changes your direction every few seconds.
This creates the famous zigzag optimization path.
Advantages
- Very fast updates
- Low memory usage
- Can escape some local minima
Disadvantages
- Noisy gradients
- Unstable convergence
- Loss fluctuates a lot
3. Mini-Batch Gradient Descent
Mini-batch combines the best parts of both approaches.
Instead of using one sample or the entire dataset, we split the data into small batches.
For example:
10,000 samples
Batch 1 = 128 samples → Update
Batch 2 = 128 samples → Update
Batch 3 = 128 samples → Update
...
Now every update is based on enough data to reduce noise, but not so much that training becomes slow.
This is why frameworks like PyTorch and TensorFlow use mini-batches by default.
Advantages
- Faster than Batch GD
- More stable than SGD
- Efficient GPU utilization
- Standard choice for deep learning
Comparison
| Method | Update Frequency | Speed | Stability |
|---|---|---|---|
| Batch Gradient Descent | After the entire dataset | Slow | High |
| Stochastic Gradient Descent | After every sample | Fast | Low |
| Mini-Batch Gradient Descent | After every small batch | Fast | High |
Final Thoughts
If you're training modern neural networks, Mini-Batch Gradient Descent is usually the best choice.
It balances speed, memory usage, and convergence, making it the default optimization strategy in most deep learning libraries.
Understanding these three optimization strategies helped me understand why neural networks train the way they do—not just how they train.
Top comments (0)