Part 13 of my "revisiting my AI/ML notes" series. Every post in this series so far used gradient descent conceptually, without asking a fairly important question: when we compute the gradient of the loss, how much of the data are we actually using at once? The answer has three different flavors, and today almost nobody uses the simplest one.
Batch Gradient Descent
The most direct approach: compute the gradient of the loss using every single data point in the training set before making one weight update.
dLoss/dw = gradient computed across all n data points
This gives a smooth, accurate estimate of the true gradient direction on every step. The catch is cost: for a large dataset, one single weight update requires a full pass over all the data, which makes training painfully slow and memory-hungry at scale.
Stochastic Gradient Descent (SGD)
The opposite extreme: compute the gradient using just one data point at a time, and update the weights immediately.
dLoss/dw = gradient computed from a single data point
This is fast per step and needs almost no memory, but the path toward the minimum is noisy — each individual data point gives a rough, sometimes-misleading estimate of where the true gradient points, so the weight trajectory jitters around rather than moving smoothly downhill.
Mini-Batch Stochastic Gradient Descent
The practical middle ground, and what's actually used almost everywhere today: compute the gradient over a small batch of k data points at a time.
dLoss/dw = gradient averaged over k data points, where 1 < k < n
This captures most of batch gradient descent's stability while keeping most of SGD's speed and memory efficiency — and it maps naturally onto GPU-parallelized computation, which is a big part of why it's the default in virtually every modern deep learning framework.
The loss landscape isn't always a simple bowl
All three variants are trying to reach the same destination: a point where the gradient is zero. But that destination isn't guaranteed to be unique or even reachable in one obvious direction, which depends on the shape of the loss function:
- A convex function has exactly one minimum — the global minimum — and no matter where you start, gradient descent reliably finds it.
- A non-convex function — which is what most deep networks actually have — can have multiple local minima and saddle points scattered across the loss surface. Critically, the gradient is exactly zero at all of these points, not just the true global minimum, which means an optimizer can genuinely get stuck somewhere that only looks optimal locally.
This is precisely why the choice between batch, SGD, and mini-batch matters beyond just speed: the noise in SGD and mini-batch updates, which looks like a downside on a simple convex bowl, actually helps the optimizer bounce out of shallow local minima and saddle points on the messier, non-convex surfaces real networks actually have.
Why this matters
"Gradient descent" isn't one algorithm with one fixed behavior — it's a family, and the choice of how much data goes into each gradient estimate directly trades off convergence speed, memory use, and the ability to escape bad local minima. Mini-batch SGD winning out as the default isn't an accident; it's the version that best fits how modern hardware and modern (non-convex) loss surfaces actually behave.
Original Notes:
Top comments (0)