DEV Community: Ali Raza

Tutorial: Padding & Edge Detection in CNNs (Hands-on)

Ali Raza — Sun, 02 Aug 2026 11:35:44 +0000

This is a hands-on tutorial that builds directly on the convolution and padding concepts from the previous article. We'll walk through normalizing filter output values and applying edge-detection filters step by step.

Step 1: Normalizing Output Values (Min-Max)

When you apply a filter to an image, the resulting values aren't always valid pixel intensities. For instance, a convolution operation might produce values like -4 or numbers greater than 255 — but a pixel can only be between 0 and 255.

This is where a min-max function comes in:

if value < 0   → set to 0
if value > 255 → set to 255

This clips the output back into a valid image range (this is conceptually very similar to what a ReLU activation does later in the network, followed by clipping at the maximum pixel value).

Step 2: Edge Detection Filters

Two of the most fundamental 3×3 filters in image processing are used to detect horizontal and vertical edges — this is essentially a simplified version of the classic Sobel operator.

Horizontal Edge Filter

 +1  +2  +1
  0   0   0
 -1  -2  -1

Vertical Edge Filter

 +1   0  -1
 +2   0  -2
 +1   0  -1

When either of these filters slides across an image, it produces a strong response wherever there's a sharp change in pixel intensity in that direction — which is exactly what an "edge" is: a sudden jump in brightness.

Step 3: Applying Padding in Practice

Continuing from our padding discussion — say we take a 6×6 grayscale image and want to preserve its size after convolution with a 3×3 filter. We calculated earlier that we need p = 1 (one row/column of zero-padding on every side).

This transforms our 6×6 image into an 8×8 padded image before the filter is applied:

Zeros are added to the top and bottom rows.
Zeros are added to the left and right columns.
The original 6×6 image data sits untouched in the center.

When the 3×3 filter now slides across this 8×8 padded image, the output comes back out as 6×6 — matching the original size, with every pixel (including the edges) properly represented in the output.

Putting It All Together

Take the raw image (grayscale or RGB).
Apply zero-padding based on your desired output size.
Slide the filter (e.g., horizontal or vertical edge detector) across the padded image, computing the convolution at each step.
Apply min-max clipping to keep pixel values within the valid 0–255 range.
The result is a feature map that highlights edges — the very first building block CNNs use to eventually recognize full objects.

This is the foundation every CNN architecture builds on — stacking many of these convolution + padding + activation steps to go from raw pixels to full image understanding.

This is Part 3 (Tutorial) of a CNN fundamentals series, based on lecture notes from a Deep Learning course.

What Is Convolution? Breaking Down How CNNs Process Images

Ali Raza — Sun, 02 Aug 2026 11:30:54 +0000

In the previous article, we saw why CNNs are inspired by the human visual cortex. Now let's get into the actual mechanics: what does "convolution" mean, and how does a CNN process an image pixel by pixel?

Images Are Just Numbers

Grayscale images

A grayscale (black & white) image is really just a grid of numbers. Each cell in the grid is a pixel, and its value ranges from 0 to 255:

0 → completely black
255 → completely white
Anything in between → a shade of gray

For example, a small 4×4 grayscale image might look like a matrix of pixel intensities. You can resize this grid — reduce or increase the number of pixels — depending on how much detail you want to keep.

Color images

A color image adds two more layers on top: Red, Green, and Blue (RGB). So instead of a single 2D grid, a color image is a 3D volume — for example, a 6×6×3 image means 6×6 pixels across 3 color channels.

The Convolution Operation

Convolution is the process of sliding a small matrix — called a filter or kernel (commonly 3×3) — across the image, and at each position, performing an element-wise multiplication followed by a sum. The result becomes one pixel in the output feature map.

This is how a CNN detects patterns: different filters are designed (or learned) to highlight specific features like edges, corners, or textures. When you apply a filter across a grayscale image, you get a new, smaller grid where certain regions light up — these are the features the filter was designed to catch.

The Problem: Shrinking Output

Here's something important to notice: after applying a filter, the output is smaller than the input. There's a formula for this:

output size = n − f + 1

Where:

n = input size (e.g., 6 for a 6×6 image)
f = filter size (e.g., 3 for a 3×3 filter)

So a 6×6 image with a 3×3 filter gives a 4×4 output — we've lost information at the borders! Pixels at the edges of the image get "seen" by the filter far fewer times than pixels in the center.

The Fix: Padding

To prevent this loss of information, we use padding — adding extra rows and columns (usually filled with zeros) around the border of the image before applying the filter.

There are two common ways to fill padding:

Fill with the value of the neighboring pixel.
Fill with zero — this is the standard, best-practice approach.

With padding added, the formula becomes:

output size = n + 2p − f + 1

Where p is the padding amount. Let's verify with an example: n = 6, f = 3, p = 1:

6 + 2(1) − 3 + 1 = 6

The output stays 6×6 — exactly the same as the input! No information is lost, because padding adds new rows and columns on the top, bottom, left, and right sides of the original image before the filter slides over it.

Coming Up Next

In the next (tutorial) article, we'll actually build edge-detection filters — horizontal and vertical — and see step by step how padding is applied in practice, along with how output pixel values are normalized back into a valid 0–255 range.

This is Part 2 of a CNN fundamentals series, based on lecture notes from a Deep Learning course.

CNN vs Human Brain — Why Convolutional Neural Networks Learn to "See"

Ali Raza — Sun, 02 Aug 2026 11:22:13 +0000

From ANN to CNN

In our earlier lectures, we studied Artificial Neural Networks (ANNs) and how they solve two broad categories of problems:

Classification — for example, predicting whether a tumor is benign or malignant (breast cancer detection).
Regression — for example, predicting a continuous value like an air quality index.

Those problems mostly dealt with structured, tabular data. Starting from this lecture, we shift focus to a different kind of data: images. This is where Convolutional Neural Networks (CNNs) come in — a specialized type of neural network built for image classification and object detection, the core tasks of computer vision.

Why "Convolutional"? A Brain-Inspired Design

Here's the interesting part: CNNs aren't just an arbitrary architecture — they take direct inspiration from how the human brain processes visual information.

At the back of our brain sits a region called the cerebral cortex. Within it lies the visual cortex — the part specifically responsible for interpreting what our eyes see. When light hits our retina, the signal isn't processed all at once. Instead, it passes through multiple layers of neurons, each layer responsible for extracting a specific kind of information — edges, shapes, textures, and eventually complete objects.

CNNs mimic this exact idea:

An image is broken down into smaller regions.
Each layer of the network is trained to detect specific patterns within those regions (like edges in earlier layers, and complex shapes/objects in deeper layers).
The layers work together, each one confirming and refining what the previous layer detected.

The Big Picture

Human Brain	CNN
Cerebral Cortex	Full Network
Visual Cortex	Convolutional Layers
Multiple processing layers of neurons	Multiple convolutional + pooling layers
Detects edges → shapes → objects	Detects edges → patterns → objects

This biological analogy is the reason CNNs are so effective at image-related tasks — they don't look at an entire image as one giant block of numbers. Instead, just like our visual cortex, they scan it piece by piece, layer by layer, building up understanding gradually.

In the next article, we'll get hands-on and answer the core question: what is convolution, mathematically? We'll break down grayscale images, RGB channels, filters, and how a CNN actually "reads" pixels.

This is Part 1 of a CNN fundamentals series, based on lecture notes from a Deep Learning course.

Batch, Stochastic, and Mini-Batch Gradient Descent: What's Actually Different

Ali Raza — Mon, 20 Jul 2026 11:51:04 +0000

Part 13 of my "revisiting my AI/ML notes" series. Every post in this series so far used gradient descent conceptually, without asking a fairly important question: when we compute the gradient of the loss, how much of the data are we actually using at once? The answer has three different flavors, and today almost nobody uses the simplest one.

Batch Gradient Descent

The most direct approach: compute the gradient of the loss using every single data point in the training set before making one weight update.

dLoss/dw = gradient computed across all n data points
This gives a smooth, accurate estimate of the true gradient direction on every step. The catch is cost: for a large dataset, one single weight update requires a full pass over all the data, which makes training painfully slow and memory-hungry at scale.

Stochastic Gradient Descent (SGD)

The opposite extreme: compute the gradient using just one data point at a time, and update the weights immediately.

dLoss/dw = gradient computed from a single data point

This is fast per step and needs almost no memory, but the path toward the minimum is noisy — each individual data point gives a rough, sometimes-misleading estimate of where the true gradient points, so the weight trajectory jitters around rather than moving smoothly downhill.

Mini-Batch Stochastic Gradient Descent

The practical middle ground, and what's actually used almost everywhere today: compute the gradient over a small batch of k data points at a time.

dLoss/dw = gradient averaged over k data points, where 1 < k < n

This captures most of batch gradient descent's stability while keeping most of SGD's speed and memory efficiency — and it maps naturally onto GPU-parallelized computation, which is a big part of why it's the default in virtually every modern deep learning framework.

The loss landscape isn't always a simple bowl

All three variants are trying to reach the same destination: a point where the gradient is zero. But that destination isn't guaranteed to be unique or even reachable in one obvious direction, which depends on the shape of the loss function:

A convex function has exactly one minimum — the global minimum — and no matter where you start, gradient descent reliably finds it.
A non-convex function — which is what most deep networks actually have — can have multiple local minima and saddle points scattered across the loss surface. Critically, the gradient is exactly zero at all of these points, not just the true global minimum, which means an optimizer can genuinely get stuck somewhere that only looks optimal locally.

This is precisely why the choice between batch, SGD, and mini-batch matters beyond just speed: the noise in SGD and mini-batch updates, which looks like a downside on a simple convex bowl, actually helps the optimizer bounce out of shallow local minima and saddle points on the messier, non-convex surfaces real networks actually have.

Why this matters

"Gradient descent" isn't one algorithm with one fixed behavior — it's a family, and the choice of how much data goes into each gradient estimate directly trades off convergence speed, memory use, and the ability to escape bad local minima. Mini-batch SGD winning out as the default isn't an accident; it's the version that best fits how modern hardware and modern (non-convex) loss surfaces actually behave.

Original Notes:

Weight Initialization: Why Random Isn't Random Enough

Ali Raza — Mon, 20 Jul 2026 11:47:43 +0000

Part 12 of my "revisiting my AI/ML notes" series. Every post so far in this series has assumed weights already have reasonable values. This one covers the step before any of that: how weights get their starting values in the first place, and why that choice matters more than it sounds like it should.

Why initialization isn't a minor detail

There are a few hard constraints on a good initialization scheme:

Weights should be small, but not vanishingly small — too small, and the earlier posts in this series on vanishing gradients explain exactly what goes wrong.
Weights should not all be the same value. If every neuron in a layer starts identical, they all compute the same gradient during backpropagation and stay identical forever — the layer never actually differentiates into useful, distinct feature detectors.
Weights need good variance — enough spread that different neurons can learn different things, without being so spread out that outputs blow up as they pass through many layers (the exploding gradient problem, also covered earlier).

Too-small variance is functionally the same starting point as the vanishing gradient problem — the signal barely moves through the network. Too-large variance heads straight toward the exploding gradient problem. The right initialization scheme depends on which activation function the layer uses.

Xavier / Glorot initialization

Designed for sigmoid and tanh activations. Two common variants:
Xavier Normal: W ~ N(0, sigma^2), where sigma^2 = 2 / (fan_in + fan_out)
Xavier Uniform: W ~ Uniform[-limit, limit], where limit = sqrt(6 / (fan_in + fan_out))
Here fan_in is the number of input connections to a neuron, and fan_out is the number of output connections. Balancing the initialization around both keeps the variance of activations (and gradients) roughly consistent as they flow forward and backward through the network — which is specifically what sigmoid and tanh, with their easily-saturating curves, need to avoid vanishing gradients from the very first forward pass.

He initialization

Designed for ReLU and its variants (Leaky ReLU, ELU, PReLU — all covered in the previous post):
He Normal: W ~ N(0, sigma^2), where sigma^2 = 2 / fan_in
He Uniform: W ~ Uniform[-limit, limit], where limit = sqrt(6 / fan_in)
He initialization only accounts for fan_in, using a larger variance than Xavier. This compensates for the fact that ReLU zeroes out roughly half its inputs (everything negative) — a larger starting variance keeps the surviving signal at a healthy scale.

Why this matters

Picking Xavier for a ReLU network, or vice versa, isn't a fatal mistake, but it does make training measurably harder — slower convergence, or a higher chance of hitting vanishing/exploding gradients early, before the optimizer has had a chance to correct course. Matching the initialization scheme to the activation function is one of the cheapest, most reliable wins available before training even starts.

Original Notes:

Beyond Sigmoid and ReLU: Leaky ReLU, ELU, Swish, and Softmax

Ali Raza — Mon, 20 Jul 2026 11:42:57 +0000

Part 11 of my "revisiting my AI/ML notes" series. Earlier in this series I covered sigmoid and plain ReLU. ReLU fixed sigmoid's vanishing gradient problem for positive inputs — but it introduced a new failure mode of its own, and there's a whole family of activation functions built to fix it.

The problem with plain ReLU: dying neurons

ReLU is max(0, z) — for any negative input, the output is exactly zero, and so is the gradient. If a neuron's weights drift into a state where its input is consistently negative, that gradient stays zero forever: the neuron stops updating completely, no matter how much more training happens. This is called the dying ReLU problem, and it can silently remove a meaningful chunk of a network's capacity.

Leaky ReLU

The fix is almost embarrassingly small: instead of outputting exactly 0 for negative inputs, output a small fraction of the input.
LeakyReLU(z) = z if z > 0, else 0.01 * z

That tiny non-zero slope for negative values is enough to keep a small gradient flowing, so a neuron that drifts negative can still recover instead of dying permanently.

ELU (Exponential Linear Unit)

ELU(z) = z if z > 0, else alpha * (e^z - 1)
ELU takes a different approach for negative inputs — an exponential curve that smoothly approaches -alpha instead of a straight line. This gives it smoother gradients around zero, which can help training stability. The tradeoff: computing e^z on every negative input is meaningfully more expensive than Leaky ReLU's simple multiplication, so it's a real speed-vs-quality tradeoff, not a strict upgrade.

PReLU (Parametric ReLU) takes Leaky ReLU one step further by making that slope (the alpha) a learnable parameter instead of a fixed 0.01 — the network figures out the best slope for itself during training.

Swish

Swish(z) = z * sigmoid(z)
Swish is a newer activation, generally recommended for very deep networks (notes mention 40+ layers). Unlike ReLU, it's smooth everywhere and slightly non-monotonic near zero, which empirically tends to help gradient flow in very deep architectures.

Softmax: for the output layer, not hidden layers

Everything above is a hidden-layer activation. Softmax is different — it's used specifically at the output layer for multi-class classification, converting a set of raw scores into a valid probability distribution that sums to 1:
softmax(z_i) = e^(z_i) / sum over j of e^(z_j)

If a model outputs raw scores across four classes and softmax turns them into something like [0.6, 0.2, 0.1, 0.1], the class with the highest probability (0.6) is the model's prediction — and unlike sigmoid, this generalizes naturally beyond two classes.

Why this matters

There's no single "best" activation function — each one trades off gradient behavior, computational cost, and training stability differently. What matters is recognizing which problem a given activation function was built to solve, so the choice for a specific architecture is deliberate rather than just "whatever the tutorial used."

Original Notes:

Dropout and Regularization: How Neural Networks Avoid Overfitting

Ali Raza — Mon, 20 Jul 2026 11:28:21 +0000

Part 10 of my "revisiting my AI/ML notes" series. Everything so far has been about getting a network to learn at all. This post is about a different problem: getting it to learn the right things, instead of just memorizing the training data.

The problem: overfitting

A network with enough capacity can, given enough training time, essentially memorize its training set — including its noise and quirks — instead of learning patterns that generalize to new data. It looks great on training metrics and falls apart on anything it hasn't seen before. Two standard techniques exist specifically to prevent this: regularization (L1, L2) and dropout.

Dropout

Dropout, introduced by Nitish Srivastava and Geoffrey Hinton in 2014, is disarmingly simple: during training, randomly "turn off" a fraction of neurons in a given layer on every forward pass.

A dropout probability like p = 0.5 means roughly 50% of neurons in that layer are randomly zeroed out on each pass.
Which specific neurons get dropped changes every single pass — no neuron can become over-reliant on always being present, and no neuron can "specialize" in memorizing one specific training quirk, because it might not even be active next time.
At test time, no neurons are dropped — the full network is used. To keep the overall signal strength consistent between training and testing, the weights (or activations, depending on implementation) are scaled by (1 - p).

This forces the network to spread useful information across many neurons instead of concentrating it in a few, which is exactly what generalization needs.

Choosing the dropout rate

The dropout probability p is a hyperparameter, and it needs tuning like any other:

Too high, and you're removing so much of the network on each pass that it struggles to learn anything useful — underfitting.
Too low, and you're barely regularizing at all — the overfitting problem you were trying to solve doesn't really go away.

Standard practice is to search over this value (hyperparameter tuning) rather than guessing, checking performance on validation data to find where the tradeoff actually lands for your specific problem.

Why this matters

Dropout and its cousin, weight regularization, are two of the most reliable tools for keeping a network's performance on unseen data close to its performance on training data. Without them, deep networks — which have enormous capacity to memorize — very often look excellent on paper and fail quietly in production.

Original Notes:

The Exploding Gradient Problem, Explained

Ali Raza — Sun, 19 Jul 2026 07:39:17 +0000

Part 9 of my "revisiting my AI/ML notes" series, and the last one in the backpropagation arc. The previous post covered gradients shrinking to nothing across many layers. This one covers the mirror-image problem: gradients that grow instead, until training becomes unstable.

The same chain rule, the opposite direction

Vanishing gradients happen when the per-layer terms in the chain rule product are consistently less than 1. Exploding gradients happen for the exact opposite reason: when those per-layer terms — usually driven by weight values — are consistently greater than 1.

gradient_at_early_layer ~= 2.5 x 2.5 x 2.5 x ... (once per layer)

Instead of shrinking toward zero, this product grows exponentially with depth. A gradient that should be a small, well-behaved correction instead becomes enormous by the time it reaches the earlier layers.

Why this makes training unstable

Recall the update rule again: w_new = w_old - lr * gradient. If the gradient is huge, the weight update is huge too — even a modest learning rate can't fully compensate for a gradient that's grown by several orders of magnitude. In practice this looks like:

Loss swinging wildly between iterations instead of decreasing smoothly
Weights growing to extremely large values, sometimes overflowing into NaN
Training that was progressing fine suddenly diverging with no clear warning

Common fixes

A few standard techniques exist specifically to keep this under control:

Gradient clipping — cap the gradient's magnitude at a fixed threshold before applying the update, regardless of how large the raw computed value is.
Careful weight initialization — techniques like Xavier/Glorot or He initialization choose starting weight scales specifically to avoid values that compound into explosion (or vanishing) across layers.
Batch normalization — normalizing activations between layers keeps values in a consistent, well-behaved range, which indirectly keeps gradients from spiraling in either direction.

Wrapping up the backpropagation arc

Across these last few posts: forward propagation produces a prediction, the loss measures how wrong it is, the chain rule propagates that error backward through every layer, and gradient descent uses it to update every weight. Vanishing and exploding gradients are what happens when that backward chain of multiplications isn't kept in a healthy range — too small and early layers stop learning, too large and the whole network becomes unstable. Every architecture that comes after this in a deep learning curriculum — CNNs, RNNs, transformers — is still built on this exact same core loop underneath.

Original Notes:

The Vanishing Gradient Problem, Explained

Ali Raza — Sun, 19 Jul 2026 07:36:13 +0000

Part 8 of my "revisiting my AI/ML notes" series. The last post ended on a warning: multiplying several small numbers together, as the chain rule does, shrinks the result fast. This post is about what happens when that shrinking gets out of hand — the vanishing gradient problem.

Where the shrinking comes from

Sigmoid is the culprit here. Its derivative — the local gradient the chain rule multiplies at every layer it passes through — has a maximum value of exactly 0.25, and drops toward zero for large positive or negative inputs.

Now apply the chain rule across a deep network. If every layer contributes a local derivative that's at best 0.25 (and realistically often smaller, once you multiply in typically-small weight values too), then the gradient reaching a weight near the input is the product of many of these small numbers, layer after layer:

gradient_at_layer_1 ~= 0.25 x 0.25 x 0.25 x ... (once per layer)

That product shrinks toward zero shockingly fast as depth increases.

[PASTE DIAGRAM IMAGE HERE - SECOND HALF]

The practical consequence

Recall the weight update rule: w_new = w_old - lr * gradient. If the gradient is effectively zero by the time it reaches an early layer, the update is effectively zero too — that weight barely changes, no matter how many training iterations run. The layers closest to the input stop learning, while the layers closest to the output keep updating more or less normally. The network ends up with a large chunk of its capacity permanently under-trained.

This is a genuinely serious problem for deep networks — it's part of why very deep sigmoid-based networks were historically hard to train, and it's a big part of why ReLU (covered earlier in this series) became the default for hidden layers: ReLU's derivative is either 0 or exactly 1, so it doesn't shrink the gradient the way sigmoid does for positive inputs.

Why this matters

Vanishing gradients are a direct, mechanical consequence of the chain rule and the shape of sigmoid — not a mysterious training instability. Once you see it as "multiplying many numbers less than 1 together," several standard deep learning fixes suddenly make sense: switching activation functions, using residual/skip connections, and normalization techniques all exist specifically to keep this product from collapsing to zero.

Next up: the opposite failure mode — what happens when that same chain of multiplications grows instead of shrinks.

Original Notes:

The Chain Rule: How Backpropagation Reaches Every Layer

Ali Raza — Sun, 19 Jul 2026 07:32:25 +0000

Part 7 of my "revisiting my AI/ML notes" series. The last post left off with a question: gradient descent needs the slope of the loss with respect to a weight, but in a multilayer network, most weights aren't directly connected to the loss at all — they're several layers removed from it. This is exactly what the chain rule solves.

The problem: weights are buried inside the network

In a single-neuron example, computing dLoss/dw is direct — the weight feeds straight into the output, which feeds straight into the loss. But in a multilayer network, an early weight influences the loss only indirectly: it affects a neuron's output, which affects the next layer's neurons, which eventually affects the final output, which determines the loss.

The chain rule is exactly this: multiply the local effects together

The chain rule from calculus says that if w1 affects a1, a1 affects a2, and a2 affects the loss, then the total effect of w1 on the loss is the product of each individual step's effect:
dLoss/dw1 = (dLoss/da2) x (da2/da1) x (da1/dw1)

Each term on the right is a local derivative — easy to compute on its own, because it only involves two directly-connected quantities. The chain rule is what lets you stitch these small, local calculations together into the one number you actually need: how much this specific weight, buried deep in the network, is responsible for the final error.

Why this is the actual mechanism behind "backpropagation"

This is also where the name comes from. To compute dLoss/dw1 for a weight near the input, you need dLoss/da2 first — which itself needs to be computed from the output side. So the computation naturally flows backward, from the output toward the input, computing each local derivative and multiplying it into the running product as it goes. That backward flow of derivatives, layer by layer, is backpropagation.

Why this matters

This chain-multiplication is elegant, but it has a real consequence worth flagging early: if each individual local derivative is small, multiplying several of them together makes the result very small very fast. If each one is large, the product explodes just as fast. That single observation is the root cause behind two of the most well-known training problems in deep learning — vanishing and exploding gradients — which are exactly what the next two posts in this series cover.

Original Notes:

Multilayer Neural Networks and Gradient Descent, Explained

Ali Raza — Sun, 19 Jul 2026 07:29:07 +0000

Part 6 of my "revisiting my AI/ML notes" series. Everything so far in this series used a single hidden layer as an example. Real networks almost always stack multiple hidden layers — this post covers what changes when you do that, and revisits gradient descent from a more visual angle.

Going from one hidden layer to many

A multilayer network just repeats the same input -> weighted sum -> activation pattern, but chains multiple hidden layers back to back before reaching the output. The structure looks like:
Input layer -> Hidden layer 1 -> Hidden layer 2 -> Output layer

Why the number of weights grows so fast

Every layer is fully connected to the next, so the number of weights is the product of the neuron counts on each side. With 4 input neurons feeding into a hidden layer of 3 neurons, that's 4 x 3 = 12 weights just for that one connection. If that hidden layer of 3 then feeds into a second hidden layer of 2 neurons, that's another 3 x 2 = 6 weights.

This is worth sitting with for a second: weight count isn't linear in the number of neurons, it's closer to combinatorial across layer boundaries. This is exactly why deep networks have millions (or billions) of parameters — it's not that any single layer is huge, it's that the connections between layers multiply fast.

Reading the slope to know which way to move

Gradient descent works by looking at the slope (derivative) of the loss curve at the weight's current position, and using that slope to decide which direction to move the weight:

If the slope is negative (the curve is heading downward to the right), the weight needs to move up to get closer to the minimum.
If the slope is positive (the curve is heading upward to the right), the weight needs to move down

This is exactly why the update rule has a minus sign in it — w_new = w_old - lr * gradient — subtracting a positive gradient moves the weight down, and subtracting a negative gradient moves it up. The sign of the gradient is doing the steering; the minus sign is what makes the steering point toward lower loss instead of higher loss.

Why this matters

A single neuron's math (covered earlier in this series) is simple enough to compute by hand. A real network with multiple hidden layers is not — and that's precisely why backpropagation needs the chain rule to work, which is the topic of the next post. Understanding gradient descent visually first, as a ball rolling down a curve based on the local slope, makes the chain-rule math that follows feel like a natural extension rather than an arbitrary formula.

The Original Notes:

Backpropagation Explained: How a Neural Network Actually Learns

Ali Raza — Sat, 18 Jul 2026 11:30:59 +0000

Part 5 of my "revisiting my AI/ML notes" series. So far this series has covered the ANN structure, forward propagation, and activation functions — all the machinery for a network to produce a prediction. This post covers the other half: how the network actually learns from being wrong.

The problem forward propagation doesn't solve

Forward propagation gives you a prediction. On its own, that prediction is basically a random guess — the weights start out essentially arbitrary. Backpropagation is the algorithm that takes the error in that prediction and uses it to adjust every single weight in the network, so the next prediction is a little bit better.

Step 1: measure how wrong the prediction is

This is the loss function. A simple and common one is squared error:
Loss = (y - y_hat)^2

Where y is the actual/true value and y_hat is the network's prediction. We square the difference for two reasons: it makes the loss positive regardless of whether the prediction was too high or too low, and it penalizes bigger errors disproportionately more than smaller ones.

Across a full dataset of n examples, the total loss is just the sum of this across every example:

Loss = sum over i=1 to n of (y_i - y_hat_i)^2

A worked example from my notes

Say a model is predicting whether someone passes an exam based on hours spent on Play (2h), Study (4h), and Sleep (8h), where 1 = pass. The true label is y = 1, but the network's current (untrained) prediction is y_hat = 0.

A worked example from my notes

Loss = (1 - 0)^2 = 1

A loss of 1 here is about as bad as this particular setup gets — the prediction is completely wrong. That's the signal backpropagation uses to correct the weights.

Step 2: figure out which direction to adjust each weight

This is the actual "backward" part. For every weight in the network, backpropagation computes the partial derivative of the loss with respect to that weight — written dLoss/dw. This derivative tells you two things: which direction increasing the loss lies in, and how sensitive the loss is to that particular weight.

The update rule is the same for every weight in the network:

w_new = w_old - learning_rate * (dLoss / dw)

Notice the minus sign — you always move the weight in the opposite direction of the gradient, because you want to walk downhill toward lower loss, not uphill toward more of it.

Step 3: repeat, many times

One update rarely gets a weight anywhere close to optimal. Backpropagation runs this update for every weight, over and over across many training examples and many passes through the dataset, gradually walking every weight toward the values that minimize total loss — what's usually called the global minimum of the loss function.

Why the learning rate matters so much

The learning rate (lr in the formula above) controls how big each step is. This is a genuine balancing act:

Too high, and updates overshoot the minimum — the loss can bounce around or even diverge instead of settling down.
Too low, and training crawls: technically correct, but painfully slow, and it can get stuck in a shallow local dip instead of finding the true global minimum.

A small, carefully chosen learning rate is what lets training descend smoothly toward the minimum instead of overshooting past it.

Why this matters

Backpropagation is the algorithm that makes "deep learning" actually mean something — it's the mechanism that turns a randomly initialized network into one that's genuinely learned patterns from data. Every architecture covered so far in this series (and every architecture still to come — CNNs, RNNs, transformers) relies on this exact same core loop: predict, measure the error, propagate it backward, update the weights, repeat.

Original Note: