DEV Community

Cover image for Understanding Backpropagation with Python Examples — Part 3
Rijul Rajesh
Rijul Rajesh

Posted on

Understanding Backpropagation with Python Examples — Part 3

#ai #machinelearning

In the previous article – Part 2 of this series, we plotted the graph of Sum of Squared Residuals (SSR) vs the bias, so that we could determine the ideal bias value.

Let’s continue from there.

We stopped here last time.

Let’s draw a curve over this.

From this, we can find the lowest point.

Instead of plugging in many values to find the lowest point on this pink curve, we can use gradient descent to find it quickly.

For that, we need to take the derivative of SSR with respect to b3.

To calculate this, we will use the chain rule.

By taking the derivative of SSR with respect to the predicted values, and the derivative of the predicted value with respect to b3, we get the following:

We have to plug this into gradient descent to find the optimal value for b3.

Let’s expand the summation:

-2 × (0 − Predicted1) × 1
+ -2 × (1 − Predicted2) × 1
+ -2 × (0 − Predicted3) × 1
Enter fullscreen mode Exit fullscreen mode

When we input 0, 1, and 0, we get -2.6, -1.6, and -2.61 as the predicted values.

-2 × (0 − -2.6) × 1
+ -2 × (1 − -1.6) × 1
+ -2 × (0 − -2.61) × 1
= -15.7
Enter fullscreen mode Exit fullscreen mode

This corresponds to the slope when b3 = 0.

We plug this into the gradient descent equation:

Step size = slope × learning rate
Step size = -15.7 × 0.1 = -1.57
Enter fullscreen mode Exit fullscreen mode 
New b3 = old b3 − step size
New b3 = 0 − (-1.57) = 1.57
Enter fullscreen mode Exit fullscreen mode

Now let’s see how the green squiggle has improved.

We now get the predicted values as -1.03, -0.03, and -1.04.

We plug these into the equation and get the new step size:

-2 × (0 − -1.03) × 1
+ -2 × (1 − -0.03) × 1
+ -2 × (0 − -1.04) × 1
= -6.26
Enter fullscreen mode Exit fullscreen mode 
Step size = -6.26 × 0.1 = -0.626
Enter fullscreen mode Exit fullscreen mode 
New b3 = 1.57 − (-0.626) = 2.19
Enter fullscreen mode Exit fullscreen mode

Changing b3 to 2.19 shifts it up further.

We take more steps until the step size is close to 0.

When b3 = 2.61, the step size is close to 0, so 2.61 is the optimal value.

So that’s how you optimize biases.

We’ll explore more in the coming articles.

You can try this out in the Colab notebook.

Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia — a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run:

ipm install repo-name
Enter fullscreen mode Exit fullscreen mode

…and you’re done!

Installerpedia Screenshot

🔗 Explore Installerpedia here

Top comments (0)