Understanding Backpropagation with Python Examples — Part 1

#ai #machinelearning

In this article, we will be trying to understand Backpropagation.

Backpropagation mainly involves the following

Using the chain rule to calculate derivatives
Plugging the derivatives into gradient descent to optimize parameters

We will start a demo on backpropagation with the following example. Assume these values

Weights

w1 = 3.34
w2 = -3.53
w3 = -1.22
w4 = -2.30

Biases

b1 = 1.43
b2 = 0.57
b3 = 2.61

Conceptually, backpropagation starts with the last parameters and works its way backwards to estimate the other parameters.

We can discuss all concepts of backpropagation by guessing the last value.

In order to start from the back, assume we have optimal values for all parameters except for the last bias, b3.

Let’s apply input 0 to 1 through the upper path.
Slope = 3.34 and bias = 1.43.

Here is a snippet to visualize this

import numpy as np
import matplotlib.pyplot as plt

def softplus(x):
    return np.log(1 + np.exp(x))

x = np.linspace(0, 1, 100)

w1, b1, w3 = 3.34, 1.43, -1.22

z_upper = w1 * x + b1
y_upper = softplus(z_upper)
y_upper_final = w3 * y_upper

plt.plot(x, y_upper_final)
plt.title("Upper Path (Blue Curve)")
plt.xlabel("Input")
plt.ylabel("Output")
plt.show()

For getting the y-axis coordinates we are
Applying it through softplus.

As a result, we get a curve. Then multiply the y-axis coordinates on the blue curve by -1.22.

We get the final blue curve.

Now for the bottom node, if we try:

Slope = -3.53 and bias = 0.57.

Input 0 to 1, we get a curve.

Then multiply it by -2.30.

w2, b2, w4 = -3.53, 0.57, -2.30

z_lower = w2 * x + b2
y_lower = softplus(z_lower)
y_lower_final = w4 * y_lower

plt.plot(x, y_lower_final)
plt.title("Lower Path (Orange Curve)")
plt.xlabel("Input")
plt.ylabel("Output")
plt.show()

We get the final curve (orange).

Now we add both blue and orange together to get the green squiggle.

Now we are ready to add the final bias b3.

We don’t know the value of b3, so we give an initial value of 0.

b3 = 0

y_combined = y_upper_final + y_lower_final + b3

plt.plot(x, y_combined)
plt.title("Combined Output (Green Squiggle)")
plt.xlabel("Input")
plt.ylabel("Output")
plt.show()

Adding 0 to all y-axis coordinates on the squiggle gives no changes.

We need to calculate the error and work our way towards finding the correct bias value, which we will explore in Part 2 of this article.

You can try out the examples on My Colab Notebook

If you’ve ever struggled with repetitive tasks, obscure commands, or debugging headaches, this platform is here to make your life easier. It’s free, open-source, and built with developers in mind.