Nothing but NumPy: Understanding & Creating Neural Networks with Computational Graphs from Scratch

Rafay Khan — Wed, 17 Jul 2019 13:53:05 +0000

Understanding new concepts can be hard, especially these days when there is an avalanche of resources with only cursory explanations for complex concepts. This blog is the result of a dearth of detailed walkthroughs on how to create neural networks in the form of computational graphs.

In this, and some following, blog posts, I will consolidate all that I have learned as a way to give back to the community and help new entrants. I will be creating common forms of neural networks all with the help of nothing but NumPy.

This blog post is divided into two parts, the first part will be understanding the basics of a neural network and the second part will comprise the code for implementing everything learned from the first part.

Part Ⅰ: Understanding a Neural Network

Let's dig in🍽️

Neural networks are a model inspired by how the brain works. Similar to neurons in the brain, our 'mathematical neurons' are also, intuitively, connected; they take inputs(dendrites), do some simple computation on them and produce outputs(axons).

The best way to learn something is to build it. Let's start with a simple neural network and hand-solve it. This will give us an idea of how the computations flow through a neural network.

Fig 1. Simple input-output only neural network

As in the figure above, most of the time you will see a neural network depicted similarly. But this succinct and simple looking picture hides a bit of the complexity. Let's expand it out.

Fig 2. Expanded neural network

Now, let's go over each node in our graph and see what it represents.

Fig 3. Inputs nodes x₁ and x₂

These nodes represent our inputs for our first and second features, x₁ and x₂, that define a single example we feed to the neural network, thus called “Input Layer”

Fig 4. Weights

w₁ and w₂ represent our weight vectors (in some neural network literature it is denoted with the theta symbol, θ). Intuitively, these dictate how much influence each of the input features should have in computing the next node. If you are new to this, think of them as playing a similar role to the ‘slope’ or ‘gradient’ constant in a linear equation.

Weights are the main values our neural network has to “learn”. So initially, we will set them to random values and let the “learning algorithm” of our neural network decide the best weights that result in the correct outputs.

Why random initialization? More on this later.

Fig 5. Linear operation

This node represents a linear function. Simply, it takes all the inputs coming to it and creates a linear equation/combination out of them. ( By convention, it is understood that a linear combination of weights and inputs is part of each node, except for the input nodes in the input layer, thus this node is often omitted in figures, like in Fig.1. In this example, I’ll leave it in)

Fig 6. Linear operation

This σ node takes the input and passes it through the following function, called the sigmoid function(because of its S-shaped curve), also known as the logistic function:

Fig 7. Sigmoid(Logistic) function

Sigmoid is one of the many “activations functions” used in neural networks. The job of an activation function is to change the input to a different range. For example, if z > 2 then, σ(z) ≈ 1 and similarly, if z < -2 then, σ(z) ≈ 0. So, the sigmoid function squashes the output range to (0, 1) (this ‘()’ notation implies exclusive boundaries; never completely outputs 0 or 1 as the function asymptotes, but reaches very close to boundary values)

In our above neural network since it is the last node, it performs the function of output. The predicted output is denoted by ŷ. (Note: in some neural network literature this is denoted by ‘h(θ)’, where ‘h’ is called the hypothesis i.e. this is the hypothesis of the neural network, a.k.a the output prediction, given parameter θ; where θ are weights of the neural networks)

Now that we know what each and everything represents let’s flex our muscles by computing each node by hand on some dummy data.

Fig 8. OR gate

The data above represents an OR gate(output 1 if any input is 1). Each row of the table represents an ‘example’ we want our neural network to learn from. After learning from the given examples we want our neural network to perform the function of an OR gate; given the input features, x₁ and x₂, try to output the corresponding y(also called ‘label’). I have also plotted the points on a 2-D plane so that it is easy to visualize(green crosses represent points where the output(y) is 1 and the red dot represents the point where the output is 0).

This OR-gate data is particularly interesting, as it is linearly separable i.e. we can draw a straight line to separate the green cross from the red dot.

Fig 9. Showing that the OR gate data is linearly separable

We’ll shortly see how our simple neural network performs this task.

Data flows from left-to-right in our neural network. In technical terms, this process is called ‘forward propagation’; the computations from each node are forwarded to the next node, it is connected to.

Let’s go through all the computations our neural network will perform on the given the first example, x₁=0, and x₂=0. Also, we’ll initialize weights w₁ and w₂ to w₁=0.1 and w₂=0.6 (recall, these weights a have been randomly selected)

Fig 10. Forward propagation of the first example from OR table data

With our current weights, w₁= 0.1 and w₂ = 0.6, our network’s output is a bit far from where we’d like it to be. The predicted output, ŷ, should be ŷ≈0 for x₁=0 and x₂=0, right now it's ŷ=0.5.

So, how does one tell a neural network how far it is from our desired output? In comes the Loss Function to the rescue.

Loss Function

The Loss Function is a simple equation that tells us how far our neural network’s predicted output(ŷ) is from our desired output(y), for ONE example, only.

The derivative of the loss function dictates whether to increase or decrease weights. A positive derivative would mean decrease the weights and negative would mean increase the weights. The steeper the slope the more incorrect the prediction was.

Fig 11. Loss function visualized

The Loss function curve depicted in Figure 11 is an ideal version. In real-world cases, the Loss function may not be so smooth, with some bumps and saddles points along the way to the minimum.

There are many different kinds of loss functions each essentially calculating the error between predicted output and desired output. Here we’ll use one of the simplest loss functions, the Squared-Error Loss function. Defined as follows:

Fig 12. Loss Function. Calculating error for a single example

Taking the square keeps everything nice and positive and the fraction (1/2) is there so that it cancels out when taking the derivative of the squared term (it is common among some machine learning practitioners to leave the fraction out).

Intuitively, the Squared Error Loss function helps us in minimizing the vertical distance between our predictor line(blue line) and actual data(green dot). Behind the scenes, this predictor line is our z(linear function) node.

Fig 13. Visualization of the effect of the Loss function

Now that we know the purpose of a Loss function let’s calculate the error in our current prediction ŷ=0.5, given y=0

Fig 14. Loss calculated for 1ˢᵗ example

as we can see the Loss is 0.125. Given this, we can now use the derivative of the Loss function to check whether we need to increase or decrease our weights.

This process is called backpropagation, as we’ll be doing the opposite of the forward phase. Instead of going from input to output we’ll track backward from output to input. Simply, backpropagation allows us to figure out how much of the Loss each part of the neural network was responsible for.

To perform backpropagation we’ll employ the following technique: at each node, we only have our local gradient computed(partial derivatives of that node), then during backpropagation, as we are receiving numerical values of gradients from upstream, we take these and multiply with local gradients to pass them on to their respective connected nodes.

Fig 15. Gradient Flow

This is a generalization of the chain rule from calculus.

Since ŷ(predicted label) dictates our Loss and y(actual label) is constant, for a single example, we will take the partial derivative of Loss with respect to ŷ

Fig 16. The partial derivative of Loss w.r.t ŷ

Since the backpropagation steps can seem a bit complicated I’ll go over them step by step:

Fig 17.a. Backpropagation

For the next calculation, we’ll need the derivative of the sigmoid function, since it forms the local gradient of the red node. Let’s derive that.

Fig 18. The derivative of the Sigmoid function.

Let’s use this in the next backward calculation

Fig 17.b. Backpropagation

The backward computations should not propagate all the way to inputs as we don’t want to change our input data(i.e. red arrows should not go to green nodes). We only want to change the weights associated with inputs.

Fig 17.c. Backpropagation

Notice something weird? The derivatives to the Loss with respect to the weights,w₁ & w₂, are ZERO! We can’t increase or decrease the weights if their derivatives are zero. So then, how do we get our desired output in this instance if we can’t figure out how to adjust the weights? The key thing to note here is that the local gradients (∂z/∂w₁ and ∂z/∂w₂) are x₁ and x₂, both of which, in this example, happens to be zero (i.e. provide no information)

This brings us to the concept of bias.

Bias

Recall equation of a line from your high school days.

Fig 19. Equation of a Line

Here b is the bias term. Intuitively, the bias tells us that all outputs computed with x(independent variable) should have an additive bias of b. So, when x=0(no information coming from the independent variable) the output should be biased to just b.

Note that without the bias term a line can only pass through the origin(0, 0) and the only differentiating factor between lines would then be the gradient m.

Fig 20. Lines from origin

So, using this new information let’s add another node to a neural network; the bias node. (In neural network literature, every layer, except the input layer, is assumed to have a bias node, just like the linear node, so this node is also often omitted in figures.)

Fig 21. Expanded neural network with a bias node

Now let’s do a forward propagation with the same example, x₁=0, x₂=0, y=0 and let’s set bias, b=0 (initial bias is always set to zero, rather than a random number), and let the backpropagation of Loss figure out the bias.

Fig 22. Forward propagation of the first example from OR table data with a bias unit

Well, the forward propagation with a bias of “b=0” didn’t change our output at all, but let’s do the backward propagation before we make our final judgment.

As before let’s go through backpropagation in a step by step manner.

Fig 23.a. Backpropagation with bias

Fig 23.b. Backpropagation with bias

Fig 23.c. Backpropagation with bias

Hurrah! we just figured out how much to adjust the bias. Since the derivative of bias(∂L/∂b) is positive 0.125, we will need to adjust the bias by moving in the negative direction of the gradient(recall the curve of the Loss function from before). This is technically called gradient descent, as we are “descending” away from the sloping region to a flat region using the direction of the gradient. Let’s do that.

Fig 24. Calculated new bias using gradient descent

Now, that we’ve slightly adjusted the bias to b=-0.125, let’s test if we’ve done the right thing by doing a forward propagation and checking the new Loss.

Fig 25. Forward propagation with newly calculated bias

Fig 26. Loss after newly calculated bias

Now our predicted output is ŷ≈0.469(rounded to 3 decimal places), that’s a slight improvement from the previous 0.5 and Loss is down from 0.125 to around 0.109. This slight correction is something that the neural network has ‘learned’ just by comparing its predicted output with the desired output, y, and then moving in the direction opposite of the gradient. Pretty cool, right?

Now you may be wondering, this is only a small improvement from the previous result and how do we get to the minimum Loss. Two things come into play: a) how many iterations of ‘training’ we perform (each training cycle is forward propagation followed by backward propagation and updating the weights through gradient descent) b) the learning rate.

Learning rate??? What’s that? Let’s talk about it

Learning Rate

Recall, how we calculated the new bias, above, by moving in the direction opposite of the gradient(i.e. gradient descent).

Fig 27. The equation for updating bias

Notice that when we updated the bias we moved 1 step in the opposite direction of the gradient.

Fig 28. The equation for updating bias showing "step"

We could have moved 0.5, 0.9, 2, 3 or whatever fraction of steps we desired in the opposite direction of the gradient. This ‘number of steps’ is what we define as the learning rate, often denoted with α(alpha).

Fig 29. The general equation for gradient descent

Learning rate defines how quickly we reach the minimum loss. Let’s visualize below what the learning rate is doing:

Fig 30. Visualizing the effect of learning rate.

As you can see with a lower learning rate(α=0.5) our descent along the curve is slower and we take many steps to reach the minimum point. On the other hand, with a higher learning rate(α=5) we take much bigger steps and reach the minimum point much faster.

The keen-eyed may have noticed that gradient descent steps(green arrows) keep getting smaller as we get closer and closer to the minimum, why is that? Recall, that the learning rate is being multiplied by the gradient at that point along the curve; as we descend away from sloping regions to flatter regions of the u-shaped curve, near the minimum point, the gradient keeps getting smaller and smaller, thus the steps also get smaller. Therefore, changing the learning rate during training is not necessary(some variations of gradient descent start with a high learning rate to descend quickly down the slope and then reduce it gradually, this is called “annealing the learning rate”)

So what’s the takeaway? Just set the learning rate as high possible and reach the optimum loss quickly. NO. Learning rate can be a double-edged sword. Too high a learning rate and the parameters(weights/biases) don’t reach the optimum instead start to diverge away from the optimum. To small a learning rate and the parameters take too long to converge to the optimum.

Fig 31. Visualizing the effect of very low vs. very high learning rate.

Small learning rate(α=5*10⁻¹⁰) resulting is numerous steps to reach the minimum point is self-explanatory; multiply gradient with a small number(α) results in a proportionally small step.

Large learning rate(α=50) causing gradient descent to diverge may be confounding, but the answer is quite simple; note that at each step gradient descent approximates its path downward by moving in straight lines(green arrows in the figures), in short, it estimates its path downwards. When the learning rate is too high we force gradient descent to take larger steps. Larger steps tend to overestimate the path downwards and shoot past the minimum point, then to correct the bad estimate gradient descent tries to move towards the minimum point but again overshoots past the minimum due to the large learning rate. This cycle of continuous overestimates eventually cause the results to diverge(Loss after each training cycle increase, instead of decrease).

Learning rate is what’s called a hyper-parameter. Hyper-parameters are parameters that the neural network can’t essentially learn through backpropagation of gradients, they have to be hand-tuned according to the problem and its dataset, by the creator of the neural network model. (The choice of the Loss function, above, is also hyper-parameter)

In short, the goal is not the find the “perfect learning rate ” but instead a learning rate large enough so that the neural network trains successfully and efficiently without diverging.

So, far we’ve only used one example(x₁=0 and x₂=0) to adjust our weights and bias(actually, only our bias up till now🙃) and that reduced the loss on one example from our entire dataset(OR gate table). But we have more than one example to learn from and we want to reduce our loss across all of them. Ideally, in one training iteration, we would like to reduce our loss across all the training examples. This is called Batch Gradient Descent(or full batch gradient descent), as we use the entire batch of training examples per training iteration to improve our weights and biases. (Others forms are mini-batch gradient descent, where we use a subset of the data set in each iteration and stochastic gradient descent, where we only use one example per training iteration as we’ve done so far).

A training iteration where the neural network goes through all the training examples is called an Epoch. If using mini-batches than an epoch would be complete after the neural network goes through all the mini-batches, similarly for stochastic gradient descent where a batch is just one example.

Before we proceed further we need to define something called a Cost Function.

Cost Function

When we perform “batch gradient descent” we need to slightly change our Loss function to accommodate not just one example but all the examples in the batch. This adjusted Loss function is called the Cost Function.

Also, note that the curve of the Cost Function is similar to the curve of the Loss function(same U-Shape).

Instead of calculating the Loss on one example the cost function calculates average Loss across ALL the examples.

Fig 32. Cost function

Intuitively, the Cost function is expanding out the capability of the Loss function. Recall, how the Loss function was helping to minimize the vertical distance between a single data point and the predictor line(z). The Cost function is helping to minimize the vertical distance(Squared Error Loss) between multiple data points, concurrently.

Fig 33. Visualization of the effect of the Cost function

During batch gradient descent we’ll use the derivative of the Cost function, instead of the Loss function, to guide our path to minimum cost across all examples. (In some neural network literature, the Cost Function is at times also represented with the letter ‘J’.)

Let’s take a look at how the derivative equation of the Cost function differs from the plain derivative of the Loss function.

The derivative of Cost Function

Fig 34. Cost function showing it takes input vectors

Taking the derivative of this Cost function, which takes vectors as inputs and sums them, can be a bit dicey. So, let’s start out on a simple example before we generalize the derivative.

Fig 35. Calculation of Cost on a simple vectorized example

Nothing new here in the calculation of the Cost. Just as expected the Cost, in the end, is the average of the Loss, but the implementation is now vectorized (we performed vectorized subtraction followed by element-wise exponentiation, called Hadamard exponentiation). Let’s derive the partial derivatives.

Fig 36. Calculation of Jacobian on the simple example

From this, we can generalize the partial derivative equation.

Fig 37. Generalized partial derivative equation

Right now we should take a moment to note how the derivative of the Loss is different for the derivative of the Cost.

Fig 38. Comparison between the partial derivative of Loss and Cost with respect to(w.r.t) ŷ⁽ⁱ⁾

We’ll later see how this small change manifests itself in the calculation of the gradient.

Back to batch gradient descent.

For each training iteration create separate temporary variables(capital deltas, Δ) that will accumulate the gradients(small deltas, δ) for the weights and biases from each of the “m” examples in our training set, then at the end of the iteration update the weights using the average of the accumulated gradients. This is a slow method. (for those familiar time complexity analysis you may notice that as the training data set grows this becomes a polynomial-time algorithm, O(n²))

Fig 39. Batch Gradient Descent slow method

The quicker method is similar to above but instead uses vectorized computations to calculate all the gradients for all the training examples in one go, so the inner loop is removed. Vectorized computations run much quicker on computers. This is the method employed by all the popular neural network frameworks and the one we’ll follow for the rest of this blog.

For vectorized computations, we’ll make an adjustment to the “Z” node of the neural network computation graph and use the Cost function instead of the Loss function.

Fig 40. Vectorized implementation of Z node

Note that in the figure above we take dot-product between W and X which can be either an appropriate size matrix or vector. The bias, b, is still a single number(a scalar quantity) here and will be added to the output of the dot product in an element-wise fashion. The predicted output will not be just a number, but instead a vector, Ŷ, where each element is the predicted output of their respective example.

Let’s set up out data(X, W, b & Y) before doing forward and backward propagation.

Fig 41. Setup data for vectorized computations.

We are now finally ready to perform forward and backward propagation using Xₜᵣₐᵢₙ, Yₜᵣₐᵢₙ, W, and b.

(NOTE: all the results below are rounded to 3 decimal points, just for brevity)

Fig 42. Vectorized Forward Propagation on OR gate dataset

How cool is that we calculated all the forward propagation steps for all the examples in our data set in one go, just by vectorizing our computations.

We can now calculate the Cost on these output predictions. (We’ll go over the calculation in detail, to make sure there is no confusion)

Fig 43. Calculation of Cost on the OR gate data

Our Cost with our current weights, W, turns out to be 0.089. Our Goal now is to reduce this cost using backpropagation and gradient descent. As before we’ll go through backpropagation in a step by step manner

Fig 44.a. Vectorized Backward on OR gate data

Fig 44.b. Vectorized Backward on OR gate data

Fig 44.c. Vectorized Backward on OR gate data

Voila, we used a vectorized implementation of batch gradient descent to calculate all the gradients in one go.

(Those with a keen eye may be wondering how are the local gradients and the final gradients are being calculated in this last step. Don’t worry, I’ll explain the derivation of the gradients in this last step, shortly. For now, its suffice to say that the gradients defined in this last step are an optimization over the naive way of calculating ∂Cost/∂W and ∂Cost/∂b)

Let’s update the weights and bias, keeping learning rate same as the non-vectorized implementation from before i.e. α=1.

Fig 45. Calculated new Weights and Bias

Now that we have updated the weights and bias lets do a forward propagation and calculate the new Cost to check if we’ve done the right thing.

Fig 46. Vectorized Forward Propagation with updated weights and bias

Fig 47. New Cost after updated parameters

So, we reduced our Cost(Average Loss across all examples) from an initial Cost of around 0.089 to 0.084. We will need to do multiple training iterations before we can converge to a low Cost.

At this point, I would recommend that you perform backpropagation step yourself. The result of that should be (rounded to 3 decimal places): ∂Cost/∂W = [-0.044, -0.035] and ∂Cost/∂b = [-0.031].

Recall, before we trained the neural network, how we predicted the neural network can separate the two classes in Figure 9, well after about 5000 Epochs(full batch training iterations) Cost steadily decreases to about 0.0005 and we get the following decision boundary :

Fig 48. Cost curve and Decision boundary after 5000 epochs

The Cost curve is basically the value of Cost plotted after a certain number of iterations(epochs). Notice that the Cost curve flattens after about 3000 epochs this means that the weights and bias of the neural network have converged, so further training will only slightly improve our weights and bias. Why? Recall the u-shaped Loss curve, as we descend closer and closer the minimum point(flat region) the gradients become smaller and smaller thus the steps gradient descent takes are very small.

The Decision Boundary shows at the line along which the decision of the neural network changes from one output to the other. We can better visualize this by coloring the area below and above the decision boundary.

Fig 49. Decision boundary visualized after 5000 epochs

This makes it much clearer. The red shaded area is the area below the decision boundary and everything below the decision boundary has an output(ŷ) of 0. Similarly, everything above the decision boundary, shaded green, has an output of 1. In conclusion, our simple neural network has learned a decision boundary by looking at the training data and figuring out how to separate its two output classes(y=1 and y=0)🙌. Now the output neuron fires up🔥(produces 1) whenever x₁ or x₂ or both are 1.

Now would be a good time to see how the “1/m” (“m” is the total number of examples in the training dataset) in the Cost function manifested in the final calculation of the gradients.

Fig 50. Comparing the effect of derivative w.r.t Cost and Loss on parameters of the neural network

From this, the most important point to know is that the gradient used to update our weights, using the Cost function, is the average of all the gradients calculated during a training iteration; same applies to bias. You may want to confirm this yourself by checking the vectorized calculations.

Taking the average of all the gradients has some benefits. Firstly, it gives us a less noisy estimate of the gradient. Second, the resultant learning curve is smooth helping us easily determine if the neural network is learning or not. Both of these features come in very handy when training neural networks on much trickier datasets, such as those with wrongly labeled examples.

This is great and all but how did you calculate the gradients ∂Cost/∂W and ∂Cost/∂b?🤔

Neural network guides and blog posts I learned from often omitted complex details or gave very vague explanations for them. Not in this blog we’ll go over everything leaving no stone unturned.

First, we’ll tackle ∂Cost/∂b. Why did we sum the gradients?

To explain this I employ our computational graph technique on three very simple equations.

Fig 51. Computational graph of simple equations

I am particularly interested in the b node, so let’s do backpropagation on this.

Fig 52. Backpropagation on the computational graph of simple equations

Note that the b node is receiving gradients from two other nodes. So the total of the gradients flowing into node b is the sum of the two gradients flowing in.

Fig 53. Sum of gradients flowing into node b

From this example, we can generalize the following rule: Sum all the incoming gradients to a node, from all the possible paths.

Let’s visualize how this rule is used in the calculation of the bias. Our neural network can be seen as doing independent calculations for each of our examples but using shared parameters for weights and bias, during a training iteration. Below bias(b) is visualized as a shared parameter for all individual calculations our neural network performs.

Fig 54. Visualizing bias parameter being shared across a training epoch.

Following the general rule defined above, we will sum all the incoming gradients from all the possible paths to the bias node, b.

Fig 55. Visualizing all possible backpropagation paths to shared bias parameter

Since the ∂Z/∂b (local gradient at the Z node) is equal to 1, the total gradient at b is the sum of gradients from each example with respect to the Cost.

Fig 56. Proof that ∂Cost/∂b is the sum of upstream gradients

Now that we’ve got derivative of the bias figured out let’s move on to derivative of weights, and more importantly the local gradient with respect to weights.

How is the local gradient(∂Z/∂W) equal to transpose of the input training data(X_train)?

This can be answered in a similar way to the above calculation for bias, but the main complication here is the calculating the derivative of the dot product between the weight matrix(W) and the data matrix(Xₜᵣₐᵢₙ), which forms our local gradient.

Fig 57.a. Figuring out the derivative of the dot product.

This derivative of the dot product is a bit complicated as we are no longer working with scalar quantities, instead, both W and X are matrices and the result of W⋅X is also a matrix. Let’s dive a bit deeper using a simple example first and then generalizing from it.

Fig 57.b. Figuring out the derivative of the dot product.

Let’s calculate the derivative of the A with respect to W.

Fig 57.c. Figuring out the derivative of the dot product.

Let us visualize this in case of a training iteration where multiple examples are being processed at the same time. (Note that input examples are column vectors.)

Fig 58. Visualizing weights being shared across a training epoch

Just as the bias(b) was being shared across each calculation in a training iteration, weights(W) are also being shared. We can also visualize the gradient flowing back to the weights, as follows (note that the local derivative of each example w.r.t to W results in a row vector of the input example i.e. transpose of input):

Fig 59. Visualizing all possible backpropagation paths to shared weights parameter

Again, following the general rule defined above, we will sum all the incoming gradients from all the possible paths to the weights node, W.

Fig 60. Derivation of ∂Cost/∂W after visualization.

Up till now what we’ve done to calculate, ∂Cost/∂W, though is correct and serves as a good explanation however, it is not an optimized calculation. We can vectorize this calculation, too. Let’s do that next

Fig 61. Proof that ∂Cost/∂W is the dot product between the Upstream gradient and the transpose of Xₜᵣₐᵢₙ

Is there an easier way of figuring this out, without the math?

Yes! Use dimension analysis.

In our OR gate example we know that the gradient flowing into node Z is a (1 × 4) matrix, Xₜᵣₐᵢₙ is a (2 × 4) matrix and the derivative of Cost with respect to the W needs to be of the same size as W, which is (1 × 2). So, the only way to generate a (1 × 2) matrix would be to take the dot product of between Z and transpose of Xₜᵣₐᵢₙ.

Similarly, knowing that bias, b, is a simple (1 × 1) matrix and the gradient flowing into node Z is (1 × 4), using dimension analysis we can be sure that the gradient of Cost w.r.t b, also needs to be a (1 × 1) matrix. The only way we can achieve this, given the local gradient(∂Z/∂b) is just equal to 1, is by summing up the upstream gradient.

On a final note when deriving derivative expressions work on small examples and then generalize from there. For example here, while calculating the derivative of the dot product w.r.t to W, we used a single column vector as a test case and generalized from there, if we would have used the entire data matrix then the derivative would have resulted in a (4 × 1 × 2) tensor (multidimensional matrix), calculation on which can get a bit hairy.

Before concluding this section lets go over a slightly more complicated example.

Fig 62. XOR gate data

Figure 62, above, represents an XOR gate data. Looking at it note that the label, y, is equal to 1 only when one of the values x₁ or x₂ is equal to 1, not both. This makes it a particularly challenging dataset as the data is not linearly separable, i.e. there is no single straight line decision boundary that can successfully separate the two classes(y=1 and y=0) in the data. XOR used to be the bane of earlier forms of artificial neural networks.

Fig 63. Some linear decision boundaries that are wrong

Recall that our current neural network was successful only because it could figure out the straight line decision boundary that could successfully separate the two classes of the OR gate dataset. A straight line won’t cut it here. So, how do we get a neural network to figure this one out?

Well, we can do two things:

Amend the data itself, so that in addition to features x₁ and x₂ a third feature provides some additional information to help the neural network decide on a good decision boundary. This process is called feature engineering.
Change the architecture of the neural network, making it deeper.

Let's go over both and see which one is better.

Feature Engineering

Let’s look at a dataset similar looking to the XOR data that will help us in making an important realization.

Fig 64. XOR-like data in different quadrants

The data in Figure 64 is exactly like the XOR data except each data point is spread out in different quadrants. Notice that in the 1ˢᵗ and 3ʳᵈ quadrant all the values are positive and in the 2ⁿᵈ and 4ᵗʰ all the values are negative.

Fig 65. Positive and negative quadrants

Why is that? In the 1ˢᵗ and 3ʳᵈ quadrants the signs of values are being squared, while in the 2ⁿᵈ and 4ᵗʰ quadrants the values are a simple product between a negative and positive number resulting in a negative number.

Fig 66. Result of the product of features

So this gives us a pattern to work with using the product of two features. We can even see a similar pattern in the XOR data, where each quadrant can be identified in a similar way.

Fig 67. Quadrant-pattern in XOR data plot

Therefore, a good third feature, x₃, would be the product of features x₁ and x₂(i.e. x₁*x₂).

Product of features is called a feature cross and results in a new synthetic feature. Feature crosses can be either the feature itself(eg. x₁², x₁³,…), a product of two or more features(eg. x₁*x₂, x₁*x₂*x₃, …) or even a combination of both(eg. x₁²*x₂). For example, in a housing dataset where the input features are the width and length of houses in yards and label is the location of the house on the map, a better predictor for this location could be the feature cross between width and length of houses, giving us a new feature of “size of house in square yards”.

Let’s add the new synthetic feature to our training data, Xₜᵣₐᵢₙ.

Fig 68. New training data

Using this feature cross we can now successfully learn a decision boundary without changing the architecture of the neural network significantly. We only need to add an input node for x₃ and a corresponding weight(randomly set to 0.2) to the input layer.

Fig 69. Neural Network with feature cross(x₃) as input

Fig 70. Expanded neural network with feature cross(x₃) as input

Given below is the first training iteration of the neural network, you may go through the computations yourself and confirm them as they make for a good exercise. Since we are already familiar with this neural network architecture, I will not go through all the computations in a step-by-step by step manner, as before.

(All calculations below are rounded to 3 decimal places)

Fig 71. Forward Propagation in the first training iteration

Fig 72. Backpropagation in the first training iteration

Fig 73. Gradient descent update for new weights and bias, in the first training iteration

After 5000 epochs, the learning curve, and the decision boundary look as follows:

Fig 74. Learning Curve and Decision Boundary of the neural net with a feature cross

As before, to visualize better we can shade the regions where the decision of the neural network changes from one to the other.

Fig 75. Shaded Decision Boundary for better visualization

Note that feature engineering allowed us to create a decision boundary that is nonlinear. How did it do that? We just need to take a look at what function the Z node is computing.

Fig 76. Node Z is computing a polynomial after adding a feature cross

Thus, feature cross helped us to create complex non-linear decision boundary.

This is a very powerful idea!

Changing Neural Network Architecture

This is the more interesting approach as it allows us to bypass the feature engineering ourselves and lets the neural network figure out the feature crosses itself!

Let’s take a look at the following neural network:

Fig 77. Neural network with one hidden layer.

So we’ve added a bunch of new nodes in the middle of our neural network architecture from the OR gate example, keeping the input layer and the output layer the same. This column of new nodes in the middle is called a hidden layer. Why hidden layer? Because after defining it we don’t have any direct control over how the neurons in the hidden layers learn, unlike the input and output layer which we can change by changing the data; also since the hidden layers neither constitute as the output or the input of the neural network they are in essence hidden from the user.

We can have an arbitrary number of hidden layers with an arbitrary number of neurons in each layer. This structure needs to be defined by the creator of the neural network. Thus, the number of hidden layers and the number of neurons in each of the layers are also hyper-parameters. The more hidden layers we add the deeper our neural network architecture becomes and the more neurons we add in the hidden layers the wider the network architecture becomes. The depth of a neural net model is where the term “Deep learning” comes from.

The architecture in Figure 77 with one hidden layer of three sigmoid neurons, was selected after some experimentation.

Since this is a new architecture I’ll go over the computations step-by-step.

First, let’s expand out the neural network.

Fig 78. Expanded neural network with one hidden layer

Now let’s perform a forward propagation:

Fig 79.a. Forward propagation on the neural net with a hidden layer

Fig 79.b. Forward propagation on the neural net with a hidden layer

We can now calculate the Cost:

Fig 80. Cost of the neural net with one hidden layer after first forward propagation

After the calculation of Cost, we can now do our backpropagation and improve the weights and biases.

Fig 81.a. Backpropagation on the neural net with a hidden layer

Fig 81.b. Backpropagation on the neural net with a hidden layer

Fig 81.c. Backpropagation on the neural net with a hidden layer

Fig 81.d. Backpropagation on the neural net with a hidden layer

Fig 81.e. Backpropagation on the neural net with a hidden layer

Whew 😅! That was a lot, but it did a great deal to improve our understanding. Let’s perform the gradient descent update:

Fig 82. Gradient descent update for the neural net with a hidden layer

At this point, I would encourage all readers to perform one training iteration themselves. The resultant gradients should be approximately(rounded to 3 decimal places):

Fig 83. Derivatives computed during 2ⁿᵈ training iteration

After 5000 epochs the Cost steadily decreases to about 0.0009 and we get the following Learning Curve and Decision Boundary:

Fig 84. Learning Curve and Decision boundary of the neural net with one hidden layer

Let’s also visualize where the decision of the neural network changes from 0(red) to 1(green):

Fig 85. Shaded decision boundary of the neural net with one hidden layer

This shows that the neural network has in fact learned where to fire-up(output 1) and where to lay dormant(output 0).

If we add another hidden layer with maybe 2 or 3 sigmoid neurons we can get an even more complex decision boundary that may fit our data even more tightly, but let’s leave that for the coding section.

Before we conclude this section I want to answer some remaining questions:

1- So, which one is better Feature Engineering or a Deep Neural Network?

Well, the answer depends on many factors. Generally, if we have a lot of training data we can just use a deep neural net to achieve acceptable accuracy, but if data is limited we may need to perform some feature engineering to extract more performance out of our neural network. As you saw in the feature engineering example above, to make good feature crosses one needs to have intimate knowledge of the dataset they are working with.

Feature engineering along with a deep neural network is a powerful combination.

2- How to count the number of layers in a Neural Network?

By convention, we don’t count layers without tunable weights and bias. Therefore, though the input layer is a separate “layer” we don’t count it when specifying the depth of a neural network.

So, our last example was a “2 layer neural network” (one hidden + output layer) and all the examples before it just a “1 layer neural network” (output layer, only).

3- Why use Activation Functions?

Activation functions are nonlinear functions and add nonlinearity to the neurons. The feature crosses are a result of stacking the activation functions in hidden layers. The combination of a bunch of activation functions thus results in a complex non-linear decision boundary. In this blog, we used the sigmoid/logistic activation function, but there are many other types of activation functions(ReLU being a popular choice for hidden layers) each providing a certain benefit. The choice of the activation function is also a hyper-parameter when creating neural networks.

Without activations functions to add nonlinearity, no matter how many linear functions we stack up the result of them will still be linear. Consider the following:

Fig 86. Showing that stacking linear layers/functions results in a linear layer/function

You may use any nonlinear function as an activation function. Some researchers have used even cos, sin, and tan functions.

4- Why Random Initialization of Weights?

This question is much easier to answer now. Note that if we had set all the weights in a layer to the same value than the gradient that passes through each node would be the same. In short, all the nodes in the layer would learn the same feature about the data. Setting the weights to random values helps in breaking the symmetry of weights so that each node in a layer has the opportunity to learn a unique aspect of the training data

There are many ways to set weights randomly in neural networks. For small neural networks, it is ok to set the weights to small random values. For larger networks, we tend to use "Xavier" or "He" initialization methods(will be in the coding section). Both these methods still set weights to random values but control their variance. For now, its suffice to say use these methods when the network does not seem to converge and the Cost becomes static or reduces very slowly when using the "plain" method of setting weights to small random values. Weight initialization is an active research area and will be a topic for a future "Nothing but Numpy" blog.

Biases can be randomly initialized, too. But in practice, it does not seem to have much of an effect on the performance of a neural network. Perhaps this is because the number of bias terms in a neural network is much fewer than the weights.

The type of neural network we created here is called a "fully-connected feedforward network" or simply a "feedforward network".

This concludes Part Ⅰ.

Part Ⅱ: Coding a Modular Neural Network

The implementation in this part follows OOP principals.

Let's first see the Linear Layer class. The constructor takes as arguments: the shape of the data coming in(input_shape), the number of neurons the layer outputs(n_out) and what type of random weight initialization need to be performed(ini_type="plain", default is "plain" which is just small random gaussian numbers).

The initialize_parameters is a helper function used to define weights and bias. We'll look at it separately, later.

Linear Layer implements the following functions:

forward(A_prev): This function allows the linear layer to take in activations from the previous layer(the input data can be seen as activations from the input layer) and performs the linear operation on them.
backward(upstream_grad): This function computes the derivative of Cost w.r.t weights, bias, and activations from the previous layer(dW, db &dA_prev, respectively)
update_params(learning_rate=0.1): This function performs the gradient descent update on weights and bias using the derivatives computed in the backward function. The default learning rate(α) is 0.1

Fig 87. Linear Layer Class

Now let's see the Sigmoid Layer class, its constructor takes in as an argument the shape of data coming in(input_shape) from a Linear Layer preceding it.

Sigmoid Layer implements the following functions:

forward(Z): This function allows the sigmoid layer to take in the linear computations(Z) from the previous layer and perform the sigmoid activation on them.
backward(upstream_grad): This function computes the derivative of Cost w.r.t Z(dZ).

Fig 88. Sigmoid Activation Layer class

The initialize_parameters function is used only in the Linear Layer to set weights and biases. Using the size of the input(n_in) and output(n_out) it defines the shape the weight matrix and bias vector need to be in. This helper function then returns both the weight(W) and bias(b) in a python dictionary to the respective Linear Layer.

Fig 89. Helper function to set weights and bias

Finally, the Cost function compute_cost(Y, Y_hat) takes as argument the activations from the last layer(Y_hat) and the true labels(Y) and computes and returns the Squared Error Cost(cost) and its derivative(dY_hat).

Fig 90. Function to compute Squared Error Cost and derivative

At this point, you should open the 2_layer_toy_network_XOR Jupyter notebook from this repository in a separate window and go over this blog and the notebook side-by-side.

Now we are ready to create our neural network. Let's use the architecture defined in Figure 77 for XOR data.

Fig 91. Defining the layers and training parameters

Now we can start the main training loop:

Fig 92. The training loop

Running the loop in the notebook we see that the Cost decreases to about 0.0009 after 4900 epochs

...
Cost at epoch#4600: 0.001018305488651183
Cost at epoch#4700: 0.000983783942124411
Cost at epoch#4800: 0.0009514180100050973
Cost at epoch#4900: 0.0009210166430616655

The Learning curve and Decision Boundaries look as follows:

Fig 93. The Learning Curve, Decision Boundary, and Shaded Decision Boundary.

The predictions our trained neural network produces are accurate.

The predicted outputs:
 [[ 0.  1.  1.  0.]]
The accuracy of the model is: 100.0%

Make sure to check out the other notebooks in the repository. We'll be building upon the things we learned in this blog in future Nothing but NumPy blogs, therefore, it would behoove you to create the layer classes from memory as an exercise and try recreating the OR gate example from Part Ⅰ.

This concludes the blog🙌🎉. I hope you enjoyed.
For any questions feel free to reach out to me on twitter @RafayAK

This blog would not have been possible without following resources and people:

Andrej Karpathy's(@karpathy) Stanford course
Christopher Olah's(@ch402) blogs
Andrew Trask's(@iamtrask) blogs
Andrew Ng(@AndrewYNg) and his Coursera courses on deep learning and machine learning
Terence Parr(@the_antlr_guy) and Jeremy Howard (@jeremyphoward)(https://explained.ai/matrix-calculus/index.html)
Ian Goodfellow(@goodfellow_ian) and his amazing book
Finally, Hassan-uz-Zaman(@OKidAmnesiac) and Hassan Tauqeer for invaluable feedback.

DEV Community: Rafay Khan