DEV Community

Cover image for Pytorch for Neural Networks Part 2: Initializing Weights and Biases
Rijul Rajesh
Rijul Rajesh

Posted on

Pytorch for Neural Networks Part 2: Initializing Weights and Biases

#machinelearning #ai

In the previous article, we got started with expressing a neural network in the form of Python code.

In this article, we will continue building on that.

This is the neural network that we will recreate using code.

You can see the weights and biases shown in the diagram above. Let us now add them to our code.

We start with the basic neural network class:

class MyBasicNN(nn.Module):
    def __init__(self):
        super().__init__()
Enter fullscreen mode Exit fullscreen mode

Our first weight has the value 1.70.

We can represent it like this:

class MyBasicNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.w00 = nn.Parameter(torch.tensor(1.7), requires_grad=False)
Enter fullscreen mode Exit fullscreen mode

Here, we initialize a new variable called w00 and make it a neural network parameter.

When we define a weight as a parameter, PyTorch treats it as part of the neural network and gives us the option to optimize it during training.

Since this value is stored as a tensor, the neural network can take advantage of features such as:

  • automatic differentiation
  • accelerated mathematical operations

If you are unfamiliar with tensors, check out my earlier article on tensors.

Since we do not need to optimize this weight, we set:

requires_grad=False
Enter fullscreen mode Exit fullscreen mode

requires_grad is short for requires gradient.

By setting it to False, we tell PyTorch that this parameter should not be updated during optimization.

In a similar way, we can define the rest of the weights and biases.

class MyBasicNN(nn.Module):
    def __init__(self):
        super().__init__()

        self.w00 = nn.Parameter(torch.tensor(1.7), requires_grad=False)
        self.b00 = nn.Parameter(torch.tensor(-0.85), requires_grad=False)
        self.w01 = nn.Parameter(torch.tensor(-40.8), requires_grad=False)

        self.w10 = nn.Parameter(torch.tensor(12.6), requires_grad=False)
        self.b10 = nn.Parameter(torch.tensor(0.0), requires_grad=False)
        self.w11 = nn.Parameter(torch.tensor(2.7), requires_grad=False)

        self.final_bias = nn.Parameter(torch.tensor(-16.), requires_grad=False)
Enter fullscreen mode Exit fullscreen mode

Now that we have initialized the weights and biases, the next step is to create a forward pass through the neural network.

The forward pass defines how the input moves through the network using these weights and biases.

To handle this logic, we need to define another method called forward().

We will cover that in the next article.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Top comments (0)