DEV Community

Cover image for Pytorch for Neural Networks Part 9: Taking Steps Toward Better Predictions
Rijul Rajesh
Rijul Rajesh

Posted on

Pytorch for Neural Networks Part 9: Taking Steps Toward Better Predictions

#ai #machinelearning

In the previous article, we went through the optimization loop and passed all three training inputs through the model.

In this article, we will explore the additional steps we need to take when the total loss is not yet small enough.

Taking a Step Toward a Better Bias

If total_loss is still too large, we need to take a small step toward a better value for final_bias.

We do this using:

optimizer.step()
Enter fullscreen mode Exit fullscreen mode

In the previous article, we saw that:

loss.backward()
Enter fullscreen mode Exit fullscreen mode

calculates derivatives and stores them inside the model parameters.

The optimizer can then use these stored derivatives to determine the correct direction to update the parameter.

Clearing Old Derivatives

After updating the model, we need to clear the stored derivatives.

We do this using:

optimizer.zero_grad()
Enter fullscreen mode Exit fullscreen mode

Here is the updated training loop:

for epoch in range(100):
    total_loss = 0

    for iteration in range(len(inputs)):
        input_i = inputs[iteration]
        label_i = labels[iteration]

        output_i = model(input_i)

        loss = (output_i - label_i) ** 2

        loss.backward()

        total_loss += float(loss)

    if total_loss < 0.0001:
        print("Num steps: " + str(epoch))
        break

    optimizer.step()
    optimizer.zero_grad()

    print(
        "Step: "
        + str(epoch)
        + " Final Bias: "
        + str(model.final_bias.data)
        + "\n"
    )
Enter fullscreen mode Exit fullscreen mode

Why Do We Need zero_grad()?

We clear the derivatives because of how PyTorch works.

Earlier, we saw that:

loss.backward()
Enter fullscreen mode Exit fullscreen mode

accumulates derivatives.

This means that if we do not clear them, then the next time we enter the loop, the new derivatives will be added to the old derivatives from the previous step.

That would lead to incorrect updates.

So after every optimization step, we reset the stored derivatives using:

optimizer.zero_grad()
Enter fullscreen mode Exit fullscreen mode

Tracking the Final Bias

At the end of each loop, we print:

  • the current epoch number
  • the current value of final_bias

This allows us to track how final_bias changes during training.

The process continues until:

  • the total loss becomes very small, or
  • we finish all 100 epochs.

Printing the Final Optimized Bias

Once training is complete, we can print the final optimized value for final_bias.

print(
    "Final bias, after optimization: "
    + str(model.final_bias.data)
)
Enter fullscreen mode Exit fullscreen mode

In the next article, we will see how this training process actually runs and how the value of final_bias changes over time, to get our final result from the script.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Top comments (0)