Our inputs are 2 and 1, and the correct answer is 3. But the network initially predicts 0.7.
Why? A neural network doesn’t automatically know the rules of addition. Its prediction depends on its current weights and biases.
- The forward pass
A forward pass means sending inputs through the network to produce a prediction.
Each neuron multiplies its inputs by weights, adds a bias, and applies an activation function.
For example, a hidden neuron might calculate:
(2 × 0.5) + (1 × -1) + 0.5 = 0.5
If we use ReLU as the activation function, positive values stay unchanged and negative values become zero.
In our example, two hidden neurons each produce 0.5. The output neuron combines them:
Prediction = (0.5 × 0.8) + (0.5 × 0.4) + 0.1
= 0.7
The forward pass gives us a prediction. It does not update the weights.
- Calculating the loss
Now we compare the prediction with the correct answer:
Correct answer = 3
Prediction = 0.7
Error = 0.7 − 3 = -2.3
Using squared-error loss:
Loss = (Prediction − Correct answer)²
= (0.7 − 3)²
= 5.29
Loss is a number that measures how wrong the prediction is. For this loss function, a smaller value means a closer prediction.
- Learning from the mistake
Calculating loss alone doesn’t teach the network.
Next, backpropagation calculates gradients: how changes to each weight and bias would affect the loss. An optimizer uses those gradients to update the parameters.
Training repeats this process across many examples:
Make a prediction
Calculate the loss
Calculate gradients
Update weights and biases
Repeat
One example isn’t enough to show that the network has learned addition. We also need to test it on input pairs it hasn’t trained on.
The key distinction
Forward pass: What does the network predict?
Loss calculation: How wrong is that prediction?
Training: How should its weights and biases change?
That’s the foundation of neural network training.
For more such posts, visit EngineeringDepth.com.
Follow Planet AI & Tech for simple explanations of AI and technology.
Top comments (0)