Forward and Backward Propagation In Neural Networks

If you’re new to neural networks, two key concepts you’ll hear are forward propagation and backward propagation. Don’t worry — they sound complicated, but they’re really just the way information flows in and out of the network. Let’s break them down step by step.

---

🌱 What is Forward Propagation?

Forward propagation is how a neural network makes predictions. Think of it like making coffee in a machine:

1. You put in inputs (water, coffee powder).
2. The machine applies weights and biases (how strong the coffee should be, how much water).
3. The machine applies a function (brewing).
4. You get an output (a cup of coffee).

In neural networks, inputs are numbers, weights and biases are adjustable values, and the function is called an activation function.

---

☕ Forward Propagation in a Single Layer

Imagine a single neuron:

• Inputs: (x_1, x_2, x_3)
• Weights: (w_1, w_2, w_3)
• Bias: (b)

The neuron calculates:

Then applies an activation function:

👉 This is forward propagation in a single layer: inputs → weighted sum → activation → output.

---

Simple Python Example

import numpy as np

# Inputs
x = np.array([2, 3, 5])

# Weights
w = np.array([0.4, 0.3, 0.2])

# Bias
b = 0.5

# Weighted sum
z = np.dot(x, w) + b

# Activation (ReLU)
a = np.maximum(0, z)

print("Output after forward propagation:", a)

---

🔄 General Implementation of Forward Propagation

When we have multiple layers, forward propagation means repeating this process layer by layer:

1. Layer 1: Take inputs, multiply by weights, add bias, apply activation.
2. Layer 2: Take outputs from Layer 1 as inputs, repeat the process.
3. Final Layer: Produce the final prediction.

General formula for each layer (l):

---

🔁 What is Backward Propagation?

Forward propagation makes predictions. Backward propagation is how the network learns from mistakes.

Think of it like tasting the coffee you just brewed:

1. You sip the coffee (prediction).
2. You compare it to what the customer wanted (actual label).
3. If it’s too strong or too weak, you adjust the recipe (weights and biases).
4. Next time, the coffee tastes closer to what the customer wants.

In neural networks, backward propagation uses gradients (mathematical slopes) to adjust weights and biases so the predictions get better over time.

---

How Backward Propagation Works

1. Calculate error: Compare predicted output with actual output using a loss function.
2. Find gradients: Measure how much each weight contributed to the error.
3. Update weights: Adjust weights slightly in the opposite direction of the error.
4. Repeat: Do this for many epochs until the network learns.

---

Simple Python Illustration

# Imagine predicted vs actual
predicted = 0.7
actual = 1.0

# Error (loss)
error = actual - predicted

# Learning rate
lr = 0.1

# Weight before update
w = 0.5

# Backward propagation: update weight
w = w + lr * error

print("Updated weight:", w)

Here, the weight is nudged in the right direction to reduce error next time.

---

📊 Text‑Based Diagram

Forward Propagation:
Inputs → Weighted Sum → Activation → Output → Prediction

Backward Propagation:
Prediction → Compare with Actual → Calculate Error → Adjust Weights → Better Prediction Next Time

---

🎯 Wrapping Up

• Forward propagation = how the network makes predictions.
• Backward propagation = how the network learns by adjusting weights and biases.
• Together, they form the learning cycle: predict → compare → adjust → improve.

---