Neural Network Training - Simply Explained with a Mental Model

A neural network learns by repeatedly making predictions, measuring how wrong it is, and nudging its internal weights to do better. This cycle - forward pass, loss, backpropagation, gradient descent - is the engine behind every modern AI system.

Diagram

Concepts

• Neural Network Training [Concept] The process of adjusting a network's weights by repeatedly showing it examples until it learns to make accurate predictions
  • Network Structure [Concept] Layers of neurons connected by weights - input, hidden, and output layers
  • Input Layer [Concept] Raw data fed into the network - pixels, words, numbers
  • Hidden Layers [Concept] Where patterns are learned - each neuron applies a weight and activation function
  • Output Layer [Concept] The final prediction - a class, a number, or the next token
  • Training Loop [Process] The 4-step cycle repeated millions of times to tune the network's weights
  • 1. Forward Pass [Process] Feed input through each layer to produce a prediction
  • 2. Calculate Loss [Process] Measure how wrong the prediction is compared to the correct answer
  • 3. Backpropagation [Process] Work backwards through the network to find which weights caused the error
  • 4. Gradient Descent [Process] Nudge each weight slightly in the direction that reduces the loss: weight = weight - (lr × gradient)
  • Epoch [Concept] One full pass through the entire training dataset
  • Weights [Concept] Tunable numbers on each connection - the memory of the network
  • Learning Rate [Concept] Controls how large each weight adjustment step is - too high diverges, too low crawls

Relationships

• Neural Network Training → built from → Network Structure
• Neural Network Training → trained via → Training Loop
• Neural Network Training → parameterized by → Weights
• Network Structure → starts with → Input Layer
• Network Structure → learns in → Hidden Layers
• Network Structure → ends with → Output Layer
• 1. Forward Pass → produces prediction for → 2. Calculate Loss
• 2. Calculate Loss → triggers → 3. Backpropagation
• 3. Backpropagation → computes gradients for → 4. Gradient Descent
• 4. Gradient Descent → updates → Weights
• Weights → used in next → 1. Forward Pass
• Learning Rate → scales → 4. Gradient Descent
• Epoch → counts iterations of → Training Loop

Real-World Analogies

Training Loop ↔ Learning to throw darts

You throw (forward pass), see how far off you are (loss), figure out what went wrong - too much wrist, wrong angle (backprop), then adjust slightly next time (gradient descent). After thousands of throws you hit the bullseye consistently.

Backpropagation ↔ A manager tracing a bug back through a team

When the final output is wrong, backprop works backwards layer by layer - like a manager asking 'who made this decision?' at each step - assigning blame proportionally to each weight's contribution to the error.

Learning Rate ↔ Adjusting a shower temperature

Too big a turn (high learning rate) and you overshoot from freezing to scalding. Too small (low learning rate) and it takes forever to warm up. The right learning rate finds the comfortable temperature efficiently.