Part 3 of my "revisiting my AI/ML notes" series. Last post covered the ANN structure — input, hidden, and output layers. This one goes into what actually happens inside a neuron: the math behind forward propagation.
The core idea
Every neuron in a network does exactly three things:
- Multiplies each input by a corresponding weight
- Sums all of those up and adds a bias
- Passes that sum through an activation function to produce the final output
In formula form:
- Weighted sum:
y = x1.w1 + x2.w2 + x3.w3 + bias - Activation:
z = sigmoid(y) = 1 / (1 + e^-y)
What weights and bias actually mean
- Weights control how much influence each input has on the final decision. A higher weight means that input matters more. During training, these are exactly the numbers backpropagation adjusts.
- Bias is a constant that shifts the output, independent of the inputs. It lets the neuron activate even when all inputs are zero, and generally gives the model more flexibility to fit the data.
Why sigmoid specifically
The raw weighted sum y can be any real number, positive or negative, large or small. That's not directly useful for a decision like "is this a cat or a dog" — we want something bounded and interpretable. Sigmoid squashes any input into a value between 0 and 1, which is why it shows up constantly in classification-style problems: the output can be read almost like a probability.
Walking through a real example
Here are the numbers I worked through by hand:
x1 = 2, w1 = 0.5
x2 = 4, w2 = 1.0
x3 = 1, w3 = 0.7
bias = 0.5
Step 1 — weighted sum:
x1w1 = 2 * 0.5 = 1.0
x2w2 = 4 * 1.0 = 4.0
x3*w3 = 1 * 0.7 = 0.7
y = 1.0 + 4.0 + 0.7 + 0.5 = 6.2
Step 2 — activation:
z = sigmoid(6.2) = 1 / (1 + e^-6.2) ~= 0.998
A correction worth calling out: in my original notes I'd written 1 x 0.7 = -0.7, which doesn't check out — there's no negative sign anywhere in that multiplication, so it should be +0.7. That one sign flip changed my total from 6.2 down to 4.8, and the sigmoid output from ~0.998 down to ~0.991. The values are close enough that it's an easy slip to miss, but it's a good reminder to actually re-derive worked examples rather than trusting old notes at face value — which, honestly, is the whole point of this series.
Why this matters beyond one neuron
This single-neuron computation is the atomic unit of every neural network, no matter how deep or complex. A full forward pass is just this same weighted-sum-then-activate process, repeated across every neuron in every layer, until the input reaches the output layer as a final prediction. Understanding this one calculation cold makes everything downstream — backpropagation, different activation functions, why certain networks struggle with vanishing gradients — much easier to reason about.
Next up: how the network learns from its mistakes — an introduction to backpropagation.
Original Notes:
Top comments (0)