If you've ever looked at a neural network diagram and every wondered what is actually happening in these networks, this post is just for you. I pulled this post together after researching the properties of neural networks, and writing out what is finally there to click. No analogies that lead to no explanations, just the math behind neural networks, one piece at a time.
The Simplest Possible Net——A Perceptron
Before neural networks got keep, they started shallow. A perceptron is a neural net with zero hidden layers, invented back in 1943. Perceptions work when data is linearly separable, meaning you could draw a line and cleanly split the classes on either side of it.
Let's take a simple example with an equation: D=y-2x-3. Everything where D>=0 is Class 1 and everything where D<0 is Class 2. This is basically how a model works: a single linear boundary that decides which side the example falls on.
However, real data is rarely this cooperative. The moment your classes tangle in a way no straight line can separate, you will need more than one layer. This is really where the deep learning begins.
What A Neuron is Actually Computing
Zoom into any single neuron in a network, and this is what entire computation occuring inside of it:
S = b∑ w_i * x_i
x_i are the inputs coming in, such as the pixel values and whatever your data is.
w_i are the weights (how much the network currently believes each input matters.
b is the bias that acts like a nudge that shifts the output independent of the inputs.
S is the logit, which is a unbounded value. The S value either shoots in the negative or positive direction indefinetely.
Stack raw logits like this across layers with nothing else added, and the whole network has a mathematical collapse into a linear function. Depth alone buys you nothing. You need a twist.
The Non-Linear Twist That Makes Depth Matter
That very twist is the activiation function. This is applied to every logit before it moves onto the next layer. One of them is sigmoid:
ø(S) = 1 / (1+e)^-s
Just as the equation says, the ø(s) pushes the S value closest to 1 as the ø(s) value is positive. If ø(s) is negative, then the S value moves towards 0.
This nonlinearity is the backbone of how depth works. With nonlinearity, each added layer can fold the decision boundary into complex shapes, which is exactly what is required to separate tangled data.
Stacking Neurons into Layers
A layer is just a bank of neurons, each computing its own weighted sum form the same inputs, and each with its own weights. The formula goes like this:
z_i = ϕ (j∑ x_j * w_ij)
The middle layers are called the hidden layers. They are called hidden layers because the values are not directly obseved; only the input and the output (final value) are observed. The early layers tend to pick up the simple patterns while the deeper layers form abstract connection with the simple patterns made by the early layers.
The final output layer turns those hidden features into a prediction:
y_i = ϕ(j∑ z_j * w_ij)
For a K-way classification, the K output neurons are used. These neurons are used as one confidence score per class.
Softmax: Turning Raw Scores into Real Probabilities
Say that the output layer displays three logits: (0.01, -3.8, 4.2). These numbers are all currently unbounded and do not hold any significant meaning at the moment. Softmax fixes this by taking any of those three logits and converting them into probabilities between 0 and 1 that sum up to exactly 1.0. The formula for softmax is:
p_j = (e^z_j) / (∑_k * e^zk)
In the example above, the 4.2 value would come out closer to a 90% or higher confidence score, while -3.8 will approach closer to 0, a lower confidence score.
Loss Functions: Scoring the Damage
One the network makes a prediction, there must be a way to measure how inaccurate it was. For the numerical predictions, it is normally considered the Mean Squared Error. For classification, it is almost always Cross-Entropy. This is how the formula goes:
CE = -n∑i=1 [y_i * log(p(yi)) + (1-yi) * log(1-p(yi))]
The intuition is quite simple: if the true label is 1 and the model confidently predicts closer to the O, the log term blows up and there are severe penalties. However, if the model was correct, the penalty shrinks towards zero. Cross-entropy rewards confident correctness and punishes the confident mistakes. This is exactly what you want out of a loss function.
What the Network Actually Learns: Backdrops and Gradient Descent
Here's the loop that turns a "useless" network into a "useful" network:
- Feed a batch of training samples through the network.
- Compute the predictions as the output layer.
- Measure losses and how inaccurate the predictions were.
- Backpropagate the error backwards through every layer so you can calculate the weight contributed to the error.
- Nudge every weight slightly in the direction that reduces the layer.
That fifth step is called the gradient descent:
(w_ij)^new = w_ij - α * (∂E) / (∂w_ij)
α holds the learning rate.
Just picture the loss as a landscape. Training is descending that landscape, one gradient-calculated step at a time, trying to reach a low point. In a network with millions of billions of weights, that landscape has many dimensions with multiple valleys. This is part of why training is not guaranteed to find the global best solution, just a good one.
One pass through the entire training is called an epoch. Networks are trained across many epochs because each pass leaves weights in a sligtly different place. This is why seeing the same data again isn't wasted effort.
Watching It Happen Through a PyTorch Example
Due to restrictions, the code commensts will be in []. Here's a real and complete neural network trained on MNIST in PyTorch.
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
[Load Data]
convertToTensor = transforms.ToTensor()
train_data = datasets.MNIST(root='./data', train=True, download=True, transform=convertToTensor)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
[Define the network: 784 input pixels -> 128 hidden neurons -> 10 output classes]
model = nn.Sequential(
nn.Flatten(),
nn.Linear(28*28, 128),
nn.ReLU(),
nn.Linear(128, 10)
)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)
[Train]
for epoch in range(5):
for images, labels in train_loader:
optimizer.zero_grad()
predictedLabels = model(images)
loss = criterion(predictedLabels, labels)
loss.backward()
optimizer.step()
print(f"Epoch {epoch}, Loss: {loss.item():.4f}")
Every concept from this post is sitting amongst these lines of code: nn.linear is the weighted sum, nn.reLU is the activation function, criterion computers cross-entropy loss, loss.backward() runs backpropagation, and optimizer.step() is the gradient descent update. This is basically just theory executed.
The One-Sentence Summary
A neural network is a stack of weighted sums broken up by non-linear activations, trained by repeatedly measuring how wrong its predictions are and nudging every weight a tiny bit in the direction that makes it less wrong — and that exact loop, scaled up, is what's running behind every AI tool you've ever used.
Top comments (0)