DEV Community: Aahan-Chauhan

Neural Networks: Weights, Activation, and Backpropagation

Aahan-Chauhan — Sun, 06 Sep 2026 22:51:56 +0000

If you've ever looked at a neural network diagram and every wondered what is actually happening in these networks, this post is just for you. I pulled this post together after researching the properties of neural networks, and writing out what is finally there to click. No analogies that lead to no explanations, just the math behind neural networks, one piece at a time.

The Simplest Possible Net——A Perceptron

Before neural networks got keep, they started shallow. A perceptron is a neural net with zero hidden layers, invented back in 1943. Perceptions work when data is linearly separable, meaning you could draw a line and cleanly split the classes on either side of it.

Let's take a simple example with an equation: D=y-2x-3. Everything where D>=0 is Class 1 and everything where D<0 is Class 2. This is basically how a model works: a single linear boundary that decides which side the example falls on.

However, real data is rarely this cooperative. The moment your classes tangle in a way no straight line can separate, you will need more than one layer. This is really where the deep learning begins.

What A Neuron is Actually Computing

Zoom into any single neuron in a network, and this is what entire computation occuring inside of it:

S = b∑ w_i * x_i

x_i are the inputs coming in, such as the pixel values and whatever your data is.
w_i are the weights (how much the network currently believes each input matters.
b is the bias that acts like a nudge that shifts the output independent of the inputs.
S is the logit, which is a unbounded value. The S value either shoots in the negative or positive direction indefinetely.

Stack raw logits like this across layers with nothing else added, and the whole network has a mathematical collapse into a linear function. Depth alone buys you nothing. You need a twist.

The Non-Linear Twist That Makes Depth Matter

That very twist is the activiation function. This is applied to every logit before it moves onto the next layer. One of them is sigmoid:

ø(S) = 1 / (1+e)^-s

Just as the equation says, the ø(s) pushes the S value closest to 1 as the ø(s) value is positive. If ø(s) is negative, then the S value moves towards 0.

This nonlinearity is the backbone of how depth works. With nonlinearity, each added layer can fold the decision boundary into complex shapes, which is exactly what is required to separate tangled data.

Stacking Neurons into Layers

A layer is just a bank of neurons, each computing its own weighted sum form the same inputs, and each with its own weights. The formula goes like this:

z_i = ϕ (j∑ x_j * w_ij)

The middle layers are called the hidden layers. They are called hidden layers because the values are not directly obseved; only the input and the output (final value) are observed. The early layers tend to pick up the simple patterns while the deeper layers form abstract connection with the simple patterns made by the early layers.

The final output layer turns those hidden features into a prediction:

y_i = ϕ(j∑ z_j * w_ij)

For a K-way classification, the K output neurons are used. These neurons are used as one confidence score per class.

Softmax: Turning Raw Scores into Real Probabilities

Say that the output layer displays three logits: (0.01, -3.8, 4.2). These numbers are all currently unbounded and do not hold any significant meaning at the moment. Softmax fixes this by taking any of those three logits and converting them into probabilities between 0 and 1 that sum up to exactly 1.0. The formula for softmax is:

p_j = (e^z_j) / (∑_k * e^zk)

In the example above, the 4.2 value would come out closer to a 90% or higher confidence score, while -3.8 will approach closer to 0, a lower confidence score.

Loss Functions: Scoring the Damage

One the network makes a prediction, there must be a way to measure how inaccurate it was. For the numerical predictions, it is normally considered the Mean Squared Error. For classification, it is almost always Cross-Entropy. This is how the formula goes:

CE = -n∑i=1 [y_i * log(p(yi)) + (1-yi) * log(1-p(yi))]

The intuition is quite simple: if the true label is 1 and the model confidently predicts closer to the O, the log term blows up and there are severe penalties. However, if the model was correct, the penalty shrinks towards zero. Cross-entropy rewards confident correctness and punishes the confident mistakes. This is exactly what you want out of a loss function.

What the Network Actually Learns: Backdrops and Gradient Descent

Here's the loop that turns a "useless" network into a "useful" network:

Feed a batch of training samples through the network.
Compute the predictions as the output layer.
Measure losses and how inaccurate the predictions were.
Backpropagate the error backwards through every layer so you can calculate the weight contributed to the error.
Nudge every weight slightly in the direction that reduces the layer.

That fifth step is called the gradient descent:

(w_ij)^new = w_ij - α * (∂E) / (∂w_ij)

α holds the learning rate.

Just picture the loss as a landscape. Training is descending that landscape, one gradient-calculated step at a time, trying to reach a low point. In a network with millions of billions of weights, that landscape has many dimensions with multiple valleys. This is part of why training is not guaranteed to find the global best solution, just a good one.

One pass through the entire training is called an epoch. Networks are trained across many epochs because each pass leaves weights in a sligtly different place. This is why seeing the same data again isn't wasted effort.

Watching It Happen Through a PyTorch Example

Due to restrictions, the code commensts will be in []. Here's a real and complete neural network trained on MNIST in PyTorch.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

[Load Data]
convertToTensor = transforms.ToTensor()
train_data = datasets.MNIST(root='./data', train=True, download=True, transform=convertToTensor)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)

[Define the network: 784 input pixels -> 128 hidden neurons -> 10 output classes]
model = nn.Sequential(
nn.Flatten(),
nn.Linear(28*28, 128),
nn.ReLU(),
nn.Linear(128, 10)
)

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

[Train]
for epoch in range(5):
for images, labels in train_loader:
optimizer.zero_grad()
predictedLabels = model(images)
loss = criterion(predictedLabels, labels)
loss.backward()
optimizer.step()
print(f"Epoch {epoch}, Loss: {loss.item():.4f}")

Every concept from this post is sitting amongst these lines of code: nn.linear is the weighted sum, nn.reLU is the activation function, criterion computers cross-entropy loss, loss.backward() runs backpropagation, and optimizer.step() is the gradient descent update. This is basically just theory executed.

The One-Sentence Summary

A neural network is a stack of weighted sums broken up by non-linear activations, trained by repeatedly measuring how wrong its predictions are and nudging every weight a tiny bit in the direction that makes it less wrong — and that exact loop, scaled up, is what's running behind every AI tool you've ever used.

Entropy and Cross-Entropy, Explained

Aahan-Chauhan — Tue, 25 Aug 2026 23:43:19 +0000

If you've spent any time around machine learning the familiarizing yourself with the concept, you have seen the term "entropy" and "cross entropy" show up everywhere. These two distinct terms often show up in topics regarding: classification models, neural networks, and understanding language models. But if you are like me a few weeks ago, you would not be able to write down the formulas without feeling the meaning of these terms.

This post is my attempt to build this intuition of yours from the ground up. This blog will cover topics ranging from entropy to cross-entropy. As you read this post, these topics will finally connect to why they are the loss functions of choice in machine learning.

PART 1: What Entropy Actually Measures

Forget about the neural networks for a second. Entropy comes from the information theory and, at its core, it answers one main question: How surprised should you be by the outcome of a random event?

Now let's image two coins:

Coin A is fair with 50% heads and 50% tails.
Coin B is rigged with 99% head and 1% tails.

If I flip Coin B and tell you the result, you will almost never be surprised if the outcome is heads. However, if I flip Coin A, you will have no idea what is coming. This is because every flip of Coin A is a genuine mystery that is only uncovered once the coin is flipped.

Entropy is a formal way of measuring this same "amount of surprise." A fair coin like Coin A has high entropy (maximum uncertainity) while a rigged coin like Coin B has low entropy (easily predictable).

Mathematically, the entropy for a probabilitiy distribution is: H(P)=−x∑p(x)log(p(x)).

Now, lets unpack this formula and understand why it makes sense.

log(p(x)) basically represents the "surprise." Rare events—small p(x)—have a more negative log value, meaning higher surprise rates. Common events—large p(x)—have log values that are closer to 0, meaning lower surprise rates.
The negative sign flips to a positive sign, knows as a surprise score, since the probabilities are between 0 and 1 with the log being a negative.
Multiplying by the p(x) value and summing it up gives you the expected value of the surprise. The expected value of surprise is the average surprise you'd experience sampling from this distribution over and over.

So basically: Entropy is the average amount of "surprise" backed into a distribution.

Part 2: From Entropy to Cross-Entropy

Entropy already assumes that you know the true distribution of the outcomes. However, in machine learning, we rarely know the true distribution. We are trying to predict the distribution.

Here is the mathematical definition of cross-entropy: H(P,Q)=−x∑p(x)log(q(x)).

This is where cross-entropy plays a major role. It measures the surprise you experience when:

The true outcomes come from distribution P (considered the actual label of an image).
But you are using a predicted distribution Q (your model's guess) to measure that particular surpise.

Notice the subtle but crucial difference from entropy: we still weight by the true probabilitiy p(x), but we take the log of the predicted probability q(x).

Here is the intuition: if your model's predicted distribution Q is close to the true distribution P, then the cross-entropy will be low while the actual entropy is close to P. However, if the model is confidently wrong, the log term completely explodes and the cross-entropy shoots up.

This is exactly something what we want in a loss function:

Confident and correct predictions are low loss.
Confident and high predictions are high loss.
Uncertain predictions are the moderate losses.

Part 3: Why Does This Matter For Neural Networks?

When a classifier is trained, the model puts out a probability distribution over classes. Cross-entropy loss compares that predicted distribution against the true label, usually a one-hot vector.

Since the true label is one-hot, the formula is properely simplified as almost every term in the sum becomes zero except for the one corresponding to the correct class. The equation goes like: Loss = -log(q(correct class)).

During the training, the model is just trying to push the predicted probability of the correct class as close to the value 1 as possible. This is because -log(1)=0 and -log(x) has x approaching 0.

Conclusion

I am a high school senior that is interested in learning ML from the ground up and this is one of the first concepts that I have thoroughly researched in detail. I hope this blog was helpful to other students and adults like me that have a fond interest in ML.