DEV Community: Nilavukkarasan R

Attention Mechanisms: Stop Compressing, Start Looking Back

Nilavukkarasan R — Sun, 19 Apr 2026 05:32:31 +0000

"The art of being wise is the art of knowing what to overlook."
— William James

The Bottleneck We Didn't Notice

In my last post, we gave networks memory. An LSTM reads a sentence word by word, maintaining a hidden state that carries context forward. It solved the forgetting problem that plagued vanilla RNNs.

But there are three problems LSTM still doesn't solve. And I didn't fully understand them until I thought about my own experience learning English.

I studied in Tamil medium all the way through school. English was a subject, not a language I lived in. When I started my first job 20 years ago, I had to learn to actually speak it and more terrifyingly, write it. Client emails. Professional communication. Things that would be read, judged, and replied to.

My strategy was the only one I knew: compose the sentence in Tamil first, then translate it word by word into English.

It worked for simple things. It broke down in three very specific ways. Those three breakdowns map exactly onto the three problems that attention was built to solve.

Problem 1: The Compressed Summary

The first breakdown happened with long emails.

I'd compose a full paragraph in Tamil mentally: three or four sentences, a complete thought. Then I'd try to hold that entire paragraph in my head while translating it into English. By the time I was writing the third sentence in English, the first one had blurred. I'd lose the subject I'd introduced. I'd forget the condition I'd set up. The English output would drift from the original Tamil thought.

The problem wasn't that I forgot individual words. It was that I was trying to carry a compressed summary of a long paragraph in my working memory and that summary wasn't big enough to hold everything.

This is exactly what an RNN encoder does.

It reads the entire input sequence and compresses it into a single fixed-size vector, the final hidden state. Then the decoder uses only that compressed summary to generate the output. For short sentences, fine. For long ones, that summary has to hold everything: the subject, the verb, the object, the tone, the nuance. Something always gets lost.

Bahdanau's Fix (2014)

The fix came from Bahdanau, Cho, and Bengio. The idea is simple in principle: don't compress. Keep every hidden state the encoder produced, one per input word, and let the decoder look back at any of them when needed.

Instead of one compressed summary, the decoder has access to the full sequence of encoder states. When generating each output word, it computes a weighted sum over all of them attending more to the ones that are relevant right now, less to the ones that aren't.

Without attention:  decoder sees only h_final (compressed summary of everything)
With attention:     decoder sees h₁, h₂, ..., hₙ and decides what to focus on

Bahdanau's original formulation used a small neural network to compute how well each encoder state matched the decoder's current need - a learned compatibility function. It worked remarkably well. Translation quality on long sentences improved dramatically.

Your brain does this too. When you're answering a question about something you read, you don't reconstruct a compressed summary, rather you mentally flip back to the relevant section. The original is still accessible. Attention gives the network the same ability.

Problem 2: Word Order

The second breakdown was more embarrassing. It happened in individual sentences, not long paragraphs.

Tamil is a verb-final language. The verb comes at the end. When I wanted to write "Can you send the report by tomorrow?", the Tamil structure in my head was roughly: "நாளைக்குள் அந்த report-ஐ அனுப்ப முடியுமா?" — "Tomorrow-by that report send can-you?" Subject implied. Object before verb.

I'd start translating from the beginning of the Tamil sentence. "Tomorrow-by" → "By tomorrow". OK so far. "That report" → "the report". Fine. "Send" → "send". And then I'd realize I'd already written "By tomorrow the report send" and I was confused where to put "Can you."

What appeared perfectly correct in Tamil didn't map cleanly to English word by word. The structures are different. A literal left-to-right translation produces nonsense.

This is the word order problem — and it's where attention does its real work.

An RNN decoder, even with access to all encoder states, still generates output left to right, one word at a time. But attention lets the decoder look at any encoder position in any order. When generating "Can", it attends to the Tamil modal at position 5. When generating "send", it attends to the Tamil verb at position 4. When generating "tomorrow", it attends back to position 1.

Tamil:    நாளைக்குள்  அந்த  report-ஐ  அனுப்ப  முடியுமா
              h₁        h₂      h₃       h₄       h₅
           (by tmrw)  (that) (report)  (send)  (can you?)

English output → attention focus:
"Can"      → h₅  (முடியுமா — the modal)
"you"      → h₅
"send"     → h₄  (அனுப்ப — the verb)
"the"      → h₂ + h₃
"report"   → h₃  (report-ஐ — the object)
"by"       → h₁  (நாளைக்குள் — the time marker)
"tomorrow" → h₁

The attention weights form a matrix, one row per English output word, one column per Tamil input word. You can literally see the reordering: the decoder jumping from position 5 back to position 4, then to 3, then to 1. It's not following the Tamil order. It's following the English order, looking back at whatever Tamil position it needs.

This is what the Q/K/V formulation captures cleanly:

Query (Q): what the decoder is currently asking — "what do I need to generate this word?"
Key (K): what each encoder position offers — a description of what's available there
Value (V): the actual content retrieved when you attend to that position

Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V

The √d scaling keeps dot products in a stable range as dimension grows without it, softmax saturates and gradients vanish. Same instability problem we saw in deep networks, same fix.

Problem 3: Speed

The third breakdown was the slowest to notice, because it wasn't about a single sentence. It was about conversation.

Word-by-word translation is sequential by nature. I'd think in Tamil, translate, speak. Then listen to the reply in English, translate it back to Tamil to understand it, formulate a Tamil response, translate that to English, speak. Every exchange had this full round-trip happening in my head.

For a simple two-line exchange, manageable. For a fast-moving technical discussion with multiple people, completely unworkable. By the time I'd finished translating the last thing someone said, the conversation had moved on two turns.

The bottleneck wasn't comprehension. It was that the process was sequential. Each step had to wait for the previous one to finish.

This is the parallelism problem — and it's what self-attention solves.

An RNN processes a sequence one step at a time. Step 2 can't start until step 1 is done. For a sentence of length 100, that's 100 sequential operations. You can't parallelize across time steps because each hidden state depends on the previous one.

Self-attention breaks this dependency entirely. Instead of processing word by word, it computes relationships between all positions simultaneously in a single matrix operation. There's no sequential chain. The entire sequence is processed at once.

When you start thinking directly in English, something similar happens. Its not a sequential process anymore. Grammar, meaning, and context were being processed in parallel, automatically, without conscious effort. It's parallel processing.

Self-attention is the architectural version of that shift.

Self-Attention: Every Word Sees Every Other Word

So far, attention was between two sequences: Tamil input, English output. The decoder attends to the encoder. But the same mechanism applies within a single sequence and this turns out to be even more powerful.

Consider: "The report that the client who called yesterday requested is ready."

What is "ready"? The report. Which report? The one the client requested. Which client? The one who called yesterday. These connections span many positions in the same sentence. An RNN would need to carry all of this through its hidden state, step by step, hoping nothing gets lost.

Self-attention resolves them in one shot every word attends to every other word in the same sequence, regardless of distance.

"ready"     → attends back to "report" (subject of the predicate)
"requested" → attends to "client" (who did the requesting)
"who"       → attends to "client" (relative clause anchor)

No sequential processing. No hidden state bottleneck. One operation, all connections at once.

Your brain does this effortlessly when reading fluently. It's only when you're translating word by word processing sequentially, one token at a time that you lose these long-range connections.

Multi-Head Attention: Noticing Multiple Things at Once

There's one more piece. A single attention operation computes one set of weights. It can only "look for" one type of relationship at a time. But language has many simultaneous relationships.

In "The cat sat on the mat because it was tired", the word "it" has:

A syntactic relationship with "sat" (subject of the clause)
A coreference relationship with "cat" (what "it" refers to)
A semantic relationship with "tired" (property being attributed)

A single attention head would have to pick one. Multi-head attention runs several attention operations in parallel, each with different learned projections:

head_i = Attention(Q·Wᵢ_Q, K·Wᵢ_K, V·Wᵢ_V)
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W_O

Each head learns to notice different relationships simultaneously. One head might track grammatical alignment. Another might track semantic similarity. Another might track coreference, which pronoun refers to which noun.

The standard Transformer uses 8 heads. Each head operates on a smaller slice of the representation (dimension d/8 instead of d), so the total computation is the same as a single large attention — but the network gets 8 different perspectives instead of one.

What Clicked for Me

The compressed summary problem is the bottleneck of trying to hold a whole paragraph in working memory before translating. The word order problem is the mismatch between SOV and SVO that makes literal translation fail. The sequential processing problem is the reason real-time conversation was impossible while I was still translating word by word.

The shift from "translate word by word" to "think in English" is the shift from RNN to attention. It's not an optimization. It's a different way of processing.

Interactive Playground

cd 09-attention
streamlit run attention_playground.py

GitHub Repository

This playground is different from the previous ones. No training loops, no waiting. Five concept demos that follow the blog post narrative — every slider updates instantly because it's all just matrix math under the hood.

What's Next

Attention solves the bottleneck. But the architecture we've built so far still has an RNN encoder underneath — it's still sequential at its core.

Post 10 asks: what if we removed the RNN entirely? What if the whole architecture was just attention, stacked?

That's the Transformer. Attention without recurrence. Parallel processing of the entire sequence at once. Positional encodings to restore order information. And a feed-forward network to add non-linearity between attention layers.

It's the architecture behind every modern language model — GPT, BERT, T5, and everything that came after. And it's built entirely from pieces we already understand.

Deep Dive

For the full mathematical treatment — dot-product attention, scaled attention, the Q/K/V framework, self-attention, multi-head attention, masking, gradient flow, and worked numerical examples — see ATTENTION_MATH_DEEP_DIVE.md.

References

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
Luong, M., Pham, H., & Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation
Vaswani, A (2017). Attention Is All You Need

Understanding Recurrent Neural Networks: From Forgetting to Remembering

Nilavukkarasan R — Mon, 13 Apr 2026 14:43:48 +0000

"The present contains nothing more than the past, and what is found in the effect was already in the cause."
— Henri Bergson

Everything We Built Assumed a Snapshot

Look back at every network we've built so far.

A perceptron takes a fixed input vector and draws a line. An MLP stacks layers to bend that line into curves. A CNN slides filters across an image to detect spatial patterns. Even with all that sophistication, the convolutions, the pooling, the skip connections, every single one treats the input as a static snapshot. Feed it in, get a prediction out. The order of inputs doesn't matter. There's no before or after.

That assumption works perfectly for images. A digit is a digit regardless.

But language isn't a snapshot. Neither is audio, or time series, or any signal where what came before changes the meaning of what comes after.

"My teacher said I was slow, but he didn't know I was just getting started."

What does "he" refer to? The teacher, obviously. But only because you held "my teacher" in mind while reading the rest. You carried context forward — unconsciously, effortlessly.

Every architecture we've built so far would fail this. It has no mechanism for carrying anything forward.

That's the gap RNNs were built to fill.

Learning to Read — Letter by Letter

I remember learning to read. Not the fluent reading I do now, the early, effortful kind.

Each letter had to be identified consciously. Then combined with the next to form a sound. Then sounds stitched into a word. Then words assembled into meaning. It was slow, sequential, and exhausting. And crucially, by the time I reached the end of a long sentence, I'd often forgotten how it started.

That's a vanilla RNN.

It processes sequences one step at a time, maintaining a hidden state, a running summary of everything seen so far and updating it at each step:

# At each step t:
hidden(t) = tanh( W_h × hidden(t-1) + W_x × input(t) )
output(t) = W_o × hidden(t)

The hidden state is the memory. It blends the new input with what came before. The same weights are reused at every step, the network doesn't learn separate rules for position 1 vs position 50. One set of weights, applied repeatedly across time.

h(0) ──► h(1) ──► h(2) ──► h(3) ──► ...
  ▲         ▲         ▲         ▲
  │         │         │         │
x(0)      x(1)      x(2)      x(3)

Elegant. And it works, for short sequences. Just like the early reader who handles a short word fine but loses the thread of a long sentence.

Training It: Backprop Through Time

Training uses the same backpropagation from Post 3 — unrolled across time steps. To compute how much each weight contributed to the final loss, you trace gradients backward through every step:

x(0)→[RNN]→h(0)→[RNN]→h(1)→[RNN]→h(2)→[RNN]→h(3)→ Loss
                                                        │
                              gradients flow backward ◄─┘
                              through every time step

Same chain rule. Just applied across time instead of across layers. The depth is now temporal rather than architectural.

And here's where the familiar problem returns.

The Long Sentence Problem

Remember the vanishing gradient from Post 7? Gradients shrink as they travel backward through many layers, multiply enough numbers less than 1 together and you get zero.

The same thing happens here, but across time steps instead of layers.

At each step backward, the gradient gets multiplied by the weight matrix W_h. For a sequence of 50 words, that's 50 multiplications. The gradient reaching step 1 is effectively zero.

Like my early reading days: by the end of a long sentence, I'd forgotten how it started.

In Post 7, skip connections fixed vanishing gradients by adding a direct additive path that bypassed the layers. We need the same idea, but for time.

LSTM: Learning to Read Fluently

Think about what changes when reading becomes fluent.

You stop processing letter by letter. You chunk into words, phrases, meaning. More importantly, you become selective. You don't hold every word in memory with equal weight. You retain what matters: the subject, the tension, the unresolved question. You discard the filler. And you do this automatically, without thinking.

That selectivity is exactly what the Long Short-Term Memory network (Hochreiter & Schmidhuber, 1997) introduced.

An LSTM has two states instead of one: a hidden state h (what it's currently working with) and a cell state c (long-term memory). The cell state is the key innovation, it runs through the sequence with only small, controlled modifications. Like the skip connection in ResNets, it's an additive path that lets gradients flow backward without decaying at every step.

Three gates control what happens to memory at each step:

Forget gate:  should I clear out old memory?
              f = sigmoid( W_f × [h(t-1), x(t)] )

Input gate:   is this new input worth remembering?
              i = sigmoid( W_i × [h(t-1), x(t)] )
              candidate = tanh( W_c × [h(t-1), x(t)] )

Output gate:  what should I act on right now?
              o = sigmoid( W_o × [h(t-1), x(t)] )

Update:
  cell:    c(t) = f × c(t-1)  +  i × candidate
  hidden:  h(t) = o × tanh( c(t) )

The sigmoid gates output values between 0 and 1 — soft switches. A forget gate near 1 means "keep everything." Near 0 means "wipe it." The network learns when to remember and when to forget, based on what the task requires.

The cell state update — c(t) = f × c(t-1) + i × candidate is additive. Old memory plus new information. That additive structure is what saves the gradient. Instead of multiplying through a squashing function at every step, gradients flow backward through the cell state with far less decay.

Same intuition as the ResNet skip connection. Different problem, same fix.

GRU: Fluency With Less Overhead

Once reading becomes fluent, you don't consciously run through all three questions at every word. Most decisions are automatic, keep reading, update the picture, move on.

The Gated Recurrent Unit is that streamlined version. It merges the cell state and hidden state into one, and uses two gates instead of three:

Reset gate:   how much past context to use for the new candidate
              r = sigmoid( W_r × [h(t-1), x(t)] )

Update gate:  how much to blend old state with new candidate
              z = sigmoid( W_z × [h(t-1), x(t)] )

Update:
  candidate: h̃ = tanh( W × [r × h(t-1), x(t)] )
  hidden:    h(t) = (1-z) × h(t-1)  +  z × h̃

Fewer parameters, similar performance. The update gate does double duty controlling both forgetting and writing in one operation. In practice, LSTMs and GRUs perform comparably. GRUs train faster; LSTMs have slightly more expressive memory. Most practitioners try both.

Layer Normalization: The Normalization That Fits Sequences

In Post 7, batch normalization stabilized deep networks by normalizing across the batch. But RNNs have a problem with batch norm. Sequences have variable lengths, and the hidden state carries information across steps. Normalizing across a batch of sequences at each time step is unstable.

Layer normalization fixes this by normalizing across the features of each individual sample, not across the batch. Same idea, different axis. Completely independent of batch size and sequence length.

This is why layer norm became the standard for all sequence models and why every modern LLM uses it. When we get to Transformers in Post 10, it'll be everywhere.

What Clicked for Me

The reading analogy didn't just help me explain RNNs — it helped me understand what the hidden state actually is.

It's not a recording of the past. It's a compressed summary the parts of history that seem relevant for predicting what comes next. Just like a fluent reader doesn't remember the exact words from three pages ago, but does remember that the detective is suspicious of the butler.

Interactive Playground

cd 08-rnn
streamlit run rnn_playground.py

GitHub Repository

Train both models, then pick a sentence length and watch the confidence bars update word by word, you'll see exactly the step where Vanilla RNN changes its mind and LSTM doesn't.

What's Next

RNNs gave networks memory. But they process sequences step by step - slow, sequential, and still limited by how far gradients can travel even with LSTM.

There's a deeper problem too. The hidden state has to compress everything seen so far into a fixed-size vector. For long sequences, that bottleneck loses information no matter how good the gating is.

Post 9 introduces Attention Mechanisms: a way for the network to directly look back at any part of the input sequence it needs, regardless of distance. No compression bottleneck. No sequential processing. No hoping, the gradient survives 100 time steps.

It's the idea that made RNNs obsolete — and made Transformers possible.

References

Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780.
Cho, K., et al. (2014). Learning Phrase Representations using RNN Encoder-Decoder. EMNLP.

Understanding Internal Covariate Shift and Residual Connections: Beyond Activation Functions and Optimizers

Nilavukkarasan R — Sat, 11 Apr 2026 14:46:14 +0000

"No man ever steps in the same river twice, for it's not the same river and he's not the same man" - Heraclitus

When Going Deeper Made Things Worse

In my last post, we built CNNs that could see. Filters learned edges. Pooling built spatial tolerance. Stack enough layers and the network recognizes digits, faces, objects.

So the obvious next move: go deeper. More layers, more capacity, more power.

But there is a catch.

Researchers took a 20-layer network and added 36 more layers. The 56-layer network should have been better. More parameters, more room to learn. Instead, it was worse. Not just on test data, But on training data as well.

That's not overfitting. Overfitting means you're too good on training data. This was the opposite: a bigger network that couldn't even fit the data it was trained on.

Two things were broken. And fixing them required two elegant ideas.

The Noisy Room Problem

Imagine you're at a loud party, trying to follow a conversation. The room is packed, music is blasting, five other conversations are happening around you. Your brain doesn't give up, it does something remarkable. It filters out the noise, locks onto the voice you care about, and normalizes the signal so you can follow along.

You do this automatically, without thinking. But a neural network? It has no such mechanism.

Here's what actually happens inside a deep network during training. Each layer transforms its input and passes it to the next. A small shift in one layer's output gets amplified by the next layer, which gets amplified again, and again. After 20 layers, the signal has either exploded into enormous numbers that saturate neurons, or collapsed into near-zero values that carry no information.

The network is trying to learn in a room that keeps getting louder.

That's the internal covariate shift problem. The distribution of each layer's input keeps changing as weights update. Every layer is chasing a moving target.

Batch normalization is the fix to it.

Batch Normalization: Tuning Out the Noise

Before each layer processes its input, normalize it. Force it to have zero mean and unit variance. Then let the network re-scale with two learned parameters: γ (gamma) and β (beta).

# For each mini-batch:
compute mean and variance of the inputs
normalize: x_norm = (x - mean) / sqrt(variance)

# Then re-scale with learned parameters:
output = γ * x_norm + β

The network can undo the normalization if it needs to. γ and β are learned. But now every layer starts from a stable, predictable baseline. The moving target stops moving.

Going back to the party analogy: batch norm is your brain's noise-cancellation. It doesn't remove the signal, it strips out the irrelevant variation so the important information comes through clearly.

The effect on training is immediate:

Without batch norm:
  Layer 5 output:  mean=2.3,  std=4.7
  Layer 10 output: mean=18.4, std=31.2   ← signal exploding
  Layer 20 output: mean=NaN              ← training collapsed

With batch norm:
  Layer 5 output:  mean≈0, std≈1
  Layer 10 output: mean≈0, std≈1
  Layer 20 output: mean≈0, std≈1         ← stable all the way down

Three things happen when you add batch norm:

Activations stay stable — no more explosions or collapses
You can use much higher learning rates — the stable baseline means bigger steps are safe
Weight initialization matters less — you no longer need to be as careful about starting values

One subtle thing worth knowing: batch norm uses statistics computed across the current mini-batch. At inference time, you might be predicting on a single example and no batch to compute statistics from. So during training, batch norm accumulates running averages of the mean and variance. At inference, it uses those instead.

The Vanishing Gradient: A Deeper Problem

Batch norm stabilizes the forward pass. But there's a second problem, and it lives in the backward pass.

Backpropagation multiplies derivatives together as it moves backward through the network. Each layer contributes a factor. If those factors are consistently less than 1, which they often are, the gradient shrinks with every layer it passes through.

By the time it reaches layer 1 of a 50 layer network, the gradient might be effectively zero. The early layers stop learning entirely.

This is why the 56-layer network performed worse than the 20-layer one. It wasn't a capacity problem. The early layers simply weren't getting any useful gradient signal. They were frozen.

Residual Connections: The Shortcut

"If I have seen further, it is by standing on the shoulders of giants."
— Isaac Newton

Instead of learning a full transformation, a residual block learns the difference from identity:

# Normal layer:
output = transform(x)

# Residual block:
output = transform(x) + x    ← just add the input back

That + x is the skip connection. The input bypasses the learned transformation and gets added back at the output.

How it changes the chain rule.

In a normal layer, backprop applies the chain rule like this:

∂L/∂x = ∂L/∂output × F'(x)

The gradient gets multiplied by F'(x) at every layer. If that's 0.1, after 50 layers you're multiplying fifty 0.1s together, the gradient reaches layer 1 as essentially zero.

With a residual block, output = F(x) + x, so the chain rule becomes:

∂L/∂x = ∂L/∂output × (F'(x) + 1)

That + 1 comes from differentiating the skip connection x with respect to x, the derivative of a straight passthrough is always 1. Now instead of multiplying fifty 0.1s, you're multiplying fifty 1.1s. The gradient stays alive all the way back to layer 1.

Before ResNets, the practical limit for trainable networks was around 20 layers. After ResNets, researchers trained a 1,202-layer network. Not because they needed 1,202 layers, but to prove they could.

That distinction: capacity vs. trainability is one of the most important ideas in deep learning.

The Bigger Picture: How Everything Fits Together

At this point in the series, it's worth stepping back. A lot of concepts been introduced, and it can start to feel like an ever growing list of tricks. It's not. Each one solved a specific, concrete failure mode:

Problem	What Goes Wrong	Solution
Dying neurons	Neurons output zero forever, stop learning	ReLU
Vanishing gradients	Gradients too small to reach early layers	ReLU + careful init
Exploding gradients	Gradients too large, training diverges	Gradient clipping, Adam
Slow convergence	Hard to find a good learning rate	Adam optimizer
Internal covariate shift	Each layer's inputs keep shifting distribution	Batch Norm
Degradation problem	Deeper networks perform worse than shallow ones	Skip connections (ResNet)

These aren't redundant, they're complementary. ReLU keeps neurons alive. Adam navigates the loss landscape efficiently. Batch norm stabilizes the signal between layers. Skip connections ensure gradients reach the beginning. Each one patches a gap the others can't cover.

Together, they form the foundation that makes modern deep networks trainable. You'll see all of them again — in every architecture from here on out.

What's Next

We can now train deep networks. But depth alone doesn't solve every problem.

Images have spatial structure, CNNs exploit that. But what about sequences? Text, audio, time series, data where order matters and context can span hundreds of steps?

Post 8 introduces Recurrent Neural Networks: architectures with memory, where the output at each step depends on everything that came before. And you'll see immediately why the vanishing gradient problem, which we just solved for depth, comes back with a vengeance for long sequences.

References

Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML.
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. CVPR 2016.

From Generalists to Specialists: The CNN Shift

Nilavukkarasan R — Tue, 31 Mar 2026 13:28:17 +0000

"Vision is the art of seeing what is invisible to others." Jonathan Swift

When Regularization Wasn't Enough

In my last post, I showed you how dropout and weight decay stop a network from memorizing training data. We trained on MNIST, closed the generalization gap, and got a network that actually works in the real world.

It felt like we'd finally solved it.

But then i tried it on a real photograph. Not 28×28 grayscale digits. A 224×224 color image.

The math was brutal.

224 × 224 × 3 = 150,528 inputs
Connect those to 1,000 neurons: 150 million parameters
Just for the first layer. Before learning anything useful.

We needed a different idea entirely. And it came, as the best ideas often do, from biology.

The Visual Cortex Moment

In 1959, neuroscientists David Hubel and Torsten Wiesel did something remarkable. They inserted electrodes into a cat's visual cortex and projected shapes onto a screen. They were trying to find what made individual neurons fire.

Most shapes did nothing. Then, almost by accident, they moved a glass slide and cast a thin line of light across the screen.

One neuron went wild.

Not all neurons. Not the whole cortex. One specific neuron, responding to one specific edge, at one specific orientation, in one specific region of the visual field.

They kept experimenting. Different neurons responded to different orientations, horizontal edges, vertical edges, diagonal edges. Each neuron had a small receptive field: a limited patch of the visual field it paid attention to. Neurons in later areas responded to more complex patterns, corners, curves, eventually whole shapes.

The visual cortex isn't a fully connected blob. It's a hierarchy of local detectors, each building on the one before it.

That insight, decades later, became the blueprint for CNNs.

The Problem with Fully Connected Networks on Images

Here's what a fully-connected network does to an image:

FC Network sees an image as:
┌─────────────────────────────────────────────────────┐
│  pixel_1, pixel_2, pixel_3, ..., pixel_150528       │
│  (all spatial structure destroyed, every pixel      │
│   connected to every neuron with separate weights)  │
└─────────────────────────────────────────────────────┘

Every neuron: "I must learn about ALL 150,528 pixels equally."

This is wasteful in two ways. First, a pixel in the top-left corner has almost nothing to do with a pixel in the bottom-right corner, but the network treats them as equally related. Second, if a cat's ear appears in the top-left of one image and the top-right of another, the network needs separate neurons to detect it in each location.

A CNN thinks differently:

CNN sees an image as:
┌──────────────────────────────────────────────────────┐
│  ┌───┐  ┌───┐  ┌───┐                                │
│  │ F │  │ F │  │ F │  ← same filter, sliding across │
│  └───┘  └───┘  └───┘    "Is there an edge here?"    │
│     ↘      ↓      ↙                                  │
│      [feature map]                                   │
└──────────────────────────────────────────────────────┘

Every filter: "I detect ONE pattern, ANYWHERE in the image."

Same filter, applied everywhere. One set of weights to detect vertical edges across the entire image. This is weight sharing, and it's the core reason CNNs work.

The parameter comparison is stark:

	FC Network	CNN (typical)
Input	224×224×3 = 150K	224×224×3 = 150K
First layer params	150K × 1000 = 150M	96 filters × 11×11×3 = 35K
Assumption	all pixels equally related	nearby pixels are related

For the full mathematical breakdown of parameter counts across FC, LeNet, AlexNet, VGG, and ResNet, see CNN_ARCHITECTURE_DEEP_DIVE.md.

Key Concepts, Grounded in Biology

Local Receptive Field

In the visual cortex, each neuron only responds to a small patch of the visual field. It's local. It doesn't see the whole image—just its neighborhood.

In a CNN, each filter application does the same thing. A 3×3 filter looks at a 3×3 patch of the image. That's its receptive field. It asks: "Does my pattern exist in this small region?"

As you go deeper in the network, receptive fields grow. A neuron in layer 3 has seen the outputs of layer 2, which saw layer 1, which saw the raw pixels. So it effectively "sees" a larger region, just like neurons deeper in the visual cortex respond to larger, more complex patterns.

Learnable Filter (The Edge Detector)

A filter is just a small grid of numbers—say 3×3. During training, backpropagation (the same algorithm from Post 3) adjusts these numbers until the filter detects something useful. One filter might learn to detect vertical edges. Another learns horizontal edges. Another learns a specific texture.

Learned vertical edge filter:    Learned horizontal edge filter:
[-1  0  1]                        [-1 -2 -1]
[-2  0  2]                        [ 0  0  0]
[-1  0  1]                        [ 1  2  1]

The network learns these automatically, no human designs them. That's the power.

Feature Map

When you slide a filter across an image, you get a feature map: a 2D grid showing where that filter's pattern was detected and how strongly.

Think of it like a heat map. Apply a vertical-edge filter to a photo of a face, and the feature map lights up along the sides of the nose, the edges of the eyes, the outline of the jaw. Dark where there are no vertical edges. Bright where there are.

Stack 32 filters and you get 32 feature maps, 32 different "views" of the same image, each highlighting a different pattern.

Padding

Here's a practical problem: if you slide a 3×3 filter across a 5×5 image, the filter can't be centered on the edge pixels. You lose a border of information, and the output shrinks.

Padding adds a ring of zeros around the image before applying the filter. This lets the filter visit every pixel, including the edges, and preserves the spatial dimensions.

It's like giving your peripheral vision a bit of extra context at the boundary of your visual field.

Pooling (Spatial Summarization)

After detecting features, we don't need to track exactly where they appeared—just roughly where. This is pooling.

Max pooling takes a small window (say 2×2) and keeps only the strongest activation. It's like asking: "Did this feature appear anywhere in this region?" The exact location doesn't matter.

Feature map:          After 2×2 max pooling:
[1  3  2  4]          [6  4]
[5  6  1  2]    →     [8  7]
[3  8  4  7]
[1  2  6  3]

This does three things: reduces the spatial size (fewer parameters downstream), makes the network tolerant to small shifts in position (translation invariance), and forces the network to summarize rather than memorize exact locations.

Your visual cortex doesn't care if a face is shifted 5 pixels left. You still recognize it. Pooling builds that tolerance in.

ReLU — Still Here

The activation function hasn't changed. After each convolution, we still apply ReLU (from Post 2):

output = max(0, convolution_result)

Negative activations become zero. Positive ones pass through. Same reason as before: it introduces non-linearity and avoids the vanishing gradient problem we discussed in Post 3. The building blocks carry forward.

Putting It Together: The CNN Pipeline

A CNN is just these ideas stacked in sequence:

Input Image
    ↓
[Conv → ReLU] × N     ← learn local patterns (like V1 cortex)
    ↓
[Pooling]             ← summarize, reduce size
    ↓
[Conv → ReLU] × N     ← learn combinations of patterns (like V2/V4)
    ↓
[Pooling]
    ↓
Flatten
    ↓
[Fully Connected]     ← classify based on learned features
    ↓
Softmax → Prediction

The early layers learn edges and textures. Middle layers combine those into shapes. Deep layers combine shapes into objects. It's the same hierarchy Hubel and Wiesel found in the cat's brain, just learned from data instead of evolution.

Training still uses backpropagation and Adam. The same gradient flow, the same weight updates. CNNs didn't replace what we built, they extended it with a smarter architecture.

CNN Architectures: A Brief Lineage

The core ideas — local receptive fields, learnable filters, pooling stayed constant. What changed over the years was how deep, how wide, and how trainable.

LeNet (1998) was the proof of concept. Two conv layers, two pooling layers, three fully-connected layers. Trained on handwritten digits. ~60K parameters. It worked, but the hardware and data of the time couldn't push it further.

AlexNet (2012) was the moment everything changed. Five conv layers, three FC layers, ~60M parameters — trained on GPUs for the first time. It won ImageNet by a margin that shocked the field. The key additions: ReLU activations (faster training), dropout (regularization), and data augmentation. The deep learning era started here.

VGG (2014) asked: what if we just go deeper, but keep it simple? Only 3×3 filters, stacked in blocks. 16–19 layers, ~138M parameters. It showed that depth itself was the driver of accuracy, but the three large FC layers at the end were a parameter bottleneck.

ResNet (2015) solved the problem VGG exposed: beyond ~20 layers, adding more layers actually hurts accuracy. Not from overfitting — from gradients vanishing before they reach early layers. ResNet's fix was elegant: skip connections that let gradients bypass layers entirely. Suddenly 50-layer, 152-layer networks were trainable. ResNet-50 achieves better accuracy than VGG-16 with 5× fewer parameters.

LeNet (1998)    →  AlexNet (2012)   →  VGG (2014)      →  ResNet (2015)
~60K params        ~60M params         ~138M params        ~25M params
proof of concept   GPU + ReLU          depth matters       skip connections
                   revolution          but costly          solve depth

CNNs Are Built for Images — Not Text

This is worth saying explicitly, because it's easy to assume CNNs are a general-purpose upgrade to fully-connected networks. They're not.

CNNs work because of two assumptions baked into the architecture:

Nearby inputs are related — a pixel's neighbors matter more than distant pixels
The same pattern can appear anywhere — weight sharing makes sense because an edge in the top-left is the same edge as one in the bottom-right

Images satisfy both assumptions perfectly. So do audio spectrograms and video frames.

Text doesn't. The word "not" next to "good" completely changes the meaning — but "not" next to "bad" means something different again. Context in language isn't local and positional the way it is in images. The same word means different things in different positions. Weight sharing across positions doesn't make the same kind of sense.

That's why text needs different architectures — RNNs that process sequences step by step, and eventually Transformers that learn which words to pay attention to regardless of distance. CNNs are a specialized tool, and their specialization is spatial data.

What Clicked for Me

I kept thinking about Hubel and Wiesel's cat. One neuron, one edge orientation. Seemed pointless.

Then it clicked: a neuron that responds to everything is useless. A neuron that responds to exactly one pattern, reliably, anywhere—that's signal. That's a building block.

Fully-connected networks try to be generalists from pixel one. CNNs start with specialists and compose up.

Interactive Playground

I've built an interactive playground where you can watch CNN in action. It has two tabs

Tab 1: FC Network vs CNN

Both models are trained from scratch on the same 1,000-sample MNIST subset using pure NumPy and Adam — the same setup from Post 4. You can adjust the FC hidden layer size, the number of CNN filters, the number of epochs (up to 20), and the batch size, then hit Train both models to run real training.

Tab 2: CNN Layer Explorer

Pick any of the digits 0, 1, 6, or 8 and explore three views:

What each filter detects — shows the raw filter weights (3×3 grid) alongside the response heatmap on your chosen digit. Bright yellow means "this pattern is strongly present here." You can see how a vertical-edge filter lights up along strokes, while a blob filter responds to filled regions.
Layer-by-layer pipeline — traces your digit through the full network: Conv1+ReLU → MaxPool → Conv2+ReLU → MaxPool → Flatten → FC → Softmax. Each stage shows the actual feature map image with a caption explaining what happened and why. A dimension table below tracks the shape at every step.
MaxPool zoom-in — takes a 4×4 patch from the conv output and shows the actual numerical values, then shows the 2×2 result after pooling. You can see exactly which values survived and why — the maximum in each 2×2 block wins.

What This Unlocked

Before CNNs, computer vision meant handcrafting features. Researchers spent years designing SIFT descriptors, HOG features, edge detectors, all by hand. Then AlexNet (2012) showed that a CNN trained on enough data could learn better features automatically, and it wasn't close. The error rate dropped from 26% to 15% in one year.

That was the moment the field changed.

Every modern vision system—object detection, medical imaging, autonomous driving, face recognition—runs on some variant of this idea. Local receptive fields. Learnable filters. Hierarchical features. Weight sharing.

All of it traced back to a cat, an electrode, and a sliding glass slide in 1959.

What's Next

We can now train CNNs that learn to see. But there's a catch: go deep enough, and training breaks down. Gradients vanish. Accuracy plateaus. Adding more layers actually hurts.

Post 7 covers the two innovations that fixed this: Batch Normalization (stabilize the activations between layers) and Residual Connections (let gradients skip layers entirely). Together, they made 50-layer, 100-layer networks trainable—and unlocked the modern era of deep learning.

References

LeCun et al. (1998). Gradient-Based Learning Applied to Document Recognition.
Krizhevsky et al. (2012). ImageNet Classification with Deep Convolutional Neural Networks.
Simple explanation of Convolutional Neural Network

Regularization: Fighting Overfitting

Nilavukkarasan R — Fri, 13 Mar 2026 14:35:02 +0000

"Learning without thought is labor lost" --Confucius

When Your Network Becomes a Memorizer

In my last post, I showed you how to train a network on MNIST. Adam optimizer, mini-batches, 100 epochs. Training accuracy climbed over 99%.

It felt like we'd solved it.

But here's what I didn't show you: what happens if you keep training.

Running the network for 200 epochs drives training accuracy over 99%. But test accuracy tells a different story, it hits 97% around epoch 50, then slowly drops as training continues.

Why? Imagine studying for an exam by memorizing that "Question 5 is always B" instead of understanding why B is correct. You'd ace the practice test but fail when questions are reordered or rephrased. Neural networks do the same thing. They memorize training data so well they hit 99% accuracy, yet struggle with new examples because they never learned the underlying patterns.

The network isn't learning anymore. It's memorizing. That's overfitting, and it's one of the core challenges in training neural networks.

The Generalization Gap

Here's what happens when you train a network without any safeguards:

Epoch 1:   Train Acc: 87.3%  Test Acc: 86.9%  Gap: 0.4%
Epoch 10:  Train Acc: 97.2%  Test Acc: 96.8%  Gap: 0.4%
Epoch 50:  Train Acc: 99.1%  Test Acc: 97.2%  Gap: 1.9%
Epoch 100: Train Acc: 99.7%  Test Acc: 96.8%  Gap: 2.9%
Epoch 200: Train Acc: 99.95% Test Acc: 96.1%  Gap: 3.85%

See that gap? It starts small, the network learns generalizable patterns. But as training continues, the gap widens. The network is still improving on training data, but test accuracy stalls and drops.

This gap is the generalization gap. It's the difference between what your network learned and what it actually understands.

Why Does This Happen?

A network has three things working against it:

1. Capacity: Your network has 100,000 weights. Your training set has 60,000 examples. Mathematically, the network has enough capacity to memorize every single example.

2. Time: Every epoch, the network sees the same training examples again. It gets more chances to memorize. After 200 epochs, it's seen each example 200 times. Memorization becomes easier than learning.

3. No penalty for complexity: The network doesn't care if it uses simple patterns or complex ones. Both reduce training loss equally. So it drifts toward complexity overfitting.

The solution? Force the network to generalize instead of memorize.

The Regularization Toolkit: A Big Picture View

Before we dive into specific techniques, let's zoom out and see the full landscape of solutions to overfitting. There are actually many ways to fix it:

1. Reduce Model Capacity

Use smaller networks (fewer neurons, fewer layers)
Prune weights after training
Use simpler architectures

The idea: if your network is smaller, it simply can't memorize as much. But this is a blunt instrument, you might lose the ability to learn complex patterns.

2. Increase Training Data

Collect more real data
Use data augmentation (rotations, crops, noise for images; paraphrasing for text)
Use synthetic data generation

The idea: with more diverse examples, memorization becomes harder. The network has to learn generalizable patterns to cover all the data.

3. Stop Training Early

Monitor test accuracy during training
Stop when test accuracy starts declining
This is called "early stopping"

The idea: overfitting gets worse over time. Stop before it happens.

4. Ensemble Methods

Train multiple networks and average their predictions
Use techniques like boosting or bagging

The idea: multiple imperfect models often generalize better than one perfect model.

5. Architectural Innovations

Skip connections (ResNets) allow training deeper networks that generalize better
Attention mechanisms focus on relevant parts of the input
Inductive biases (like convolutions for images) reduce the effective capacity

The idea: design the architecture to match the problem structure.

Our Focus: Regularization Techniques

In this post, we're going to deep-dive into two regularization techniques - dropout and weight decay.

Why these two? Because they represent two different philosophies:

Dropout prevents co-adaptation (neurons learning to work together in ways that only make sense for training data)
Weight decay encourages simplicity (smaller weights = simpler decision boundaries)

Together, they form a powerful one-two punch against overfitting. And understanding them deeply will help you understand other regularization techniques.

Dropout: An Ensemble of Smaller Networks

Here's an idea: what if we randomly disabled neurons during training?

Not permanently. Just during each forward pass.

This sounds like sabotage. Why would we intentionally break our network?

Because it forces the network to learn redundant representations.

Think of it like this: imagine you're building a team to solve problems. If you always have the same 10 people, they'll specialize and depend on each other. Person A always handles data, Person B always handles logic. If Person A gets sick, the team fails.

But if you randomly remove people from the team each day, they can't specialize. Everyone learns to do everything. The team becomes robust.

That's dropout.

When we randomly disable neurons during training, the network can’t rely on specific neurons to make a prediction. Instead, it must learn multiple pathways to the same answer. This redundancy prevents co-adaptation. i.e. neurons relying on each other in ways that only work for the training data.

If a layer has n neurons, there are 2ⁿ possible subnetworks, depending on which neurons are active or dropped. During training, each mini-batch randomly samples one of these subnetworks.

Imagine a hidden layer with 256 neurons and 50% dropout.

Each mini-batch activates a different random subset of neurons:

Mini-batch 1 trains with neurons {1, 3, 5, 7, ..., 255}
Mini-batch 2 trains with neurons {2, 4, 6, 8, ..., 256}
Mini-batch 3 trains with neurons {1, 2, 4, 7, ..., 254}

Each subset forms a slightly different network. Over training, the model samples from an enormous space of subnetworks and learns weights that perform well across many of them.

Modern implementations use inverted dropout.

During training, we randomly drop neurons and scale the activations so that their expected value stays the same. At test time, we simply run the full network without any dropout.

# Training: randomly disable neurons
if training:
    mask = np.random.binomial(1, 1 - dropout_rate, X.shape)
    X_dropped = X * mask / (1 - dropout_rate)  # Scale to maintain expected value
else:
    X_dropped = X  # No dropout at test time

The scaling factor 1 / (1 - dropout_rate) is crucial.

Without it, the magnitude of activations during training would be smaller than during inference, causing inconsistent predictions.

By scaling during training, the expected activation remains the same whether dropout is active or not.

Dropout forces the network to learn robust representations. No neuron can assume another neuron will always be present, so useful features must be distributed across the network.

The result:

Less memorization
Better generalization
A model that performs well on unseen data, not just the training set.

Weight Decay: Occam's Razor for Neural Networks

Dropout prevents co-adaptation. But there's another approach: what if we penalize large weights?

This idea is called weight decay, also known as L2 regularization.

The idea is simple: add a penalty to the loss function proportional to the magnitude of weights.

Total Loss = Cross-Entropy Loss + λ * (sum of squared weights)

The λ (lambda) parameter controls how much we penalize large weights. Higher λ means stronger penalty.

Why does this work? Large weights tend to make a network very sensitive to small changes in the input, producing sharp decision boundaries that can fit noise in the training data. Smaller weights produce smoother functions that change more gradually.

Consider two networks that both achieve 95% training accuracy:

Network A: Has weights like [0.1, 0.2, -0.15, 0.08, ...]. Small adjustments to inputs.

Network B: Has weights like [5.2, -8.7, 12.3, -6.1, ...]. Large adjustments to inputs.

Both fit the training data. But Network B's large weights create sharper, more extreme responses to inputs, which increases the risk of overfitting. Weight decay prefers Network A because its weights have smaller magnitude.

Without weight decay, the optimizer only cares about minimizing training loss.

With weight decay, the optimizer faces a trade-off:

Reduce training loss
Keep weights small

During backpropagation, the regularization term adds an extra component to the gradient:

gradient = original_gradient + λ * w

This gently pulls weights toward zero during training. The result is not that the network learns less, but that it learns more restrained solutions that tend to generalize better.

Weight decay doesn't restrict learning.
It simply nudges the model toward simpler explanations.

The Tuning Process:

Train without regularization. Measure the train-to-test gap.
If gap < 1%, you're good. No regularization needed.
If gap is 1-3%, add dropout=0.2. Retrain and measure.
If gap is still > 2%, add weight_decay=0.0001. Retrain and measure.
If gap is still > 2%, increase dropout to 0.3 or 0.4.
If gap is > 3%, you might need more data or a smaller network.

The key is experimentation. Every dataset is different. What works for MNIST might not work for ImageNet. Start conservative, measure, adjust.

Interactive Exploration

This is where the playground comes in. I've built a Streamlit app that lets you experiment with these techniques in real-time. It covers two parts overfitting and weight distribution to explore with.

What Clicked for Me

Regularization is a trade-off. You're not trying to achieve 100% training accuracy. You're trying to maximize test accuracy. I used to think "higher training accuracy = better network." Now I know better.

Dropout is elegant. It's not a hack. It's a principled way to train an ensemble of networks simultaneously.

Each breakthrough solved a problem the previous one created:

Perceptrons couldn't learn complex patterns → Multi-layer networks
Multi-layer networks couldn't learn efficiently → Backpropagation
Backpropagation was slow on large datasets → Optimization (mini-batches, Adam)
Optimization worked but overfitted → Regularization (dropout, weight decay)

We're building a complete system. Each piece is necessary.

What's Next

We can now train networks that actually work in the real world. They learn patterns, not memorize data. They generalize to new examples.

For now, we're still limited to fully connected networks on small images. MNIST is 28×28. Real images are 1000×1000 or larger. And fully connected networks don't scale, a 1000×1000 image would require 1 million input neurons.

We need a different architecture. One designed specifically for images.

Enter convolutional networks.

The jump from fully connected to convolutional is as big as the jump from perceptrons to multi-layer networks. It's a completely different way of thinking about neural networks.

And it's the next breakthrough in our journey.

References

Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. Improving neural networks by preventing co-adaptation of feature detectors
Ng, A. Y. Feature selection, L1 vs. L2 regularization, and rotational invariance.

Tags: #MachineLearning #AI #DeepLearning #Regularization #Dropout #WeightDecay #Overfitting #MNIST #NeuralNetworks

Code: GitHub Repository

Neural Network Optimizers: From Baby Steps to Intelligent Learning

Nilavukkarasan R — Wed, 04 Mar 2026 15:06:50 +0000

Adapt what is useful, reject what is useless, and add what is specifically your own.
-- Bruce Lee

In my last post, I showed you how backpropagation could learn the weights for XOR automatically. No more hand-crafting. No more trial and error. Just set a learning rate, run the algorithm, and watch the loss curve drop.

It felt like magic. Almost too easy.

But here's what I glossed over: XOR has just 4 training examples. With 4 examples, you compute the gradient using all of them at once. Every weight update sees the complete picture.

But XOR is a toy problem. Let me tell you about a real dataset.

MNIST: The "Hello World" of Deep Learning

MNIST is a collection of 70,000 handwritten digit images—60,000 for training, 10,000 for testing. Each image is 28×28 grayscale pixels.

The task: look at an image and predict which digit (0-9) it represents.

Trivial for humans. Genuinely hard for 1990s computers. It became the standard benchmark for machine learning.

Each image has 784 pixels (28×28). To classify them, we need 784 inputs, a hidden layer (say, 128 neurons), and 10 outputs. That's roughly 100,000 weights to learn.

Here's the problem: computing the gradient using all 60,000 examples—like we did with XOR—requires 60,000 forward and backward passes per weight update. On my laptop, that's 30 seconds per epoch.

Training for 100 epochs? 50 minutes total.

Not terrible. But remember, this is just MNIST.

Now consider:

GPT-2 trained on 40 billion tokens
GPT-3 trained on 300 billion tokens
GPT-4? Trillions of tokens, hundreds of billions of weights

If full-batch gradient descent takes 50 minutes for MNIST, modern LLMs would take thousands of years.

This is the "Scale Wall." Nobody trains this way anymore.

The Leap from Toy to Real

The jump from XOR to MNIST isn't just more data—it's a fundamental shift in thinking. At scale, learning can't be perfect. It has to be incremental, approximate, adaptive. Just like human learning.

This is where optimisers enter.

Mini-Batches: Learning from Subsets. The insight that changed everything - you don't need all 60,000 examples to know which direction to move your weights.

Think about cooking. You don't taste every grain of rice to know if you need more salt. You taste,a spoonful, a small sample tells you enough.

That's mini-batch stochastic gradient descent. Instead of using all 60,000 examples:

The algorithm is simple:

for each epoch:
    shuffle training data
    divide into mini-batches
    for each mini-batch:
        forward pass (compute predictions)
        compute loss (average over batch)
        backward pass (compute gradients)
        update weights

One complete pass through all data is an epoch. With 60,000 examples and batch size 64, that's 937 mini-batches per epoch.

The magic? Each mini-batch gradient is noisy—not the exact gradient from all 60,000 examples. But it points roughly in the right direction. And it's fast.

On my laptop:

Full-batch: 30 seconds per update
Mini-batch (size 64): 0.03 seconds per update

That's 1000× faster. Training for 100 epochs drops from 50 minutes to 5 minutes.

The Three Key Concepts

Mini-batch: A small subset of training data for one gradient update. Common sizes: 16, 32, 64, 128. Smaller batches are noisier but faster. Larger batches are more accurate but slower.

Epoch: One complete pass through your training dataset. With 60,000 examples and batch size 64: one epoch = 937 updates.

Stochastic: Means "random." We shuffle data before each epoch, so mini-batches differ every time. This randomness helps—it prevents the network from memorising example order.

Optimizers as Human Learners

Just like human learners, no single optimizer is perfect. Each optimizer is a different learning strategy, a different way to navigate the loss landscape.

I’ve put together a simple summary of three optimizers — SGD, Momentum, and Adam — and how they approach learning. Do take a look OPTIMIZERS_SIMPLIFIED, if you’re interested

There are many more optimizers in the wild: AdaGrad (adapts per-parameter but burns out on long training runs), RMSProp (fixes AdaGrad's aggressive decay), AdaDelta (removes the need for a learning rate), NAdam (Nesterov-accelerated Adam), L-BFGS (second-order method for smaller datasets), and newer variants like AdamW (Adam with weight decay done right), RAdam (rectified Adam with warmup), and Lookahead (maintains fast and slow weights).

I'll cover these in future posts when the context is right, showing you not just what they do, but when and why to use them.

For now, Pick your learner, Start training.

Time to Experiment

Let's see this in action! I've built a neural network playground on MNIST with three optimizers— Batch SGD, Momentum, and Adam. Experiment with each and watch how they differ in training time and convergence speed.

Sample screenshot from the playground:

What Clicked for Me

Scale changes everything. Full-batch works for 4 examples, collapses at 60,000. Mini-batching is survival, not cleverness.

Adaptive learning makes sense. Not all weights should move equally. Adam adjusts per-parameter instead of treating everything the same.

And the progression is elegant:
Perceptron → multi-layer networks → backpropagation → scalable optimizers.

Each breakthrough made the next one possible.

What's Next

We can now train on real data. Backprop computes gradients, Adam updates weights. 99% accuracy on MNIST in seconds.

Everything works. Until it doesn't.

Train longer: training accuracy climbs to 99.8%, but test accuracy stalls and drops. The model isn't learning—it's memorising.

Next: why overfitting happens and how dropout and weight decay force networks to generalise instead of memorise.

Training a network is easy. Making it work in the real world? That's the challenge.

References

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536.
Tieleman, T., & Hinton, G. (2012). Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.
Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.

Tags: #MachineLearning #AI #DeepLearning #Optimization #SGD #Adam #Momentum #MNIST #NeuralNetworks

Backpropagation: Errors Flow Backward, Knowledge Flows Forward

Nilavukkarasan R — Thu, 19 Feb 2026 14:37:13 +0000

"The backpropagation algorithm was a key historical step in demonstrating that deep neural networks could be trained effectively."

-- Geoffrey Hinton

The Weight of the Problem

In my last post, I showed you how a multi-layer perceptron could solve XOR—something a single perceptron couldn't do. The network had 2 inputs, 2 hidden neurons, and 1 output. It worked beautifully.

But here's the catch: I hand-crafted those weights.

I sat there, adjusting numbers, testing combinations, until I found values that worked. For a tiny 2-2-1 network with 9 weights total, it took me hours of trial and error—and probably three cups of coffee I shouldn't have had.

Now imagine GPT-4. It has 1.76 trillion parameters.

If I spent one second per weight, it would take me 55,000 years to hand-craft GPT-4's weights. And that's assuming I got each one right on the first try (spoiler: I wouldn't).

This is the problem that haunted neural networks in the 1980s. We knew multi-layer networks could solve complex problems. We just didn't know how to train them.

Then, in the mid-1980s, came backpropagation. And here's the magic: it doesn't require you to know the right weights ahead of time. It learns them automatically.

The Breakthrough: Learning from Mistakes

Here's the beautiful insight: networks can learn from their mistakes. And that changes everything.

Think about learning to throw darts. Your first throw misses the bullseye by a foot. You don't randomly try a completely different throw. You adjust—a little less force, slightly different angle. You use the error (how far you missed) to guide your correction.

That's exactly what backpropagation does.

The process is simple:

Forward pass: Make a prediction with current weights
Calculate error: How wrong was the prediction?
Backward pass: Figure out which weights caused the error
Update weights: Adjust them to reduce the error
Repeat: Do this thousands of times

The magic is in step 3—figuring out which weights to blame. That's where the "backpropagation" name comes from: we propagate the error backward through the network.

The Chain Rule: Error Flows Backward

Backpropagation is often described as "just the chain rule from calculus." And it is! But let me make it concrete.

Imagine you're hiking and you want to go downhill (minimize loss). You're standing on a slope, and you need to know which direction is down.

Gradient descent is the strategy: always step in the direction that goes downhill.

Backpropagation is how you figure out which direction that is for every weight in your network.

Here's the flow:

Forward Pass (Making Predictions):
Input → Hidden Layer → Output → Loss
  x   →      h       →   ŷ    →  L

Backward Pass (Computing Gradients):
Loss → Output Error → Hidden Error → Weight Updates
  L  →      δ₂      →      δ₁      →   Δw

The error at the output layer is easy to compute: prediction - target. But how do we know how much each hidden neuron contributed to that error?

That's where the chain rule comes in. The error flows backward through the network, multiplied by weights and activation derivatives at each step. Each weight gets a gradient that tells it: "If you increase by a tiny amount, the loss will change by this much."

For the mathematically curious: I've written a detailed walkthrough with concrete numerical examples in BACKPROP_CALCULUS_EXPLAINED.md. It shows the full chain rule derivation with a 2-2-1 network solving XOR, complete with actual numbers flowing through each calculation.

Learning Rate: The Step Size

Once we know which direction to adjust each weight, we need to decide how big a step to take. That's the learning rate.

Think of it like adjusting the volume on a stereo:

Too high (learning rate = 1.0): You overshoot. The volume jumps from 2 to 10, then back to 1, then to 8. You never settle on the right level.
Too low (learning rate = 0.01): You're turning the knob so slowly it takes forever to reach the right volume.
Just right (learning rate = 0.3): You make steady progress toward the perfect volume.

In the playground, you can experiment with different learning rates and watch what happens. Too high and the loss bounces around. Too low and training crawls. Just right and you see that beautiful downward curve.

For practical tips: Check out HYPERPARAMETER_INSIGHTS.md for a deep dive into learning rates, architecture choices, and why some random seeds get stuck in local minima.

What Clicked for Me

After implementing backpropagation and watching it train on XOR, here's what became clear:

The loss curve tells the story. In Post 2, I hand-crafted weights and got 100% accuracy immediately. With backpropagation, I watched the loss start high (the network is guessing randomly) and gradually decrease as it learned. That curve going down? That's learning happening in real-time.

Initialization matters. I tried different random seeds and got wildly different results. Some converged to 100% accuracy in 2000 epochs. Others got stuck at 75% accuracy forever. The starting point matters—it's like starting a hike from different locations on a mountain. Some paths lead to the summit, others to local valleys.

It's the same algorithm everywhere. Whether it's XOR with 9 weights or GPT-4 with 1.76 trillion parameters, the algorithm is identical: forward pass, compute loss, backward pass, update weights. The scale changes, but the principle doesn't.

Automatic beats manual. Hand-crafting weights for XOR took me hours. Backpropagation learned them in seconds. For anything beyond toy problems, automatic learning isn't just better—it's the only option.

Watch It Learn: The Interactive Playground

I've built an interactive playground where you can watch backpropagation in action. It has two tabs, each showing a different aspect of learning.

GitHub Repository: perceptrons-to-transformers - 03-backpropagation

Tab 1: Training Visualization

Watch the network learn XOR from scratch. You'll see:

Loss curve decreasing over epochs (learning in action!)
Decision boundary evolving from random to correct
Final accuracy reaching 100% (when it works!)

Try this:

Train with learning rate 0.3 and seed 123 (recommended) - watch it converge smoothly
Try seed 42 with 2-2-1 architecture - watch it get stuck at 75% accuracy (local minimum!)
Switch to 2-4-1 architecture - notice how it's more robust to bad initialization

Tab 2: Gradient Flow Visualization

See the backward pass in action. This tab shows:

Forward pass step-by-step (input → hidden → output)
Backward pass step-by-step (error flowing backward)
Gradient magnitudes at each layer
Weight updates before and after one training step

This is where the "backpropagation" name becomes concrete. You literally see the error propagating backward through the network, computing gradients for each weight.

Try this:

Select different XOR test cases and watch how gradients change
Notice how gradients get smaller in earlier layers (vanishing gradient effect)
Compare gradient magnitudes with different learning rates

Running the Playground

# Clone the repository
git clone https://github.com/rnilav/perceptrons-to-transformers.git
cd perceptrons-to-transformers/03-backpropagation

# Install dependencies (if needed)
pip install -r ../requirements.txt

# Run the playground
streamlit run backprop_playground.py

Then open your browser and explore all two tabs. The playground is designed to make the abstract concrete—you can see learning happen, watch gradients flow, and understand why backpropagation works.

What This Unlocked

When Rumelhart, Hinton, and Williams published their backpropagation paper in 1986, it changed everything.

Before backpropagation, neural networks were theoretical curiosities. We knew multi-layer networks could solve complex problems, but we couldn't train them. It was like having a Ferrari with no key.

After backpropagation, neural networks became practical. Suddenly, we could:

Train networks with multiple hidden layers
Learn from large datasets automatically
Solve problems that were previously impossible

The progression is beautiful:

1958: Perceptron learns linear boundaries

1969: Minsky proves perceptrons can't solve XOR (AI winter begins)

1986: Backpropagation enables training multi-layer networks (AI winter thaws)

2012: Deep learning revolution (ImageNet breakthrough)

2017: Transformers architecture (foundation for GPT)

2023: ChatGPT and the LLM explosion

Every single one of these breakthroughs builds on backpropagation. GPT-4 is trained using backpropagation. DALL-E is trained using backpropagation. Every modern neural network you've ever used learned its weights through backpropagation.

The algorithm that learns 9 weights for XOR is the same algorithm that learns 1.76 trillion parameters for GPT-4. The scale changed, but the principle didn't.

What's Next

We can now train neural networks automatically. But there's still a lot to explore:

We can train networks now. But training them well—at scale, reliably, without overfitting—that's a different challenge. Next post, we'll see how modern optimisation algorithms solve this puzzle.

References

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536.
Nielsen, M. (2015). Neural Networks and Deep Learning. Determination Press. Available at: http://neuralnetworksanddeeplearning.com/
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Available at: http://www.deeplearningbook.org/

Tags: #MachineLearning #AI #DeepLearning #Backpropagation #NeuralNetworks #GradientDescent

Series: From Perceptrons to Transformers

Code: GitHub Repository

Multi Layer Perceptron: From Lines to Curves - The Hidden Layer

Nilavukkarasan R — Tue, 17 Feb 2026 15:14:10 +0000

"The perceptron has many limitations... the most serious is its inability to learn even the simplest nonlinear functions."

--Marvin Minsky

The Problem That Stumped AI

In my last post, I mentioned that the perceptron could learn AND, OR, and NAND gates perfectly. But there was one simple logic gate it couldn't learn, no matter how much you trained it.

That gate was XOR (exclusive-or).

XOR Truth Table:
┌─────────┬─────────┬────────┐
│ Input 1 │ Input 2 │ Output │
├─────────┼─────────┼────────┤
│    0    │    0    │   0    │
│    0    │    1    │   1    │
│    1    │    0    │   1    │
│    1    │    1    │   0    │
└─────────┴─────────┴────────┘

When Marvin Minsky and Seymour Papert published their book "Perceptrons" in 1969, they proved mathematically that single-layer perceptrons couldn't solve XOR. This revelation triggered the first "AI winter" - funding dried up, research stalled, and neural networks were largely abandoned for over a decade.

But why? What makes XOR so special?

The Geometry of Impossibility

Here's the thing: a perceptron draws a straight line to separate classes. That's it. One straight line.

For XOR, you need the output to be 1 when inputs are different, and 0 when they're the same:

Visual representation:
    Input 2
      ↑
    1 │  [1]    [0]
      │
    0 │  [0]    [1]
      └──────────────→ Input 1
         0       1

[0] = Output 0 
[1] = Output 1

Try drawing a single straight line that separates the class [1] from class [0]. You can't. The pattern is diagonal - you'd need two lines, or a curve.

This is what "not linearly separable" means.

I spent hours staring at this diagram when I first learned about it. I tried every angle, every position for that line. Nothing worked. And that's exactly the point - it's mathematically impossible. The perceptron's limitation isn't a bug, it's a fundamental constraint of linear classifiers.

For AND and OR gates, the pattern is simple - all the 1s are on one side, all the 0s on the other. But XOR? The classes are interleaved. You need a more sophisticated approach.

The Breakthrough: Hidden Layers

When I was a kid learning math, adding single-digit numbers was simple. 3 + 5 = 8. I just did it. One step, done.

But then came the leap to multi-digit addition: 27 + 15.

I kept getting it wrong. I'd add 2 + 1 = 3, then 7 + 5 = 12, and write 312. Completely wrong. My brain was treating it like two separate single-digit problems mashed together. I was missing something invisible.

Then came the breakthrough: 7 + 5 doesn't just equal 12. It creates a 1 that carries over to the next column. That invisible 1 moving from ones to tens column—that was the missing piece. Once I understood the carry, it clicked.

The carry was an intermediate step that transformed the problem.

It sounds trivial now. But back then? It was a massive leap for me. I couldn't see why single-digit rules didn't just scale up. I needed something new—not more of the same.

That's the hidden layer. But here's the catch:

If I just wrote down two addition problems and stacked them on top of each other, nothing changes. 2 + 1, then 7 + 5. That's still just two separate additions. Adding more steps doesn't help if each step is the same linear operation.

But the carry isn't linear. When 7 + 5 = 12, something special happens: the 1 doesn't stay in that column. It transforms—it becomes a 1 in a different column, changing what comes next. That transformation—that non-linearity—is what makes the whole system work.

Without the carry's transformation, stacking problems is useless. With it, multi-digit addition becomes possible.

That's exactly what non-linear activation functions do in neural networks.

A single-layer perceptron is like single-digit addition—inputs straight to output, no transformation. If you just stack more linear layers, you still have the same problem: one straight line, no matter how many you combine.

But add non-linear activation functions (like sigmoid or ReLU)—the carry transforms the space. Now XOR becomes solvable.

*Simple shallow network with hand-crafted weights and biases *

Solving XOR: The Aha Moment

With a 2-2-1 network (2 inputs, 2 hidden neurons, 1 output), we can finally solve XOR:

How it works:
┌──────────────────────────────────────┐
│ Hidden Neuron 1: Learns OR pattern   │
│   (fires when x₁ OR x₂ is 1)         │
│                                      │
│ Hidden Neuron 2: Learns AND pattern  │
│   (fires when x₁ AND x₂ are 1)       │
│                                      │
│ Output: Combines them                │
│   (OR but NOT AND = XOR)             │
└──────────────────────────────────────┘

The hidden layer isn't just adding complexity - it's transforming the problem into something solvable.

A comparative snapshot generated from playground

Try it yourself: Run the interactive playground to see the curved decision boundary in action. Adjust the weight slider to see how the boundary changes from weak to strong. Compare it with a perceptron's straight line attempt. The visualisation makes it clear why hidden layers are the breakthrough.

GitHub Repository: perceptrons-to-transformers - 02-multi-layer-perceptron

What you'll find:

02-multi-layer-perceptron/mlp.py - Clean MLP implementation
02-multi-layer-perceptron/mlp_playground.py - Interactive Streamlit app

The playground lets you:

See the curved decision boundary that solves XOR
Adjust weights and watch the boundary change in real-time
View the network architecture with all weights labeled
Compare perceptron's straight line vs MLP's curve

What This Unlocked

Solving XOR might seem trivial now. But it was the breakthrough that unlocked everything.

The problem wasn't just XOR. It was the realisation: hidden layers don't just add complexity—they enable non-linear thinking. Once researchers understood this, the floodgates opened.

In the 1980s, Geoffrey Hinton, David Rumelhart, and Ronald Williams proved you could actually train these multi-layer networks with backpropagation. Suddenly, problems that seemed impossible became solvable. The AI winter thawed.

The progression is beautiful:

Perceptrons learned to draw lines (linear boundaries)
MLPs learned to draw curves (non-linear boundaries)
Deep networks learned hierarchies (edges → shapes → objects → concepts)

Today's neural networks—from image classifiers to GPT-4—all follow the same principle: stack layers with non-linear activations to transform data into increasingly meaningful representations.

It all started with one insight: add a hidden layer. All from adding that first hidden layer.

What's Next:

We can now build networks that solve XOR. But there's one crucial question: How do we learn the weights?

The XOR network I showed you uses hand-crafted weights—I manually set values that worked. But for real problems with thousands of inputs and millions of weights, we can't do that by hand.

We need an algorithm that automatically learns the right weights from examples.

That algorithm is called backpropagation, and it's what makes neural networks practical. It's how networks learn from their mistakes and gradually improve.

In the next post, we'll dive into backpropagation—the algorithm that ties everything together. It involves calculus, but I promise to make it intuitive.

References

Minsky, M., & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press.
Nielsen, M. (2015). Neural Networks and Deep Learning. Determination Press. Available at: http://neuralnetworksanddeeplearning.com/

Tags: #MachineLearning #AI #DeepLearning #NeuralNetworks #MLP

Series: From Perceptrons to Transformers

Code: GitHub Repository

Perceptron: The Foundation of Modern AI

Nilavukkarasan R — Sun, 15 Feb 2026 08:40:21 +0000

"We now have a new kind of programming paradigm. Instead of telling the computer what to do, we show it examples of what we want, and it figures out how to do it."

-- Michael Nielsen

My Journey Back to the Beginning

My first encounter with Artificial Intelligence was during my college days. I had memorised more than I understood, but none of what I studied appeared in the exam, so I wrote whatever I could, and I’m quite certain the professor didn’t understand my answers either.

Fast forward 20 years of building software systems. In all that time, I barely touched AI/ML. Sure, I designed applications that integrated with black box, AI/ML systems for OCR, but that was it.

Then ChatGPT happened.

Like many of you, I started with the ChatGPT web interface, learning prompt engineering. Then I began experimenting—building RAG chatbots, exploring chunking strategies, testing different embedding models and retrieval techniques. I experimented with agents, explored MCPs and agentic patterns. I was learning these tools, building with them—but something bothered me.

I didn't understand how any of it actually worked.

So I decided to go back. Not to the latest paper or the newest framework, but to the very beginning. To the first artificial neuron.

Why This Matters

You might wonder why bother to learn about decades-old concept when we have ChatGPT, Claude and countless AI tools at our fingertips.

Here's why: Every single neuron in GPT-4, in every transformer, in every neural network you've ever used, works on the same basic principles as that first artificial neuron. The perceptron isn't history-It's the foundation.

Understanding it means understanding what's actually happening when you call an LLM API. It means knowing why things work, not just that they work.

If you've felt this same curiosity and want to truly understand the foundations beneath the tools we use every day, join me. Learning from first principles, one concept at a time.

From Biology to Silicon

In 1943, Warren McCulloch and Walter Pitts created the first mathematical model of a neuron. But it was Frank Rosenblatt in 1958 who built the perceptron, the first artificial neuron that could actually learn.

Rosenblatt's breakthrough came from mimicking nature. He studied how biological neurons work and translated that logic into mathematics. Here's how they compare:

Biological Neuron:

Dendrites  →  Cell Body  →  Threshold Check  →  Axon
(receive)     (process)     (fire if met)        (output)

Artificial Neuron (Perceptron):

Inputs     →  Weighted Sum  →  Threshold Check  →  Output
x₁,x₂,...     Σ(xᵢ × wᵢ)       (≥ threshold?)       0 or 1

The key insight: Learning happens by adjusting the weights.

How a Perceptron Works

Let's break it down to basics.

A perceptron takes inputs, multiplies each by a weight, adds them up, and makes a decision.

def perceptron_forward(inputs, weights, bias):
    # Multiply each input by its weight
    weighted_sum = sum(x * w for x, w in zip(inputs, weights))

    # Add bias (shifts the decision boundary)
    weighted_sum += bias

    # Activation: output 1 if positive, 0 otherwise
    return 1 if weighted_sum > 0 else 0

That's it. That's the core of a perceptron.

What's happening:

Each input has a weight (how important is this input?)
We sum up: (input₁ × weight₁) + (input₂ × weight₂) + ... + bias
If the sum is positive, output 1. Otherwise, output 0.

Example: AND gate

Let's say we want to implement the AND logic gate:

Input: [0, 0] → Output: 0
Input: [0, 1] → Output: 0
Input: [1, 0] → Output: 0
Input: [1, 1] → Output: 1

Traditional way (if/else):

def and_gate_traditional(input1, input2):
    if input1 == 1 and input2 == 1:
        return 1
    else:
        return 0

Perceptron way (learned weights):

With the right weights ([0.5, 0.5] and bias -0.7), the perceptron can solve this:

[0, 0]: 0×0.5 + 0×0.5 - 0.7 = -0.7 → Output: 0 ✓
[0, 1]: 0×0.5 + 1×0.5 - 0.7 = -0.2 → Output: 0 ✓
[1, 0]: 1×0.5 + 0×0.5 - 0.7 = -0.2 → Output: 0 ✓
[1, 1]: 1×0.5 + 1×0.5 - 0.7 = 0.3 → Output: 1 ✓

The difference? The traditional way is hardcoded. The perceptron learns these weights from examples. That's the new programming paradigm Nielsen talked about.

What Clicked for Me

After implementing and testing the perceptron, here's what became clear:

Weights are just numbers. There's no magic. A weight of 0.5 means "this input matters half as much as an input with weight 1.0."

The bias shifts the boundary. Without bias, the decision boundary always goes through the origin. Bias lets it move anywhere.

Learning is adjustment. When the perceptron makes a mistake, we adjust the weights. That's learning.

It's a linear classifier. The perceptron draws a straight line (or hyperplane) to separate classes. This is both its power and its limitation.

Explore the Code

I've implemented a complete perceptron from scratch with visualizations:

Here is the sample visualization screenshot from the playground

GitHub Repository: perceptrons-to-transformers

What you'll find:

01-perceptron/perceptron.py - Full implementation with learning algorithm
01-perceptron/perceptron_playground.py - Streamlit app to play with it

What's Next

The perceptron can learn AND, OR, and NAND gates perfectly. But it has a fundamental limitation.

No matter how you adjust the weights, there's one simple logic gate it cannot learn. This limitation exposed a critical weakness in single-layer networks.

In the next post, we'll explore this limitation and see why it led to the invention of multilayer networks.

Spoiler: The problem is called XOR, and solving it ultimately enabled path to modern deep learning.

References

Nielsen, M. (2015). Neural Networks and Deep Learning. Determination Press. Available at: http://neuralnetworksanddeeplearning.com/

Tags: #MachineLearning #AI #DeepLearning #Perceptron #NeuralNetworks

Series: From Perceptron to Transformers

Code: GitHub Repository