DEV Community

Cover image for The Multilayer Perceptron: How Backprop Cracked XOR
Muhammad Saud
Muhammad Saud

Posted on Originally published at logicstack.org

The Multilayer Perceptron: How Backprop Cracked XOR

In Part 1, one neuron drew one straight line, and XOR broke it. The fix was simple to state and hard to train: stack neurons into layers, and teach every layer at once. In 1986, Rumelhart, Hinton and Williams showed how.

1. Stack the neurons

A multilayer perceptron adds one or more hidden layers between the inputs and the output. Each hidden unit draws its own line, and the output combines them.

2. XOR, finally solved

Two hidden units draw two lines. The output fires only in the band between them, which is exactly where the two XOR 1s sit.

3. Backprop: blame travels backwards

Backprop runs the chain rule in reverse. The error at the output flows back through every layer, and each weight moves in proportion to its share of the mistake.

4. The idea waited twelve years

Werbos derived backprop in his 1974 thesis, but the field was not listening. The 1986 Nature paper made it impossible to ignore, and Cybenko proved in 1989 that one hidden layer can approximate any continuous function.

Key takeaway

Layers plus backprop turned a toy that could only draw straight lines into a machine that can learn almost any function. Every deep network we train today is this idea, scaled up.

What is Next in This Series

Next is Part 3, CNNs, Visualized, where the same layers learn to see. To see where it all started, read Part 1, The Perceptron, Visualized. For the deeper story of how these ideas grew into modern LLMs, see How LLMs Actually Work.


Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Top comments (0)