In Part 1, one neuron drew one straight line, and XOR broke it. The fix was simple to state and hard to train: stack neurons into layers, and teach every layer at once. In 1986, Rumelhart, Hinton and Williams showed how.
1. Stack the neurons
A multilayer perceptron adds one or more hidden layers between the inputs and the output. Each hidden unit draws its own line, and the output combines them.
2. XOR, finally solved
Two hidden units draw two lines. The output fires only in the band between them, which is exactly where the two XOR 1s sit.
3. Backprop: blame travels backwards
Backprop runs the chain rule in reverse. The error at the output flows back through every layer, and each weight moves in proportion to its share of the mistake.
4. The idea waited twelve years
Werbos derived backprop in his 1974 thesis, but the field was not listening. The 1986 Nature paper made it impossible to ignore, and Cybenko proved in 1989 that one hidden layer can approximate any continuous function.
Key takeaway
Layers plus backprop turned a toy that could only draw straight lines into a machine that can learn almost any function. Every deep network we train today is this idea, scaled up.
What is Next in This Series
Next is Part 3, CNNs, Visualized, where the same layers learn to see. To see where it all started, read Part 1, The Perceptron, Visualized. For the deeper story of how these ideas grew into modern LLMs, see How LLMs Actually Work.
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)