Quick recap, if you're just arriving: Blog 1 showed that every core idea in reinforcement learning was discovered by someone watching animals, not computers. Thorndike's cats taught us that learning is selection, not insight. Pavlov's dogs taught us that a signal can carry information about the future. Skinner's pigeons taught us that big behaviors are built from small rewarded steps. Hebb's neurons taught us that learning lives in the strength of a connection.
Four beautiful observations. Not one of them tells you how to build something that does this.
This is the decade where people tried to build it anyway. What followed was ten years of beautiful, clunky, brilliant hardware — machines that stumbled through mazes, adjusted their own wiring, and learned, in the most primitive sense, from their mistakes. None of the people building them knew they were inventing Reinforcement Learning. They were just trying to make something move on its own.
The Missing Link: Why "a signal predicts the future" Needed a Number
Before we jump to the machines, one 20-second detour — because it's the cleanest bridge between Blog 1 and everything below.
Pavlov noticed his dogs salivated at footsteps, before food even showed up. Nice story. But it doesn't tell you how much the dog should update its expectation each time, or why the association eventually stops getting stronger.
Decades later — 1972, well after everything in this post — psychologists Robert Rescorla and Allan Wagner finally wrote that down as a formula:
In plain words: how much you learn from an event equals how surprising it was. V is what you currently expect, λ is what actually happened, and (λ - V) is the surprise — the gap between prediction and reality. If a bell always predicts food, eventually there's no surprise left, and no more learning happens. That gap term is the whole engine.
Why mention a 1972 psychology paper in a post about 1948–1954 engineering? Because the people you're about to meet — Wiener, Turing, Minsky, Shannon — independently backed into the exact same idea from a completely different direction: engineering, not biology. None of them had read Pavlov's fine print. They just kept discovering, on their own, that intelligence seems to boil down to measuring a gap and closing it. Keep that gap-closing idea in your head — it's the thread connecting every section below.
The Cybernetic Spark: Wiener's Feedback Loops (1948)
In 1948, mathematician Norbert Wiener published Cybernetics: Or Control and Communication in the Animal and the Machine, and made a claim that was radical for its time: the same mathematical principle governs both living organisms and machines — negative feedback.
Forget dogs and food for a second. Think of a thermostat. It doesn't "know" temperature in any deep sense — it just tracks one number obsessively: the gap between where it is and where it's supposed to be.
Where T is the target state and S is the current state. If D ≠ 0, the system fires a correction to close the gap. That's it. That's the whole idea.
This is exactly the Rescorla-Wagner gap from above, just wearing an engineer's clothes instead of a psychologist's. D here is the direct ancestor of the Temporal Difference (TD) error you'll see in modern RL — the signal that tells an agent "something's off, adjust." Every gradient update in every deep RL system running today is, underneath the math, a Wiener-style error minimization.
Wiener's punchline: Intelligence isn't magic. It's error minimization on a loop. If you can define a goal and measure your distance from it, a machine can "learn" to close that distance.
So Wiener gave the field a drive — a reason to move, a number to chase to zero. But a drive isn't a mechanism. It doesn't tell you what happens inside the machine when it's wrong versus when it's right. That's exactly what Alan Turing was working out, in the very same year, from a completely different angle.
The Turing Blueprint: Pain, Pleasure, and the First Neural Network on Paper (1948)
Turing's 1948 paper Intelligent Machinery introduced the P-type (Pain-type) Unorganized Machine, and reading it today feels like finding the first design document for a neural network, forty years too early.
Turing's argument was blunt: a human infant's cortex is mostly a random, disorganized network that only becomes useful through experience. So why not build the machine equivalent? He proposed a network built from NAND gates — think of these as the smallest possible decision units, each taking two inputs and spitting out one output based on a simple rule. Each gate could sit in one of two internal "configuration states," and that state is what determined how the gate behaved.
The learning happened through two signals, and this is the part that matters:
Pain — when the machine got something wrong, the gates that had just fired got scrambled at random:
Pleasure — when the machine got something right, those same configurations froze in place. No more random reshuffling:
So: wrong answer, shake things up. Right answer, lock it in. It's almost embarrassingly simple — which is exactly why it's important.
Turing also spotted the obvious problem immediately: pure random shuffling doesn't scale. If a machine has n units, each with k possible states, the number of configurations to search through is k^n — a number that explodes into uselessness almost immediately, even for tiny machines.
His fix: bring in a teacher. Instead of letting the machine flail randomly forever, a teacher hands out Pleasure signals early to nudge it toward useful sub-behaviors, and then Pain prunes away the remaining mistakes. That's not just a clever patch — it's the seed of Supervised Fine-Tuning (SFT) and Imitation Learning, ideas that wouldn't get their modern names for another fifty-plus years. Turing had quietly moved the burden of "being smart" off the programmer's shoulders and onto the environment's.
His paper sat mostly unread for decades. But look at the shape of it: a randomly wired network, shaped over time by reward and punishment, converging on useful behavior. Every RLHF pipeline running behind a modern chatbot is, structurally, still running Turing's 1948 loop.
Wiener gave us the why — close the gap. Turing gave us the what happens inside — freeze on success, scramble on failure. What nobody had shown yet was a machine that could do this outside of a chalkboard, with actual physical parts that changed themselves. That's where Marvin Minsky comes in.
Minsky's SNARC: The First Machine That Rewired Itself (1951–1954)
While Turing's ideas stayed on paper, Marvin Minsky and his Princeton colleague Dean Edmonds went to the workshop and actually built something.
In 1951, they built the SNARC — Stochastic Neural Analog Reinforcement Calculator — the first machine to physically implement Hebbian learning (remember Blog 1: neurons that fire together, wire together) combined with reinforcement, in real hardware. Minsky later wrote it up in his 1954 thesis.
The SNARC simulated a rat finding its way through a maze using 40 artificial synapses. Each "synapse" was a vacuum tube paired with a potentiometer — picture a volume knob, a simple dial that changes how much resistance flows through a circuit. That dial's position was the connection's strength, standing in for what we'd now call a "weight."
Here's the learning loop, in plain terms:
- A signal representing the "rat" travels through the network, picking a path.
- If that path leads to reward, the potentiometers ("volume knobs") along that path get turned to lower resistance — meaning that path becomes electrically easier to travel next time.
- If the path leads nowhere, those knobs stay put or drift the other way.
Notice what's happening here: this is credit assignment, done with actual electricity. The machine had no rule anywhere that said "this specific connection caused the reward." It used the reward signal traveling backward through whichever path was active to figure out which knobs to turn. That's structurally the same job that backpropagation does in a modern neural network — just with copper and vacuum tubes instead of calculus.
The SNARC couldn't scale — 40 synapses is barely a network by today's standards, and analog parts drift and degrade over time. But it proved something that had never been physically shown before: a machine can change its own internal wiring in response to reward, with no one programming the change directly.
That single idea — connections that adjust themselves because of reward — is the founding premise of every neural network trained with RL today.
So now we've got a drive (Wiener), a mechanism (Turing), and proof that self-adjusting hardware works (Minsky). One year later, Claude Shannon showed what happens when you point that same idea at something a person can actually watch move.
Claude Shannon's Theseus: Learning You Can See (1952)
Shannon's contribution was the most tangible of the whole decade. In 1952, he built Theseus — a copper mouse that could navigate a 5×5 maze, controlled by a cabinet of 75 telephone relays sitting underneath.
Here's the trick: the mouse itself had no brain. The relays — think of a relay as a mechanical switch that can remember whether it's "on" or "off" — were the brain. Each of the 25 cells in the maze corresponded to a relay setup that remembered which directions had already failed from that spot. Hit a wall, and the relay for that direction-at-that-position flipped off, so the mouse tried something else next time. Find a working path, and those relay settings locked in for good.
The payoff: drop Theseus anywhere in a maze it had already solved, and it walked straight to the goal, no hesitation. Drop it somewhere new, and it explored from scratch — then remembered that too.
What Shannon had actually built was a physical lookup table: a direct map from state (position in the maze) to action (which way to go), stored as the mechanical position of 75 relays. This is the hardware great-grandparent of what's now called a Q-table.
But the genuinely clever part was the "anti-neurotic" circuit. Shannon noticed that a strategy based purely on "eliminate what failed" could trap the mouse in an infinite loop — try North, fail, try East, fail, loop back to North forever, repeat. His fix: a circuit that detected this looping behavior and injected a random direction change to break it.
This is, as far as anyone can tell, the first deliberate hardware solution to exploration vs. exploitation — the tension that sits at the center of nearly every RL algorithm ever written. In plain terms: should the mouse keep using the move it already knows works (exploit), or try something new in case there's a better option (explore)? Shannon's answer was a random kick when stuck. Modern RL answers with names like epsilon-greedy (occasionally pick a random action on purpose) or entropy bonuses (reward the agent slightly for staying unpredictable). Same problem, seventy years of better tools.
Theseus proved something important: "learning" could be reduced entirely to hardware state changes. No probability distributions, no gradients — just relays flipping. Intelligence as a lookup table.
The Bridge (1954–1957): From Wires to Equations
By the mid-1950s, the hardware approach was hitting a wall. Shannon's mouse could solve one specific maze. Minsky's SNARC topped out at 40 synapses. Turing's ideas mostly lived on paper. Every one of these machines was an impressive, custom-built solution to one specific problem — and none of them generalized.
Richard Bellman, among others, got impatient with that. Building a maze-solving mouse was a great parlor trick, but it wasn't a theory. What he wanted was a general rule — something that could handle any problem made of a long chain of decisions, no matter what the environment looked like underneath.
The question shifted from "can we make this specific thing work?" to "can we prove the mathematically best way to act, in general?"
Bellman's answer became the Principle of Optimality and the Bellman Equation: the idea that the value of being in a state is just the immediate reward, plus the (slightly discounted) value of wherever you end up next. He took the mechanical "vibes" of the SNARC and Theseus and translated them into a single recursive formula — one equation that compresses an agent's entire future into a present-tense number.
That equation, and the computational nightmare it revealed (Bellman's own term for it: the Curse of Dimensionality), is where Blog 3 picks up. For now, the point is simpler: after a decade of wiring things together by hand, the field realized it needed math to go any further.
The wires were beautiful. But equations scale.
The Decade in One Frame
Zooming out, here's what this era actually built — not a summary, a thesis:
Wiener (1948) gave learning a drive: minimize the gap on a feedback loop.
Turing (1948) gave learning a mechanism: reward freezes, punishment randomizes.
Minsky (1951/54) gave learning physical weight: the first adjustable synapses that existed as hardware, not theory.
Shannon (1952) gave learning memory: state-action mappings stored in the physical position of relays.
Bellman (1957) gave learning math: a recursive formula that would eventually unify everything above it — that's Blog 3.
In under a decade, "learning" went from a biological observation about cats and pigeons to a wired, probabilistic, physically real process you could point to on a workbench. The machines were small, fragile, and each one could only do exactly one thing. But the ideas inside them were load-bearing — every one of them is still holding up the RL systems built today.
Top comments (0)