A Hopfield network from scratch, and why the textbook rule fails

A Hopfield network is the simplest model of memory that actually works. You give it a few patterns to remember, hand it back a corrupted version of one, and it settles, neuron by neuron, into the clean original. No training loop, no gradients. It does this by rolling downhill on an energy landscape where each stored memory sits at the bottom of its own valley. John Hopfield shared the 2024 Nobel Prize in Physics for this idea, forty years after he published it.

It is also a great thing to build from scratch, because the whole model is about thirty lines of numpy, and because the obvious way to build it is wrong in an instructive way. I stored five letters with the textbook rule and the network recalled nonsense. This is the story of why, and the one-line change that fixes it.

The model in one screen

A neuron is a single number, plus one or minus one. The network is a vector s of N of them, fully connected by a symmetric weight matrix W. Two pieces define the dynamics.

The first is an energy, a single number that measures how unhappy the current state is:

import numpy as np

def energy(s, W):
    return -0.5 * s @ W @ s

The second is an update. One neuron looks at every other neuron through the weights, takes their weighted vote, and flips to match its sign:

def update(s, W, i):
    s[i] = 1.0 if W[i] @ s >= 0 else -1.0

Here is the only fact that makes the whole thing work: because W is symmetric and we zero its diagonal, a single neuron flip can never raise the energy. You can prove it in two lines. Flip neuron i from s_i to s_i'. With a zero diagonal the only energy terms that change are the ones coupling i to the rest, and the change comes out to dE = -(s_i' - s_i) * (W_i . s). The update sets s_i' to the sign of W_i . s, so whenever the neuron actually flips, (s_i' - s_i) and (W_i . s) share a sign and their product is non-negative. dE is at most zero, always.

So the energy is a Lyapunov function. It only ever goes down, it is bounded below, and there are finitely many states, so the network cannot wander forever. It has to settle. This is also why we update one neuron at a time. Update all of them at once, synchronously, and the guarantee breaks: a synchronous Hopfield network can fall into a two-cycle, flapping between two states forever. Asynchronous updates, one random neuron per step, can only descend. Recall is just running the update on random neurons until a full sweep changes nothing:

def recall(s, W):
    s = s.copy()
    while True:
        before = s.copy()
        for i in np.random.permutation(len(s)):
            update(s, W, i)
        if np.array_equal(s, before):
            return s

That settled state is the answer. The interesting part is making the valleys land where you want them.

Hebb's rule, and the bug

The 1949 rule from Donald Hebb is "neurons that fire together wire together." In code it is an outer product per pattern, summed:

def hebb(patterns):           # patterns: (k, N) of +-1
    N = patterns.shape[1]
    W = sum(np.outer(p, p) for p in patterns) / N
    np.fill_diagonal(W, 0.0)
    return W

For a couple of random, roughly balanced patterns this works beautifully. So I stored five 14x14 letters, the characters I, W, T, L, P, corrupted the W by flipping 28 percent of its pixels, and ran recall.

It converged to a clean, stable pattern. The wrong one. The energy had dropped well below the energy of the stored W. The network had confidently recalled a memory that was not any of the five letters.

This is not a coding mistake. It is a real property of Hebbian storage, and the reason is worth understanding.

Why it fails

Letters are sparse. A 14x14 glyph is maybe 15 to 40 percent ink and the rest background, so as plus-one and minus-one vectors the five letters are not balanced and they are highly correlated with each other: they all agree on the large blank border. Hebb's rule builds W from sum(p p^T), and that sum is dominated by the direction all the patterns share, the mostly-blank background.

The consequence is that the deepest valleys in the energy landscape are not the letters. They are blends and the near-blank state, because those align best with the bias the sum baked in. The clean letters are still roughly fixed points, but they are shallow, and a 28 percent corruption is enough to slide out of a shallow basin into a deeper, spurious one next door. You can see it directly: feed the network a perfect, uncorrupted letter and sweep once. With five sparse letters stored, it drifts. The pattern you stored is not even stable.

Hebbian capacity is famously about 0.14 N for random balanced patterns. Five letters in 196 neurons is nowhere near that limit on paper. Correlation, not count, is what broke it.

The one-line fix

If the problem is that the stored patterns are not clean fixed points, then make them clean fixed points. We want a W such that W p = p for every stored p, that is, every memory is an eigenvector with eigenvalue one. The matrix that does this is the projector onto the span of the patterns, and there is a closed form, the projection rule (Personnaz et al., 1985):

def store(patterns):          # patterns: (k, N) of +-1
    P = patterns
    W = P.T @ np.linalg.pinv(P @ P.T) @ P
    np.fill_diagonal(W, 0.0)
    return W

P.T (P P^T)^{-1} P is exactly the orthogonal projection onto the row space of P. Any stored pattern lies in that space, so the projector leaves it untouched: W p = p. Each stored pattern is now an eigenvector of W with eigenvalue one, so the update leaves it fixed by construction, regardless of how correlated the patterns are.

The one requirement is that the patterns be linearly independent. Store more than N of them, or linearly dependent ones, and P P^T is no longer invertible; np.linalg.pinv quietly switches to its least-squares best, and that is exactly where the projection rule begins to degrade. Within that limit it is exact, which is all five letters needed.

Same five letters, same code everywhere else, swap hebb for store. Now every letter is stable, and the corrupted W heals back to a perfect W with the energy falling monotonically to the floor. The whole demo that did not work, works.

The cost is honesty about what changed. Hebb's rule is local and biologically suggestive: each weight only needs the two neurons it connects. The projection rule needs a matrix inverse, a global computation, which is not something a synapse can do on its own. You trade a plausible learning rule for a correct one. For a from-scratch demo that is the right trade, and it is worth saying out loud rather than hiding the pinv and pretending Hebb was enough.

What you actually built

A Hopfield network is a content-addressable memory. A normal array is addressed by index: you ask for slot 7. This is addressed by content: you hand it a smudged, partial version of a thing and it returns the nearest complete thing it knows. That is a different and more brain-like way to store information, and it falls out of nothing more than a symmetric weight matrix and an energy that only goes down.

Two honest limits worth keeping in mind. First, capacity: push past it and the valleys merge, and the network returns ghost patterns, stable states you never stored. These are not random noise. The classic ones are mixture states, the sign of an odd combination like p1 + p2 + p3, which are genuine fixed points of the dynamics, real valleys in the landscape that you simply never asked for. Second, the basins are finite: corrupt a pattern far enough and it settles into the wrong memory, which is a feature if you call it nearest-neighbor recall and a bug if you expected magic.

The idea did not stay in the 1980s. The "modern Hopfield networks" of 2020 replace the quadratic energy with a log-sum-exp over the stored patterns, which sharpens every valley enough that exponentially many patterns can coexist without merging. Work out the update rule that descends that energy and a softmax falls out: the stored patterns play the role of keys and values, the corrupted probe you hand in is the query, and a single step of recall is one layer of attention. The dot-product attention inside a transformer is a Hopfield network with a better energy. The line from this thirty-line toy to the models running today is shorter than it looks.

If you want to build it yourself, the satisfying version is: rasterize a few letters into plus-one and minus-one grids, store them with both rules, and watch the difference. The failure is the part that teaches you something, so do not skip it.