I built an attention head out of a 1982 associative memory and measured where it breaks

The second-biggest story on Hacker News this morning (590 points) is titled "You could have invented transformers from memory" — the argument, in one line, is that attention is what you get when you take a classical associative memory and make its retrieval rule differentiable. It's a lovely essay. It's also a claim with a number attached, and claims with numbers attached are my department. So I built the thing: a modern Hopfield network — the 1982 idea, updated — and an attention head, in the same file, and measured both where they are identical and where the memory breaks.

The equivalence is not approximate. It's exact.

A modern Hopfield update takes a cue and returns a weighted mix of stored patterns, with the weights given by a softmax over how well the cue matches each pattern. A single attention head takes a query and returns a weighted mix of values, weights given by a softmax over query-key matches. Set keys = values = the stored patterns, query = the cue, and these stop being two mechanisms that resemble each other:

import numpy as np

def softmax(x):
    x = x - x.max(); e = np.exp(x); return e / e.sum()

def hopfield_step(X, cue, beta):
    return X.T @ softmax(beta * (X @ cue))   # retrieve: mix patterns by match

def attention(X, cue, beta):
    return softmax((X @ cue) * beta) @ X     # attend: mix values by match

equivalence: max|hopfield - attention| = 0.000e+00

Not 1e-9. Zero. The two implementations produce bit-identical output on the same inputs, because they are the same computation written in two dialects, forty years apart. If you have ever implemented attention, you have implemented an associative memory; the transformer paper's contribution was noticing you could stack the retrieval, learn what gets stored, and do it for every position in parallel. The essay's premise survives contact with numpy.

Where the memory breaks is more interesting

An associative memory you can't overload is a database with extra steps. So I swept it: store N random unit patterns in dimension 64, corrupt one with Gaussian noise, run one retrieval step, and count a success only if the output lands on the right pattern with cosine similarity above 0.9. Two hundred trials per cell, same seed throughout:

retrieval success (d=64, noise=0.3, beta=16):
  N=  10  success=0.90
  N=  25  success=0.86
  N=  50  success=0.79
  N= 100  success=0.66
  N= 200  success=0.59
  N= 400  success=0.55
  N= 800  success=0.39

First surprise: there is no cliff. The classic Hopfield rule of thumb says the old binary network stores about 0.14 × d patterns — at d=64, roughly 9 patterns — before it falls apart. The continuous, softmax version is still retrieving the right pattern more than half the time at N=200, twenty times the classic capacity, and degrades like a slope instead of a wall. That's the entire reason attention scales to long contexts: the failure mode is graceful.

Second surprise: what kills it isn't how much you store. It's how dirty the cue is:

noise sweep (d=64, N=100, beta=16):
  noise=0.1  success=1.00
  noise=0.3  success=0.69
  noise=0.5  success=0.29
  noise=0.8  success=0.10
  noise=1.2  success=0.03

Going from 10 stored patterns to 800 costs you about half your accuracy. Going from a clean cue to a noisy one costs you essentially all of it. And the sharpness knob — beta, the inverse temperature, the thing attention implements as the 1/√d scale — helps, but with diminishing returns:

beta sweep (d=64, N=100, noise=0.5):
  beta= 1.0  success=0.00
  beta= 4.0  success=0.18
  beta=16.0  success=0.28
  beta=64.0  success=0.36

At beta=1 the softmax is so flat that retrieval never once cleared my 0.9 bar in 200 trials. Sharpening four-fold from 16 to 64 buys eight points. Temperature is a real lever; it's just a weak one compared to cue quality.

Why I care about a 1982 memory model

Read those three tables as a description of attention in a production model and they stop being history. Long context is the N sweep: stuffing more into the window degrades retrieval gradually, not catastrophically — which matches everyone's lived experience that models get vaguely worse, not suddenly broken, as context fills. Prompt quality is the noise sweep: a sloppy, contradictory cue doesn't cost you a little accuracy, it costs you almost all of it, and no amount of temperature tuning buys it back. And the flat-softmax failure at beta=1 is why the scaling factor in attention is not decoration.

The essay says you could have invented transformers from memory. After running it, I'd put it differently: the transformer is the memory, and its known weaknesses — dilution over long contexts, sensitivity to prompt noise — are not mysteries of deep learning. They're the measured failure modes of an associative memory, visible in 60 lines of numpy, in a model old enough to be the transformer's parent.

The full script is the three functions and sweeps above; every number in this post reproduces from seed 7.