Someone in a thread says attention is just a Hopfield network, and the next reply runs with it: so the model already has associative memory, so an agent does not need anything else to remember. Both sentences sound like the same claim. Only the first one is true, and only in a narrow sense that is worth stating exactly, because the narrow version is useful and the loose version leads people to wrong architecture decisions.
One line to carry: one update step of a modern continuous Hopfield network equals scaled dot-product attention under a specific identification of queries, keys and values; that is an identity of operation, not a history of how transformers were built, and it does not turn a context window into memory that outlives the conversation.
The identity, stated precisely
The result is from Ramsauer and colleagues, in a 2020 paper titled Hopfield Networks is All You Need that appeared at ICLR 2021. The abstract states it in one sentence:
The new update rule is equivalent to the attention mechanism used in transformers.
The update rule of their modern Hopfield network is ξ_new = X softmax(β Xᵀξ). Read it as: compare the current state with every stored pattern, turn the similarities into softmax weights, return the weighted sum of patterns. Now identify the stored patterns with keys, the state with a query, add the value projection, and set β = 1/√d_k. The update becomes softmax(QKᵀ/√d_k)V, which is scaled dot-product attention.
Under those substitutions the equality is exact, at the level of that one operation. A query attends over keys and returns a weighted blend of values. A Hopfield state updates by attending over stored patterns and returning their weighted blend.
What the identity does not say
It is one update step. The paper's own text says that "retrieval with one update is compatible with activating the layers of deep networks", and its theorems bound how close a single update gets to the stored pattern when patterns are well separated. It is a statement about a step, not about a network iterating to convergence.
It is not a lineage. Attention was introduced in Attention Is All You Need in 2017, three years before the equivalence was shown. The paper's first figure uses an equality sign and explains it in the caption: the sign means "keeps the properties". The two were found to be the same operation after the fact.
It covers the operation and nothing around it. Residual connections, layer normalisation, the feed-forward blocks, causal masking and training dynamics are outside it. Later work narrows the conditions under which the identity holds rather than overturning it.
The title itself is a nod to the 2017 paper, and it belongs to a genre large enough to have been measured: a December 2025 preprint, All You Need is Not All You Need for a Paper Title, counts 717 arXiv titles containing the phrase between 2009 and 2025, 200 of them in 2025 alone, and argues the format favours "memorability over precision". This is one of the rare cases where the memorable title is also the theorem.
Three capacity numbers that get merged into one
Ask how many patterns a Hopfield network can store and you will see three different answers quoted as if they were the same. They answer three different questions.
| source | capacity | what it tolerates |
|---|---|---|
| Hopfield, 1982, simulations at N = 30 and 100 | about 0.15N | until recall errors become severe |
| Amit, Gutfreund and Sompolinsky, 1985 and 1987 | αc ≈ 0.138, often rounded to 0.14, so about 0.138N | a small fraction of bit errors, under 1.5 % at zero temperature |
| McEliece, Posner, Rodemich and Venkatesh, 1987 | n/(2 ln n) | none: exact recall of most memories |
There is no contradiction between them. The first is an empirical rule of thumb from small simulations. The second is a statistical mechanics result that allows a little noise in what comes back. The third demands exact recovery, so it is smaller and grows more slowly. The modern variants changed the question altogether: dense associative memories raised capacity to polynomial and then exponential scaling, and Ramsauer and colleagues prove exponential capacity in the dimension of the space for continuous states. If a post quotes 0.14N as the capacity of the network inside a transformer, it has taken the number from the wrong row of a table that stopped at 1987.
A vocabulary for what retrieval returns
The most practical part of the paper is not the equivalence but the taxonomy of fixed points it gives for the update. A query can settle on a single stored pattern, the clean case, when one pattern is close and well separated. It can settle on a metastable state that averages a subset of similar patterns. Or, when nothing is well separated, it can settle on a global average of everything stored.
The authors then used that lens on trained models. The abstract reports that attention heads "perform in the first layers preferably global averaging and in higher layers partial averaging via metastable states". A blend, in this reading, is an operating regime of the retrieval step and not automatically a malfunction: when the stored patterns are not separated enough, the step returns an average of a subset rather than one of them. That is a statement about one retrieval step. It is useful vocabulary, and it is not an explanation of every wrong answer a model gives.
What this means for an agent's memory
It gives a clean boundary. Attention retrieves associatively over what is in the context right now. It does that very well, and the Hopfield reading explains why. It does not keep anything once the context is gone.
Google Research's Titans paper draws the same line in its abstract:
From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory.
So the reply in the opening thread has the direction backwards. Attention being a Hopfield update is a statement about how a model retrieves inside a window. What an agent needs to have in the window next week, after a restart, in a different session or a different tool, has to be stored somewhere that survives the session and put back in front of the model when it matters. The identity tells you what the retrieval step does with whatever gets there. It says nothing about getting it there.
Where did you last see the capacity of a Hopfield network quoted, and which of the three rows was it actually from?
Disclosure: I work on Mnemoverse, a memory service for AI agents, which is the kind of system the last section is about, so weigh that section accordingly. The longer version, with every paper and the full capacity table, is on our library, and the MCP server is open source (MIT): github.com/mnemoverse/mcp-memory-server.
Top comments (6)
The three-row capacity table is the most clarifying thing in this post. 0.15N from 1982 simulations, 0.138N from the 1985/87 statistical mechanics result, n/(2 ln n) from the exact-recall bound — these answer three different questions and are not in conflict, but they get merged constantly because readers take whichever number appeared in the text they read last. The modern continuous Hopfield exponential capacity result is the fourth row most people are missing entirely.
The "direction backwards" correction in the last section is the payoff. Attention being a Hopfield update explains what the retrieval step does with what's in the context window now. It says nothing about how information gets into the window next session, after a restart, or across tool boundaries. Those require a different kind of storage. The Titans framing (attention as short-term, neural memory as long-term) is the cleaner practical distinction.
The taxonomy of retrieval fixed points — single pattern, metastable subset average, global average — is also useful vocabulary for reasoning about why attention behaves the way it does in practice, especially the global-averaging finding for early layers.
Three numbers answering three different questions, and readers taking whichever one they read last, is exactly the failure the table was built to stop. The fourth row you name is the one I left out, and leaving it out was a choice I would make again for a different reason than length: the modern exponential result changes what the model can hold, not what happens to it when the process ends, and the article is about the second thing.
Your reading of the last section is the one I hoped for. An update step being a Hopfield update explains the retrieval that happens inside the window that already exists. It says nothing about how anything gets into the next window, and the gap between those two sentences is where most of the confusion in this area lives. The Titans framing is cleaner, and the reason it is cleaner is that it names the boundary rather than blurring it.
On the fixed-point taxonomy, the global-averaging finding for early layers is the part I keep coming back to, because it gives a mechanical reason for behaviour people describe in vague terms. An early layer that averages across a metastable subset is not attending to a fact, and a prompt that assumes it is will read as ignored in exactly the way the rules articles keep describing.
@izgorodin the identity clicked. The leap right after it didn't. Ran into this building agent memory for CogniRunner, a colleague swore a long context window already covered it because attention does the retrieval. It doesn't survive a restart. That's the part the loose reading drops every time. One update step equal to attention says nothing about what happens once the process ends. Good you pinned the exact substitution. Most threads just wave at the paper title.
The colleague argument is the common one, and the reason it survives is that it is half right. Attention does do the retrieval, and inside one context window it does it well enough that nothing looks missing. The half that fails is not retrieval quality, it is that the window is bounded by the life of the process.
A long window makes the boundary later, which reads as if it made the boundary go away. That is why the substitution matters: one update step equals attention is a statement about what happens to state that is already in the window, and it carries no claim at all about state that has to exist before the window is built. Restart, new machine, different tool, the identity is still true and completely unhelpful.
CogniRunner will hit the same line at the point where a fact has to survive something the process does not survive, and that is where the storage question starts rather than ends.
The "not automatically a malfunction" framing is the part worth sitting with -- if metastable averaging only shows up when stored patterns aren't separated enough, that's a testable claim about the store, not just the retrieval step. It reads directly onto agent memory systems that let near-duplicate facts (the same thing restated three different ways across sessions) sit in the same index: that's precisely the insufficiently-separated precondition the paper describes, so dedup/embedding-separation at write time is doing real work, not just tidiness.
Didn't know the 717-title stat either -- good to have that one sourced properly for once instead of repeated as a joke.
Reading it onto the store rather than the retrieval step is the useful move, and it turns the paper's precondition into something you can measure on your own index. If metastable averaging needs patterns that are not separated enough, then near-duplicate facts written across sessions are that precondition built on purpose, one restatement at a time.
The check is cheap: take the embeddings of the facts in one topic and look at the pairwise similarity of things that should be distinct. If restatements of one fact sit closer to each other than distinct facts do, dedup at write time is doing real work. If distinct facts sit as close as restatements, the problem is not duplication, it is that the embedding cannot tell them apart, and deduplication will merge things that should stay separate.