The enormous internal values that appear inside every large language model, and the attention sinks they produce, can be traced to one specific layer. Researchers including Ruixiang Tang have identified what they call the Massive Emergence Layer, a layer consistently observed across different model families where these outsized activations first appear before propagating deeper through residual connections. Loosening the rigidity of the token that carries them improves instruction following and math reasoning, in both training-free and fine-tuned settings.
Key facts
- The paper names a Massive Emergence Layer, present consistently across model families, where massive activations first appear.
- Inside that layer, RMSNorm and feed-forward parameters jointly produce the effect, rather than either one alone.
- The proposed fix improves performance on instruction following and math reasoning in both training-free and fine-tuning settings.
- Posted 8 May 2026, revised 12 May, as arXiv:2605.08504, titled "A Single Layer to Explain Them All: Understanding Massive Activations in Large Language Models."
- It builds on two earlier landmarks: Efficient Streaming Language Models with Attention Sinks, which named the phenomenon, and Massive Activations in Large Language Models, which documented the outsized values themselves.
Open up a working language model and look at the numbers flowing through it, and you find something that looks like a bug. In a sea of values clustered around zero, a handful are thousands of times larger, concentrated in a few dimensions of a few token positions. They have been documented for years. They are not noise, because deleting them wrecks the model. And they show up in essentially every large transformer regardless of who trained it.
They come with a companion phenomenon. Attention sinks are tokens, usually at or near the start of a sequence, that absorb a large share of the model's attention no matter what they contain. The first token of a prompt is often mostly ignored semantically and heavily attended to mechanically. This is inconvenient in practice, because it constrains how you can trim, cache or slide a context window without breaking the model, and it is the reason several long-context techniques have to keep the first few tokens around forever. We cover the phenomenon from the ground up in our lesson on attention sinks.
The prevailing explanation has been functional: the model needs somewhere to park attention when no token is especially relevant, so it designates a scratch position and dumps attention there. Useful as a description, unsatisfying as a mechanism, because it does not say where the giant numbers come from or why they appear in the same place in models trained independently by different labs.
This paper answers that structurally. There is a specific layer where the massive activations emerge, it is consistent across families, and within it the responsibility is shared: both the normalization parameters and the feed-forward parameters contribute jointly to producing them. Once created, they ride downstream through residual connections, which is why they appear to be everywhere at once when you look at a late layer.
The finding that turns this from a curiosity into an intervention is what happens next. Once formed, the massive-activation token's representation stays largely invariant across layers. It stops changing. That is the mechanism connecting the two phenomena, and it is also a cost: a token that never updates reduces the diversity of hidden representations reaching the attention module, so a slice of the model's capacity is spent carrying a constant. Imagine one instrument in an orchestra holding the same fortissimo note for the entire piece. Everyone can hear it, everyone plays around it, and it contributes almost no information.
Motivated by exactly that, the authors propose a method to reduce the rigidity of the massive-activation token. It improves performance on instruction following and math reasoning across multiple tasks, and it works without retraining as well as with fine-tuning. It also, in their words, "mitigates attention sinks by selectively weakening their influence," which they present as evidence that sinks originate at the hidden-state level rather than in the attention computation itself.
The caveat is that massive activations have been explained several times already, and each account has isolated a real contributor without turning out to be the whole story. "A single layer to explain them all" is a confident title for a result demonstrated on a set of open model families, and the reported performance gains are modest enough that a competing or additional mechanism is not excluded. Independent replication on models outside the tested set is the thing to watch for.
Even discounted, it is a good week for structural interpretability. This result and MIT's finding that large language models organize into brain-like functional modules make the same kind of argument from opposite ends of the model: something everyone had filed under "weird artifact" turns out to be load-bearing structure with a traceable origin and a causal test. That is a meaningfully higher standard than the correlational probing that dominated interpretability work two years ago.
Background: our lessons on mechanistic interpretability and why LLM inference is memory-bound.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)