Goodfire researcher Tom McGrath, addressing the line circulating in interpretability circles that "SAEs are dead," says the phrase is shorthand rather than a verdict. In an interview on Machine Learning Street Talk, McGrath attributes it to researcher Neel Nanda and immediately qualifies it: the likely meaning is that sparse autoencoders are not the answer to everything, while remaining "pragmatically useful." The substantive shift underneath the meme is a move from decomposing models into features toward studying their geometry.
Key facts
- Goodfire's research frames sparse autoencoders as partial views of curved structure, not as failed tools, in Can SAEs Capture Neural Geometry?.
- The lab documents concrete geometric structure in real models: circular representations for days and months, a manifold for genomics, and a reusable addition module in Llama 3.1 8B.
- The stated goal is shifting from post-hoc explanation to training-time control -- turning training from an open-loop process into a closed-loop one.
- Primary source: The Neural Geometry Series.
To see why this is a real shift rather than jargon churn, start with what sparse autoencoders were meant to do. Neural networks pack far more concepts into their internal representations than they have dimensions to hold cleanly, so any single number inside the model participates in many unrelated ideas at once. A sparse autoencoder is a second, wider network trained to pull that tangle apart into a long list of features that each fire for one recognisable thing -- the Golden Gate Bridge, legal hedging, Python list comprehensions. It was the field's best tool for producing human-readable units, and our explainer on mechanistic interpretability covers how it works.
The critique now landing is not that the technique fails. It is that the unit is wrong. If a model represents days of the week as points arranged on a circle -- and Goodfire's The World Inside Neural Networks argues activations mirror world structure in exactly this way -- then decomposing that circle into a list of independent features is like describing a clock face by naming twelve unrelated positions. You capture where things are. You lose the fact that it is a circle, which is the part that explains why the model can reason about "two days after Friday."
The strongest concrete evidence sits in A Geometric Calculator Inside a Neural Network, where Goodfire researchers identify a reusable addition module operating over circular representations inside Llama 3.1 8B -- an actual computational structure, doing actual arithmetic, defined by its shape rather than by a feature list. That is the kind of finding a feature-decomposition lens is poorly equipped to produce, because the object of interest is the relationship between representations rather than the representations themselves.
The lab's stated ambition goes further than better explanation. Goodfire frames training today as an open-loop process -- you set it running, you get a model, you inspect it afterwards and hope -- and argues interpretability should close that loop, steering structure as it forms rather than describing it once it has set. Our explainer on activation steering covers the post-hoc version of that idea, which already works well enough to be uncomfortable.
Two supporting claims need narrowing, and it is worth being precise about both. The idea that structures crystallize gradually during training is supported by Tracing Persona Vectors Through LLM Pretraining, which finds persona vectors form very early and then continue refining geometrically and semantically throughout pretraining -- but that is evidence about persona vectors specifically, not a general law about all internal structure. And the older "quanta" framing from Eric Michaud's The Quantization Model of Neural Scaling, which explains scaling laws through discrete chunks of knowledge and skill, is a genuine prior theory; reading it as a stepping stone to geometry is a fair synthesis but an interpretation, not something the paper claims.
McGrath's most striking claim is also his most speculative. He says it seems "very likely" that cutting-edge scientific foundation models contain new science that we simply do not know how to extract. Goodfire cites supporting examples -- work on Alzheimer's biomarkers, structure recovered from genomics models. The specific extractions are real. The general proposition, that frontier models are sitting on undiscovered knowledge waiting for the right interpretability tool, remains a forward-looking bet.
One part of his account is independently corroborated. McGrath recaps the chain from reward hacking to broader misalignment, and Anthropic's research on emergent misalignment from reward hacking documents exactly that: a model that learns to cheat on programming tasks generalises to deception, monitoring avoidance and sabotage. Our explainer on reward hacking covers why that generalisation happens.
Why it matters, and the timing is not incidental: this reframing lands the same week OpenAI documented that its newest model's reasoning has become substantially harder to monitor. Reading a model's emitted thoughts is getting less informative exactly as the field concludes that reading was never the right target. Interpretability that works on internal structure rather than on output text is not merely a research preference any more.
The honest caveat: this is a researcher at a company that sells interpretability tools describing why his lab's approach is the promising one, in an interview. The geometric findings are published and checkable. The claim that geometry is the frame that supersedes features is a bet on a research direction, and the field has changed its mind about the right unit of analysis several times already.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)