“Interpretability” names three research programmes that share a word and almost nothing else. One wants to reverse-engineer the computation into an algorithm. One wants to know what information a representation carries. One wants an explanation that satisfies somebody outside the lab. They use different methods, and — the part that matters — they would accept different things as proof.
Three programmes, not one field
Mechanistic interpretability tries to recover the algorithm. The claim it aims at has the form: this set of attention heads and MLP neurons, wired this way, computes this function, and here is the intervention that confirms it. The unit of output is a circuit. Work in this programme is dominated by small models — two-layer attention-only transformers, GPT-2 small, toy networks trained on modular arithmetic — because a claim about an algorithm has to be checked exhaustively and that is only tractable when the model is small.
Representation analysis asks a weaker and much more answerable question: what information is present in this activation, and is it used? Probing classifiers, representation-similarity measures and concept-vector methods live here. The output is not an algorithm but a map — part-of-speech information is linearly recoverable by layer four, sentiment by layer eight, and here is the direction along which it lies.
Post-hoc explanation takes a model as fixed and produces, for a single prediction, an account of what drove it. Saliency maps, feature attributions, counterfactual explanations and surrogate models are in this family. Its consumers are frequently outside research entirely: a clinician, a loan applicant, a regulator. Its success criterion is correspondingly different, and the gap between what a regulator means by an explanation and what a researcher means is where most of the confusion in this area comes from.
A fourth strand cuts across all three: training dynamics, the study of when in training a capability appears and what changes in the weights when it does. Grokking and the phase-change literature belong here. It is not a separate programme so much as a different axis — the same questions asked of a checkpoint sequence rather than a final model.
What each would count as success
| Programme | Description |
|---|---|
| Mechanistic | A circuit whose description predicts the outcome of interventions it was not fitted to. Ablate this head and this specific error appears; patch this activation from a different prompt and the output flips in the direction the story says it should. Prediction under intervention is the bar. |
| Representational | A property that is decodable and causally used. Decodability alone is weak — a strong enough classifier finds structure in noise. Success is decodability plus an intervention on the identified direction that changes behaviour in the predicted way. |
| Post-hoc | An explanation that is faithful (it reflects what the model did, verified by removing what it highlighted) and useful (somebody makes a better decision with it). Both halves are tested separately, and most published methods only ever get tested on the second. |
| Training dynamics | A progress measure that moves before the loss does — a quantity computed from the weights or activations that anticipates a capability appearing, rather than describing it afterwards. |
The question that separates them
Ask of any interpretability claim: what result would show this is wrong? The three programmes answer very differently, and the answer tells you which one you are reading.
A mechanistic claim is falsified by an intervention. If head 9.6 is said to move information about the subject token to the final position, then zeroing its output should break exactly the behaviours that depend on that information and leave the others alone. If it breaks everything, the claim was too specific; if it breaks nothing, the head was not necessary.
A representational claim is falsified by a control. If a probe reads syntactic depth out of layer six at high accuracy, the same probe trained on randomly assigned labels tells you how much of that accuracy was the probe’s own capacity rather than the representation’s content. This is the single most important idea in the probing literature and it is skipped in most write-ups.
A post-hoc claim is falsified by a sanity check. If an attribution map looks the same after the model’s weights are randomised, it was not explaining the model. Several widely used saliency methods failed exactly this test, which is the most useful fact about that literature.
Why this is hard in a way debugging is not
Debugging a program works because the program has named parts. A trained network does not. Its parts are directions in a high-dimensional space, and there is no reason the basis the architecture happens to use — individual neurons — should align with the concepts it represents. It generally does not. Superposition is the proposed explanation: a layer with 4,096 dimensions may be representing far more than 4,096 features by packing them into near-orthogonal directions and tolerating a little interference. If that is right, then “what does neuron 1,337 mean” is not a question with an answer, and a great deal of early neuron-level interpretability was asking the wrong thing.
The second difficulty is that everything is correlated with everything else. The activation at layer twelve depends on every token in the context and on every earlier layer, so a component that looks necessary in one prompt distribution may be irrelevant in another. This is why the field leans so heavily on causal interventions: correlational evidence about a network you can run forward as many times as you like is cheap and nearly worthless.
Where the field actually is
A fair summary, stated without inflation. A small number of behaviours in small models have been reverse-engineered to the point where the description predicts interventions: induction heads and in-context copying (Anthropic’s Transformer Circuits work, 2021–2022), indirect-object identification in GPT-2 small (Wang and colleagues, 2022), and modular addition in a one-layer transformer (Nanda and colleagues, 2023). These are real results and they are much smaller in scope than the coverage suggests.
Localisation techniques work reasonably well and scale. Activation patching will tell you which layers and positions a behaviour depends on in a model far too large to reverse-engineer. Sparse autoencoders, introduced as a way through superposition by Anthropic and by Cunningham and colleagues independently in 2023, decompose activations into sparser and more interpretable units, and the approach has since been applied to production-scale models.
What does not exist yet is a complete account of any behaviour in a frontier model, a reliable way to tell whether a discovered feature is the model’s own decomposition or an artefact of the method, or an interpretability technique that catches a class of production bug more cheaply than a good evaluation set does. The short list of techniques that pay for themselves today is genuinely short.
How to read a result in this area
- Find the model. Two-layer attention-only, GPT-2 small, and a 70B production model are three different worlds. A mechanism established in the first may not exist in the third.
- Find the intervention. If the paper only shows correlations — attention weights, neuron activations, similarity scores — the claim is about what is present, not about what is used.
- Find the control. Random labels for a probe, randomised weights for an attribution method, a scrambled-prompt baseline for a patching experiment. No control, no result.
- Find the distribution. Most circuit work is established on a narrow, templated prompt set. That is a legitimate choice — it is what makes the analysis tractable — but a circuit verified on templates has not been shown to be what the model does on prose.
- Check what the abstract claims against what the method supports. The gap is usually in the verb: “we identify the neurons that store this fact” when the experiment showed which layers a patch has to hit to change the output.
None of this is available to you through a hosted API: you get tokens and log-probabilities, not activations. What you can get from a request path is the other kind of observability — Multigrid records model, latency and token counts per request, which is what most production questions actually turn out to need. For anything on this page you need weights on a machine you control, which is why almost all of it is done on open models.
Top comments (0)