Almost every production LLM bug is diagnosed from inputs, outputs and logs. Interpretability earns its keep in a narrow set of cases, and knowing which ones is worth more than knowing the techniques. This page is that list, with the cases where it is the wrong tool named first.
The honest starting position
If your model is producing bad output, the ordered list of things to do begins: read the actual prompt that was sent, including everything your framework appended; check the retrieved context for the failing case; check whether the failure reproduces at temperature 0; and check whether it started at a specific deploy.
Those four steps resolve the overwhelming majority of real incidents. The internals of the model are the last place to look, not the first, because they are the part that did not change — and the systematic version of that checklist is a better use of an afternoon than any technique on this page.
What follows is for the residue: problems that survive that pass, and where a signal from inside the model is genuinely the cheapest way forward.
The gate: do you have the weights?
Almost everything in this cluster requires local weights. A hosted API returns tokens and, sometimes, log-probabilities. It does not return activations, and it will not.
So the practical question is not “is this technique good” but “is this problem worth self-hosting an open model to investigate”. Usually it is not. Occasionally — a classifier at the centre of a product, a safety-relevant behaviour you must characterise, a model you fine-tuned and now cannot explain — it is, and the self-hosting trade-off changes when internals access is part of what you are buying.
One technique survives the gate: log-probability analysis works through any API that exposes it, and it is the cheapest useful signal about a model’s internal state that exists.
What pays for itself
Log-probabilities, on any model
The token distribution tells you whether the model was confident or picking between near-ties. A wrong answer at probability 0.95 and a wrong answer at 0.31 are different bugs: the first is a representation problem, the second is often a prompt or retrieval problem where the right answer was in contention and lost.
For any constrained output — classification, routing, extraction from a fixed set — reading the distribution over the permitted tokens gives you a usable confidence signal and an abstention threshold. That is a production feature, not an analysis: abstaining below a threshold converts a class of silent wrong answers into a handled case. See log-probabilities for the mechanics.
Embedding-space diagnosis for retrieval
When a retrieval system returns the wrong documents, the internals in question are the embeddings, and they are fully available to you. Check the similarity between the query embedding and the embedding of the document that should have won, then check the documents that actually won. This distinguishes three failures that look identical from the outside: the document was not in the index at all, it was there but scored below the cut, or it scored well and was displaced by near-duplicates. Each has a different fix, and no amount of prompt iteration separates them.
A probe as a runtime classifier
If you self-host, a linear probe on a mid-layer hidden state is a genuinely practical component. It costs a dot product on a tensor you already computed, it runs on the prompt before generation, and it can flag categories — this request looks like the class we route elsewhere — more cheaply than a second model call.
The evaluation obligations from the probing page apply in full: a control task, a held-out split at document level, and a measured false positive rate on real traffic. A probe deployed on the strength of its training accuracy is a liability.
Attention patterns for long-context routing bugs
Narrow but real. If a model ignores instructions placed in the middle of a long document, the attention pattern shows the positional structure directly. This is a legitimate use because the claim being made is about routing, which is what attention reports, rather than about causation. The fix is usually moving the instruction, and having seen the pattern makes that a decision rather than a guess.
Before reaching for any of this, the request path usually has the answer. Multigrid records the model, the token counts and the latency for every request, which is what tells you whether a regression started at a specific deploy, whether it correlates with prompt length, and whether it is one model or all of them. A behaviour that appeared on a Tuesday is a deployment question, and no activation will tell you that.
Triage table: bug to technique
| Symptom | Description |
|---|---|
| answers got worse after a deploy | Not an interpretability problem. Diff the prompts, the model version and the retrieval index. Model internals did not change; something around them did. |
| confidently wrong on a known class of input | Build an eval set for that class first. If it reproduces at temperature 0 and survives prompt changes, and you have the weights, a probe on the failing distinction tells you whether the model represents it at all. |
| right answer sometimes, wrong sometimes, same input | Sampling. Check temperature and top-p before anything else. If it persists at temperature 0, look for non-determinism in the serving stack rather than in the model. |
| retrieval returns irrelevant documents | Embedding-space diagnosis. Compare query-to-target similarity against query-to-retrieved. Fully solvable with tools you already have. |
| model ignores an instruction in a long prompt | Attention pattern over positions, then move the instruction. One of the few cases where an attention map answers the question being asked. |
| fine-tune regressed something unrelated | Layer-wise representation similarity against the base model localises where the adaptation landed. See the representation similarity page. |
| model refuses things it should not | Prompt and system-prompt work first. Refusal behaviour is heavily shaped by alignment training and is unusually amenable to prompt-level fixes; internals are for when you must characterise it rather than change it. |
What does not pay yet
- Circuit analysis of a production model. Months of work on a narrow behaviour in a small model. There is no version of this that fits inside an incident.
- Steering vectors as a behaviour control. Real, and almost always beaten by a system prompt plus an output check, which are auditable, portable across model versions and do not need a coefficient sweep. See the steering page for the cases where it does win.
- Sparse autoencoder features as monitoring. The research is promising. Operationally it means training and maintaining a second model to interpret the first, and then validating that its features mean what their labels say. Watch it; do not build on it yet.
- Attribution maps as user-facing explanations. Covered on the saliency page: the most convincing maps were the ones that failed the sanity checks. Showing users an explanation you cannot verify is worse than showing none.
If you do build it, build it small
- Write the eval set first. Twenty to fifty cases that fail, with expected outputs. Without it you cannot tell whether anything you learn helps, and building it frequently solves the problem outright.
- Reproduce on an open model. If the bug does not reproduce on a model whose weights you have, internals are not available and the investigation ends here.
- Instrument once, cheaply. Capture hidden states for your failing and passing cases in a single pass and save them. Everything after that is analysis on stored tensors rather than repeated inference.
- Probe, with a control task. Ask whether the model represents the distinction it is getting wrong. A negative answer is informative: it says the fix is data or retrieval, not prompting.
- Intervene only to confirm. Patching or ablation to test the story you formed. If you cannot state what result would falsify it, you are not ready to run it.
Top comments (0)