Co-authored by Sourav Nandy and Rudrendu Paul.
Repo: https://github.com/RudrenduPaul/neuronscope, An open-source CLI and MCP server for mechanistic interpretability.
You ask a language model why it gave you a particular answer, and it hands back a confident, plausible-sounding explanation. That explanation is generated by the same model that produced the original answer, so it's not a report on what actually happened inside it. It's a second guess dressed up as a first-hand account.
NeuronScope is our attempt at the alternative: a CLI and MCP server that traces which attention heads and MLP neurons actually drove a model's output, instead of asking the model to narrate itself. It's built on top of an existing open-source library for hooking into model internals, ships a versioned JSON schema on every command, and exposes the same four operations to an AI agent over MCP. MIT-licensed, pip install neuronscope-cli, runs small models like GPT-2 on CPU.
Real output: neuronscope trace gpt2 "The capital of France is Paris. The capital of Japan is" ranking the attention heads and neurons that produced the prediction Tokyo.
Gartner expects explainable-AI tooling to account for half of all LLM observability investment by 2028, up from 15% today (Gartner: Explainable AI Will Drive 50% of LLM Observability Investment by 2028). Almost all of the funding chasing that shift is going into hosted platforms: interpretability research lab Goodfire raised a $150M Series B at a $1.25B valuation in February (Goodfire: Our Series B), and Apollo Research converted from a philanthropy-funded model to a VC-backed public benefit corporation the same month (Apollo Research Is Becoming a PBC). Almost none of that money is landing on a command you can run against an open-weight model on your own laptop. That's the gap we're trying to close. It's also three weeks old, and we already broke its most-marketed feature once. That's most of this article.
What mechanistic interpretability actually measures
Ask a language model why it produced a given output and it will happily generate a plausible explanation. Because that explanation is a second generation from the same model, it's disconnected from what actually happened inside the first one. Mechanistic interpretability is the alternative: you instrument the forward pass and measure which components moved the prediction.
Three terms carry most of the weight here. Attention heads move information between token positions; ranking them by direct logit attribution tells you which head's output pushed the final prediction the hardest. MLP neurons fire on specific features inside a layer; ranking them by activation magnitude tells you which ones were most active at the position that mattered. Activation patching (or ablation) is the causal test: zero out one component and watch how much the prediction actually changes, which is a stronger claim than "this component was active" on its own.
The tooling gets honest about its own limits right at that distinction. High logit attribution only tells you a component was correlated with the output: two components can be redundant, so ablating either one individually barely moves the prediction even though both ranked high in isolation. NeuronScope's own circuit command, which chains ranking and single-component ablation into an automated sketch, states this explicitly in its JSON output's method field, so a caller can't assume more rigor than the method provides. A tool that can't name where its own technique breaks down is a tool you should be skeptical of, and that applies as much to NeuronScope as to anything else in this space.
The CLI-shaped hole
The underlying interpretability library most of this space is built on gives you a real, capable Python API: load a model, register hooks, run a forward pass, read the activations back as tensors. That's the right interface for a research notebook where you're iterating interactively. It's the wrong interface for two things that are increasingly common in 2026: a CI check that needs a JSON exit code, and an agent that needs to call a tool over MCP and get back an already-serialized, structured result.
NeuronScope exists to be that second interface. Here's the actual command and its real output:
neuronscope trace gpt2 "The capital of France is Paris. The capital of Japan is" --top-k 3 --json
{
"schema_version": 1,
"operation": "trace",
"model": {
"requested_name": "gpt2",
"resolved_name": "gpt2",
"backend": "transformer_lens",
"device": "cpu",
"n_layers": 12,
"n_heads": 12,
"d_model": 768,
"d_mlp": 3072
},
"prompt": "The capital of France is Paris. The capital of Japan is",
"predicted_token": " Tokyo",
"predicted_token_id": 11790,
"top_neurons": [
{ "layer": 10, "neuron_index": 97, "activation": 7.839381217956543 },
{ "layer": 11, "neuron_index": 611, "activation": 4.695372581481934 },
{ "layer": 11, "neuron_index": 2997, "activation": 4.646785736083984 }
],
"top_heads": [
{ "layer": 9, "head_index": 8, "logit_attribution": 4.067923545837402 },
{ "layer": 8, "head_index": 11, "logit_attribution": 2.9028172492980957 },
{ "layer": 10, "head_index": 7, "logit_attribution": -1.4781968593597412 }
]
}
gpt2 predicts Tokyo, and head L9H8 is the single largest contributor. That's the same document, byte-for-byte, that an MCP client gets back when it calls the equivalent tool, no CLI shell-out required. One schema, two callers.
For a team deciding where to put interpretability tooling budget, the split is straightforward: computing the activations, the hard part, is a solved research problem with mature open libraries behind it. Integration was the unsolved part: getting that computation into a shape a script or an agent can consume without hand-written glue code per model architecture. That's a tooling investment. It's a much smaller check to write than a research one.
The mistake that took down the flagship feature
Three weeks after the first release, the MCP server, the single most heavily marketed capability in the README, stopped working on every fresh install. It broke immediately, for anyone who ran pip install neuronscope-cli after a specific date.
The cause was a dependency line in pyproject.toml: mcp>=1.0, with no upper bound. That's a reasonable-looking constraint right up until the mcp package ships a 2.0.0 release that removes the exact module (mcp.server.fastmcp) NeuronScope's server code imports. It did. A fresh install resolved mcp==2.0.0, and neuronscope mcp-server crashed on startup with ModuleNotFoundError: No module named 'mcp.server.fastmcp'.
We found this the way you'd want to, and the way a lot of teams don't: an independent repo audit ran the documented quickstart command in a clean virtual environment before anything shipped further. It didn't lean on a CI run from a week earlier. That run was already stale, since CI had last gone green before the breaking mcp release even existed. The fix was one line, mcp>=1.0,<2.0, plus a version bump and a changelog entry. The lesson generalizes past this one dependency: an unbounded floor on any library your integration touches directly, especially one still moving fast enough to ship breaking majors, is a live outage waiting for someone else's release cadence to trigger it. We now pin every dependency NeuronScope imports directly, by name, with both a floor and an upper bound.
For an engineering lead, the actual takeaway is narrower than "pin your dependencies," which everyone already knows: the dependencies worth bounding aggressively are the ones backing your most-marketed, least-tested code path. That's exactly where a silent break does the most damage to a first impression, and it's the break that gets caught last.
The feature we decided not to build
Not every engineering decision in this project was a bug fix. One was a deliberate no. The obvious move for reaching the Node/TypeScript agent-tooling crowd would've been a thin npm package that shells out to the PyPI one, so npx neuronscope-cli works without anyone needing Python on their PATH. We scoped it, then skipped it for v1.
The reasoning: the tool's actual work always requires a Python runtime with a multi-hundred- megabyte machine learning dependency installed. A Node wrapper doesn't remove that cost, it just adds a second package to keep in sync, a second place for bugs to hide (subprocess invocation, PATH resolution, version drift between the shim and the thing it wraps), and a convenience that mostly matters to someone who wasn't going to get value from the tool anyway, since they'd still need Python and PyTorch installed to run anything past the wrapper itself. We'll build it if real users ask for it post-launch. We won't build it speculatively because it looked easy.
The failure mode worth naming explicitly for anyone maintaining a CLI with a heavy native dependency: a second distribution channel isn't free just because the wrapper code is short. The maintenance cost comes from the surface area it adds, no matter how few lines the wrapper itself takes to write.
Why the backend is an interface
The one piece of architecture built ahead of an immediate need: Backend is an abstract interface (load_model, get_activations, patch_activations, list_supported_architectures), and the interpretability library NeuronScope launched with is the only concrete implementation of it in v1. That's a small amount of upfront indirection to pay for a project that, on day one, has exactly one backend to support.
The reason it's worth that cost here specifically: the domain has more than one credible backend already in view, each with a different trade-off (fixed-model-family coverage versus arbitrary PyTorch model support versus deeper feature-level analysis). Adding a second one later means writing a new class against an existing interface: the CLI, the MCP layer, and the JSON schema stay untouched. Hardcoding the first backend's specific calls directly into the command layer would have turned that into a rewrite. The failure mode to watch for on the other side of this decision: building the interface before you have a second real implementation to validate it against is a bet. It's only a good one when you're confident enough in the domain's shape to lose a little now for a lot less later.
Where NeuronScope fits, and where it doesn't
Four categories of existing approaches are relevant here, and it's worth being precise about where each one actually sits: treating them as interchangeable hides real differences.
Research-grade Python libraries for hooking into a model's internals are the most mature part of this space: mainstream, actively maintained, and capable of real depth. They're built for a notebook-and-script workflow, though, which means a non-Python process can't shell out to them and an agent can't call them directly.
A newer wave of hosted, enterprise-funded interpretability platforms is where most of the capital described above is actually going. These offer browsable feature databases and managed infrastructure, real value for teams that want a dashboard. They're a different product category from a lightweight, scriptable CLI, and they often carry deployment requirements (a database, a container orchestrator) that a solo developer running a one-off trace doesn't want.
A third category offers exactly the CLI-plus-structured-output shape this article is arguing for. Its trade-off: model support is locked to a fixed, curated list of families, and generic coverage of whatever an underlying library happens to support isn't part of the deal. That's a legitimate trade-off, depth of analysis against breadth of model coverage. Because of it, the "CLI with JSON output" problem already has real competition claiming that niche. What's still comparatively open is pairing model-agnostic coverage with a native, first-class MCP surface: elsewhere, that surface shows up only as JSON export bolted on afterward.
A fourth category focuses specifically on the training and analysis of sparse autoencoders, a deeper and more specialized technique for decomposing model internals into interpretable features. That's complementary to component-level tracing, and NeuronScope doesn't attempt to replace it.
NeuronScope's own position in that space is narrow on purpose: a CLI and MCP layer that works across whatever model families the library underneath it supports, skipping the fixed allowlist other tools rely on. That trades some analytical depth for breadth, and for being callable by a script or an agent without hand-written integration code. It's trying to make the research those four categories already do reachable from a terminal or a tool call, without claiming to out-research any of them.
What this means if you're deciding where to spend on interpretability
If you're an engineering leader weighing interpretability tooling spend, the practical split is between two different problems that get talked about as one. Interpretability research capacity, training new methods, discovering new circuit-analysis techniques, is a genuinely hard, ongoing problem, and it's where the funding above is correctly flowing. Interpretability tooling integration, getting an existing, working method into a shape your CI pipeline or your agent harness can actually call, is a much smaller, mostly-solved software engineering problem that happens to be sitting mostly unclaimed in open source rather than requiring a platform subscription. Knowing which one you actually need before you buy either is the whole decision.
Honest limitations
We'd rather list these than have you find them yourself. The circuit command is an approximation: it ranks components by logit attribution, then measures each one's individual causal effect through single-component ablation, and it says so directly in its own --json output's method field. It doesn't do full path-patching with clean and corrupted prompt pairs, and it won't catch interaction effects between components that only show up when you remove two of them together.
NeuronScope's MCP server currently has no built-in resource limit on model size or forward-pass time. If you expose it to an untrusted agent, bounding it with a container or process limit is on you: the tool doesn't enforce that yet. And the underlying library's model-loading function is already flagged deprecated upstream, in favor of a newer API we haven't migrated to. It still works, every command in this article ran on it, but it's a tracked, open item in the repo. A tool that hides gaps like these doesn't deserve the trust the rest of this article is asking for.
Try it against your own model
pip install neuronscope-cli
neuronscope trace gpt2 "The capital of France is Paris. The capital of Japan is" --top-k 5
No GPU required for small models. neuronscope mcp-server starts the MCP server over stdio if you want to wire it into Claude Code, Claude Desktop, or any other MCP host.
We have one open, genuinely undecided question: the next backend to add behind the same interface. Arbitrary PyTorch model support with no fixed list, or deeper, feature-level circuit discovery through sparse autoencoders. They pull in different directions, and we haven't built either yet. Which one would you actually reach for first?
If this is useful to you, a star on the repo helps other people working on interpretability tooling find it.
GitHub: github.com/RudrenduPaul/NeuronScope
PyPI: pypi.org/project/neuronscope-cli
Co-authored by Sourav Nandy and Rudrendu Paul.
Sourav Nandy and Rudrendu Paul build open-source developer tools for the AI agent ecosystem. They are the co-authors of NeuronScope, a model-agnostic CLI and MCP server for mechanistic interpretability, along with a related set of AI-agent infrastructure projects. Find the code at github.com/RudrenduPaul.
Top comments (0)