J-space in practice: using Anthropic's Jacobian lens to decide what an LLM can forget

Anish Shrestha — Wed, 29 Jul 2026 12:07:06 +0000

Anthropic published Verbalizable Representations Form a Global Workspace in Language Models on July 6, and the vocabulary it introduced is suddenly everywhere: J-space, the Jacobian lens, a global workspace inside Claude. Most of the discussion so far is about interpretability and alignment auditing, which is fair, since that is what the paper is about. I had a narrower and more mercenary question: can the workspace tell an inference runtime which parts of the KV cache it is safe to throw away?

Three days after the paper landed, the first pre-registered gate on that question passed. As of this week the signal has replicated on three models and ships inside EVOKE, my KV cache memory manager built on a forked llama.cpp. This post covers what J-space is, why it makes a good KV cache eviction signal, the numbers across Qwen2.5-7B, Qwen3-8B, and Qwen3-4B, and the caveat that comes with them.

What J-space is, in one paragraph

The Jacobian lens is the instrument and J-space is the phenomenon. The lens isolates directions in a model's residual stream that encode a token the model could verbalize next, and those directions form a low-dimensional workspace: roughly 10% of activation variance, concentrated in the middle layers, carrying whatever the model is "holding in mind" at each position. Anthropic's headline application is alignment auditing, reading reasoning the model never voices. What makes independent work possible is that they released companion code under Apache-2.0 along with fitted lens matrices for open Qwen models on Hugging Face, so anyone can apply the lens to an open-weights model on a single GPU.

The systems problem: KV cache eviction

Every long-running LLM session eventually outgrows its KV cache budget. An agent session in a coding harness crosses tens of thousands of cached tokens within a few turns, and something has to decide which entries stay in GPU memory. The standard answers, H2O and SnapKV, rank cache blocks by accumulated attention history: keep what the model has been attending to, evict the rest. StreamingLLM adds protected sink tokens and a recency window.

These policies share a structural weakness. They are backward-looking, so they can only rank blocks the model has already used. A fact planted at turn 1 that will matter at turn 14 looks cold the whole time, and it gets evicted exactly when the budget tightens.

The hypothesis: workspace content predicts KV reuse

The lens readout suggested a forward-looking alternative. If a position carries workspace content, meaning the model is holding something in mind there, then its KV entries should be disproportionately likely to be read by later reasoning. That is a content signal, computable at prefill time from the residual stream alone, before any attention history exists.

The offline validation lives in j-space: lens application, a family of workspace statistics over the readout (kurtosis of block means, transport ratios, energy concentration, computed across layers), block aggregation, eviction simulation, and probe distillation. Because a signal this convenient deserved suspicion, I pre-registered three kill gates before running anything:

Fact ranking. The best workspace statistic has to beat SnapKV at ranking the task-critical block, by a pre-set AUC margin, or the project dies.
Masked eviction. With only 25% of the cache retained under the workspace policy, the planted fact has to stay answerable.
Probe distillation. Applying the full lens at inference time is too expensive, so a ridge probe distilled from the workspace readout has to reproduce the ranking at Spearman 0.8 or better. The deployable artifact is one dot product per position.

Results across three Qwen models

All three gates passed on all three models, with no hand-tuning on the replications.

Model	Fact-AUC (workspace)	Fact-AUC (SnapKV)	Eviction @ 25% budget	Probe Spearman
Qwen2.5-7B-Instruct	0.891	0.622	35/36 vs 25/36	0.959
Qwen3-8B	0.865	0.563	35/36 vs 14/36	0.944
Qwen3-4B	0.939	0.541	36/36 vs 13/36	0.946

The eviction column reads: episodes where the planted fact survived eviction and was answered correctly, workspace policy vs SnapKV at the same budget. On Qwen3-4B the workspace policy saturates at 36/36 at every budget tested.

The distilled probe also holds up live. Inside EVOKE's agent bench, the workspace signal passes the fact probe at every cache budget (512, 1024, and 2048 tokens) with recovery disabled, while H2O fails all three budgets and SnapKV passes only at 2048. Measured decode overhead was -0.6%, which is noise; the probe is a dot product per position at prefill, and nothing runs during decode.

What transfers and what does not

Here is the caveat, and I think it is the most interesting finding in the project. The winning statistic is model-dependent. On Qwen2.5-7B and Qwen3-8B it is kurtosis of block-mean workspace scores at deep layers (layer 23 and layer 30 respectively). On Qwen3-4B it is a transport-ratio statistic at layer 12 of 36. If I had frozen the 7B statistic and shipped it as "the" signal, the 4B replication would have failed. What transfers is the statistic family plus the automated procedure that selects the winner per model on held-out episodes, not any single magic number.

The other scope limits are stated plainly in the paper: three models so far, all Qwen; the probes are fit on the benchmark's own episode family; and the task family is long-horizon fact recall. LongBench-style suites and non-Qwen architectures are open work. As far as I can tell this is the first interpretability-derived KV eviction signal, and I would much rather over-state the limits than the result.

Where it ships: EVOKE v2.0

EVOKE treats the KV cache the way an operating system treats memory. Blocks are evicted under budget pressure and recovered recompute-free through save/restore primitives added to a forked llama.cpp, 20 to 32 times faster than re-prefilling the same tokens. Version 2.0 adds the workspace probe as a first-class eviction signal alongside live attention capture, harness priority tags, task-focus coherence, and recency, and includes the paper PDF with the full three-model evaluation.

You can watch the signal work in the live demo on Hugging Face Spaces, which runs three arms side by side on Qwen3-4B: a plain baseline, EVOKE with its attention scorer, and EVOKE with workspace eviction. There is also a benchmark walkthrough on my learning lab.

DEV Community: Anish Shrestha