DEV Community: Anish Shrestha

Running a 26B MoE on an 8 GB Jetson by streaming experts from SSD

Anish Shrestha — Sat, 01 Aug 2026 14:00:35 +0000

My Jetson Orin Nano has 8 GB of unified memory. Gemma 4 26B-A4B is a 13.3 GiB download at Q4. This post is about the llama.cpp patch that lets the first thing run the second thing anyway, producing logits that are bit-for-bit identical to the standard code path, and why mixture-of-experts models make that possible when dense models never could.

The idea is borrowed, and worth borrowing

Credit where it belongs: the architecture comes from TurboFieldfare, a Swift and Metal runtime that runs Gemma 4 26B-A4B in about 2 GB of RAM on Apple Silicon. Its trick is to notice that an MoE model only touches a few experts per token. The dense weights (attention, router, shared expert) stay resident, and the routed experts, which are most of the model, live on the SSD. A small per-layer cache of expert slots is filled by parallel reads the moment the router announces which experts the next token needs, and the reads hide behind compute the GPU was doing anyway. The related academic thread is Eliseev and Mazur's expert-offloading work (arXiv:2312.17238), which established that MoE routing is stable enough between tokens for caching to pay off.

TurboFieldfare proves the idea beautifully, but it is a bespoke runtime: two supported models, Apple platforms only, custom kernels for everything. I wanted the same idea for the other cheap 8 GB machine on my desk, a Jetson Orin Nano, and I wanted it for any MoE model I could quantize. So instead of porting the runtime, I grafted the idea into llama.cpp, which already runs on the Jetson and already has Ampere-tuned CUDA kernels.

Streaming without touching a single GPU kernel

The part I like most about the design is that no CUDA kernel changes were needed. llama.cpp evaluates the routed-expert matmul with ggml_mul_mat_id, which takes the expert weight tensor plus a small tensor of selected expert ids and addresses each expert as base + id * stride on the device. The kernel does not care whether base points at 128 experts or at 32; it just multiplies ids by strides.

So the patch allocates, per MoE layer, a persistent pool tensor holding N expert-sized slots plus a tiny slot-id tensor, and at load time it records where every expert's bytes sit in the GGUF file. During decode, a scheduler callback fires on the router's top-k node, reads the chosen expert ids, and consults a per-layer LFU cache: experts already in slots are hits, misses are pread directly from the file into their assigned slots by a small thread pool. The callback writes the slot indices into the slot-id tensor and lets the graph continue. The matmul reads the pool through slot ids and gets exactly the bytes it would have read from the full tensor.

There is one place to be genuinely careful, and it is the reason this patch has a paranoid test. The same expert ids feed several consumers in the graph: the main matmul, the per-expert scale vectors that quantization-aware checkpoints ship, LoRA adapters, and expert biases. Only the main matmul may see remapped slot ids. Everything else must keep the real expert ids, because those tensors stay fully resident and are indexed by actual expert number. Get this wrong and nothing crashes; the model just gets quietly stupider. The test that guards against it greedy-decodes the same prompt with streaming off and on and compares every logit of every step with memcmp. Same kernels, same bytes, different addressing, therefore bitwise identity is the correct bar, not "close enough".

On the Jetson there is a bonus that TurboFieldfare's mmap trick foreshadowed: the Orin's GPU shares memory with the CPU, so the slot pools live in pinned host memory that the disk reads land in directly and the GPU addresses without any copy.

For the IO-hiding part, Gemma 4's graph has a shared dense FFN that runs for every token alongside the routed experts. The patch reorders the layer so the router's top-k comes first, dispatches the disk reads, lets the GPU compute the shared FFN while the NVMe works, and joins right before the expert matmul needs the data. One synchronization per layer, exactly TurboFieldfare's overlap, expressed as a graph reordering plus a two-phase callback.

Numbers from the actual machine

Measured on a Jetson Orin Nano Super (8 GB, MAXN, NVMe) with Gemma 4 26B-A4B QAT Q4, using llama-bench tg128. The baseline is the best you could previously do in llama.cpp: dense weights on the GPU, experts mmap'd on the CPU with --cpu-moe.

Configuration	Decode tok/s
`--cpu-moe` mmap baseline	1.94
Streaming, 32 slots x 8 IO threads, overlap on	2.95
Same, overlap off	2.23

A few observations that the table compresses:

The overlap alone is worth 32 percent, which is the TurboFieldfare thesis confirmed on entirely different hardware: expert IO really does hide behind compute you were already doing.
The NVMe's random-read ceiling (about 1.27 GB/s at this block size, per fio) puts the IO-bound limit near 5 tok/s, so there is headroom left in the IO path, not a wall.
Slot count is a real tuning knob with a cliff. 32 slots per layer (a 3 GB pinned pool) is the sweet spot on 8 GB; 64 slots allocates 6 GB and starves the rest of the system.
Peak RSS stays at 6.7 GB, and the correctness bar held on device: 32 greedy steps on the real 26B, every logit bitwise identical, with 10.5 GiB streamed through the pools during the comparison.

The honest tradeoff: cold prefill drops to about a quarter of the baseline's speed, because streaming mode deliberately stops pre-faulting the expert file (pre-faulting 13 GiB into 8 GB of RAM is its own disaster). Prefill-through-the-pool is the obvious next piece of work, along with O_DIRECT reads.

The GIF at the top is one request against the production gemma4-26b entry in llama-swap. I sent a warm-up call first to get past cold-load and cold-prefill, then recorded a streamed completion with tegrastats logging GPU load and power draw as each token arrived, laid out as three regions in the same terminal so the numbers and the text are visible at once. It settled at 2.09 tok/s where the bench measured 2.95, which is about the gap I'd expect between a clean llama-bench run and one live request with a different prompt. The GPU trace is the interesting part: utilization swings between 0 and 95 percent because for stretches of every layer the GPU waits while the NVMe pulls in the next expert. That is the overlap claim from earlier, on screen.

What generalizing bought

Because the hook lives in llama.cpp's graph layer rather than in any model's code, it works for every GGUF MoE architecture. I verified bitwise identity on three families: Qwen3-MoE style graphs with per-expert scale tensors, Gemma 4's merged gate-up projection, and Qwen 3.5/3.6's hybrid DeltaNet layers. The same flag also gets a 22 GiB Qwen 3.6 35B-A3B onto the same 8 GB board.

Is 3 tokens per second fast? No. It is roughly reading speed, fine for anything asynchronous and painful for chat. But the alternative on this hardware is not a slower 26B, it is no 26B at all. The compromise the patch offers is exactly the one I wanted: model size becomes a choice you pay for in speed instead of a hard out-of-memory wall.

The work lives on the moe-stream branch of my llama.cpp fork: a standalone expert streamer, a --moe-stream flag, pool allocation at load, the dual-id graph integration, the Gemma 4 overlap, and the identity test harness, in seven commits against upstream master.

Thanks again to TurboFieldfare for the architecture. Ideas this good deserve to escape their original hardware.

The run behind that GIF is scrubbable. I put a replay of it on the Learning Lab, with GPU load and power draw lined up against each token.

My context selector beat grep. An agent with grep beat it.

Anish Shrestha — Fri, 31 Jul 2026 14:00:23 +0000

cognitive-cache started from an observation I still think is correct: every coding tool decides what goes into the context window, and almost none of them treat that decision as an explicit selection problem. Cursor greps, RAG systems embed and cosine-search, and plenty of harnesses just stuff files until the budget runs out. The OS analogy is hard to unsee once you have it. The context window is RAM, eviction and retrieval are virtual memory, and token budget allocation is an allocator that nobody has written.

So I wrote one. Six weighted signals score every file in a repo against a task description, and a two-phase greedy submodular selector picks the highest-value set that fits a token budget. Nine languages, no LLM calls, no API keys. It shipped to PyPI and I used it through MCP for a while.

Then I built a benchmark harness, and the harness spent the next two months dismantling most of what I believed about the tool. This is that sequence, in order, because the order is the interesting part.

The first honest measurement said it does not beat grep

The original benchmark was bad in the way most self-benchmarks are bad. It coupled retrieval to LLM patch generation, reported rank-blind full-set recall, had no significance test, and I almost never ran it.

The replacement is LLM-free and deterministic. For each issue it clones the repo at the pre-fix commit, produces a full file ranking per strategy, and scores that ranking against the files the gold patch actually modified, using recall@k and MRR. Every head-to-head comparison carries a paired bootstrap 95% confidence interval, so a difference that excludes zero is real and a difference that straddles zero is not a result.

On the first dataset, 23 hand-curated issues across 8 repos:

strategy	R@1	R@5	R@10	MRR
cognitive-cache	0.170	0.428	0.511	0.540
grep (keyword count)	0.170	0.409	0.559	0.533
lexical (TF-IDF)	0.159	0.362	0.496	0.487

The recall@5 difference over grep was +0.020 with a 95% CI of [-0.067, +0.109]. Grep won at recall@10. Whatever the tool was doing, at that sample size it was indistinguishable from counting keywords.

Two other things fell out of the same run. The per-signal ablation suggested graph_distance, carrying 0.20 of the weight budget, was net negative, and file_role_prior was doing nothing at 0.07. And the budget selector was leaking badly: ranking recall@5 was 0.428 while end-to-end recall under a 12k budget was 0.282, so the selector was discarding relevant files its own ranking had already surfaced.

I nearly acted on the ablation immediately. A weight sweep stopped me, because every candidate reweighting had a bootstrap lower bound of +0.000. The "graph_distance is net negative" reading was itself noise at n=23. The real blocker was not the weights, it was that 23 narrow issues cannot detect or drive an improvement in either direction.

Widening the dataset flipped the result

I moved to SWE-bench, which solves the curation problem by construction: real GitHub issues paired with the gold patch that fixed them, pulled over the HuggingFace datasets-server with no auth. Ground truth is the set of source files the patch modified, and because patches only touch files present at the base commit, every ground-truth file is retrievable.

On a balanced 78-instance subset of SWE-bench Lite across 12 repos:

strategy	R@1	R@5	R@10	MRR
cognitive-cache	0.231	0.538	0.654	0.375
grep	0.103	0.359	0.487	0.236
lexical (TF-IDF)	0.141	0.410	0.462	0.253

Recall@5 over grep: +0.179, CI [+0.064, +0.295]. Over TF-IDF: +0.128, CI [+0.013, +0.244]. Both exclude zero. On a harder 91-issue subset restricted to fixes touching two or more files, the edge held at 0.365 vs 0.252, CI [+0.056, +0.174].

The earlier negative was a small-sample artifact. That cuts both ways, and it is worth sitting with: if I had published the n=23 result I would have been wrong, and if I had published the n=78 result as the final word I would also have been wrong, for the reason in the next section.

The baseline I had not run was an agent

"Beats grep" means beats a naive keyword count. It says nothing about the thing my tool would actually be competing with, which is a coding agent running its own search loop.

So I ran that. Five single-file SWE-bench issues, one each from flask, requests, xarray, pylint, and pytest. Fresh Claude subagents, cold in the repo at the base commit, grep, glob, and read only, cognitive-cache explicitly forbidden.

The agent got 5 of 5, averaging about 3.4 tool calls and 19k tokens, and in each case it pinpointed the mechanism rather than just the file. cognitive-cache got 4 of 5, missing pylint #5859, which the agent found in two calls. Naive keyword grep got 3 of 5.

The token argument inverted at the same time. The agent read a handful of targeted files. A cognitive-cache call returns file contents up to the full token budget, so on these tasks it front-loads more context than a surgical agent consumes. It was costing context, not saving it.

That is n=5 on hand-picked single-file issues, so it is directional rather than definitive. But it moved the honest positioning from "more accurate than the agent" to "fewer roundtrips on a cold start in unfamiliar code", which is a much smaller claim.

The signal that was worth zero, and why

graph_distance measures import-graph distance from the files the task names. It carried 0.20 weight from the day I wrote it, on the reasoning that a fix often lives one hop away from the file the issue mentions.

Across both datasets it earned nothing. On the 91-issue multi-file set, zeroing it changed recall@5 by +0.000 and MRR by -0.006. That was the pre-registered kill criterion, so I was ready to retire it.

Then I pushed back on my own experiment, which turned out to be the useful move. Multi-file is not the same as multi-hop, a regex-based import graph is not a reference graph, and testing a signal as one term in a drowned weighted sum is not a fair test of the underlying idea. All three critiques were fair, so instead of assuming, I wrote a diagnostic over the 91-issue set to check whether the ground truth was even reachable.

The answer killed the revival rather than supporting it. 93% of issues have all their ground-truth files reachable from the entry points. But 87% of fix files are themselves entry points, at graph distance 0, with only about 10% sitting at one to three hops and none further out. The mechanism was never starvation, it was redundancy: real issue text names the files it touches, so symbol_overlap has already ranked those files highly and import distance adds nothing on top.

That is not a bug in the experiment. It is a fact about the task distribution, and it explains the agent result too. On issue-shaped work the target is usually named in the prompt, which is exactly the regime where an agentic grep loop is excellent and a static ranker has little room to add value. Claude Code shipping no code index looks less like an omission and more like the same finding, reached earlier.

The delivery problem, measured

One thread survived long enough to be worth reporting separately. I built a jedi-backed neighborhood tool (definition, callers, in-repo callees, with real scope-aware resolution rather than name matching) and tested it two ways.

Against agents, on two hard cross-file issues where naive grep recall was 0, I ran a 2x2: agents with grep and read, versus agents offered the neighborhood tool as well. Every arm scored 0.5, finding the file tied to the named concept and missing the cross-file one. The agents that had the tool invoked it zero times out of two, despite being told to. Checking the tool's output directly against the symbols those same agents had reasoned about, it surfaced both missed files. The data would have lifted recall from 0.5 to 1.0 if it had been used.

At scale the value is real but modest: across all 91 multi-file issues, neighborhood-of-seeds alone scores 0.164 (worse than grep's 0.252 standalone), while grep-or-neighborhood combined reaches 0.330, recovering 20 of grep's 257 missed fix files. That is +7.8 absolute points, in line with RepoGraph's own reported 9-12%.

Put the two halves together and the conclusion is about delivery, not capability. A passive tool that competes with the grep reflex for invocation loses the invocation. Value like that only lands if it is injected into results the agent is already reading, not offered as one more thing it could call.

Where it ended

I set graph_distance to weight 0, rewrote the README around the scoreboard instead of the pitch, and stopped building. The remaining pitch would have required a forced-injection layer to capture a modest gain, in a niche that already has direct clones, running against the current of agents that search well on their own.

What survives is the measurement rig: the SWE-bench importer that needs no token, the retrieval evaluator, the weight tuner, the multi-hop diagnostic, the neighbor-recovery harness, and the bootstrap CIs that make each of those trustworthy. In a category that ships on anecdotes, that is the part I would build again first.

The generalizable lesson is not "context selection is a dead end". It is that the baseline you skip is the one that decides whether your work matters. I measured against grep for two months because grep was the baseline that let the tool win. The agent was two hours of work and it reframed the entire project.

J-space in practice: using Anthropic's Jacobian lens to decide what an LLM can forget

Anish Shrestha — Wed, 29 Jul 2026 12:07:06 +0000

Anthropic published Verbalizable Representations Form a Global Workspace in Language Models on July 6, and the vocabulary it introduced is suddenly everywhere: J-space, the Jacobian lens, a global workspace inside Claude. Most of the discussion so far is about interpretability and alignment auditing, which is fair, since that is what the paper is about. I had a narrower and more mercenary question: can the workspace tell an inference runtime which parts of the KV cache it is safe to throw away?

Three days after the paper landed, the first pre-registered gate on that question passed. As of this week the signal has replicated on three models and ships inside EVOKE, my KV cache memory manager built on a forked llama.cpp. This post covers what J-space is, why it makes a good KV cache eviction signal, the numbers across Qwen2.5-7B, Qwen3-8B, and Qwen3-4B, and the caveat that comes with them.

What J-space is, in one paragraph

The Jacobian lens is the instrument and J-space is the phenomenon. The lens isolates directions in a model's residual stream that encode a token the model could verbalize next, and those directions form a low-dimensional workspace: roughly 10% of activation variance, concentrated in the middle layers, carrying whatever the model is "holding in mind" at each position. Anthropic's headline application is alignment auditing, reading reasoning the model never voices. What makes independent work possible is that they released companion code under Apache-2.0 along with fitted lens matrices for open Qwen models on Hugging Face, so anyone can apply the lens to an open-weights model on a single GPU.

The systems problem: KV cache eviction

Every long-running LLM session eventually outgrows its KV cache budget. An agent session in a coding harness crosses tens of thousands of cached tokens within a few turns, and something has to decide which entries stay in GPU memory. The standard answers, H2O and SnapKV, rank cache blocks by accumulated attention history: keep what the model has been attending to, evict the rest. StreamingLLM adds protected sink tokens and a recency window.

These policies share a structural weakness. They are backward-looking, so they can only rank blocks the model has already used. A fact planted at turn 1 that will matter at turn 14 looks cold the whole time, and it gets evicted exactly when the budget tightens.

The hypothesis: workspace content predicts KV reuse

The lens readout suggested a forward-looking alternative. If a position carries workspace content, meaning the model is holding something in mind there, then its KV entries should be disproportionately likely to be read by later reasoning. That is a content signal, computable at prefill time from the residual stream alone, before any attention history exists.

The offline validation lives in j-space: lens application, a family of workspace statistics over the readout (kurtosis of block means, transport ratios, energy concentration, computed across layers), block aggregation, eviction simulation, and probe distillation. Because a signal this convenient deserved suspicion, I pre-registered three kill gates before running anything:

Fact ranking. The best workspace statistic has to beat SnapKV at ranking the task-critical block, by a pre-set AUC margin, or the project dies.
Masked eviction. With only 25% of the cache retained under the workspace policy, the planted fact has to stay answerable.
Probe distillation. Applying the full lens at inference time is too expensive, so a ridge probe distilled from the workspace readout has to reproduce the ranking at Spearman 0.8 or better. The deployable artifact is one dot product per position.

Results across three Qwen models

All three gates passed on all three models, with no hand-tuning on the replications.

Model	Fact-AUC (workspace)	Fact-AUC (SnapKV)	Eviction @ 25% budget	Probe Spearman
Qwen2.5-7B-Instruct	0.891	0.622	35/36 vs 25/36	0.959
Qwen3-8B	0.865	0.563	35/36 vs 14/36	0.944
Qwen3-4B	0.939	0.541	36/36 vs 13/36	0.946

The eviction column reads: episodes where the planted fact survived eviction and was answered correctly, workspace policy vs SnapKV at the same budget. On Qwen3-4B the workspace policy saturates at 36/36 at every budget tested.

The distilled probe also holds up live. Inside EVOKE's agent bench, the workspace signal passes the fact probe at every cache budget (512, 1024, and 2048 tokens) with recovery disabled, while H2O fails all three budgets and SnapKV passes only at 2048. Measured decode overhead was -0.6%, which is noise; the probe is a dot product per position at prefill, and nothing runs during decode.

What transfers and what does not

Here is the caveat, and I think it is the most interesting finding in the project. The winning statistic is model-dependent. On Qwen2.5-7B and Qwen3-8B it is kurtosis of block-mean workspace scores at deep layers (layer 23 and layer 30 respectively). On Qwen3-4B it is a transport-ratio statistic at layer 12 of 36. If I had frozen the 7B statistic and shipped it as "the" signal, the 4B replication would have failed. What transfers is the statistic family plus the automated procedure that selects the winner per model on held-out episodes, not any single magic number.

The other scope limits are stated plainly in the paper: three models so far, all Qwen; the probes are fit on the benchmark's own episode family; and the task family is long-horizon fact recall. LongBench-style suites and non-Qwen architectures are open work. As far as I can tell this is the first interpretability-derived KV eviction signal, and I would much rather over-state the limits than the result.

Where it ships: EVOKE v2.0

EVOKE treats the KV cache the way an operating system treats memory. Blocks are evicted under budget pressure and recovered recompute-free through save/restore primitives added to a forked llama.cpp, 20 to 32 times faster than re-prefilling the same tokens. Version 2.0 adds the workspace probe as a first-class eviction signal alongside live attention capture, harness priority tags, task-focus coherence, and recency, and includes the paper PDF with the full three-model evaluation.

You can watch the signal work in the live demo on Hugging Face Spaces, which runs three arms side by side on Qwen3-4B: a plain baseline, EVOKE with its attention scorer, and EVOKE with workspace eviction. There is also a benchmark walkthrough on my learning lab.