DEV Community: Anish Shrestha

My coding agents keep a record of what they decided and why, not just what they built

Anish Shrestha — Sat, 29 Aug 2026 21:55:14 +0000

A session with a coding agent ends and whatever it figured out along the way goes with it. The code survives, since it's on disk. The reasoning behind the code doesn't: why this table got a composite index instead of two separate ones, why the retry logic gave up after three attempts instead of five, which approach got tried first and thrown out. Open a new session next week on the same project and the agent starts from the code alone, with no memory of any of that.

Claude Code has a memory system for this, and it's genuinely good at what it does. Cursor has nothing like it. I run the two on different machines for different kinds of work, Claude Code where I vibecode, Cursor on a separate box for the work I want to sit with, so it was never a question of getting one tool's session context into the other's. What I wanted was a record of the reasoning that outlives any single machine: an Obsidian vault, synced with the self-hosted obsidian-livesync plugin, that both machines write into and that also lands on my phone for a quick read when I'm nowhere near either one.

Three hooks doing what a prompt can't

Telling an agent in its system prompt to "keep a devlog" doesn't work reliably. It's one more instruction competing with the task itself, and the task wins almost every time, because the task is enforced by the point of the session and the devlog instruction is enforced by nothing. What actually holds is three separate hooks in the harness, each firing at a different point in the session, backing each other up.

The first hook, session-start-wiki.sh, runs when a session opens and hands the agent the last three devlog entries plus an index of every project in the wiki, so it starts already knowing what happened last time instead of re-deriving it from the code. The context it injects is one fixed block: WIKI: project={slug} vault={wiki_vault}, followed by an instruction to write plans, decisions, spikes, and devlog entries as it works.

The second, wiki-nudge.sh, checks in every fifth turn. If no wiki file has been touched yet this session, it injects a reminder pointing at the exact devlog path to write to and the index file that has to be updated alongside it, and it repeats on the same schedule until something gets written.

The third is the one that actually forces the issue. stop-wiki-enforce.sh runs when the session is about to end, counts how many Write, Edit, or NotebookEdit calls happened in the transcript, and if three or more code changes went by with no devlog or index update to match, it blocks the stop outright. In Claude Code that's a {"decision": "block"} response the hook returns; in Cursor, which doesn't expose a transcript path the same way, it falls back to checking whether anything in the wiki directory changed in the last two hours. Either way the session doesn't get to close clean. The reason for it gets re-fed as the next prompt, and the agent has to either write the entry or explain why this particular session didn't need one.

A fourth hook applies the same idea to a different failure mode. Long sessions get compacted to fit the context window, and anything unwritten before that happens disappears the same way it would at session end. pre-compact.sh fires right before that compression runs and gives the agent one explicit last chance to write down any plans, decisions, spikes, or concepts from the session, since compaction is the point where the context actually goes away.

What actually gets written

The wiki has a fixed shape: wiki/projects/<slug>/ holds an overview.md, a devlog.md, and subfolders for decisions/, plans/, and spikes/, plus a top-level index.md that lists every project. Each page type has its own template. A decision page has to name the alternatives that were on the table alongside the one that won. A spike page states the question it was answering before it gets to the finding. The devlog format is the strictest: a dated heading, two to four bullets, written as plain fact rather than a "here's what we did this session" narration, with no name attached and no hedged phrasing softening what happened. That strictness is what keeps a year of entries readable in one sitting rather than turning into the same throat-clearing paragraph with a new date stamped on it.

The hooks themselves are identical on both sides. session-start-wiki.sh, wiki-nudge.sh, and stop-wiki-enforce.sh are deployed to ~/.claude/ and ~/.cursor/ unchanged, same triggers, same thresholds, and only the JSON shape of the block response differs because the two tools expect different formats. The harness's own docs never claim Cursor needs this more because it lacks a native memory system; that's my own read of the setup, not a documented rationale. I think it holds up anyway. Claude Code's memory already softens the cost of a missed devlog entry, where Cursor has no such fallback, so the same devlog is doing more of the actual work for Cursor sessions than for Claude Code ones.

This doesn't replace what Claude Code's memory does. That system decides on its own what's worth keeping and recalls it without me asking, where the wiki only surfaces the specific file a hook decided to inject. What the wiki adds is a shared, greppable record that exists independent of either tool, readable on a phone as easily as on either machine, in a text editor, or in a completely different tool six months from now, because it's a stack of plain markdown files with a template, sitting on disk, that I could read at 2am with nothing running at all.

The harness's own README states the philosophy bluntly: "The human never writes the wiki. The LLM writes and maintains all of it." That line only holds because the enforcement backs it up. Without the stop hook it would just be an aspiration; with it, a session that closes on three or more real changes has to have written the record of them first.

Scheduling every job on a GPU that can only hold one model

Anish Shrestha — Sat, 29 Aug 2026 21:54:33 +0000

A Raspberry Pi 5 runs DNS for the network, plus dozens of other small services that don't need much CPU or memory but can't go down. A Jetson Orin Nano handles anything that needs a GPU, and its GPU and CPU share one memory pool, small enough to hold one loaded model or one active render job, never both.

One flat network

There's one LAN. Clients and services aren't split into VLANs or separated behind an internal firewall; the upstream router is the gateway, and everything else sits behind it as a peer. If one device gets compromised, there's nothing stopping it from reaching every other device on the LAN. I've thought about segmenting it and decided against it: at this size, a mesh of routing rules is more to maintain than the threat justifies. It stays flat.

The Pi carries DNS and everything that has to be up all the time. It's picked for uptime and low draw, not speed, and I don't let it run anything that could pin its CPU long enough to take DNS down with it.

One model at a time

Load two things onto the Jetson that it can't both hold, and it thrashes. So nothing talks to it directly: every job (a chat request, a background research task, a scheduled scrape) lands in a durable queue first, and a single gateway between that queue and the Jetson admits exactly one job at a time, by priority. That gateway is most of the engineering here. Skip it and two jobs landing at once both try to load into memory built for one, and now the box is fighting itself instead of doing either job.

Priority doesn't preempt. If a background scrape is already loaded and running when a chat request comes in, the chat request doesn't kill it, it queues behind it and jumps ahead of everything else waiting. The alternative, killing a running job because something more urgent showed up, throws away whatever work was already done and risks leaving the model half-loaded. Waiting for the current job to finish costs a few seconds at most; a job that has to restart from zero costs more than that most of the time.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="280" height="56" rx="2"></rect>
  <rect width="280" height="76" rx="2"></rect>
  <rect width="280" height="56" rx="2"></rect>
  <rect width="280" height="60" rx="2"></rect>
  <rect width="280" height="70" rx="2"></rect>
  <rect width="280" height="76" rx="2"></rect>
  <rect width="280" height="56" rx="2"></rect>
  <line x1="180" y1="86" x2="180" y2="126"></line>
  <line x1="180" y1="206" x2="180" y2="246"></line>
  <line x1="180" y1="306" x2="180" y2="346"></line>
  <line x1="180" y1="410" x2="180" y2="446"></line>
  <line x1="540" y1="206" x2="540" y2="446"></line>
  <path d="M 400,180 C 300,230 260,300 300,352"></path>
</g>
<g fill="currentColor">



















</g>

Everything queues through one gateway before it reaches the Jetson, except the RTX box's own render work, which never goes near it.

A second machine, with an RTX 4070 Ti Super, does rendering, and it doesn't go through this gateway for that work. The gateway exists to arbitrate the Jetson's single model slot, and the RTX box was never competing for that slot to begin with, so routing it through anyway would just add latency it doesn't need. It does share one thing with the Jetson: a single LLM call both need. For that one call, the two machines are fighting over the same resource, so it queues like everything else.

The price of running it this way

A Pi 5, a Jetson Orin Nano, and a desktop GPU: together roughly the price of a decent laptop.

hearth is what I built to schedule every job on my homelab's GPU

Anish Shrestha — Sat, 29 Aug 2026 21:54:31 +0000

Every scheduled job in my homelab runs through a platform I built called hearth, and one piece of it is a gateway that admits one job at a time to whichever machine happens to be running a model. It runs on the Pi, and it exists because nothing else that could do the job fit the hardware. Kubernetes-style orchestration (k3s, Nomad) carries its own memory tax, and the Pi was already sitting at 5 GiB used plus 4.4 GiB of swap before adding a scheduler to the pile. Plain systemd timers per service were the other option, and they don't do priority ordering, backpressure, or a shared budget across jobs, which is most of what a GPU with one model slot needs.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="110" height="50" rx="2"></rect>
  <rect width="110" height="50" rx="2"></rect>
  <line x1="170" y1="185" x2="196" y2="185"></line>
  <line x1="350" y1="185" x2="376" y2="185"></line>
  <line x1="530" y1="185" x2="556" y2="185"></line>
  <line x1="710" y1="185" x2="736" y2="185"></line>
  <line x1="245" y1="220" x2="200" y2="286"></line>
  <line x1="280" y1="220" x2="325" y2="286"></line>
</g>
<g fill="currentColor">



















</g>

The two numbers that guard the gateway, the duration estimate and the output cap, both live at the one point that can measure them.

A queue that survives a reboot for free

hearth's queue is SQLite in WAL mode, not Redis or Celery. Memory is the scarce resource on this box, not throughput, and a SQLite file on disk means the queue's state survives a reboot without anything extra. Jobs carry a priority (interactive, user, batch, best-effort), a manifest-driven schedule, retry with exponential backoff, and a dead-letter path for anything that needs a human to look at it. If a job depends on something that's down, it parks instead of burning retries against a dependency that isn't coming back soon.

Each job runs as a subprocess inside its own systemd-run scope, and every resource class maps to a systemd slice with its own memory and CPU limits. An OOM kills the one job in that slice, not the host.

</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="340" height="210" rx="2"></rect>
  <rect width="280" height="150" rx="2"></rect>
  <rect width="340" height="210" rx="2"></rect>
  <rect width="280" height="40" rx="2"></rect>
  <rect width="280" height="40" rx="2"></rect>
  <rect width="280" height="40" rx="2"></rect>
</g>
<g stroke="" stroke-width="2">
  <line x1="20" y1="50" x2="360" y2="260"></line>
  <line x1="360" y1="50" x2="20" y2="260"></line>
  <line x1="740" y1="140" x2="760" y2="170"></line>
  <line x1="760" y1="140" x2="740" y2="170"></line>
</g>
<g fill="currentColor">






</g>
<g fill="">

</g>
<g fill="currentColor">

</g>

Without a slice boundary, one job's OOM takes the whole host down. With one, it's contained to the slice that job runs in.

Adding a resource class costs two files, not one, a manifest entry and a matching slice unit, and there's a test that fails loudly if the second file is missing. A job in a class with no slice unit doesn't error at validation, it dies five seconds into every run with no output and nothing in the journal, so the two-file rule exists to keep that failure mode rare.

The gateway learns durations from the worst recent run, not the typical one

hearth's scheduler will only start a queued job ahead of a fixed time window if it can promise the job finishes before that window closes, which means it needs an honest estimate of how long the job takes. The first version learned that estimate as the median of the last five runs, on the reasoning that one bad night shouldn't poison the number.

One of the jobs that runs through hearth is bimodal: about twenty minutes when it finds real work, about five seconds when it doesn't. Two of its three recorded runs were quiet ones, so the learned median came out to six seconds against a real run that took twenty-one minutes. Under that estimate, the scheduler would have started the job thirty seconds before a workday window it had no chance of clearing. Deleting the two quiet runs fixed the number for twenty minutes, until a fresh quiet run dragged it right back down, which is what proved the fix was against the symptom, not the cause: a bimodal job produces quiet runs indefinitely, and a median can't tell a normal quiet run from an anomaly.

</defs>
<g stroke="currentColor" stroke-width="1">
  <line x1="30" y1="200" x2="790" y2="200"></line>
  <line x1="219" y1="200" x2="219" y2="206"></line>
  <line x1="452" y1="200" x2="452" y2="206"></line>
  <line x1="40" y1="40" x2="234" y2="40"></line>
  <line x1="234" y1="40" x2="234" y2="200"></line>
  <line x1="755" y1="30" x2="755" y2="200" stroke=""></line>
  <line x1="40" y1="70" x2="219" y2="70"></line>
  <line x1="40" y1="110" x2="219" y2="110"></line>
  <line x1="40" y1="150" x2="752" y2="150"></line>
</g>
<g fill="currentColor">







</g>

The median estimate sits next to the two quiet runs and ignores the one real run entirely; the max estimate sits next to the real run.

The estimate is now the maximum of the last five runs, not the median. Verified against the same history: max reads 1277 seconds where median read 641. Under-promising is the dangerous direction here, since a job that starts and gets killed partway wastes the work it did before dying, while over-promising only makes something wait a little longer, which costs nothing. A job with occasional long runs is always scheduled as if it will take the long path, so fewer jobs fit into small gaps in an otherwise empty day. For a scheduler whose entire job is filling a mostly idle day, that's a fine trade. It wouldn't be on hardware that's busy.

The output cap moved out of every caller and into the gateway

Every service calling into the GPU box's model used to hardcode its own token limit, and every single one of those numbers was wrong. Twenty-nine calls in one day landed on exactly 1024 tokens against prompts of 600 to 900 tokens, some of them for a job scoring zero results out of eighty-one it should have judged, because the model's answer got cut mid-word before it finished.

No caller can size this correctly, because the two facts that determine a correct cap don't live where the caller lives. The context window is a property of the model config, and the generation rate is a property of the hardware, and a caller has access to neither, so any number it writes down is a guess. Guesses land low, because a low guess only fails sometimes, and it takes a while to notice.

</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="51" height="24"></rect>
  <rect width="48" height="24"></rect>
</g>
<rect width="620" height="24" fill=""></rect>
<g fill="currentColor">



</g>

The guessed cap barely covered the prompt itself, let alone an answer. The derived cap comes from the timeout budget and the model's measured speed, the two facts only the gateway has.

The fix moved the cap to the one place that has both facts: the gateway derives it from the request's timeout budget times the model's measured tokens per second, and fills that number in whenever a caller sends none. One of the models running through hearth is both the slowest measured, at 13.9 tokens a second against a faster model's 17.5, and the one with the largest context window, so an unbounded generation on that model alone could hold the GPU for close to an hour under a single-slot gateway. Sizing the cap to the same time budget the timeout already enforces costs the same wall clock in the worst case, but comes back with a usable partial answer instead of nothing.

A second GPU doesn't get routed through a chokepoint built for a different GPU

There's a second machine on the network with its own GPU, a dedicated render box that runs its own copy of the render pipeline directly against its own REST API, not through hearth's gateway. The gateway's entire job is arbitrating time on the GPU box's single model slot, and the second box was never competing for that slot in the first place, so sending its traffic through anyway would add nothing but latency.

It does share one thing with the GPU box: a short script-writing call that only the GPU box can serve, since the second box has no model of its own. That one call goes through the gateway, under its own named budget, so heavy experimentation on the render side can't starve the budget an established job already depends on. The rest of that box's work gets its own resource class rather than sharing the class built to stop the GPU box double-booking itself, because serializing a render job behind the GPU box's queue would waste GPU time on hardware that was never in contention to begin with. Reachability is checked the same way hearth checks any dependency, a live probe declared once in the job's manifest, not a guard written into the job itself.

I run browser automation inside a disposable VM

Anish Shrestha — Thu, 27 Aug 2026 08:00:08 +0000

Automating a browser that holds real logins means running code that fetches arbitrary pages on the open internet. That code needs to be treated as something that can be compromised. That's the assumption behind every decision here.

Why a VM instead of a sandboxed process

The browser automation on this network runs inside a disposable virtual machine, launched on demand and torn down after each job. The reason isn't abstract: Chromium's rendering engine assumes 4 KB memory pages, and the Pi it needed to run on uses a kernel built with 16 KB pages, so navigation hung outright on the host. The direct fix, rebuilding the Pi's kernel with 4 KB pages, broke the Pi's Ethernet and took the home network's DNS down with it. That ruled out patching the host to match the browser.

Running the browser inside its own small guest, with its own kernel built the way Chromium expects, sidesteps the page-size problem entirely and never touches the host kernel, its network stack, or the DNS resolver running on that same box. The guest costs about 2 GB of memory while it's running and a few tens of seconds to boot cold, so it stays on-demand rather than always-on.

Placement: the always-on box, not the one running a model

This automation runs on the Raspberry Pi, not the GPU box. The GPU box exists to hold a model in memory, and a browser process, headless or not, competes for the same pool that model needs. The Pi already carries DNS and everything else that has to be up all the time, and a browser job launched on demand under its own memory cap costs that box far less than fighting a model for the same 8 GB would cost the other one. Same reasoning that put DNS on the low-power box in the first place: a job goes where it costs the least to be wrong about.

What the VM guarantees, and what it doesn't

A VM boundary is real protection against one specific thing: if the browser inside it gets exploited at the kernel level, the compromise is stuck inside that guest. It is not, by itself, protection against the compromised guest reaching other things on the network. The guest's outbound traffic gets network-address-translated through the host, and network address translation is a routing detail, not a security boundary, so a compromised browser inside the VM can still open a connection to anything else on the LAN unless something stops it.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="220" height="120" rx="2"></rect>
  <rect width="220" height="120" rx="2"></rect>
  <rect width="140" height="120" rx="2"></rect>
</g>
<g stroke="" stroke-width="2">
  <line x1="260" y1="90" x2="336" y2="90"></line>
  <line x1="490" y1="130" x2="490" y2="180"></line>
  <line x1="460" y1="180" x2="520" y2="180"></line>
</g>
<g stroke="currentColor" stroke-width="1.4">
  <line x1="560" y1="90" x2="636" y2="90"></line>
</g>
<g fill="currentColor">








</g>
<g fill="">


</g>

The VM stops the exploit from leaving the guest's kernel. A separate firewall rule, enforced on the host and keyed to the guest's own uid, is what stops the resulting network traffic from leaving the guest at all.

The fix is a firewall rule on the host itself, not inside the guest, keyed to the dedicated user account the guest's process runs as. It denies that uid's traffic to the local network ranges and allows everything else, loaded at boot and reasserted every time a guest starts, so it holds regardless of what happens inside the VM. Enforcing it on the host rather than trusting the guest to police itself is the whole point: a compromised guest has no way to touch a rule it was never allowed to see.

Two identities, not one guest with two exits

Some jobs need to browse anonymously; some need a real logged-in session. The obvious design puts both behind one guest with a switchable exit, direct or through a VPN tunnel. That falls apart at the browser profile: a single cookie jar that sometimes exits from the home connection and sometimes from a rotating VPN address looks exactly like the pattern session-security heuristics are built to catch, and a logged-in profile behind a hopping exit collects CAPTCHAs and forced re-logins until it locks out entirely.

</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="340" height="180" rx="2"></rect>
  <rect width="260" height="50" rx="2"></rect>
  <rect width="340" height="180" rx="2"></rect>
  <rect width="280" height="45" rx="2"></rect>
  <rect width="280" height="45" rx="2"></rect>
</g>
<g fill="currentColor">






</g>
<g>


</g>

One profile alternating between a home IP and a rotating VPN exit reads as account-takeover behavior to the sites it visits. Two profiles, each with a fixed identity, never trigger that pattern.

The fix is two separate guests, one per exit mode, each with its own user account, its own disk profile, and its own machine identity, chosen at the start of a job rather than switched mid-run. The VPN guest's profile starts empty and stays empty on purpose: anything that needs a real login has to run in the direct guest, which is the design working as intended, not a gap in it.

The kill switch blocked its own tunnel

Building the VPN path surfaced a subtler lesson worth keeping. The firewall rule meant to guarantee that guest's traffic never leaves except through the tunnel was written as: drop everything for that uid except traffic on the tunnel's own interface. It looked correct, and it passed every test against a tunnel that had never successfully connected.

The moment a real tunnel came up, all of that guest's traffic stopped dead, tunnel included. WireGuard's encrypted outer packets inherit the user ID of the socket that generated the data they're carrying, so they left the host still tagged as the browser guest's own uid, on the physical network interface rather than the tunnel interface, and the same rule that was supposed to protect the tunnel killed it outright.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="170" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="150" height="45" rx="2"></rect>
  <rect width="170" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="150" height="45" rx="2"></rect>
</g>
<g stroke="currentColor" stroke-width="1.2">
  <line x1="190" y1="52" x2="226" y2="52"></line>
  <line x1="410" y1="52" x2="446" y2="52"></line>
  <line x1="630" y1="52" x2="666" y2="52"></line>
  <line x1="190" y1="142" x2="226" y2="142"></line>
  <line x1="410" y1="142" x2="446" y2="142"></line>
  <line x1="630" y1="142" x2="666" y2="142"></line>
</g>
<g fill="currentColor">














</g>

The fix didn't change what the browser guest can do. It gave the tunnel's own transport socket a mark the guest itself has no privilege to set, so the fix can't be forged from inside the guest it protects.

The fix tags the tunnel's own outbound packets with a firewall mark that only a process with elevated network privileges can set, something the confined guest doesn't have, and lets that mark bypass the block.

Running a 26B MoE on an 8 GB Jetson by streaming experts from SSD

Anish Shrestha — Sat, 01 Aug 2026 14:00:35 +0000

My Jetson Orin Nano has 8 GB of unified memory. Gemma 4 26B-A4B is a 13.3 GiB download at Q4. This post is about the llama.cpp patch that lets the first thing run the second thing anyway, producing logits that are bit-for-bit identical to the standard code path, and why mixture-of-experts models make that possible when dense models never could.

The idea is borrowed, and worth borrowing

Credit where it belongs: the architecture comes from TurboFieldfare, a Swift and Metal runtime that runs Gemma 4 26B-A4B in about 2 GB of RAM on Apple Silicon. Its trick is to notice that an MoE model only touches a few experts per token. The dense weights (attention, router, shared expert) stay resident, and the routed experts, which are most of the model, live on the SSD. A small per-layer cache of expert slots is filled by parallel reads the moment the router announces which experts the next token needs, and the reads hide behind compute the GPU was doing anyway. The related academic thread is Eliseev and Mazur's expert-offloading work (arXiv:2312.17238), which established that MoE routing is stable enough between tokens for caching to pay off.

TurboFieldfare proves the idea beautifully, but it is a bespoke runtime: two supported models, Apple platforms only, custom kernels for everything. I wanted the same idea for the other cheap 8 GB machine on my desk, a Jetson Orin Nano, and I wanted it for any MoE model I could quantize. So instead of porting the runtime, I grafted the idea into llama.cpp, which already runs on the Jetson and already has Ampere-tuned CUDA kernels.

Streaming without touching a single GPU kernel

The part I like most about the design is that no CUDA kernel changes were needed. llama.cpp evaluates the routed-expert matmul with ggml_mul_mat_id, which takes the expert weight tensor plus a small tensor of selected expert ids and addresses each expert as base + id * stride on the device. The kernel does not care whether base points at 128 experts or at 32; it just multiplies ids by strides.

So the patch allocates, per MoE layer, a persistent pool tensor holding N expert-sized slots plus a tiny slot-id tensor, and at load time it records where every expert's bytes sit in the GGUF file. During decode, a scheduler callback fires on the router's top-k node, reads the chosen expert ids, and consults a per-layer LFU cache: experts already in slots are hits, misses are pread directly from the file into their assigned slots by a small thread pool. The callback writes the slot indices into the slot-id tensor and lets the graph continue. The matmul reads the pool through slot ids and gets exactly the bytes it would have read from the full tensor.

There is one place to be genuinely careful, and it is the reason this patch has a paranoid test. The same expert ids feed several consumers in the graph: the main matmul, the per-expert scale vectors that quantization-aware checkpoints ship, LoRA adapters, and expert biases. Only the main matmul may see remapped slot ids. Everything else must keep the real expert ids, because those tensors stay fully resident and are indexed by actual expert number. Get this wrong and nothing crashes; the model just gets quietly stupider. The test that guards against it greedy-decodes the same prompt with streaming off and on and compares every logit of every step with memcmp. Same kernels, same bytes, different addressing, therefore bitwise identity is the correct bar, not "close enough".

On the Jetson there is a bonus that TurboFieldfare's mmap trick foreshadowed: the Orin's GPU shares memory with the CPU, so the slot pools live in pinned host memory that the disk reads land in directly and the GPU addresses without any copy.

For the IO-hiding part, Gemma 4's graph has a shared dense FFN that runs for every token alongside the routed experts. The patch reorders the layer so the router's top-k comes first, dispatches the disk reads, lets the GPU compute the shared FFN while the NVMe works, and joins right before the expert matmul needs the data. One synchronization per layer, exactly TurboFieldfare's overlap, expressed as a graph reordering plus a two-phase callback.

Numbers from the actual machine

Measured on a Jetson Orin Nano Super (8 GB, MAXN, NVMe) with Gemma 4 26B-A4B QAT Q4, using llama-bench tg128. The baseline is the best you could previously do in llama.cpp: dense weights on the GPU, experts mmap'd on the CPU with --cpu-moe.

Configuration	Decode tok/s
`--cpu-moe` mmap baseline	1.94
Streaming, 32 slots x 8 IO threads, overlap on	2.95
Same, overlap off	2.23

A few observations that the table compresses:

The overlap alone is worth 32 percent, which is the TurboFieldfare thesis confirmed on entirely different hardware: expert IO really does hide behind compute you were already doing.
The NVMe's random-read ceiling (about 1.27 GB/s at this block size, per fio) puts the IO-bound limit near 5 tok/s, so there is headroom left in the IO path, not a wall.
Slot count is a real tuning knob with a cliff. 32 slots per layer (a 3 GB pinned pool) is the sweet spot on 8 GB; 64 slots allocates 6 GB and starves the rest of the system.
Peak RSS stays at 6.7 GB, and the correctness bar held on device: 32 greedy steps on the real 26B, every logit bitwise identical, with 10.5 GiB streamed through the pools during the comparison.

The honest tradeoff: cold prefill drops to about a quarter of the baseline's speed, because streaming mode deliberately stops pre-faulting the expert file (pre-faulting 13 GiB into 8 GB of RAM is its own disaster). Prefill-through-the-pool is the obvious next piece of work, along with O_DIRECT reads.

The GIF at the top is one request against the production gemma4-26b entry in llama-swap. I sent a warm-up call first to get past cold-load and cold-prefill, then recorded a streamed completion with tegrastats logging GPU load and power draw as each token arrived, laid out as three regions in the same terminal so the numbers and the text are visible at once. It settled at 2.09 tok/s where the bench measured 2.95, which is about the gap I'd expect between a clean llama-bench run and one live request with a different prompt. The GPU trace is the interesting part: utilization swings between 0 and 95 percent because for stretches of every layer the GPU waits while the NVMe pulls in the next expert. That is the overlap claim from earlier, on screen.

What generalizing bought

Because the hook lives in llama.cpp's graph layer rather than in any model's code, it works for every GGUF MoE architecture. I verified bitwise identity on three families: Qwen3-MoE style graphs with per-expert scale tensors, Gemma 4's merged gate-up projection, and Qwen 3.5/3.6's hybrid DeltaNet layers. The same flag also gets a 22 GiB Qwen 3.6 35B-A3B onto the same 8 GB board.

Is 3 tokens per second fast? No. It is roughly reading speed, fine for anything asynchronous and painful for chat. But the alternative on this hardware is not a slower 26B, it is no 26B at all. The compromise the patch offers is exactly the one I wanted: model size becomes a choice you pay for in speed instead of a hard out-of-memory wall.

The work lives on the moe-stream branch of my llama.cpp fork: a standalone expert streamer, a --moe-stream flag, pool allocation at load, the dual-id graph integration, the Gemma 4 overlap, and the identity test harness, in seven commits against upstream master.

Thanks again to TurboFieldfare for the architecture. Ideas this good deserve to escape their original hardware.

The run behind that GIF is scrubbable. I put a replay of it on the Learning Lab, with GPU load and power draw lined up against each token.

My context selector beat grep. An agent with grep beat it.

Anish Shrestha — Fri, 31 Jul 2026 14:00:23 +0000

cognitive-cache started from an observation I still think is correct: every coding tool decides what goes into the context window, and almost none of them treat that decision as an explicit selection problem. Cursor greps, RAG systems embed and cosine-search, and plenty of harnesses just stuff files until the budget runs out. The OS analogy is hard to unsee once you have it. The context window is RAM, eviction and retrieval are virtual memory, and token budget allocation is an allocator that nobody has written.

So I wrote one. Six weighted signals score every file in a repo against a task description, and a two-phase greedy submodular selector picks the highest-value set that fits a token budget. Nine languages, no LLM calls, no API keys. It shipped to PyPI and I used it through MCP for a while.

Then I built a benchmark harness, and the harness spent the next two months dismantling most of what I believed about the tool. This is that sequence, in order, because the order is the interesting part.

The first honest measurement said it does not beat grep

The original benchmark was bad in the way most self-benchmarks are bad. It coupled retrieval to LLM patch generation, reported rank-blind full-set recall, had no significance test, and I almost never ran it.

The replacement is LLM-free and deterministic. For each issue it clones the repo at the pre-fix commit, produces a full file ranking per strategy, and scores that ranking against the files the gold patch actually modified, using recall@k and MRR. Every head-to-head comparison carries a paired bootstrap 95% confidence interval, so a difference that excludes zero is real and a difference that straddles zero is not a result.

On the first dataset, 23 hand-curated issues across 8 repos:

strategy	R@1	R@5	R@10	MRR
cognitive-cache	0.170	0.428	0.511	0.540
grep (keyword count)	0.170	0.409	0.559	0.533
lexical (TF-IDF)	0.159	0.362	0.496	0.487

The recall@5 difference over grep was +0.020 with a 95% CI of [-0.067, +0.109]. Grep won at recall@10. Whatever the tool was doing, at that sample size it was indistinguishable from counting keywords.

Two other things fell out of the same run. The per-signal ablation suggested graph_distance, carrying 0.20 of the weight budget, was net negative, and file_role_prior was doing nothing at 0.07. And the budget selector was leaking badly: ranking recall@5 was 0.428 while end-to-end recall under a 12k budget was 0.282, so the selector was discarding relevant files its own ranking had already surfaced.

I nearly acted on the ablation immediately. A weight sweep stopped me, because every candidate reweighting had a bootstrap lower bound of +0.000. The "graph_distance is net negative" reading was itself noise at n=23. The real blocker was not the weights, it was that 23 narrow issues cannot detect or drive an improvement in either direction.

Widening the dataset flipped the result

I moved to SWE-bench, which solves the curation problem by construction: real GitHub issues paired with the gold patch that fixed them, pulled over the HuggingFace datasets-server with no auth. Ground truth is the set of source files the patch modified, and because patches only touch files present at the base commit, every ground-truth file is retrievable.

On a balanced 78-instance subset of SWE-bench Lite across 12 repos:

strategy	R@1	R@5	R@10	MRR
cognitive-cache	0.231	0.538	0.654	0.375
grep	0.103	0.359	0.487	0.236
lexical (TF-IDF)	0.141	0.410	0.462	0.253

Recall@5 over grep: +0.179, CI [+0.064, +0.295]. Over TF-IDF: +0.128, CI [+0.013, +0.244]. Both exclude zero. On a harder 91-issue subset restricted to fixes touching two or more files, the edge held at 0.365 vs 0.252, CI [+0.056, +0.174].

The earlier negative was a small-sample artifact. That cuts both ways, and it is worth sitting with: if I had published the n=23 result I would have been wrong, and if I had published the n=78 result as the final word I would also have been wrong, for the reason in the next section.

The baseline I had not run was an agent

"Beats grep" means beats a naive keyword count. It says nothing about the thing my tool would actually be competing with, which is a coding agent running its own search loop.

So I ran that. Five single-file SWE-bench issues, one each from flask, requests, xarray, pylint, and pytest. Fresh Claude subagents, cold in the repo at the base commit, grep, glob, and read only, cognitive-cache explicitly forbidden.

The agent got 5 of 5, averaging about 3.4 tool calls and 19k tokens, and in each case it pinpointed the mechanism rather than just the file. cognitive-cache got 4 of 5, missing pylint #5859, which the agent found in two calls. Naive keyword grep got 3 of 5.

The token argument inverted at the same time. The agent read a handful of targeted files. A cognitive-cache call returns file contents up to the full token budget, so on these tasks it front-loads more context than a surgical agent consumes. It was costing context, not saving it.

That is n=5 on hand-picked single-file issues, so it is directional rather than definitive. But it moved the honest positioning from "more accurate than the agent" to "fewer roundtrips on a cold start in unfamiliar code", which is a much smaller claim.

The signal that was worth zero, and why

graph_distance measures import-graph distance from the files the task names. It carried 0.20 weight from the day I wrote it, on the reasoning that a fix often lives one hop away from the file the issue mentions.

Across both datasets it earned nothing. On the 91-issue multi-file set, zeroing it changed recall@5 by +0.000 and MRR by -0.006. That was the pre-registered kill criterion, so I was ready to retire it.

Then I pushed back on my own experiment, which turned out to be the useful move. Multi-file is not the same as multi-hop, a regex-based import graph is not a reference graph, and testing a signal as one term in a drowned weighted sum is not a fair test of the underlying idea. All three critiques were fair, so instead of assuming, I wrote a diagnostic over the 91-issue set to check whether the ground truth was even reachable.

The answer killed the revival rather than supporting it. 93% of issues have all their ground-truth files reachable from the entry points. But 87% of fix files are themselves entry points, at graph distance 0, with only about 10% sitting at one to three hops and none further out. The mechanism was never starvation, it was redundancy: real issue text names the files it touches, so symbol_overlap has already ranked those files highly and import distance adds nothing on top.

That is not a bug in the experiment. It is a fact about the task distribution, and it explains the agent result too. On issue-shaped work the target is usually named in the prompt, which is exactly the regime where an agentic grep loop is excellent and a static ranker has little room to add value. Claude Code shipping no code index looks less like an omission and more like the same finding, reached earlier.

The delivery problem, measured

One thread survived long enough to be worth reporting separately. I built a jedi-backed neighborhood tool (definition, callers, in-repo callees, with real scope-aware resolution rather than name matching) and tested it two ways.

Against agents, on two hard cross-file issues where naive grep recall was 0, I ran a 2x2: agents with grep and read, versus agents offered the neighborhood tool as well. Every arm scored 0.5, finding the file tied to the named concept and missing the cross-file one. The agents that had the tool invoked it zero times out of two, despite being told to. Checking the tool's output directly against the symbols those same agents had reasoned about, it surfaced both missed files. The data would have lifted recall from 0.5 to 1.0 if it had been used.

At scale the value is real but modest: across all 91 multi-file issues, neighborhood-of-seeds alone scores 0.164 (worse than grep's 0.252 standalone), while grep-or-neighborhood combined reaches 0.330, recovering 20 of grep's 257 missed fix files. That is +7.8 absolute points, in line with RepoGraph's own reported 9-12%.

Put the two halves together and the conclusion is about delivery, not capability. A passive tool that competes with the grep reflex for invocation loses the invocation. Value like that only lands if it is injected into results the agent is already reading, not offered as one more thing it could call.

Where it ended

I set graph_distance to weight 0, rewrote the README around the scoreboard instead of the pitch, and stopped building. The remaining pitch would have required a forced-injection layer to capture a modest gain, in a niche that already has direct clones, running against the current of agents that search well on their own.

What survives is the measurement rig: the SWE-bench importer that needs no token, the retrieval evaluator, the weight tuner, the multi-hop diagnostic, the neighbor-recovery harness, and the bootstrap CIs that make each of those trustworthy. In a category that ships on anecdotes, that is the part I would build again first.

The generalizable lesson is not "context selection is a dead end". It is that the baseline you skip is the one that decides whether your work matters. I measured against grep for two months because grep was the baseline that let the tool win. The agent was two hours of work and it reframed the entire project.

J-space in practice: using Anthropic's Jacobian lens to decide what an LLM can forget

Anish Shrestha — Wed, 29 Jul 2026 12:07:06 +0000

Anthropic published Verbalizable Representations Form a Global Workspace in Language Models on July 6, and the vocabulary it introduced is suddenly everywhere: J-space, the Jacobian lens, a global workspace inside Claude. Most of the discussion so far is about interpretability and alignment auditing, which is fair, since that is what the paper is about. I had a narrower and more mercenary question: can the workspace tell an inference runtime which parts of the KV cache it is safe to throw away?

Three days after the paper landed, the first pre-registered gate on that question passed. As of this week the signal has replicated on three models and ships inside EVOKE, my KV cache memory manager built on a forked llama.cpp. This post covers what J-space is, why it makes a good KV cache eviction signal, the numbers across Qwen2.5-7B, Qwen3-8B, and Qwen3-4B, and the caveat that comes with them.

What J-space is, in one paragraph

The Jacobian lens is the instrument and J-space is the phenomenon. The lens isolates directions in a model's residual stream that encode a token the model could verbalize next, and those directions form a low-dimensional workspace: roughly 10% of activation variance, concentrated in the middle layers, carrying whatever the model is "holding in mind" at each position. Anthropic's headline application is alignment auditing, reading reasoning the model never voices. What makes independent work possible is that they released companion code under Apache-2.0 along with fitted lens matrices for open Qwen models on Hugging Face, so anyone can apply the lens to an open-weights model on a single GPU.

The systems problem: KV cache eviction

Every long-running LLM session eventually outgrows its KV cache budget. An agent session in a coding harness crosses tens of thousands of cached tokens within a few turns, and something has to decide which entries stay in GPU memory. The standard answers, H2O and SnapKV, rank cache blocks by accumulated attention history: keep what the model has been attending to, evict the rest. StreamingLLM adds protected sink tokens and a recency window.

These policies share a structural weakness. They are backward-looking, so they can only rank blocks the model has already used. A fact planted at turn 1 that will matter at turn 14 looks cold the whole time, and it gets evicted exactly when the budget tightens.

The hypothesis: workspace content predicts KV reuse

The lens readout suggested a forward-looking alternative. If a position carries workspace content, meaning the model is holding something in mind there, then its KV entries should be disproportionately likely to be read by later reasoning. That is a content signal, computable at prefill time from the residual stream alone, before any attention history exists.

The offline validation lives in j-space: lens application, a family of workspace statistics over the readout (kurtosis of block means, transport ratios, energy concentration, computed across layers), block aggregation, eviction simulation, and probe distillation. Because a signal this convenient deserved suspicion, I pre-registered three kill gates before running anything:

Fact ranking. The best workspace statistic has to beat SnapKV at ranking the task-critical block, by a pre-set AUC margin, or the project dies.
Masked eviction. With only 25% of the cache retained under the workspace policy, the planted fact has to stay answerable.
Probe distillation. Applying the full lens at inference time is too expensive, so a ridge probe distilled from the workspace readout has to reproduce the ranking at Spearman 0.8 or better. The deployable artifact is one dot product per position.

Results across three Qwen models

All three gates passed on all three models, with no hand-tuning on the replications.

Model	Fact-AUC (workspace)	Fact-AUC (SnapKV)	Eviction @ 25% budget	Probe Spearman
Qwen2.5-7B-Instruct	0.891	0.622	35/36 vs 25/36	0.959
Qwen3-8B	0.865	0.563	35/36 vs 14/36	0.944
Qwen3-4B	0.939	0.541	36/36 vs 13/36	0.946

The eviction column reads: episodes where the planted fact survived eviction and was answered correctly, workspace policy vs SnapKV at the same budget. On Qwen3-4B the workspace policy saturates at 36/36 at every budget tested.

The distilled probe also holds up live. Inside EVOKE's agent bench, the workspace signal passes the fact probe at every cache budget (512, 1024, and 2048 tokens) with recovery disabled, while H2O fails all three budgets and SnapKV passes only at 2048. Measured decode overhead was -0.6%, which is noise; the probe is a dot product per position at prefill, and nothing runs during decode.

What transfers and what does not

Here is the caveat, and I think it is the most interesting finding in the project. The winning statistic is model-dependent. On Qwen2.5-7B and Qwen3-8B it is kurtosis of block-mean workspace scores at deep layers (layer 23 and layer 30 respectively). On Qwen3-4B it is a transport-ratio statistic at layer 12 of 36. If I had frozen the 7B statistic and shipped it as "the" signal, the 4B replication would have failed. What transfers is the statistic family plus the automated procedure that selects the winner per model on held-out episodes, not any single magic number.

The other scope limits are stated plainly in the paper: three models so far, all Qwen; the probes are fit on the benchmark's own episode family; and the task family is long-horizon fact recall. LongBench-style suites and non-Qwen architectures are open work. As far as I can tell this is the first interpretability-derived KV eviction signal, and I would much rather over-state the limits than the result.

Where it ships: EVOKE v2.0

EVOKE treats the KV cache the way an operating system treats memory. Blocks are evicted under budget pressure and recovered recompute-free through save/restore primitives added to a forked llama.cpp, 20 to 32 times faster than re-prefilling the same tokens. Version 2.0 adds the workspace probe as a first-class eviction signal alongside live attention capture, harness priority tags, task-focus coherence, and recency, and includes the paper PDF with the full three-model evaluation.

You can watch the signal work in the live demo on Hugging Face Spaces, which runs three arms side by side on Qwen3-4B: a plain baseline, EVOKE with its attention scorer, and EVOKE with workspace eviction. There is also a benchmark walkthrough on my learning lab.