DEV Community: Anish Shrestha

My Claude Code sessions work as a team, and they message each other

Anish Shrestha — Tue, 13 Oct 2026 04:03:51 +0000

TL;DR: I run one Claude Code session per repository and let them message each other instead of editing each other's code. The messaging ships with Claude Code (ListAgents and SendMessage). Running six to nine of them at once works because of a harness that keeps cache costs visible, pushes to my phone when a session is waiting, logs every action a guard blocks, and carries context across resets. Over three and a half weeks, 81 sessions sent each other 624 messages, and cross-repository requests that would have gone through me now go straight from one session to another.

On a busy day six to nine Claude Code sessions are active on my machine in the same hour, each one sitting in its own repository: Loom, the research repos that feed it (unlearn, redraft, JEV-like-research, memories-for-llms), second-brain, a memory service, wardrowbe, the harness that configures all of them. Each session owns its repository and nothing else. When one needs something from another repository, it doesn't open those files and edit them; it messages the session that owns them, and that session does the work in its own domain and replies.

Between 18 September and 12 October, 81 sessions across 13 projects sent each other 624 messages. The busiest day was 1 October, with 200. The most sessions active in a single hour was nine, on three separate days.

One session per repository

The setup is easiest to see in how I start a day's work. This is the opening prompt of a Loom session on 29 September, verbatim:

we were working on some advanced feature for loom where we were brigining up our various research we have like evoke(unlearn), jev, memories-for-llms, redraft etc. ... This is opus main session. You can spawn other child sonnet with appropriate effort level to do your work. main session is the orchestration and managing session. you will have all the context and you will pass in around as needed. AND additionally there are other agents on each project ready to colab with you on any related tasks. you can directly contact them and make them work on their own domain if needed. good luck!

Twenty minutes later that session called ListAgents and found itself as loom-1d, with four peers that had all started 23 minutes earlier: unlearn-e6, redraft-20, jev-like-research-65 and memories-for-llms-8a. Loom is the hub of that team because it's where the research gets integrated; each research repository has its own session that knows its own code, its own experiments, and its own history.

There are two levels of delegation here and they're different things. Inside one session, the Opus main session spawns Sonnet subagents for bounded pieces of work and reviews what comes back. Across sessions, the peers are full Claude Code sessions with their own context, their own permission prompts, and their own person approving things, which happens to be me in every case.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="200" height="56" rx="2"></rect>
  <rect width="200" height="50" rx="2"></rect>
  <rect width="200" height="50" rx="2"></rect>
  <rect width="200" height="50" rx="2"></rect>
  <rect width="200" height="50" rx="2"></rect>
</g>
<g fill="none" stroke="currentColor" stroke-width="1">
  <rect width="80" height="34" rx="2"></rect>
  <rect width="80" height="34" rx="2"></rect>
  <rect width="80" height="34" rx="2"></rect>
</g>
<g fill="currentColor">





</g>
<g fill="currentColor">










</g>
<g stroke="" stroke-width="1.5" fill="none">
  <line x1="240" y1="78" x2="380" y2="160"></line>
  <line x1="720" y1="78" x2="580" y2="160"></line>
  <line x1="240" y1="282" x2="380" y2="198"></line>
  <line x1="720" y1="282" x2="580" y2="198"></line>
</g>
<g stroke="currentColor" stroke-width="1" fill="none">
  <line x1="440" y1="206" x2="390" y2="298"></line>
  <line x1="480" y1="206" x2="480" y2="298"></line>
  <line x1="520" y1="206" x2="570" y2="298"></line>
</g>

The team from the 29 September kickoff. Highlighted lines are cross-session messages between peers; the dashed boxes are subagents the hub spawns and discards inside its own session.

How they talk

The messaging itself isn't something I built. Claude Code ships two tools for it: ListAgents shows the other sessions running on the machine, and SendMessage delivers text to one of them over a local Unix socket. The receiving session sees the message wrapped in a <cross-session-message from="..." from-name="app-c6"> block at its next tool round, so a busy session finishes the step it's on before reading it, and replies by sending back to the from address. My harness doesn't wrap, extend, or configure any of this. What it does is make running this many sessions at once cheap enough and safe enough to keep doing, which comes further down.

The sessions sit in panes in herdr, a terminal multiplexer that knows which panes hold coding agents and whether each one is working, blocked, or done. On 10 October herdr agent list showed four Claude sessions in one workspace: three working, in an app repository, the harness and my home directory, and one blocked, in wardrowbe, waiting on me. Herdr's own log has up to about twenty Claude panes alive on a single day.

What they say to each other

The useful part is what the messages are for, so here are five of them as they were sent.

An API change and a release, across two repositories. A Rust memory service of mine depended on second-brain through a pinned git revision and needed a private method made public. On 18 September its session wrote to the second-brain session:

[The memory service] needs a second-brain release it can depend on from crates.io instead of its git rev pin. ... Two asks: (1) make MaintenanceOps::backfill_embeddings public, or expose it through a public method that runs only the embedding backfill, not the rest of repair() ([it] must not run the cross-project merging); (2) cut a release ... Please reply with the version number once it is published, or tell me if something blocks it. No rush over your current audit work.

The reply came fifteen seconds later, and it pushed back on the order: second-brain was midway through an audit whose fixes also affected the memory service, so the release should wait for them and would likely be 0.9.0, "because it adds public API and changes behaviour". It also said publishing to crates.io can't be undone, so it was getting my go-ahead first. Five days later the release landed:

second-brain 0.9.0 is on crates.io with a public backfill_embeddings: all six crates are published (core, adapters, sync, mcp, api, cli), tagged v0.9.0.

The memory service moved onto it, and Loom, which runs it, checked the change and confirmed it needed no changes on its side.

A turn on the GPU. One GPU box is shared by three research repositories, and on 27 September memories-for-llms needed it for an experiment window. It asked Loom to "book 100 minutes on [the GPU box]; I expect it to take 45 to 55." Loom, acting as the scheduler, told the unlearn session to step off:

Please step off [the GPU box] now. The user has cleared window 1 (memories-for-llms) to start immediately, booked for 100 minutes.

Stop your scheduled task and any process of yours using the GPU, and reply to me with "stopped" once VRAM is released (show the nvidia-smi line).

In the same minute it corrected memories-for-llms' stop script, because "killing every llama-server by image name would take down anything of anyone's on the box." The unlearn session couldn't comply right away: it was in plan mode, which blocks anything that changes the system, so it said it was asking me. I approved it, it replied stopped, VRAM released, and Loom opened the window: "unlearn-97 has stopped EVOKE, and nvidia-smi shows the GPU idle." From the booking request to the window opening took five minutes.

A merge with ground rules. On 3 October the verdict session in JEV-like-research needed work merged into Loom's main. Loom had already set the terms:

Please do it on a branch in its own worktree rather than in the [main loom] checkout: I merge several branches into main from that checkout through the day, and uncommitted edits there would collide with those merges.

Verdict did that, ran the gates, and asked Loom to merge the four commits. Loom asked it to hold its service restart until the merge was done, then reported back: "merged into loom main and pushed; gates passed. You can go ahead with the verdict restart."

Waiting for everyone before a deploy. Several sessions often work in one app repository at once. Before a production deploy on 4 October I asked one of them to wait for all the others, and it broadcast:

Please message [me] when your work is committed and finished; I'm waiting on you before a prod deploy. ... send me one line covering three things: (1) you're finished and the tree is clean of your changes, (2) your last commit hash, and (3) whether your work adds a database migration or needs anything special in prod ...

One session answered "Not finished yet", still editing UI files; thirteen minutes later it reported "Finished: every change from my session is committed and the tree has none of my changes; I won't commit again," with its last commit hash and its one migration, and the deploy went out with its commits included.

Asking a peer what it measured. On 4 October Loom wanted a seven-hour training job made faster without a bigger GPU, and asked memories-for-llms, which had trained every adapter so far, for "your measured knowledge, not guesses". Two minutes later the answer started: "almost none of the knobs you named was ever measured. Details with citations follow; I read files only and did not touch [the GPU box]."

Not every message is between related projects. On 12 October a session making a video built a new skill straight into ~/.claude/skills, then told the harness session to adopt it. The harness session moved it into the harness, ported a follow-up, and replied: "collage-film now lives in the harness at [harness]/configs/shared/skills/collage-film; please don't edit ~/.claude/skills/collage-film any more, because install.sh overwrites it from there."

The harness under it

All of these sessions run on the same harness: one repository that renders the configuration, hooks, rules, skills and MCP servers for Claude Code, Cursor, Codex and opencode. Running nine sessions at once changes what the harness has to do, and most of the pieces below exist because of a specific failure.

Cost

The costliest failure the harness has recorded is a large context whose prompt cache expired. A Claude Code main session's cache lasts an hour idle and a subagent's about five minutes; the first turn after a longer gap re-sends the whole context at the cache-write price. Orchestrating sessions had been running subagents for 8 to 24 hours while the main session waited, so the rules now cap subagent work to minutes, never resume a large-context agent after a gap, and keep builds and long suites in background shells on the main session.

The cache-guard mod, further down, puts that clock on the status line so the rule doesn't depend on anyone remembering it.

The /orchestrate command splits a plan into independent workstreams, has a bigger model review the split first, runs the streams as parallel subagents in their own worktrees where files would collide, and leaves the merge and the full test run to the main session.

Attention

With nine sessions, the bottleneck is noticing which one is waiting on me. A hook sends a push to my phone when any session has been blocked on a question or a permission for fifteen seconds, or has finished a turn for sixty, and cancels the push if I answer first. In one eighteen-hour stretch on 12 and 13 October it sent 35 blocked pushes and 16 finished ones, and 34 of the 35 blocks were answered.

Hygiene

Parallel sessions leave worktrees behind. The rule is that whoever creates a worktree removes it, and a sweep at session start removes any worktree idle for a day whose commits have all landed, saves uncommitted files to .git/worktree-salvage/, and keeps any branch with work that hasn't landed. Each session's status files, block log entries and handoff digests are keyed by its session id, so two sessions in the same repository never write over each other's state.

Guards

Hooks block specific things before they happen, and every block is logged so I can replay a guard change over real transcripts before shipping it. Between 3 and 12 October they blocked 140 actions: 45 bare rm/cp/mv calls that would have hit interactive aliases, 33 writes breaking the code-quality rules (em dashes, smart quotes, emoji, inline imports), 26 narrating code comments, 18 unbounded reads or command outputs on a main session, 17 attempts to stop a session after code changes with nothing recorded in the wiki, and one remote shutdown of the GPU box.

Memory

Two stores outlive any single session. second-brain ingests every Claude Code and Cursor transcript and consolidates it into memories any session can recall over MCP; it currently holds 11,255 live memories distilled from 4,880 transcript files. The wiki is an Obsidian vault that sessions write decisions, plans and a dated devlog into, enforced by hooks, which I wrote about in an earlier post. Between them, a session that starts cold on a repository it hasn't touched in a week has somewhere to look first.

Mods

Hooks can block an action or add text to the conversation, but they can't show anything. Claude Code mods can: a plugin can add a segment to the status line, a band above the prompt, a toast, or a slash command. The harness ships five, all added on 11 October, and with them each of the nine panes tells me at a glance which session is about to get expensive, which one hit a guard, and which one hasn't written its scope or devlog.

cache-guard learns the real cache lifetime from the hits and misses it observes, then shows the time left on the status line: cache 42m · agents 3 · 1 cold. Once the cache has expired, the segment turns yellow and shows what the next prompt will cost to re-write (cache cold ≈ $X). If an idle prompt would re-bill a dollar or more, it stops and asks whether to send anyway, compact first, or keep the prompt as a draft. After a cold turn that re-wrote 50,000 tokens or more, it raises a toast with the cost.
guard-watch counts every call a guard denied and shows N blocks on the status line. It raises a toast when a guard blocks a subagent, because those blocks never reach the main transcript, so a subagent could otherwise route around a guard without the main session knowing it had been refused.
nudge-band puts a band above the prompt when .scope.md is missing or stale, when nothing has gone into the wiki this session, or when untracked files are piling up, with buttons that pre-fill /scope, /devlog or a cleanup prompt. It replaced three shell hooks that did the same job by injecting reminders into the conversation, where every reminder cost context in every later turn.
handoff runs when a session compacts or is cleared. A shell script collects the facts (files touched, repository state, the last plan, the newest turns) without a model call, and a tool-less fork of the session writes the goal, the next step, the decisions, what was ruled out, open questions, and how to verify. The session that published this post started from a cleared context, read one of those digests, and already knew the draft had been reviewed, which four problems the review found, and that a second post was sitting uncommitted.
context-audit adds /context-audit, which totals what the context is spending on MCP servers, skills, memory files, agents and commands, and names up to five things to trim.

</defs>
<g fill="currentColor">



</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="150" height="44" rx="2"></rect>
  <rect width="150" height="44" rx="2"></rect>
  <rect width="150" height="44" rx="2"></rect>
  <rect width="150" height="44" rx="2"></rect>
  <rect width="720" height="70" rx="2"></rect>
  <rect width="230" height="48" rx="2"></rect>
  <rect width="230" height="48" rx="2"></rect>
  <rect width="64" height="44" rx="2"></rect>
</g>
<g fill="currentColor">







</g>
<g fill="currentColor">


</g>
<g stroke="" stroke-width="1.5" fill="none">
  <line x1="280" y1="72" x2="320" y2="72"></line>
  <line x1="470" y1="72" x2="510" y2="72"></line>
  <line x1="660" y1="72" x2="700" y2="72"></line>
</g>
<g stroke="currentColor" stroke-width="1.1" fill="none">
  <line x1="315" y1="210" x2="315" y2="268"></line>
  <line x1="665" y1="210" x2="665" y2="268"></line>
  <line x1="850" y1="175" x2="878" y2="175"></line>
  <line x1="205" y1="94" x2="205" y2="138"></line>
  <line x1="395" y1="94" x2="395" y2="138"></line>
  <line x1="585" y1="94" x2="585" y2="138"></line>
  <line x1="775" y1="94" x2="775" y2="138"></line>
</g>

Every session runs on the same harness and shares the same two stores. The highlighted lines between panes are Claude Code's own cross-session messages; everything else is the harness.

Where it breaks

Sessions die and their addresses go stale. A session's address is its socket, and a restarted session gets a new one. A send to the old one fails with ENOENT ... the peer process may have restarted, so this socket path is stale. The sender usually recovers by listing peers again; when the peer is gone for good, the best it can do is what one did on 30 September: "The wardrowbe-b8 session is gone (its socket no longer exists and it isn't among the live peers), so I'm saving the report where it can be picked up later." When an orchestrating session was about to be replaced, it had every peer also append its reports to a shared inbox file, so a change of name wouldn't lose anything.

A reset session forgets what it asked for. The memory service asked second-brain for the same release twice, five days apart, because its second session had no memory of the first request. Redraft reported two items as outstanding that had been done three days earlier, then corrected itself: it had "only looked at loom's committed copy and the redraft repo."

Messages go to the wrong session. On 11 October Loom sent a landing-page brief to the wrong peer and had to follow up: "Please ignore my earlier message about the Loom landing page; I sent it to the wrong session by mistake. No reply needed." The deploy broadcast also reached a Loom session, which answered that it wasn't working in that repository.

Every session commits as me. When the verdict session found a commit it hadn't made in a checkout it was working in, it asked the harness session whether it was the author. It wasn't, and nothing in git could say which session was, because all of them commit under the same name.

Two sessions can start the same work. Once, a session found a peer on the same project already building the thing it had just planned to build.

Each session still has its own human gate. unlearn couldn't stop its GPU process until I approved it, and second-brain wouldn't publish to crates.io without my go-ahead, which is how it's meant to work: a session can ask a peer to act, but it can't use the peer to get around an approval it would need itself.

What it adds up to

Commit counts don't measure this well, and I checked. My weekly commit rate across every repository more than doubled after August, but almost all of the increase is Loom, which didn't exist before September. Outside Loom the rate didn't rise, and in some repositories it fell. What changed is how much runs in parallel and who carries information between the parallel parts. On 1 October nine projects had live sessions, and they exchanged 200 messages that day: API requests, GPU bookings, merge requests, deploy barriers, measured answers with citations. I didn't relay any of them. Some started with a request from me, like the deploy wait, and some stopped for my approval partway, but what reaches me now is mostly the decisions only I can make: an approval in plan mode, a crates.io publish, a production deploy, and a phone push when one of them is waiting.

What to take from this

If you run more than one agent session, these are the parts that carry over to any setup, not only mine:

Give every repository one owner, and make cross-repository work a request. The owning session knows what a visitor doesn't: that a migration number is reserved, that the main checkout takes merges all day, that killing every llama-server by name would take down someone else's job. In four of the five exchanges above, the owner caught something the asker would have broken or got wrong.
Keep the human gate per session. A peer can ask another session to act, but it can't borrow that session's permissions. Plan mode and the crates.io approval both held while the requests came in over messages.
With many sessions, the bill is cache expiry. The expensive moment is the first prompt after an idle hour, when the whole context is re-written. Put the cache clock where you can see it, and keep subagent work to minutes.
Your attention is what runs out first, so route it. A push when a session is waiting, cancelled as soon as I answer, got 34 of 35 blocked sessions answered in one eighteen-hour stretch, without me cycling through panes to find them.
Assume sessions die. Addresses go stale and a reset session forgets what it asked for, so anything that matters has to land somewhere that outlives the session: a shared inbox file, a handoff, the wiki, second-brain.

I didn't build a framework for any of this. Claude Code supplied the messaging, and I added the harness, the mods and the rules around it until nine sessions could run side by side all day without me relaying between them.

Open Jev research: a month of chasing coverage at 5% error

Anish Shrestha — Mon, 12 Oct 2026 10:03:52 +0000

TypeSafe AI put Jev into limited early access on 15 September 2026. It is a closed "System One" model: you send it a state and a list of typed questions, and it returns a probability for every option instead of generating text. The open clones already matched its accuracy, but Kev's README reports that the best of them, Kev-9B, can answer only 0.45 to 0.57 of decisions automatically at a 5% error rate, against 0.70 for Jev. Between 23 September and 12 October I worked on that gap with open models and a single 16 GB RTX 4070 Ti Super, and when the answer for my own use turned out to be "use the models that already exist", I asked whether decisions of that quality could run on a CPU.

None of the models I built beat the released ones on the metric the series is about. On the same 1,046-question general set, every 4B to 9B model I scored lands between 0.550 and 0.557 coverage at 5% error: Kev-4B, decider-4b, Cloudflare's Clef-flash, and my llama.cpp port of Kev-4B. Every attempt to get a smaller model there (a raw MoE screen, Kev's training recipe on Granite-H-350M, distilling Kev-4B into Kev-0.8B) finished at or below Kev-0.8B's 0.276. Along the way I built a decision router that serves these models to my agent harness, an adversarial panel for shell-command risk, and three small public models that you can try side by side in a demo Space. This post is the hub for the series: what Jev is, the one metric every post uses, how the month went, and a map of the results.

Jev and its open copies

Jev exposes one endpoint, POST /v1/systemone, which takes a state (text or JSON) and a list of questions. There are three question types: Choice picks one of up to 255 options and returns per-option probabilities, Score rates against ordered levels, and Noul returns a yes/no probability. TypeSafe says the model is a transformer trained only on synthetic data and post-trained with "Reinforcement Learning for Calibrated Decisions", and it has published no paper, weights or parameter count. Its headline figures (193.6 times faster, 444.6 times cheaper) come from TypeSafe's own workflow evals against the average of two frontier LLMs, while independent tests found about 2.9 times the speed and 12 times lower cost. Archer Hume's black-box probing suggests the likeliest design: encode the shared state once, run each question as its own branch over that prefix, and read the answer straight off the option logits.

Two open projects speak the same API. Kev, by Jared Palmer, ships at 0.5B, 0.8B, 4B and 9B, and Laya, mostly written by Nandakishor of Convai Innovations (Apache-2.0), ships a server that speaks Jev's wire format. Two more arrived during the month: decider-4b (Mapika, Apache-2.0) and Cloudflare's Clef-flash (9B, post-trained from Qwen3.5-9B, Apache-2.0, released 1 October). The landscape post covers how these differ.

The metric: coverage at 5% error

A decision model is useful in an automated pipeline only if you can trust the answers it is sure of and send the rest to something slower. That is selective prediction: the model answers when it is confident and abstains otherwise. Coverage at 5% error measures how much it can answer under that rule.

For one task, take every question, read the model's confidence as the probability of its top option, and sort the questions from most to least confident. Accept answers from the top of the list. Coverage is the largest fraction of the task you can accept while the error rate among the accepted answers stays at or under 5%. A task where the 60 most confident of 100 answers contain three mistakes covers at least 0.60. Questions with the same confidence are accepted or rejected together, and a question the model failed to answer is never accepted but still counts in the total. This is coverage_at_error in Kev's kev/metrics.py, which forge's scorer reuses per task. The series reports the macro figure: coverage computed per task, then averaged over tasks with equal weight.

The metric rewards how well confidence is ordered, and that often disagrees with accuracy. Laya is more accurate than Kev-4B on the emotion task (0.613 against 0.550) yet covers nothing there, because its most confident answers are wrong. A raw Qwen3-4B-Instruct readout answered 72% of the general set correctly and covered 0.091, because its confidence does not separate right answers from wrong ones.

Most numbers in the series come from one instrument, the general set: the development split of Kev's transfer-v9 evaluation, 1,046 questions over 11 tasks that mix public benchmarks (mmlu, sciq, qnli, emotion, paws) with probes for buried evidence, rule composition and unknowable questions. The bar for the month was about 0.55, where Kev-4B (0.555) and Clef-flash (0.550) sit.

How the month went

The first four days, 23 to 26 September, went into building my own system. After surveying the open clones I chose to build forge, a synthetic decision-data engine and coverage-first evaluation, with Kev's trainer as the first training backend. forge reproduced Kev's trainer bit for bit, a leakage audit caught the engine giving answers away through look-alike sentences, and a pilot on long buried documents was registered at about 40 GPU-hours.

From 27 September the work turned toward real decisions. Loom, my agent harness, asked whether a decision model could take calls like command risk and routing off its LLM. That became verdict, a service that answers Loom over the System One API. On 1 October an experiment called K1 found that sending each task to whichever backend covers it best would gain only 1.2 points of coverage over Kev-4B alone, against a 5-point bar, and since a pipeline keeps long context in the LLM and asks the decision model short questions, the long-document pilot was solving a problem nobody had. forge was parked, and Kev and Laya became the models.

On 2 and 3 October I built a command-risk panel from public security datasets, checked a public leaderboard's ranking against my own panel, and turned verdict into a router once Clef-flash came out.

On 5 October the goal changed to decisions near Clef-flash quality on a CPU ladder, from a Raspberry Pi 5 with 8 GB up to a laptop x86 CPU. A single-pass decision is a prefill, which is compute-bound, so its cost tracks active parameters times tokens and offloading or streaming weights does not make it faster. I screened small bases, trained Kev's recipe on one, distilled Kev-4B into Kev-0.8B, published the three resulting models, and ported Kev to llama.cpp.

Results map

The posts go up one at a time, so titles without a link are not out yet.

#	Post	Headline result
2	The open Jev landscape	JevBench ranks Kev-4B 37th, but on our panel decider-4b only ties it (0.556 against 0.555 coverage)
3	forge, a synthetic decision-data engine	Replays Kev's trainer bit for bit (all 314 loss windows and 392 adapter tensors identical); K1's +1.2 pp against a 5 pp bar parked it
4	The command_risk panel and its source leak	A character n-gram logistic regression scores 0.965 AUROC, against 0.798 for Clef-flash and 0.776 for Kev
5	verdict, a decision router	Clef-flash ties Kev-4B on false alarms at full risky recall (60.5% against 61.8% on test) and now heads every chain
6	Small decision models on a CPU ladder	Kev's recipe on Granite-H-350M reaches 0.132 coverage against Kev-0.8B's 0.276
7	Distillation buys accuracy, not coverage	Soft targets add 2.4 pp accuracy over gold labels; coverage stays at 0.250 against a 0.33 kill line
8	Kev on llama.cpp	Kev-4B at Q8 covers 0.557 against 0.555 in torch, at about 4.5 GB instead of 18 GB, with no speed gain

The chart puts every general-set number from the series on one axis. The 4B to 9B models are within 0.007 of each other however they were trained or served, and below 1B nothing I tried, whether a different base, more teacher-labelled data or Kev's full recipe, moved coverage above the released Kev-0.8B. The distillation result explains part of that: the teacher's probabilities carried answer knowledge but not a better ordering of confidence, and ordering is what this metric measures.

Jev's 0.70 was reported on Kev's benchmark, not on the general set, so none of these posts measures the gap to Jev directly. What the month did establish is that the open 4B to 9B models agree at about 0.55, and that getting a sub-1B model near that number is a confidence-ranking problem the methods I tried did not solve.

The open Jev clones: three ways to read a decision, and a leaderboard that does not transfer

Anish Shrestha — Mon, 12 Oct 2026 10:03:50 +0000

Within a week of TypeSafe AI opening early access to Jev, its closed "System One" decision model, GitHub was full of open look-alikes, and by October an independent board, JevBench, was ranking more than a hundred systems on sealed items. I needed one of these models to answer decisions for Loom, my agent harness, so the question for this post is practical: which open clone should sit behind that endpoint, and can I trust the public ranking to tell me?

The clones split into three mechanism families, and the two I studied most closely, Kev and Laya, sit in different ones. JevBench ranks Kev-4B 37th with a score of 39.9, behind decider-4b at 55.8 and behind a stock Qwen3-4B-Instruct read at its option logits at 54.1. When I scored those models on my own panel, by coverage at 5% error (the share of questions a model can answer on its own while keeping its error on those under 5%; defined in the series overview), decider-4b tied Kev-4B at 0.556 against 0.555, and the stock Qwen readout covered 0.091. The ranking did not transfer, and Kev-4B stayed the pinned model.

What the clones are copying

Jev takes a state (text or JSON) and a list of typed questions, and returns probabilities instead of text. A Choice question picks one of up to 255 options and returns a probability per option, a Score question rates against ordered levels, and a Noul returns a yes/no probability. TypeSafe says it is a transformer trained on synthetic data and post-trained with "Reinforcement Learning for Calibrated Decisions" (RLCD); there is no paper, no weights and no parameter count.

The most useful public guess comes from Archer Hume's black-box probing of roughly 10,000 API calls. Latency tracks input tokens rather than option count, and content injected into one question does not leak into another, which fits a causal transformer that encodes the state once and runs each question as an isolated branch over that shared prefix (the pattern from Hydragen and DeFT), ending in a softmax over option logits with no text generated. A minority view (djev) holds that Jev is a diffusion model; Hume argues diffusion is not needed to explain what the API does.

Three ways to read a decision

Every open clone has to answer the same two questions: what hidden state does the decision head read, and how does that become a distribution over options? The answers fall into three families.

Pointer head on a causal LLM (Kev by Jared Palmer, and TinyJev). Kev packs one sequence per request: <state> ... <q> instruction <opt> option1 </opt> <opt> option2 </opt> ... <decide>, with one <q> ... <decide> block per question. The delimiters reuse rarely used Qwen special tokens, so no embeddings are resized. A block-causal mask lets each question's tokens attend to the state and to their own question block only, and position ids restart after the state for every question, so questions share the state but cannot read each other. The head has no vocabulary readout at all: two linear projections map the <decide> hidden state and each option's </opt> hidden state into a 256-dimensional space, the scaled dot products become one logit per option, and a softmax over options gives the distribution. The base (Qwen3.5 at 0.8B, 4B or 9B) is frozen with a rank-16 LoRA adapter. Qwen3.5 interleaves Gated DeltaNet layers, which are recurrent and cannot honor a block mask, so on those bases each question runs as its own causal row continuing a cached state prefix, and a cache miss costs one forward pass instead of one per question. Calibration is a single post-hoc temperature per checkpoint (2.41 for the Kev-4B I pin).

Mask-marker encoder (Laya by Nandakishor Mukkunnoth of Convai Innovations). Laya is an encoder, ModernBERT-large at 421M parameters. Each question becomes one sequence, [CLS] <qtype> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP], with a [MASK] token inserted before every option. A two-layer transformer head runs over the encoder output, and a small shared scorer turns the hidden state at each [MASK] position into one logit per option; a softmax over those, with a temperature per question type and option-count bucket, gives the probabilities. All of a request's questions are padded into one batch and answered in one call. Because every option has to fit in a fixed token budget, the option count is capped: on Banking77's 77 labels each option gets about 3 to 4 tokens. Training uses a strictly proper scoring rule (log score plus spherical score, plus a ranked probability term for ordinal questions) as a policy-gradient reward, which Laya's documentation calls RLCD.

First-token logit readout (SemIf, formerly OpenJev, and jevfire). These need no training. The options are written as single tokens (letters A, B, C and so on), a stock LLM runs one forward pass, and the decision is a softmax over the logits of those candidate tokens at the first answer position, the same trick as FIRST (arXiv 2406.15657) for listwise reranking. jevfire adds shared-prefix KV in vLLM so many questions over one state stay cheap. Two failure modes recur: ambiguous single-letter tokens (space-prefixed against byte-level) can silently corrupt the readout, and letter-slot variants run out of single tokens at about 26 to 52 options.

Kev and Laya on their own numbers

On sources it never trained on, Kev-9B reaches 0.852 test accuracy against Jev's 0.857 on the development split, a comparison Kev's own README flags as uncontrolled. The gap that remains is coverage at 5% error: Kev covers 0.45 to 0.57 against Jev's 0.70. Kev's author tried to close it during training (label smoothing, Brier and focal losses), and that round came back negative under its pre-registered gate.

Laya reports beating Jev on its own benchmark (typed-decisions accuracy 0.766 against 0.727, ECE 0.081 against 0.246), but every Jev number in those comparisons is a published third-party figure, so each one is cross-study. One independent test put Laya at 0.590 against Jev's 0.974, the lowest accuracy of the clones it compared and the fastest at about 30 ms. Laya's documentation is open about its own losses: Banking77 at 0.425 to 0.492 against Jev's 0.870, and an English checkpoint that scores 0.000 on Khmer at 0.952 mean confidence.

What JevBench says

JevBench v1.5.4 is the one board that runs many clones on the same items: 106 ranked systems, 1,624 items of which 720 are sealed, on evaluator-owned GPUs. "Intelligence" is a chance-adjusted accuracy, and the composite rank also weights speed and cost.

Rank	System	Base	Intelligence	Choice ECE
1	Cygnet	frozen Gemma-4-12B-it, one-token readout	71.1	0.041
2	Winnow-12B	Gemma-4-12B-it LoRA	74.4	0.085
3	Jev 1.13.0	closed	72.0	0.027
7	decider-4b v2	Qwen3.5-4B-Base	55.8	0.049
21	raw Qwen3-4B-Instruct-2507	stock, option logits	54.1	0.329
37	Kev-4B	LoRA + pointer head	39.9	0.278
93	Laya	ModernBERT-large	0.0	0.149

The top open system is a first-token readout on a frozen 12B model, and a 4B model read the same way with no training at all beats Kev-4B by 14 points, which on the board's face meant I was serving the wrong model.

Provenance narrowed the candidates before any measurement. Nimble's pipeline contains a bridge that calls the Jev API although its README says it was not distilled from Jev, JevK5 is reported to include GPT outputs, and several "jev-distill" models train on Jev outputs. decider-4b (by Mapika, Apache-2.0) states a local Qwen3.5-27B teacher and nothing distilled from Jev, which made it the best-placed 4B candidate. The raw Qwen readout was the training-free control.

K2: scoring the board's picks on my panel

The board's task mix is not Loom's: Loom asks short questions, and it needs a model that knows when not to answer more than one that answers most things right. The board's Kev row also describes a Qwen3-4B base, while I pin jaredpalmer/kev-4b at revision 139fdd9, so I could not tell whether it was the same model. I pre-registered a check, K2, before running anything: score decider-4b and the raw Qwen readout on the same panel Kev-4B had already been scored on, and swap only if a candidate beats Kev-4B by at least 5 points of macro coverage at 5% error with a bootstrap lower bound above zero, on both the full panel and the panel without emotion and sciq (decider's training mixture touches those two).

The panel is the development split of Kev's frozen transfer-v9 eval suite: 11 tasks and 1,046 questions. Both new arms ran on my GPU box (a 16 GB RTX 4070 Ti Super) through decider's own server, the raw Qwen arm through its chat readout layout with no temperature, so that arm measures the stock model plus decider's prompt and readout. The Kev-4B and Laya rows are reused from the earlier K1 run.

Arm	11-task macro	Difference from Kev-4B (lower bound)	9-task macro	Difference (lower bound)
Kev-4B	0.555		0.549
decider-4b	0.556	+0.1 pp (-9.1)	0.518	-3.2 pp (-13.2)
raw Qwen3-4B	0.091	-46.4 pp (-54.7)	0.000	-54.9 pp (-64.7)
Laya	0.158	-39.7 pp (-46.6)	0.131	-41.8 pp (-50.5)

Nothing came close to the swap rule, so Kev-4B stayed.

The raw Qwen readout answers 72% of the panel correctly and still covers only 0.091, because at temperature 1.0 its confidence does not separate right answers from wrong ones. No threshold keeps its error under 5% on any task except sciq, where every arm covers 1.000. JevBench's 54.1 rewards accuracy, while my swap rule rewards knowing when to answer.

decider-4b and Kev-4B tie on the macro with opposite strengths:

Task	Kev-4B coverage	decider-4b coverage	Kev-4B accuracy	decider-4b accuracy
unknowable_control	1.000	0.691	0.955	0.827
composition_holdout	1.000	0.750	0.979	0.896
buried	0.375	0.075	0.700	0.675
legacy_holdout	0.800	0.662	0.812	0.738
paws	0.275	0.188	0.787	0.713
qnli	0.825	0.925	0.912	0.925
mmlu	0.350	0.575	0.725	0.725
tweet_offensive	0.312	0.588	0.738	0.787
emotion	0.163	0.463	0.550	0.762
mmlu_pro	0.005	0.205	0.560	0.505
sciq	1.000	1.000	0.975	0.988

decider-4b is better on knowledge and classification tasks (mmlu, mmlu_pro, qnli, tweet_offensive, and emotion, which its training mixture may have seen). Kev-4B is better on buried evidence, held-out rule compositions and above all unknowable_control, the questions where the deciding evidence was removed and answering is the error. That is the task that matters most to Loom, and Kev-4B covers 1.000 of it against decider-4b's 0.691. decider-4b answered at 48 ms p50 against Kev-4B's 553 ms, but the two ran on different serving stacks, so that gap mixes model and server.

Running decider took three fixes on the GPU box: its server needs flash-linear-attention and causal-conv1d, so it ran in WSL; its CUDA-graph warm-up did not fit 16 GB, so it ran eager; and the WSL image could not live on a nearly full system drive, because filling it corrupted the distro once.

Where this leaves the pick

The readout family decides what the head reads (option tokens in the vocabulary, a pointer between two hidden states, or a marker in a bidirectional encoder) and with it the option cap and the cost per question. The top open system on JevBench is a first-token readout, yet on my panel a training-free first-token readout covered 0.091 because its confidence could not tell right from wrong. For a caller that has to abstain, what matters is whether the model's confidence orders right answers above wrong ones, and JevBench's intelligence axis does not measure that. A per-task mix of Kev-4B and decider-4b looks tempting given the opposite strengths, but the earlier K1 run found only +1.2 points of oracle headroom for per-task selection with Laya as the second model, and this panel was not built to choose between models, so any follow-up needs fresh items. K2 kept Kev-4B as the pinned model in verdict, the decision service Loom calls.

My coding agents keep a record of what they decided and why, not just what they built

Anish Shrestha — Sat, 29 Aug 2026 21:55:14 +0000

A session with a coding agent ends and whatever it figured out along the way goes with it. The code survives, since it's on disk. The reasoning behind the code doesn't: why this table got a composite index instead of two separate ones, why the retry logic gave up after three attempts instead of five, which approach got tried first and thrown out. Open a new session next week on the same project and the agent starts from the code alone, with no memory of any of that.

Claude Code has a memory system for this, and it's genuinely good at what it does. Cursor has nothing like it. I run the two on different machines for different kinds of work, Claude Code where I vibecode, Cursor on a separate box for the work I want to sit with, so it was never a question of getting one tool's session context into the other's. What I wanted was a record of the reasoning that outlives any single machine: an Obsidian vault, synced with the self-hosted obsidian-livesync plugin, that both machines write into and that also lands on my phone for a quick read when I'm nowhere near either one.

Three hooks doing what a prompt can't

Telling an agent in its system prompt to "keep a devlog" doesn't work reliably. It's one more instruction competing with the task itself, and the task wins almost every time, because the task is enforced by the point of the session and the devlog instruction is enforced by nothing. What actually holds is three separate hooks in the harness, each firing at a different point in the session, backing each other up.

The first hook, session-start-wiki.sh, runs when a session opens and hands the agent the last three devlog entries plus an index of every project in the wiki, so it starts already knowing what happened last time instead of re-deriving it from the code. The context it injects is one fixed block: WIKI: project={slug} vault={wiki_vault}, followed by an instruction to write plans, decisions, spikes, and devlog entries as it works.

The second, wiki-nudge.sh, checks in every fifth turn. If no wiki file has been touched yet this session, it injects a reminder pointing at the exact devlog path to write to and the index file that has to be updated alongside it, and it repeats on the same schedule until something gets written.

The third is the one that actually forces the issue. stop-wiki-enforce.sh runs when the session is about to end, counts how many Write, Edit, or NotebookEdit calls happened in the transcript, and if three or more code changes went by with no devlog or index update to match, it blocks the stop outright. In Claude Code that's a {"decision": "block"} response the hook returns; in Cursor, which doesn't expose a transcript path the same way, it falls back to checking whether anything in the wiki directory changed in the last two hours. Either way the session doesn't get to close clean. The reason for it gets re-fed as the next prompt, and the agent has to either write the entry or explain why this particular session didn't need one.

A fourth hook applies the same idea to a different failure mode. Long sessions get compacted to fit the context window, and anything unwritten before that happens disappears the same way it would at session end. pre-compact.sh fires right before that compression runs and gives the agent one explicit last chance to write down any plans, decisions, spikes, or concepts from the session, since compaction is the point where the context actually goes away.

What actually gets written

The wiki has a fixed shape: wiki/projects/<slug>/ holds an overview.md, a devlog.md, and subfolders for decisions/, plans/, and spikes/, plus a top-level index.md that lists every project. Each page type has its own template. A decision page has to name the alternatives that were on the table alongside the one that won. A spike page states the question it was answering before it gets to the finding. The devlog format is the strictest: a dated heading, two to four bullets, written as plain fact rather than a "here's what we did this session" narration, with no name attached and no hedged phrasing softening what happened. That strictness is what keeps a year of entries readable in one sitting rather than turning into the same throat-clearing paragraph with a new date stamped on it.

The hooks themselves are identical on both sides. session-start-wiki.sh, wiki-nudge.sh, and stop-wiki-enforce.sh are deployed to ~/.claude/ and ~/.cursor/ unchanged, same triggers, same thresholds, and only the JSON shape of the block response differs because the two tools expect different formats. The harness's own docs never claim Cursor needs this more because it lacks a native memory system; that's my own read of the setup, not a documented rationale. I think it holds up anyway. Claude Code's memory already softens the cost of a missed devlog entry, where Cursor has no such fallback, so the same devlog is doing more of the actual work for Cursor sessions than for Claude Code ones.

This doesn't replace what Claude Code's memory does. That system decides on its own what's worth keeping and recalls it without me asking, where the wiki only surfaces the specific file a hook decided to inject. What the wiki adds is a shared, greppable record that exists independent of either tool, readable on a phone as easily as on either machine, in a text editor, or in a completely different tool six months from now, because it's a stack of plain markdown files with a template, sitting on disk, that I could read at 2am with nothing running at all.

The harness's own README states the philosophy bluntly: "The human never writes the wiki. The LLM writes and maintains all of it." That line only holds because the enforcement backs it up. Without the stop hook it would just be an aspiration; with it, a session that closes on three or more real changes has to have written the record of them first.

Scheduling every job on a GPU that can only hold one model

Anish Shrestha — Sat, 29 Aug 2026 21:54:33 +0000

A Raspberry Pi 5 runs DNS for the network, plus dozens of other small services that don't need much CPU or memory but can't go down. A Jetson Orin Nano handles anything that needs a GPU, and its GPU and CPU share one memory pool, small enough to hold one loaded model or one active render job, never both.

One flat network

There's one LAN. Clients and services aren't split into VLANs or separated behind an internal firewall; the upstream router is the gateway, and everything else sits behind it as a peer. If one device gets compromised, there's nothing stopping it from reaching every other device on the LAN. I've thought about segmenting it and decided against it: at this size, a mesh of routing rules is more to maintain than the threat justifies. It stays flat.

The Pi carries DNS and everything that has to be up all the time. It's picked for uptime and low draw, not speed, and I don't let it run anything that could pin its CPU long enough to take DNS down with it.

One model at a time

Load two things onto the Jetson that it can't both hold, and it thrashes. So nothing talks to it directly: every job (a chat request, a background research task, a scheduled scrape) lands in a durable queue first, and a single gateway between that queue and the Jetson admits exactly one job at a time, by priority. That gateway is most of the engineering here. Skip it and two jobs landing at once both try to load into memory built for one, and now the box is fighting itself instead of doing either job.

Priority doesn't preempt. If a background scrape is already loaded and running when a chat request comes in, the chat request doesn't kill it, it queues behind it and jumps ahead of everything else waiting. The alternative, killing a running job because something more urgent showed up, throws away whatever work was already done and risks leaving the model half-loaded. Waiting for the current job to finish costs a few seconds at most; a job that has to restart from zero costs more than that most of the time.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="280" height="56" rx="2"></rect>
  <rect width="280" height="76" rx="2"></rect>
  <rect width="280" height="56" rx="2"></rect>
  <rect width="280" height="60" rx="2"></rect>
  <rect width="280" height="70" rx="2"></rect>
  <rect width="280" height="76" rx="2"></rect>
  <rect width="280" height="56" rx="2"></rect>
  <line x1="180" y1="86" x2="180" y2="126"></line>
  <line x1="180" y1="206" x2="180" y2="246"></line>
  <line x1="180" y1="306" x2="180" y2="346"></line>
  <line x1="180" y1="410" x2="180" y2="446"></line>
  <line x1="540" y1="206" x2="540" y2="446"></line>
  <path d="M 400,180 C 300,230 260,300 300,352"></path>
</g>
<g fill="currentColor">



















</g>

Everything queues through one gateway before it reaches the Jetson, except the RTX box's own render work, which never goes near it.

A second machine, with an RTX 4070 Ti Super, does rendering, and it doesn't go through this gateway for that work. The gateway exists to arbitrate the Jetson's single model slot, and the RTX box was never competing for that slot to begin with, so routing it through anyway would just add latency it doesn't need. It does share one thing with the Jetson: a single LLM call both need. For that one call, the two machines are fighting over the same resource, so it queues like everything else.

The price of running it this way

A Pi 5, a Jetson Orin Nano, and a desktop GPU: together roughly the price of a decent laptop.

hearth is what I built to schedule every job on my homelab's GPU

Anish Shrestha — Sat, 29 Aug 2026 21:54:31 +0000

Every scheduled job in my homelab runs through a platform I built called hearth, and one piece of it is a gateway that admits one job at a time to whichever machine happens to be running a model. It runs on the Pi, and it exists because nothing else that could do the job fit the hardware. Kubernetes-style orchestration (k3s, Nomad) carries its own memory tax, and the Pi was already sitting at 5 GiB used plus 4.4 GiB of swap before adding a scheduler to the pile. Plain systemd timers per service were the other option, and they don't do priority ordering, backpressure, or a shared budget across jobs, which is most of what a GPU with one model slot needs.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="150" height="70" rx="2"></rect>
  <rect width="110" height="50" rx="2"></rect>
  <rect width="110" height="50" rx="2"></rect>
  <line x1="170" y1="185" x2="196" y2="185"></line>
  <line x1="350" y1="185" x2="376" y2="185"></line>
  <line x1="530" y1="185" x2="556" y2="185"></line>
  <line x1="710" y1="185" x2="736" y2="185"></line>
  <line x1="245" y1="220" x2="200" y2="286"></line>
  <line x1="280" y1="220" x2="325" y2="286"></line>
</g>
<g fill="currentColor">



















</g>

The two numbers that guard the gateway, the duration estimate and the output cap, both live at the one point that can measure them.

A queue that survives a reboot for free

hearth's queue is SQLite in WAL mode, not Redis or Celery. Memory is the scarce resource on this box, not throughput, and a SQLite file on disk means the queue's state survives a reboot without anything extra. Jobs carry a priority (interactive, user, batch, best-effort), a manifest-driven schedule, retry with exponential backoff, and a dead-letter path for anything that needs a human to look at it. If a job depends on something that's down, it parks instead of burning retries against a dependency that isn't coming back soon.

Each job runs as a subprocess inside its own systemd-run scope, and every resource class maps to a systemd slice with its own memory and CPU limits. An OOM kills the one job in that slice, not the host.

</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="340" height="210" rx="2"></rect>
  <rect width="280" height="150" rx="2"></rect>
  <rect width="340" height="210" rx="2"></rect>
  <rect width="280" height="40" rx="2"></rect>
  <rect width="280" height="40" rx="2"></rect>
  <rect width="280" height="40" rx="2"></rect>
</g>
<g stroke="" stroke-width="2">
  <line x1="20" y1="50" x2="360" y2="260"></line>
  <line x1="360" y1="50" x2="20" y2="260"></line>
  <line x1="740" y1="140" x2="760" y2="170"></line>
  <line x1="760" y1="140" x2="740" y2="170"></line>
</g>
<g fill="currentColor">






</g>
<g fill="">

</g>
<g fill="currentColor">

</g>

Without a slice boundary, one job's OOM takes the whole host down. With one, it's contained to the slice that job runs in.

Adding a resource class costs two files, not one, a manifest entry and a matching slice unit, and there's a test that fails loudly if the second file is missing. A job in a class with no slice unit doesn't error at validation, it dies five seconds into every run with no output and nothing in the journal, so the two-file rule exists to keep that failure mode rare.

The gateway learns durations from the worst recent run, not the typical one

hearth's scheduler will only start a queued job ahead of a fixed time window if it can promise the job finishes before that window closes, which means it needs an honest estimate of how long the job takes. The first version learned that estimate as the median of the last five runs, on the reasoning that one bad night shouldn't poison the number.

One of the jobs that runs through hearth is bimodal: about twenty minutes when it finds real work, about five seconds when it doesn't. Two of its three recorded runs were quiet ones, so the learned median came out to six seconds against a real run that took twenty-one minutes. Under that estimate, the scheduler would have started the job thirty seconds before a workday window it had no chance of clearing. Deleting the two quiet runs fixed the number for twenty minutes, until a fresh quiet run dragged it right back down, which is what proved the fix was against the symptom, not the cause: a bimodal job produces quiet runs indefinitely, and a median can't tell a normal quiet run from an anomaly.

</defs>
<g stroke="currentColor" stroke-width="1">
  <line x1="30" y1="200" x2="790" y2="200"></line>
  <line x1="219" y1="200" x2="219" y2="206"></line>
  <line x1="452" y1="200" x2="452" y2="206"></line>
  <line x1="40" y1="40" x2="234" y2="40"></line>
  <line x1="234" y1="40" x2="234" y2="200"></line>
  <line x1="755" y1="30" x2="755" y2="200" stroke=""></line>
  <line x1="40" y1="70" x2="219" y2="70"></line>
  <line x1="40" y1="110" x2="219" y2="110"></line>
  <line x1="40" y1="150" x2="752" y2="150"></line>
</g>
<g fill="currentColor">







</g>

The median estimate sits next to the two quiet runs and ignores the one real run entirely; the max estimate sits next to the real run.

The estimate is now the maximum of the last five runs, not the median. Verified against the same history: max reads 1277 seconds where median read 641. Under-promising is the dangerous direction here, since a job that starts and gets killed partway wastes the work it did before dying, while over-promising only makes something wait a little longer, which costs nothing. A job with occasional long runs is always scheduled as if it will take the long path, so fewer jobs fit into small gaps in an otherwise empty day. For a scheduler whose entire job is filling a mostly idle day, that's a fine trade. It wouldn't be on hardware that's busy.

The output cap moved out of every caller and into the gateway

Every service calling into the GPU box's model used to hardcode its own token limit, and every single one of those numbers was wrong. Twenty-nine calls in one day landed on exactly 1024 tokens against prompts of 600 to 900 tokens, some of them for a job scoring zero results out of eighty-one it should have judged, because the model's answer got cut mid-word before it finished.

No caller can size this correctly, because the two facts that determine a correct cap don't live where the caller lives. The context window is a property of the model config, and the generation rate is a property of the hardware, and a caller has access to neither, so any number it writes down is a guess. Guesses land low, because a low guess only fails sometimes, and it takes a while to notice.

</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="51" height="24"></rect>
  <rect width="48" height="24"></rect>
</g>
<rect width="620" height="24" fill=""></rect>
<g fill="currentColor">



</g>

The guessed cap barely covered the prompt itself, let alone an answer. The derived cap comes from the timeout budget and the model's measured speed, the two facts only the gateway has.

The fix moved the cap to the one place that has both facts: the gateway derives it from the request's timeout budget times the model's measured tokens per second, and fills that number in whenever a caller sends none. One of the models running through hearth is both the slowest measured, at 13.9 tokens a second against a faster model's 17.5, and the one with the largest context window, so an unbounded generation on that model alone could hold the GPU for close to an hour under a single-slot gateway. Sizing the cap to the same time budget the timeout already enforces costs the same wall clock in the worst case, but comes back with a usable partial answer instead of nothing.

A second GPU doesn't get routed through a chokepoint built for a different GPU

There's a second machine on the network with its own GPU, a dedicated render box that runs its own copy of the render pipeline directly against its own REST API, not through hearth's gateway. The gateway's entire job is arbitrating time on the GPU box's single model slot, and the second box was never competing for that slot in the first place, so sending its traffic through anyway would add nothing but latency.

It does share one thing with the GPU box: a short script-writing call that only the GPU box can serve, since the second box has no model of its own. That one call goes through the gateway, under its own named budget, so heavy experimentation on the render side can't starve the budget an established job already depends on. The rest of that box's work gets its own resource class rather than sharing the class built to stop the GPU box double-booking itself, because serializing a render job behind the GPU box's queue would waste GPU time on hardware that was never in contention to begin with. Reachability is checked the same way hearth checks any dependency, a live probe declared once in the job's manifest, not a guard written into the job itself.

I run browser automation inside a disposable VM

Anish Shrestha — Thu, 27 Aug 2026 08:00:08 +0000

Automating a browser that holds real logins means running code that fetches arbitrary pages on the open internet. That code needs to be treated as something that can be compromised. That's the assumption behind every decision here.

Why a VM instead of a sandboxed process

The browser automation on this network runs inside a disposable virtual machine, launched on demand and torn down after each job. The reason isn't abstract: Chromium's rendering engine assumes 4 KB memory pages, and the Pi it needed to run on uses a kernel built with 16 KB pages, so navigation hung outright on the host. The direct fix, rebuilding the Pi's kernel with 4 KB pages, broke the Pi's Ethernet and took the home network's DNS down with it. That ruled out patching the host to match the browser.

Running the browser inside its own small guest, with its own kernel built the way Chromium expects, sidesteps the page-size problem entirely and never touches the host kernel, its network stack, or the DNS resolver running on that same box. The guest costs about 2 GB of memory while it's running and a few tens of seconds to boot cold, so it stays on-demand rather than always-on.

Placement: the always-on box, not the one running a model

This automation runs on the Raspberry Pi, not the GPU box. The GPU box exists to hold a model in memory, and a browser process, headless or not, competes for the same pool that model needs. The Pi already carries DNS and everything else that has to be up all the time, and a browser job launched on demand under its own memory cap costs that box far less than fighting a model for the same 8 GB would cost the other one. Same reasoning that put DNS on the low-power box in the first place: a job goes where it costs the least to be wrong about.

What the VM guarantees, and what it doesn't

A VM boundary is real protection against one specific thing: if the browser inside it gets exploited at the kernel level, the compromise is stuck inside that guest. It is not, by itself, protection against the compromised guest reaching other things on the network. The guest's outbound traffic gets network-address-translated through the host, and network address translation is a routing detail, not a security boundary, so a compromised browser inside the VM can still open a connection to anything else on the LAN unless something stops it.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="220" height="120" rx="2"></rect>
  <rect width="220" height="120" rx="2"></rect>
  <rect width="140" height="120" rx="2"></rect>
</g>
<g stroke="" stroke-width="2">
  <line x1="260" y1="90" x2="336" y2="90"></line>
  <line x1="490" y1="130" x2="490" y2="180"></line>
  <line x1="460" y1="180" x2="520" y2="180"></line>
</g>
<g stroke="currentColor" stroke-width="1.4">
  <line x1="560" y1="90" x2="636" y2="90"></line>
</g>
<g fill="currentColor">








</g>
<g fill="">


</g>

The VM stops the exploit from leaving the guest's kernel. A separate firewall rule, enforced on the host and keyed to the guest's own uid, is what stops the resulting network traffic from leaving the guest at all.

The fix is a firewall rule on the host itself, not inside the guest, keyed to the dedicated user account the guest's process runs as. It denies that uid's traffic to the local network ranges and allows everything else, loaded at boot and reasserted every time a guest starts, so it holds regardless of what happens inside the VM. Enforcing it on the host rather than trusting the guest to police itself is the whole point: a compromised guest has no way to touch a rule it was never allowed to see.

Two identities, not one guest with two exits

Some jobs need to browse anonymously; some need a real logged-in session. The obvious design puts both behind one guest with a switchable exit, direct or through a VPN tunnel. That falls apart at the browser profile: a single cookie jar that sometimes exits from the home connection and sometimes from a rotating VPN address looks exactly like the pattern session-security heuristics are built to catch, and a logged-in profile behind a hopping exit collects CAPTCHAs and forced re-logins until it locks out entirely.

</g>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="340" height="180" rx="2"></rect>
  <rect width="260" height="50" rx="2"></rect>
  <rect width="340" height="180" rx="2"></rect>
  <rect width="280" height="45" rx="2"></rect>
  <rect width="280" height="45" rx="2"></rect>
</g>
<g fill="currentColor">






</g>
<g>


</g>

One profile alternating between a home IP and a rotating VPN exit reads as account-takeover behavior to the sites it visits. Two profiles, each with a fixed identity, never trigger that pattern.

The fix is two separate guests, one per exit mode, each with its own user account, its own disk profile, and its own machine identity, chosen at the start of a job rather than switched mid-run. The VPN guest's profile starts empty and stays empty on purpose: anything that needs a real login has to run in the direct guest, which is the design working as intended, not a gap in it.

The kill switch blocked its own tunnel

Building the VPN path surfaced a subtler lesson worth keeping. The firewall rule meant to guarantee that guest's traffic never leaves except through the tunnel was written as: drop everything for that uid except traffic on the tunnel's own interface. It looked correct, and it passed every test against a tunnel that had never successfully connected.

The moment a real tunnel came up, all of that guest's traffic stopped dead, tunnel included. WireGuard's encrypted outer packets inherit the user ID of the socket that generated the data they're carrying, so they left the host still tagged as the browser guest's own uid, on the physical network interface rather than the tunnel interface, and the same rule that was supposed to protect the tunnel killed it outright.

</defs>
<g fill="none" stroke="currentColor" stroke-width="1.4">
  <rect width="170" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="150" height="45" rx="2"></rect>
  <rect width="170" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="180" height="45" rx="2"></rect>
  <rect width="150" height="45" rx="2"></rect>
</g>
<g stroke="currentColor" stroke-width="1.2">
  <line x1="190" y1="52" x2="226" y2="52"></line>
  <line x1="410" y1="52" x2="446" y2="52"></line>
  <line x1="630" y1="52" x2="666" y2="52"></line>
  <line x1="190" y1="142" x2="226" y2="142"></line>
  <line x1="410" y1="142" x2="446" y2="142"></line>
  <line x1="630" y1="142" x2="666" y2="142"></line>
</g>
<g fill="currentColor">














</g>

The fix didn't change what the browser guest can do. It gave the tunnel's own transport socket a mark the guest itself has no privilege to set, so the fix can't be forged from inside the guest it protects.

The fix tags the tunnel's own outbound packets with a firewall mark that only a process with elevated network privileges can set, something the confined guest doesn't have, and lets that mark bypass the block.

Running a 26B MoE on an 8 GB Jetson by streaming experts from SSD

Anish Shrestha — Sat, 01 Aug 2026 14:00:35 +0000

My Jetson Orin Nano has 8 GB of unified memory. Gemma 4 26B-A4B is a 13.3 GiB download at Q4. This post is about the llama.cpp patch that lets the first thing run the second thing anyway, producing logits that are bit-for-bit identical to the standard code path, and why mixture-of-experts models make that possible when dense models never could.

The idea is borrowed, and worth borrowing

Credit where it belongs: the architecture comes from TurboFieldfare, a Swift and Metal runtime that runs Gemma 4 26B-A4B in about 2 GB of RAM on Apple Silicon. Its trick is to notice that an MoE model only touches a few experts per token. The dense weights (attention, router, shared expert) stay resident, and the routed experts, which are most of the model, live on the SSD. A small per-layer cache of expert slots is filled by parallel reads the moment the router announces which experts the next token needs, and the reads hide behind compute the GPU was doing anyway. The related academic thread is Eliseev and Mazur's expert-offloading work (arXiv:2312.17238), which established that MoE routing is stable enough between tokens for caching to pay off.

TurboFieldfare proves the idea beautifully, but it is a bespoke runtime: two supported models, Apple platforms only, custom kernels for everything. I wanted the same idea for the other cheap 8 GB machine on my desk, a Jetson Orin Nano, and I wanted it for any MoE model I could quantize. So instead of porting the runtime, I grafted the idea into llama.cpp, which already runs on the Jetson and already has Ampere-tuned CUDA kernels.

Streaming without touching a single GPU kernel

The part I like most about the design is that no CUDA kernel changes were needed. llama.cpp evaluates the routed-expert matmul with ggml_mul_mat_id, which takes the expert weight tensor plus a small tensor of selected expert ids and addresses each expert as base + id * stride on the device. The kernel does not care whether base points at 128 experts or at 32; it just multiplies ids by strides.

So the patch allocates, per MoE layer, a persistent pool tensor holding N expert-sized slots plus a tiny slot-id tensor, and at load time it records where every expert's bytes sit in the GGUF file. During decode, a scheduler callback fires on the router's top-k node, reads the chosen expert ids, and consults a per-layer LFU cache: experts already in slots are hits, misses are pread directly from the file into their assigned slots by a small thread pool. The callback writes the slot indices into the slot-id tensor and lets the graph continue. The matmul reads the pool through slot ids and gets exactly the bytes it would have read from the full tensor.

There is one place to be genuinely careful, and it is the reason this patch has a paranoid test. The same expert ids feed several consumers in the graph: the main matmul, the per-expert scale vectors that quantization-aware checkpoints ship, LoRA adapters, and expert biases. Only the main matmul may see remapped slot ids. Everything else must keep the real expert ids, because those tensors stay fully resident and are indexed by actual expert number. Get this wrong and nothing crashes; the model just gets quietly stupider. The test that guards against it greedy-decodes the same prompt with streaming off and on and compares every logit of every step with memcmp. Same kernels, same bytes, different addressing, therefore bitwise identity is the correct bar, not "close enough".

On the Jetson there is a bonus that TurboFieldfare's mmap trick foreshadowed: the Orin's GPU shares memory with the CPU, so the slot pools live in pinned host memory that the disk reads land in directly and the GPU addresses without any copy.

For the IO-hiding part, Gemma 4's graph has a shared dense FFN that runs for every token alongside the routed experts. The patch reorders the layer so the router's top-k comes first, dispatches the disk reads, lets the GPU compute the shared FFN while the NVMe works, and joins right before the expert matmul needs the data. One synchronization per layer, exactly TurboFieldfare's overlap, expressed as a graph reordering plus a two-phase callback.

Numbers from the actual machine

Measured on a Jetson Orin Nano Super (8 GB, MAXN, NVMe) with Gemma 4 26B-A4B QAT Q4, using llama-bench tg128. The baseline is the best you could previously do in llama.cpp: dense weights on the GPU, experts mmap'd on the CPU with --cpu-moe.

Configuration	Decode tok/s
`--cpu-moe` mmap baseline	1.94
Streaming, 32 slots x 8 IO threads, overlap on	2.95
Same, overlap off	2.23

A few observations that the table compresses:

The overlap alone is worth 32 percent, which is the TurboFieldfare thesis confirmed on entirely different hardware: expert IO really does hide behind compute you were already doing.
The NVMe's random-read ceiling (about 1.27 GB/s at this block size, per fio) puts the IO-bound limit near 5 tok/s, so there is headroom left in the IO path, not a wall.
Slot count is a real tuning knob with a cliff. 32 slots per layer (a 3 GB pinned pool) is the sweet spot on 8 GB; 64 slots allocates 6 GB and starves the rest of the system.
Peak RSS stays at 6.7 GB, and the correctness bar held on device: 32 greedy steps on the real 26B, every logit bitwise identical, with 10.5 GiB streamed through the pools during the comparison.

The honest tradeoff: cold prefill drops to about a quarter of the baseline's speed, because streaming mode deliberately stops pre-faulting the expert file (pre-faulting 13 GiB into 8 GB of RAM is its own disaster). Prefill-through-the-pool is the obvious next piece of work, along with O_DIRECT reads.

The GIF at the top is one request against the production gemma4-26b entry in llama-swap. I sent a warm-up call first to get past cold-load and cold-prefill, then recorded a streamed completion with tegrastats logging GPU load and power draw as each token arrived, laid out as three regions in the same terminal so the numbers and the text are visible at once. It settled at 2.09 tok/s where the bench measured 2.95, which is about the gap I'd expect between a clean llama-bench run and one live request with a different prompt. The GPU trace is the interesting part: utilization swings between 0 and 95 percent because for stretches of every layer the GPU waits while the NVMe pulls in the next expert. That is the overlap claim from earlier, on screen.

What generalizing bought

Because the hook lives in llama.cpp's graph layer rather than in any model's code, it works for every GGUF MoE architecture. I verified bitwise identity on three families: Qwen3-MoE style graphs with per-expert scale tensors, Gemma 4's merged gate-up projection, and Qwen 3.5/3.6's hybrid DeltaNet layers. The same flag also gets a 22 GiB Qwen 3.6 35B-A3B onto the same 8 GB board.

Is 3 tokens per second fast? No. It is roughly reading speed, fine for anything asynchronous and painful for chat. But the alternative on this hardware is not a slower 26B, it is no 26B at all. The compromise the patch offers is exactly the one I wanted: model size becomes a choice you pay for in speed instead of a hard out-of-memory wall.

The work lives on the moe-stream branch of my llama.cpp fork: a standalone expert streamer, a --moe-stream flag, pool allocation at load, the dual-id graph integration, the Gemma 4 overlap, and the identity test harness, in seven commits against upstream master.

Thanks again to TurboFieldfare for the architecture. Ideas this good deserve to escape their original hardware.

The run behind that GIF is scrubbable. I put a replay of it on the Learning Lab, with GPU load and power draw lined up against each token.

My context selector beat grep. An agent with grep beat it.

Anish Shrestha — Fri, 31 Jul 2026 14:00:23 +0000

cognitive-cache started from an observation I still think is correct: every coding tool decides what goes into the context window, and almost none of them treat that decision as an explicit selection problem. Cursor greps, RAG systems embed and cosine-search, and plenty of harnesses just stuff files until the budget runs out. The OS analogy is hard to unsee once you have it. The context window is RAM, eviction and retrieval are virtual memory, and token budget allocation is an allocator that nobody has written.

So I wrote one. Six weighted signals score every file in a repo against a task description, and a two-phase greedy submodular selector picks the highest-value set that fits a token budget. Nine languages, no LLM calls, no API keys. It shipped to PyPI and I used it through MCP for a while.

Then I built a benchmark harness, and the harness spent the next two months dismantling most of what I believed about the tool. This is that sequence, in order, because the order is the interesting part.

The first honest measurement said it does not beat grep

The original benchmark was bad in the way most self-benchmarks are bad. It coupled retrieval to LLM patch generation, reported rank-blind full-set recall, had no significance test, and I almost never ran it.

The replacement is LLM-free and deterministic. For each issue it clones the repo at the pre-fix commit, produces a full file ranking per strategy, and scores that ranking against the files the gold patch actually modified, using recall@k and MRR. Every head-to-head comparison carries a paired bootstrap 95% confidence interval, so a difference that excludes zero is real and a difference that straddles zero is not a result.

On the first dataset, 23 hand-curated issues across 8 repos:

strategy	R@1	R@5	R@10	MRR
cognitive-cache	0.170	0.428	0.511	0.540
grep (keyword count)	0.170	0.409	0.559	0.533
lexical (TF-IDF)	0.159	0.362	0.496	0.487

The recall@5 difference over grep was +0.020 with a 95% CI of [-0.067, +0.109]. Grep won at recall@10. Whatever the tool was doing, at that sample size it was indistinguishable from counting keywords.

Two other things fell out of the same run. The per-signal ablation suggested graph_distance, carrying 0.20 of the weight budget, was net negative, and file_role_prior was doing nothing at 0.07. And the budget selector was leaking badly: ranking recall@5 was 0.428 while end-to-end recall under a 12k budget was 0.282, so the selector was discarding relevant files its own ranking had already surfaced.

I nearly acted on the ablation immediately. A weight sweep stopped me, because every candidate reweighting had a bootstrap lower bound of +0.000. The "graph_distance is net negative" reading was itself noise at n=23. The real blocker was not the weights, it was that 23 narrow issues cannot detect or drive an improvement in either direction.

Widening the dataset flipped the result

I moved to SWE-bench, which solves the curation problem by construction: real GitHub issues paired with the gold patch that fixed them, pulled over the HuggingFace datasets-server with no auth. Ground truth is the set of source files the patch modified, and because patches only touch files present at the base commit, every ground-truth file is retrievable.

On a balanced 78-instance subset of SWE-bench Lite across 12 repos:

strategy	R@1	R@5	R@10	MRR
cognitive-cache	0.231	0.538	0.654	0.375
grep	0.103	0.359	0.487	0.236
lexical (TF-IDF)	0.141	0.410	0.462	0.253

Recall@5 over grep: +0.179, CI [+0.064, +0.295]. Over TF-IDF: +0.128, CI [+0.013, +0.244]. Both exclude zero. On a harder 91-issue subset restricted to fixes touching two or more files, the edge held at 0.365 vs 0.252, CI [+0.056, +0.174].

The earlier negative was a small-sample artifact. That cuts both ways, and it is worth sitting with: if I had published the n=23 result I would have been wrong, and if I had published the n=78 result as the final word I would also have been wrong, for the reason in the next section.

The baseline I had not run was an agent

"Beats grep" means beats a naive keyword count. It says nothing about the thing my tool would actually be competing with, which is a coding agent running its own search loop.

So I ran that. Five single-file SWE-bench issues, one each from flask, requests, xarray, pylint, and pytest. Fresh Claude subagents, cold in the repo at the base commit, grep, glob, and read only, cognitive-cache explicitly forbidden.

The agent got 5 of 5, averaging about 3.4 tool calls and 19k tokens, and in each case it pinpointed the mechanism rather than just the file. cognitive-cache got 4 of 5, missing pylint #5859, which the agent found in two calls. Naive keyword grep got 3 of 5.

The token argument inverted at the same time. The agent read a handful of targeted files. A cognitive-cache call returns file contents up to the full token budget, so on these tasks it front-loads more context than a surgical agent consumes. It was costing context, not saving it.

That is n=5 on hand-picked single-file issues, so it is directional rather than definitive. But it moved the honest positioning from "more accurate than the agent" to "fewer roundtrips on a cold start in unfamiliar code", which is a much smaller claim.

The signal that was worth zero, and why

graph_distance measures import-graph distance from the files the task names. It carried 0.20 weight from the day I wrote it, on the reasoning that a fix often lives one hop away from the file the issue mentions.

Across both datasets it earned nothing. On the 91-issue multi-file set, zeroing it changed recall@5 by +0.000 and MRR by -0.006. That was the pre-registered kill criterion, so I was ready to retire it.

Then I pushed back on my own experiment, which turned out to be the useful move. Multi-file is not the same as multi-hop, a regex-based import graph is not a reference graph, and testing a signal as one term in a drowned weighted sum is not a fair test of the underlying idea. All three critiques were fair, so instead of assuming, I wrote a diagnostic over the 91-issue set to check whether the ground truth was even reachable.

The answer killed the revival rather than supporting it. 93% of issues have all their ground-truth files reachable from the entry points. But 87% of fix files are themselves entry points, at graph distance 0, with only about 10% sitting at one to three hops and none further out. The mechanism was never starvation, it was redundancy: real issue text names the files it touches, so symbol_overlap has already ranked those files highly and import distance adds nothing on top.

That is not a bug in the experiment. It is a fact about the task distribution, and it explains the agent result too. On issue-shaped work the target is usually named in the prompt, which is exactly the regime where an agentic grep loop is excellent and a static ranker has little room to add value. Claude Code shipping no code index looks less like an omission and more like the same finding, reached earlier.

The delivery problem, measured

One thread survived long enough to be worth reporting separately. I built a jedi-backed neighborhood tool (definition, callers, in-repo callees, with real scope-aware resolution rather than name matching) and tested it two ways.

Against agents, on two hard cross-file issues where naive grep recall was 0, I ran a 2x2: agents with grep and read, versus agents offered the neighborhood tool as well. Every arm scored 0.5, finding the file tied to the named concept and missing the cross-file one. The agents that had the tool invoked it zero times out of two, despite being told to. Checking the tool's output directly against the symbols those same agents had reasoned about, it surfaced both missed files. The data would have lifted recall from 0.5 to 1.0 if it had been used.

At scale the value is real but modest: across all 91 multi-file issues, neighborhood-of-seeds alone scores 0.164 (worse than grep's 0.252 standalone), while grep-or-neighborhood combined reaches 0.330, recovering 20 of grep's 257 missed fix files. That is +7.8 absolute points, in line with RepoGraph's own reported 9-12%.

Put the two halves together and the conclusion is about delivery, not capability. A passive tool that competes with the grep reflex for invocation loses the invocation. Value like that only lands if it is injected into results the agent is already reading, not offered as one more thing it could call.

Where it ended

I set graph_distance to weight 0, rewrote the README around the scoreboard instead of the pitch, and stopped building. The remaining pitch would have required a forced-injection layer to capture a modest gain, in a niche that already has direct clones, running against the current of agents that search well on their own.

What survives is the measurement rig: the SWE-bench importer that needs no token, the retrieval evaluator, the weight tuner, the multi-hop diagnostic, the neighbor-recovery harness, and the bootstrap CIs that make each of those trustworthy. In a category that ships on anecdotes, that is the part I would build again first.

The generalizable lesson is not "context selection is a dead end". It is that the baseline you skip is the one that decides whether your work matters. I measured against grep for two months because grep was the baseline that let the tool win. The agent was two hours of work and it reframed the entire project.

J-space in practice: using Anthropic's Jacobian lens to decide what an LLM can forget

Anish Shrestha — Wed, 29 Jul 2026 12:07:06 +0000

Anthropic published Verbalizable Representations Form a Global Workspace in Language Models on July 6, and the vocabulary it introduced is suddenly everywhere: J-space, the Jacobian lens, a global workspace inside Claude. Most of the discussion so far is about interpretability and alignment auditing, which is fair, since that is what the paper is about. I had a narrower and more mercenary question: can the workspace tell an inference runtime which parts of the KV cache it is safe to throw away?

Three days after the paper landed, the first pre-registered gate on that question passed. As of this week the signal has replicated on three models and ships inside EVOKE, my KV cache memory manager built on a forked llama.cpp. This post covers what J-space is, why it makes a good KV cache eviction signal, the numbers across Qwen2.5-7B, Qwen3-8B, and Qwen3-4B, and the caveat that comes with them.

What J-space is, in one paragraph

The Jacobian lens is the instrument and J-space is the phenomenon. The lens isolates directions in a model's residual stream that encode a token the model could verbalize next, and those directions form a low-dimensional workspace: roughly 10% of activation variance, concentrated in the middle layers, carrying whatever the model is "holding in mind" at each position. Anthropic's headline application is alignment auditing, reading reasoning the model never voices. What makes independent work possible is that they released companion code under Apache-2.0 along with fitted lens matrices for open Qwen models on Hugging Face, so anyone can apply the lens to an open-weights model on a single GPU.

The systems problem: KV cache eviction

Every long-running LLM session eventually outgrows its KV cache budget. An agent session in a coding harness crosses tens of thousands of cached tokens within a few turns, and something has to decide which entries stay in GPU memory. The standard answers, H2O and SnapKV, rank cache blocks by accumulated attention history: keep what the model has been attending to, evict the rest. StreamingLLM adds protected sink tokens and a recency window.

These policies share a structural weakness. They are backward-looking, so they can only rank blocks the model has already used. A fact planted at turn 1 that will matter at turn 14 looks cold the whole time, and it gets evicted exactly when the budget tightens.

The hypothesis: workspace content predicts KV reuse

The lens readout suggested a forward-looking alternative. If a position carries workspace content, meaning the model is holding something in mind there, then its KV entries should be disproportionately likely to be read by later reasoning. That is a content signal, computable at prefill time from the residual stream alone, before any attention history exists.

The offline validation lives in j-space: lens application, a family of workspace statistics over the readout (kurtosis of block means, transport ratios, energy concentration, computed across layers), block aggregation, eviction simulation, and probe distillation. Because a signal this convenient deserved suspicion, I pre-registered three kill gates before running anything:

Fact ranking. The best workspace statistic has to beat SnapKV at ranking the task-critical block, by a pre-set AUC margin, or the project dies.
Masked eviction. With only 25% of the cache retained under the workspace policy, the planted fact has to stay answerable.
Probe distillation. Applying the full lens at inference time is too expensive, so a ridge probe distilled from the workspace readout has to reproduce the ranking at Spearman 0.8 or better. The deployable artifact is one dot product per position.

Results across three Qwen models

All three gates passed on all three models, with no hand-tuning on the replications.

Model	Fact-AUC (workspace)	Fact-AUC (SnapKV)	Eviction @ 25% budget	Probe Spearman
Qwen2.5-7B-Instruct	0.891	0.622	35/36 vs 25/36	0.959
Qwen3-8B	0.865	0.563	35/36 vs 14/36	0.944
Qwen3-4B	0.939	0.541	36/36 vs 13/36	0.946

The eviction column reads: episodes where the planted fact survived eviction and was answered correctly, workspace policy vs SnapKV at the same budget. On Qwen3-4B the workspace policy saturates at 36/36 at every budget tested.

The distilled probe also holds up live. Inside EVOKE's agent bench, the workspace signal passes the fact probe at every cache budget (512, 1024, and 2048 tokens) with recovery disabled, while H2O fails all three budgets and SnapKV passes only at 2048. Measured decode overhead was -0.6%, which is noise; the probe is a dot product per position at prefill, and each decode step adds only a 28 KB host copy and one dot product per block.

What transfers and what does not

Here is the caveat, and I think it is the most interesting finding in the project. The winning statistic is model-dependent. On Qwen2.5-7B and Qwen3-8B it is kurtosis of block-mean workspace scores at deep layers (layer 23 and layer 30 respectively). On Qwen3-4B it is a transport-ratio statistic at layer 12 of 36. If I had frozen the 7B statistic and shipped it as "the" signal, the 4B replication would have failed. What transfers is the statistic family plus the automated procedure that selects the winner per model on held-out episodes, not any single magic number.

The other scope limits are stated plainly in the paper: three models so far, all Qwen; the probes are fit on the benchmark's own episode family; and the task family is long-horizon fact recall. LongBench-style suites and non-Qwen architectures are open work. As far as I can tell this is the first interpretability-derived KV eviction signal, and I would much rather over-state the limits than the result.

Where it ships: EVOKE v2.0

EVOKE treats the KV cache the way an operating system treats memory. Blocks are evicted under budget pressure and recovered recompute-free through save/restore primitives added to a forked llama.cpp, 20 to 32 times faster than re-prefilling the same tokens. Version 2.0 adds the workspace probe as a first-class eviction signal alongside live attention capture, harness priority tags, task-focus coherence, and recency, and includes the paper PDF with the full three-model evaluation.

You can watch the signal work in the live demo on Hugging Face Spaces, which runs three arms side by side on Qwen3-4B: a plain baseline, EVOKE with its attention scorer, and EVOKE with workspace eviction. There is also a benchmark walkthrough on my learning lab.