DEV Community: Helgard

Running the 510 GB DeepSeek-V4.1-Flash on an 8 GB GPU — and three bugs that never raise an error

Helgard — Sat, 03 Oct 2026 12:25:53 +0000

My home machine for local models is modest: an RTX 5060 with 8 GB, a Core Ultra 5 225F, 31 GiB of RAM and a Gen5 NVMe drive used only for model files. DeepSeek-V4.1-Flash is 510 GB on disk. It now runs on that box as a normal chat model in Open WebUI: ~1.6 tokens/s reading experts from disk only, ~2.4 tokens/s with a 16 GB RAM cache. That is slow, and this post is honest about why — but it works, and every speed-up keeps the output token-for-token identical to a plain version that does all the math with DeepSeek's own code.

Code: https://github.com/helgard-orlm/deepseek-v41-flash-8gb

Who did what, up front: I set the goal, chose the model, proposed some of the ideas (predicting the next layer's experts among them) and made the calls. The engine, the server and the bug hunts were done by Claude (Anthropic) in my sessions; the overlap scheme and the one-token short path came from Codex (OpenAI) reviewing the engine. "We" below means the three of us.

This is the second model we run this way. The first, a 133 GB Qwen at 11 tokens/s, is described in the previous post. The comparison between the two is the most useful part of this one.

Bytes per token decide everything

For a mixture-of-experts model, the size on disk barely matters. What matters is how many expert bytes one token pulls in.

DeepSeek-V4.1-Flash: 40 layers, 384 experts per layer, 6 chosen per token, each expert 17.93 MiB in FP4 with scales.

6 experts × 40 layers × 17.93 MiB ≈ 4.2 GiB per token

The Qwen model needs 1.27 GiB per token. Same machine, same disk, same method — 3.3× more bytes, and that alone explains most of the speed gap.

Where the 475 GiB go:

part	size	where it lives
attention (FP8), shared experts, norms, router	~6.7 GiB	GPU
routed experts (15,360 of them)	269 GiB	NVMe → RAM cache → GPU
Engram tables (a per-token memory, FP8)	189 GiB	NVMe, ~48 rows per token, 5 ms
embeddings + output head		RAM, head computed on the CPU in FP32 like the reference

The approach: don't rewrite the model, only move its storage

DeepSeek publishes its reference inference code (model.py + kernel.py, MIT). We use it as is for all the arithmetic and replace only where the weights come from. That has a nice consequence: there is no conversion step. In the original safetensors files each expert's three matrices already sit next to each other (17.7 MB) and its scales are one more block (1.1 MB), so an expert is two pread calls with O_DIRECT straight from the Hugging Face files. A read benchmark showed this already uses 94–97% of what the drive can do, so re-laying-out the data on disk had nothing to give.

We first measured the whole thing on a rented RTX 5090 with VRAM artificially capped at 7.3 GiB, before buying the Gen5 drive. Once the numbers looked worth it, the model moved home.

Three bugs on RTX 50xx that never raise an error

This is the part I'd want to read before trying anything similar on a Blackwell consumer card.

1. TileLang 0.1.8 computes garbage on sm_120, silently. The reference pins this version. On our card the FP4 matrix multiply had cosine 0.0006 to a plain torch computation (i.e. unrelated numbers), the FP8 one returned NaN, and the model happily wrote "athaatha Stone". The reference self-test passed — it checks shapes, not values. TileLang 0.1.9: cosine 0.9996.

2. A race in the reference act_quant kernel. With DeepSeek's scale format (ue8m0, always on in this model) the kernel is built with num_stages = 0. On the RTX 5060, for more than 64 rows, part of the FP8 output comes out NaN — and differently every time: the same input five times gave 367, then up to 2065 NaN values. Generating one token at a time is clean, so only prompts were hit. It wasn't the compiler (CUDA 12.8 did the same). num_stages = 2 gives zero NaN and is bit-exact with a torch implementation at 1, 17, 300 and 1024 rows. That one line is the only change the setup script makes to DeepSeek's code.

3. The sparse attention kernel wants 141 KB of shared memory; consumer Blackwell has ~99 KB. Heads are independent, so we call the same kernel in groups of 16 heads. Small trap inside: .contiguous() on a head slice of a 1-token tensor doesn't change its strides, and the kernel checks strides — clone(memory_format=torch.contiguous_format) does.

The common lesson: on new hardware, "it ran without errors" says nothing. Compare every kernel with a dumb torch version on real weights before trusting a single generated word.

Making it faster: v1 to v4

The rule for every step: the 136 generated tokens of a fixed 4-question check must stay identical to the plain version (v1). Anything that changed a token was rejected or fixed. To be precise about what that proves: the speed-ups changed nothing. It does not prove v1 equals DeepSeek's untouched program run side by side — that program can't load the model on 8 GB, so we never ran it. What backs v1 instead: it calls DeepSeek's own model and kernel code for every computation, the individual kernels were compared with plain torch on real weights (above), the expanded wo_a matches DeepSeek's convert.py bit for bit, and the answers are right (Canberra, a correct TCP/UDP explanation, working merge code).

v1 — plain streaming: 0.89 tokens/s from disk, 1.17 with a 14 GB RAM cache. Read the layer's 6 experts, copy them to the GPU, wait, compute. Disk, bus and compute simply added up.

v2 — overlap (Codex's scheme): 1.28. A separate copy thread with CUDA events instead of a global synchronize. The surprise: issuing all 6 expert reads at once was slower. They share the drive, so they all finish together at the end of the layer and computing can't start any earlier. What worked was a deep queue but experts one after another: at most 2 in flight, each read in 4 parallel pieces. Waiting for the disk went from 0.65 to 0.31 s per token. (The first version of this had a race of its own — a ring buffer overwrote an expert that hadn't reached the GPU yet, and token 29 changed. The token check caught it.)

v3 — a short path for one token: +5.7%. The reference finds each expert's tokens with torch.where(indices == e) — a CPU↔GPU sync, 240 times per token, and each one delays the next copy. For a single token the indices are now moved to the CPU once per layer. Same arithmetic.

v4 — prefetch two experts of the next layer into the gap: 1.64 from disk (+21%), 2.3 with RAM. Apply layer i+1's router to layer i's input (rescaled by the two layers' norm weights) and you get the right top-6 71% of the time. Once layer i's real reads are in flight, read the top 2 guesses that aren't already in RAM. 94% of them get used. Three or four guesses are worse: they start competing with the reads that are actually needed.

One thing we got wrong along the way: I tried to reduce "bytes read in vain" by counting guesses that were already in RAM against the budget. Fewer wasted bytes, lower speed. Reads that land in the gap, while the disk would otherwise idle, are free; the only thing that matters is how many experts arrive on time.

version	disk only	with RAM cache
v1 plain	0.89	1.17
v2 overlap	1.28	1.42–1.67
v3 short path	1.35	1.95
v4 prefetch	1.64	2.28–2.34 (14 GB)
live chat, RAM 16 GB		2.43

A 311-token prompt takes 18 s (27 s in v1). Peak VRAM 7.06 GiB. Follow-up messages don't re-read the conversation: the server snapshots the full model state (30 MiB — KV, the sliding window, compression tails, indexer cache, Engram history) after each answer and processes only the new tail.

What we measured and dropped

Some of these were my ideas, some Claude's; all were measured on the real weights rather than argued about.

Store deltas between experts or layers, they'll compress better. The FP4 codes have 3.895 bits of entropy; the difference to the same expert in the next layer, or to a neighbour, has 3.997 — worse than the original. Even after the best neuron permutation and per-neuron scaling, the remaining difference is 0.999–1.000 of the original, exactly what two random matrices give.
Skip "silent" neurons and don't read their part of the down projection. To keep each expert within 5% error you still need 61–75% of the neurons: ~12% fewer bytes, and the read would have to wait for the first half of the computation. This model's SwiGLU neurons simply don't go quiet the way ReLU² ones do.
Smarter cache policies. A simulation over recorded routes: LRU beat LFU with ageing, per-layer LRU and SLRU. The theoretical optimum is ~15 points higher, but nothing simple gets near it.
More reader threads, splitting reads into smaller pieces — the drive was already near its limit.

Why it stops here

From disk, a token takes about 0.6 s, and just reading its 4.2 GiB at the 8.2 GiB/s the drive delivers on these reads is ~0.5 s of that. There's very little software left between us and that number. The only ways to go faster are fewer bytes (more RAM for the cache — the whole expert set is 269 GiB, so a cache is always partial) or a second drive. With Qwen the same method gives 11 tokens/s because each token needs a third of the bytes. If you're choosing a MoE model to stream on a home PC, look at bytes per token before parameter count.

Try it

You need a fast NVMe drive with ~520 GB free and a Blackwell GPU (the fixes above are for sm_120; other cards may need fewer of them). setup.sh downloads the exact model revision we used if it isn't there, checks that kernel.py is the expected file, and builds a copy of DeepSeek's code with the one-line fix. Before publishing, we cloned the repository fresh on the same machine, ran the setup against the downloaded model and started the server from the clone: it loaded with every parameter accounted for and answered correctly. The engine and server files are byte-identical to what runs at home. The full 510 GB download from scratch was not repeated for this test.

Engine, server, bug hunts and this write-up: Claude (Anthropic). Overlap scheme and the one-token short path: Codex (OpenAI). Goal, model choice, the expert-prediction idea and decisions: me. All numbers are from logs on the machines described above.

Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe

Helgard — Sat, 03 Oct 2026 11:46:52 +0000

I run local models on one home machine: an RTX 5060 with 8 GB, a Core Ultra 5 225F, 31 GiB of RAM and a Gen5 NVMe drive used only for model files. When NVIDIA published Qwen3.8-Flash-Next in NVFP4 (133 GB on disk), the obvious answer was "it doesn't fit". It does now: it runs as a normal chat model in Open WebUI at 9–12 tokens/s, and its output matches the Hugging Face reference implementation.

Code: https://github.com/helgard-orlm/qwen-flash-next-8gb

Who did what, up front: I set the goal, chose the model and made the calls along the way. The engine, kernels and server were written by Claude (Anthropic) in my sessions; the CUDA-graph speed-up was written by Codex (OpenAI) in a parallel session, and Claude then found and fixed a crash in it. "We" below means the three of us.

This post is about the method, the measurements, and what didn't work.

Why this model, of all models

Most of the weights of a mixture-of-experts model are experts, and each token uses only a few of them. So the question is not "how big is the model" but how many expert bytes one token needs.

Qwen3.8-Flash-Next: 48 layers, 512 experts per layer, top-10 routing, and each expert is tiny — three 2560×640 matrices, 2.70 MiB in NVFP4 with scales.

10 experts × 48 layers × 2.70 MiB ≈ 1.27 GiB per token

For comparison, DeepSeek-V4.1-Flash, which we run on the same box with the same trick, needs about 4.2 GiB per token. Same disk, a third of the bytes. (How DeepSeek went — 2.4 tokens/s and three silent bugs on RTX 50xx — is in a separate post, code at https://github.com/helgard-orlm/deepseek-v41-flash-8gb.)

Everything else (embeddings, attention, the 36 Gated DeltaNet layers, router, shared expert, lm_head) is about 7.2 GB in BF16. Storing the big DeltaNet matrices in FP8 with one scale per row brings the resident part to 6.13 GiB of VRAM. The 63 GiB of experts live on the NVMe drive.

The engine

There was no runtime for this architecture that could stream experts, so it's a from-scratch PyTorch + Triton engine. The math was ported from transformers 5.18 (qwen4_exp): DeltaNet, full attention with a block indexer that kicks in beyond ~2048 tokens, four hyper-connection streams, and an n-gram embedding table (PLE, used in layer 1) with 51 billion parameters.

The pieces, in the order they mattered:

1. Repack experts for the disk. In the original files an expert's six tensors are scattered. We rewrote them into one 2,764,800-byte block per expert (675 pages of 4 KiB): gate | up | down | gate_scale | up_scale | down_scale. One expert = one pread with O_DIRECT into page-aligned pinned memory = one copy to the GPU.

2. A RAM cache in front of the disk. LRU over pinned memory (16 GB holds about a quarter of all experts), a pool of reader threads, and two banks of GPU slots so the next layer can load while this one computes.

3. Prefetch into the gap. After a layer's required reads are issued, guess the next layer's experts and read two of them while the GPU is busy (91% of guesses are used). The first version started the prefetch at the same time as the required reads — they shared the disk and everything got slower. It has to go into the gap, not next to the real work.

4. Own kernels. An NVFP4 SwiGLU expert kernel for a single token using Blackwell's hardware cvt.rn.f16x2.e2m1x2 instruction (115 µs per layer, 3.6e-7 from a torch reference), and an FP8 row-scaled matrix-vector kernel. The generic unpack path for the FP8 DeltaNet matrices had been costing ~106 ms per token on its own.

5. Don't read 66 GB per prompt chunk. The first prompt path processed 512 tokens at a time, and each chunk needed nearly every expert, so the whole expert set was read again for every chunk. Running the whole prompt through each layer at once, with experts read in batches of 32, took a 2451-token prompt from 71.6 s to 23.3 s.

6. Read only the PLE rows you need. The n-gram table is a 53.7 GB file; each token needs a handful of rows. Random reads from NVMe: 0.3–0.5 ms per token. (On the HDD it would be a disk seek per row — a second AI acting as reviewer caught that the table was still on the HDD in the first run.)

Where the time actually goes

With every expert already in RAM, a token took 80–88 ms. Splitting it:

39 ms — Python issuing GPU work (the GPU was waiting on Python, not the other way round);
47 ms — moving 1.33 GB of experts over PCIe (measured 28.6 GB/s, PCIe 5.0 ×8);
and the two ran one after the other.

There are 52 CPU↔GPU syncs per token (mostly reading the router's chosen experts). We expected them to be the problem; measured, they are cheap. They only hurt because the expert copy for layer L can't start until layer L's router has answered. That pointed to three changes:

Speculative copies. Run layer L+1's router on layer L's MoE input. It picks the right top-10 65% of the time — enough to start most copies one layer early. (Prefetching top-16 to raise the hit rate moved more bytes over the bus than it saved.)
CUDA graphs for the 36 one-token DeltaNet layers (Codex). One shared memory pool: 36 private pools ran out of memory on an 8 GB card. Graphs are dropped before any prompt ≥1024 tokens.
Pin the decode thread to a P-core. On a hybrid CPU the unpinned thread wandered between P and E cores: 93–108 ms per token, jumping around. Pinned to core 0: a flat 81.6 ms.

Results

configuration	tokens/s
first version, experts from disk, no cache	2.68
disk only + prefetch	4.72–4.75
RAM cache 16 GB + prefetch	8.1
live chat, 1000-token answer, before the last three changes	9.71
live chat, 1000-token answer, after	11.65

Prompt processing: 24 tokens in 4.5 s, 2451 in 23.3 s, 7182 in 98 s. Follow-up messages are fast because the server snapshots the whole recurrent state (DeltaNet matrices + KV + indexer keys) in RAM and only processes the new tail.

How we know it's right

Speed without correctness is a random number generator, so every change was checked against the transformers reference fed with the same quantized weights:

next-token argmax: 32/32, mean |Δlogit| 0.077;
a 2451-token prompt, long enough for the attention indexer to start selecting blocks: 11/11, and the needle-in-a-haystack fact comes back right;
perplexity EN 2.21 / RU 2.22 — plus control runs that must break: with nibbles swapped it's 2016 / 33,269, with experts removed 588 / 5,351. That checks the NVFP4 unpacking independently of the reference.

What didn't work

A fused DeltaNet kernel (one kernel for the recurrence, 4.3 → 0.43 ms per token across 36 layers). 5 of 256 tokens came out different from the reference. Rejected — 4 ms isn't worth a model that says different things.
The 8192-token context didn't actually work, in the old version either: a 7182-token prompt ran out of VRAM in attention. Fixed by sizing prompt pieces so that piece × (position + piece) ≤ 3072².
A crash only after long prompts: CUDA graph capture in the default global mode was invalidated by expert-reader threads calling event.synchronize(). capture_error_mode="thread_local" fixed it.

Try it

The repository has the engine, the server (OpenAI-compatible, streaming, a thinking-mode model id), the repack and reference-check tools, and a setup.sh that prepares a working directory from a Hugging Face snapshot. Before publishing, we ran it the way a stranger would: a fresh git clone of the repo on the same box, setup.sh into an empty directory (15 min 48 s, mostly reading ~120 GB from an HDD; spot check of 204 repacked experts against the originals: 0 differences), then the server from the clone. The repacked experts and the extracted non-expert weights came out byte-identical to the production directory, and three greedy test prompts gave identical answers from both.

You need a Blackwell GPU (the NVFP4 kernel uses the sm_120a e2m1 instruction), a fast NVMe drive with ~130 GB free, and patience for a first-time setup that reads most of the 133 GB.

The bigger point: on a mixture-of-experts model, "does it fit in VRAM" is the wrong question. The right one is bytes per token, and a home PC with a fast drive can serve a lot more of them than it looks.

Engine, kernels, server and this write-up: Claude (Anthropic). CUDA graphs and the speculation/pinning A/B: Codex (OpenAI). Goal, model choice and decisions: me. All numbers are from logs on the machine described above.

An AI hardware advisor that walks every path — and gets its math checked by the rules in Sanity

Helgard — Sat, 03 Oct 2026 05:49:32 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

An agent that answers "what computer do I need to run this AI model?" — including "don't buy, rent" and
"you already have enough". It reads a Sanity dataset of criteria, not a catalog: laws (formulas for weights
memory, KV cache, speed ceiling, own-vs-rent breakeven), machine-checkable rules, 11 solution paths
(keep what you have · upgrade · used · one GPU · multi-GPU · unified memory · MoE experts in system RAM ·
CPU-only small model · smaller model · cloud API · rented GPU), run reports and reference hardware.
New models appear every week, so anything not in the base is looked up on the web by the base's own checklist.

Sanity project: onwa0wvs, public dataset v2 (200 documents), Studio: https://hw-for-ai-lab-v2.sanity.studio
Agent connection: Sanity Context MCP on the full dataset with embeddings enabled (/context/mcp/onwa0wvs/v2); all numbers below are measured through this connection
Second entry: a Knowledge Base (kbW7wsbtJQkl, built from the core 138 documents of the same dataset, refreshed weekly) behind its own Context MCP endpoint in KB mode — created and connected, not measured separately
Demo — try it with your own agent: https://hw-advisor.helgardorlm.tech (MCP https://hw-advisor.helgardorlm.tech/mcp, no login) · Replay of all 27 measured runs: https://helgard-orlm.github.io/sanity-ai-hardware-advisor/
Repo: https://github.com/helgard-orlm/sanity-ai-hardware-advisor

Demo

Live — bring your own agent (no login, no keys): https://hw-advisor.helgardorlm.tech · MCP endpoint https://hw-advisor.helgardorlm.tech/mcp
Replay of all 27 measured runs: https://helgard-orlm.github.io/sanity-ai-hardware-advisor/

Code

https://github.com/helgard-orlm/sanity-ai-hardware-advisor (the public proxy with check_answer is in public_mcp/)

Sanity Project Details

Project ID onwa0wvs, public dataset v2 — 200 documents, 15 types (law, rule, solutionPath, aiModel, gpu, cpu, runReport, offer, cloudOffer, situationTemplate, …)
Query it directly: https://onwa0wvs.api.sanity.io/v2025-01-01/data/query/v2?query=*[_type=="law"]
Studio: https://hw-for-ai-lab-v2.sanity.studio

Try it with your own agent (no keys, no login)

The demo is a public MCP endpoint, not a hosted chatbot: you bring the agent, the base brings the knowledge and the checking.
Add https://hw-advisor.helgardorlm.tech/mcp to ChatGPT (developer mode), Claude, Claude Code, Codex, Cursor or VS Code.
There are one-click buttons for Cursor and VS Code on the page. Or paste one line into any agent that can run commands:
"Read https://hw-advisor.helgardorlm.tech/setup.md and follow it."

Behind it is a ~230-line stdlib proxy. It passes the four Sanity Context MCP tools through unchanged and keeps the token
server-side. It puts today's date and the advisor's work order in front of initial_context. It adds one tool, check_answer,
which is the same validator the app uses: it recomputes every calculation with the law.formula stored in Sanity.
In the first outside test (Codex, its own subscription), check_answer rejected the first draft with 4 errors. The second draft passed:
5 calculations recomputed, 11/11 solution paths walked. A test from ChatGPT (developer mode) also went through to a passing check. Agents that can only browse get the same tools as plain HTTPS links
(/api/context, /api/query?q=…, /api/check), listed in /llms.txt.

Known limit: the check covers numbers, completeness, sources and identity. It does not cover whether a verdict makes sense. In one test an agent said yes to "MoE experts in RAM" for a dense model, and the check passed.

How I Used Sanity

Sanity Context MCP tools the agent calls: initial_context, groq_query, schema_explorer, array_field_reader (plus our check_answer). What it does with the content it retrieves — the structure is used, not just searched:

Paths are data. The agent must give a yes/no/maybe with a number for every solutionPath document. Before this, the same model answered a 321B-MoE question with "4×H200" and never mentioned the one-GPU + large-RAM path.
Rules are the search checklist. For any model/card/CPU that is not in the base, the app requires the fields that the base's rule documents test for that type (e.g. cpu.instructionSets, aiModel.variants.fileGb).
Laws are recomputed by code. The agent writes every calculation with its inputs into a hidden check block; the app evaluates the law.formula from Sanity and sends the answer back if the number is wrong.
The person's situation is stored, not remembered: country, budget, existing PC survive a change of mind.

The same code checks the base itself: every law variable and rule path must be a real schema field
(coverage check), and laws get property tests. That caught the filler model writing an own-vs-rent formula that
added electricity instead of subtracting it and billed rent for 24 h/day — 12.3 vs 110 months on the same inputs.

A real trace (GLM-5.3-Flash, not in the base) — open it in the replay

GROQ over the base → no aiModel document → 8 web searches by the base's own checklist: the model card and config.json (layers, KV heads, head dim), file sizes, and the GPU fields the base's rules test (slots, bandwidth, compute capability).
Laws from Sanity: weights 320 × 4 / 8 = 160 GB (NVFP4), KV for 8k context 22.5 GiB (flagged as rough for hybrid attention).
Validator round 1 sent the answer back: the speed ceiling was written as 19.91 tok/s — code evaluated the law.formula from Sanity with the agent's own inputs and got 199.1 (a 10× slip); plus three entities missing fields that the base's rules check (e.g. variants.fileGb, slots).
Round 2: no errors. All 11 solution paths get a verdict — multi-GPU = yes, MoE-in-RAM / unified memory = maybe, cloud = no (the person ruled it out) — and the build is 3 × RTX PRO 6000 (288 GB) with the numbers above.

Other catches by the same recomputation in the measured run: KV cache 0.949 → 6.75 GiB (A4), monthly electricity cost 0.675 → 2.025 (A5).

Measured, honestly

Blind grader (a different model, sees only the dialogue and a frozen checklist), 9 scenarios × 3 runs,
78 checklist items, same base state for all arms:

version	score	median time
v2: base + instructions	47/78	62 s
v3: + solution paths, validator	53/78	104 s
v3.1 via Sanity Context MCP: + rules-as-checklist, recomputed laws, dated prices	59/78	181 s

What this does not show: that structured retrieval beats plain text for reading. The same 200 documents
pasted as text notes scored the same as v3 (53/78) — at this size the model reads either. Where structure
mattered in our runs is checking: fewer first answers rejected by the validator with structured queries
(8/27 vs 14/27), and the gains of v3.1 are exactly in the scenarios where code recomputes Sanity's laws
(KV for 8 users 7/9 → 9/9, own vs rent 5/12 → 7/12, "don't buy" 4/6 → 6/6, v2 → v3.1). The price: answers take ~3× longer.
An earlier full run of v3.1 scored 62/78 with two bugs in my own checker (nested fields, European number format) that sent
correct answers back for rework; after fixing them the clean run scored 59/78 — with N=3 per scenario, 59 vs 62 is noise.

How it was built

Architect method: a risk catalog (16 classes of agent failure seen in earlier sessions) → acceptance scenarios
written before changes → build → blind grading. The base was filled by a model (GPT Luna) from an empty schema and
two prompts; I never wrote data by hand. An independent judge model watched the session and corrected the method
several times (frozen checklist, snapshot baseline, text control).

Known limits

Answers are slow (minutes) because the validator can send them back up to twice.
Ratings N=3 per scenario; differences of 1–2 points are noise.
"unknown after search" is allowed and can be abused for vague entities.
In the measured run 12 of 27 answers came out in Russian to English questions (an account language setting leaked in); the app now checks the answer's language in code and sends it back (3/3 English on a re-test).

We create a way to unload Qwen2.5 KV cache to RAM.

Helgard — Thu, 11 Jun 2026 20:14:41 +0000

Two separate problems with local LLMs, one mechanism fixes both:

Long context doesn't fit. The KV-cache grows linearly with every token; on 8 GB it dies around 110k.
Memory doesn't survive a restart. Kill the process and the cache is gone — next session re-reads everything from scratch (that's all RAG is: re-reading text, every query).
What the KV-cache actually is: as the model reads, each layer stores a key/value vector per token — this is the model's working state, the thing it attends over to pick the next token. It's not text, it's computed internal state. Normally it lives entirely in VRAM and only exists while the process is alive.

The mechanism:

Keep only the last N tokens' KV in VRAM (the hot window).
Stream everything older to CPU RAM (8-bit), then disk. It's not deleted, just moved somewhere 10× cheaper.
Keep a tiny index in VRAM: one vector per sentence. On a query, that index finds the few relevant chunks and copies only those KV vectors back into VRAM for the attention step.
So VRAM stays flat — it holds the hot window plus a small retrieval budget, regardless of whether total context is 50k or 800k. The cap moves from VRAM to your RAM stick.

The part that makes it memory and not just offload: you write that cold KV to disk and reload it in a different process days later. The model resumes from its own stored state — a few ms of memcpy, no re-reading the source. Two things make the positions line up across the gap: keys are stored already rotary-rotated at their absolute position, and the query is injected at its true position, so relative distances stay exact even between tokens 700k apart.

Result (RTX 5060 8 GB, 4-bit): a planted fact is the top prediction from 120k–500k tokens, top-2 at 800k, VRAM flat ~6 GB — where a normal full cache OOMs at 65k. The live store behind my assistant is 2.5M tokens of past sessions in 467 MB, answered in ~1s on CPU. Miss → LOW CONFIDENCE, not a hallucination.

Why Qwen2.5-7B-1M and not anything else — two independent requirements, most models fail one:

Architecture must keep a growing KV. Mamba/SSM and linear-attention models (Granite-4, RWKV, Qwen3.5's hybrid layers) compress history into one fixed-size state — there's no per-token KV to stream out or pull back. The whole method is inapplicable to them.
The model must actually read at that range. Effective context is ~50–70% of the advertised number (RULER). MiniCPM-1B (128k nominal) goes to garbage by ~120k even with a perfect full cache in VRAM — that's the model's limit, not the cache's. No infrastructure fixes it.
Qwen2.5-7B-1M passes both and fits 8 GB in 4-bit. Substrate alone OOMs at 65k; the model alone can't persist across restarts. The combination is the whole point.

Limits: confidence drops with distance (p 0.98 → 0.21 at 800k); many near-identical facts confuse the cheap index (need full attention there); on messy logs generation can override the retrieved KV with its own prior, so I quote stored text verbatim instead. Research PoC — the core (spilling KV to CPU) has precedent (ArkVale, RetrievalAttention), credited in the README.

Code (MIT): https://github.com/helgard-orlm/living-kv-cache