DEV Community: Helgard

Running NVIDIA's new 120B Nemotron-3.5-Super on an 8 GB GPU — skipping the zeros, and finding the real bottleneck

Helgard — Sat, 17 Oct 2026 12:42:22 +0000

NVIDIA released Nemotron-3.5-Super on October 15: a 120B mixture-of-experts model, 75 GB even in their 4-bit NVFP4 format. The usual answer for an 8 GB card is "not on your hardware". Two days later it runs on my home PC — one RTX 5060 with 8 GB — as an ordinary chat model in Open WebUI at ~8 tokens/s, it reads images, and its answers match the Hugging Face reference implementation.

Code: https://github.com/helgard-orlm/nemotron-super-8gb

Who did what, up front: I picked the model, set the goal and made the calls (including the idea to exploit the zeros after ReLU², below). The engine, kernels, server and measurements were written by Claude (Anthropic) in my sessions. "We" below means the two of us.

This is the third model in a series on the same box — after Qwen3.8-Flash-Next, 133 GB, 11 tok/s and DeepSeek-V4.1-Flash, 510 GB, 2.4 tok/s. Same idea, different model, and two new tricks.

The machine

RTX 5060 8 GB (Blackwell, PCIe 5.0 ×8), Core Ultra 5 225F, 32 GB DDR5, a Gen5 NVMe drive used only for model files. Nothing exotic.

The only number that matters: bytes per token

In a mixture-of-experts model most of the weights are experts, and each token uses only a few of them. So the question isn't "does 75 GB fit in 8 GB" — it's how many expert bytes does one token need.

Nemotron-3.5-Super: 88 layers (40 Mamba-2, 40 MoE, 8 attention), 512 experts per MoE layer, top-22. That sounds expensive, but NVIDIA made the experts tiny: they work in a 1024-wide "latent" space instead of the full 4096. One expert is up 2688×1024 + down 1024×2688, ReLU², no gate — 3.10 MB in NVFP4.

22 experts × 40 layers × 3.10 MB ≈ 2.7 GB per token

Between Qwen (1.3 GB/token) and DeepSeek (4.2 GB/token). Everything that isn't a routed expert — Mamba, attention, shared experts, router, latent projections, output head — is 6.6 GiB on the GPU, kept in the checkpoint's own FP8/NVFP4 formats and multiplied by our own Triton kernels. The 59 GiB of routed experts stay on the NVMe drive.

The engine, step by step

No existing runtime streams experts for this architecture, so it's a PyTorch + Triton engine with the transformers NemotronH code as the skeleton (Mamba mixers, attention, router). We replaced the experts, the dense layers and the per-token MoE step.

1. One block per expert on disk. Each expert's four tensors (up, its scales, down, its scales) are rewritten into one 3,096,576-byte block — exactly 756 pages of 4 KiB. One expert = one pread with O_DIRECT into pinned RAM. Repacking 59 GiB takes about 2.5 minutes.

2. A RAM cache in front of the disk. LRU over pinned memory, 32 reader threads. Routing is concentrated: with 24% of the experts in RAM, 81–84% of lookups hit.

3. Prefetch into the gap. Run the next MoE layer's router on the current input: it guesses the next layer's experts ~72% right. The missing guesses are read while the GPU is busy.

4. Skip the zeros (new). After ReLU², 92.5% of an expert's neurons are exactly zero for a given token. A zero neuron doesn't need its column of down — so why move it across PCIe? The catch: NVFP4 shares one scale per 16 consecutive values, so you can't just pick neurons out of the original layout. We re-laid down on disk once (a pure byte permutation, same file size): each neuron's 4-bit values are contiguous, scales grouped per 16 neurons. Now only up (1.55 MB) is copied to the GPU; the down kernel reads directly from pinned RAM over PCIe, and only the 16-neuron groups that contain a live neuron. +12%, 6.6 → 7.3–7.6 tok/s.

5. Chat server. OpenAI-compatible /v1/chat/completions with streaming, a -think model id for reasoning mode, and dialog snapshots (~170 MB: 8 attention KV layers + 40 Mamba states) so the next message only processes its new tail.

Where the time actually went — and where I was wrong

At 7.3 tok/s a token takes ~137 ms. The plan was CUDA graphs: thousands of small kernel launches per token, surely Python overhead is the problem. Before writing graphs, we profiled with Nsight Systems:

GPU busy ~92 ms, idle ~54 ms per token;
of the idle time, gaps between launches (what a CUDA graph would fix) were only ~13 ms.

Then we tried two "obvious" fixes for the remaining waits: reordering work so the shared expert runs before the router result comes back, and copying the predicted experts early. Both were bit-for-bit correct. Neither helped — the speculative copies made it 7% slower, because wrong guesses queued on the bus ahead of the right ones.

So we timed what the CPU was waiting for:

wait per token	ms
NVMe reads (experts missing from the RAM cache)	50–55
Python cache bookkeeping	17
waiting for the router's choice	11

The disk was the biggest wait. An earlier measurement had said "disk ~3 ms" — but that run had GPU copies switched off, which hid the disk behind nothing. The first README said that; it's corrected now. (The bad number was pointed out by a second agent reviewing the README, not by me.)

The fix was boring: a bigger RAM cache. Before touching the live server we recorded the routing of 5 conversations and replayed an LRU cache offline. The simulation matched live miss counts within ±0.4 reads per token, so we could pick a size on paper:

RAM cache	disk reads / token
14 GB	156
18 GB	117
20 GB	102
24 GB	78

20 GB live: 136.9 → 121.5 ms per token, 7.3 → 8.2 tokens/s. That leaves ~7 GB of RAM for the rest of a 32 GB system, which is why it stops there.

Results

version	decode, tokens/s
first run, no prefetch	6.1
+ prefetch	6.5
+ skip ReLU² zeros	7.3–7.6
+ 20 GB RAM cache (current)	8.2

A 32–40 token prompt: ~5 s. Load time: 7–27 s.

It reads images too

Nemotron-3.5-Super-VL ships a vision tower (C-RADIO, 1.3 GB). The engine's 6.6 GiB plus the tower don't fit on the card at once, so the tower's weights live in pinned RAM and are streamed to the GPU layer by layer while it runs (0.4–1.6 s per image, vs 10–31 s on the CPU). In Open WebUI you just attach a photo.

A test that's easy to fake and hard to pass: an invoice of 60 lines in a 15-px font, asked for the code, quantity and price of lines 21–30. At normal resolution (1047 image tokens) it got 7–9 of 10 per field — swapped letters, one made-up line. At NVIDIA's maximum resolution (2204 tokens, a separate -doc model id): 10/10 on all three fields, 116 s total. The vision code is in the repo too (vision/, tests with the invoice and its ground truth in tests/vision/).

How we know it's right

Speed is meaningless if the model says different things. The reference is transformers NemotronHForCausalLM in fp32 on the CPU, with weights read from the original checkpoint — not our repack — and unpacked with transformers' own FP4 code:

text	PPL reference	PPL engine	engine's greedy answer = reference
English paragraph	5.233	5.280	—
chat answer, TCP vs UDP	6.592	6.714	63/64
chat answer, Russian	5.873	5.854	64/64

Mean KL ≤ 0.003. The one differing token is a real tie: the reference itself scores -layer and layer 0.11 apart, less than one BF16 step.

Plus checks that must fail: swap the two 4-bit halves of each byte and PPL jumps to 18,784 — so the unpacking order is tested, not assumed. Repack and re-layout were compared byte-for-byte with the original on 200 experts each.

One bug of ours was caught by running the same input three times: perplexity drifted between 5.19 and 5.29. Cause: index_add_ with repeated indices uses atomics in random order. Fixed with a per-(token, expert) buffer summed in a fixed order. A second source of run-to-run difference wasn't a bug: mamba-ssm Triton kernels pick configs by timing, so separate processes rounded differently; TRITON_CACHE_AUTOTUNING=1 makes them bit-identical.

What didn't work

Putting the prompt's experts into the RAM cache — prompts got slower, decode didn't gain.
Prefetching 6 guesses instead of 4 — more wasted reads.
Skipping zeros per single neuron — re-reading scales per neuron was slower than copying the whole block. Per 16-neuron group worked.
Speculative early copies (above) — correct, slower.
Keeping recently used experts on the GPU — 40% of a layer's experts repeat on the next token, but holding them needs ~2.7 GB of VRAM that an 8 GB card doesn't have next to the 6.6 GB dense part.

Try it

The repo has the engine, the Triton kernels, the server, the repack/verify tools and every measurement script behind the numbers above. Honest caveat: it hasn't been run end-to-end from a fresh clone yet — the README lists the exact steps used on my machine. You need a Blackwell GPU (the kernels use the NVFP4 hardware path), ~70 GB free on a fast NVMe drive, and 32 GB of RAM.

The point is the same as in the earlier posts: for a mixture-of-experts model, "does it fit in VRAM" is the wrong question. Count bytes per token — and then measure where the time goes before you fix it, because the obvious culprit (here: Python launch overhead) can be the wrong one.

Engine, kernels, server, measurements and this write-up: Claude (Anthropic). Model choice, goal, the ReLU² idea and the decisions: me. All numbers are from logs on the machine described above.

I taught a local vision model to find the same AliExpress item cheaper — variant and shipping included

Helgard — Wed, 14 Oct 2026 21:01:27 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

The problem: the price you see on AliExpress is rarely the price you pay

You're searching for a phone charger on AliExpress and find it for $1.09. You click on it and realize $1.09 is a short cable price for the same listing. The actual price of the charger is $2.55. And you pay for shipping at the checkout.

It happens quite often. On 14 October 2026 I checked out 3 "chargers" from search results:

What the listing says	Price in the search results	What that price actually buys	Real price of the charger
"40W Fast Charger Type C Dual Port PD"	$1.09	a 25 cm USB-C cable	$2.55
"120W Fast Charger 4 Port USB Type C"	$2.28	a 25 cm cable	$4.09
"Original Samsung 25W Charger PD"	$1.15	a cable	$3.16

The usual tricks:

On search it shows the cheapest in the list. However, often the product is different - cable, adapter, instead of EU - US plug etc.
the price doesn't include shipping;
some prices are 'welcome deals' which are only available when you first buy something;
"sort by price" is not really sorting at all.

So if you want to get it cheaper, you got to open 20 tabs, click on each thing from every listing, calculate the item + shipping costs yourself. That's what I do every week, buying electronics parts from AliExpress.

What I Built

pricehunt is a free Chrome extension that automates all that for you.

Go to the item on AliExpress that you want to buy. Select your specific options (power, color, plug, size etc.) Then click on Find the same thing cheaper.

This extension then diligently searches AliExpress like someone would, by picture, by the seller's title, clicking on the listing one by one and checking all the options. And you finish with a list of offers of the same thing, same option you chose, shipping included.

Genuine search. Found item at $12.43. 2 minutes later the extension finds the same charger (same model, black, EU plug, free shipping) at $8.42. Manually I could find only $8.89 as a best option.

Each result plainly says what it is:

Exact match, Similar or Different and you can't fake a look-alike
the price is split into item + shipping;
like "$8.28 is the US plug price, not the one you selected", "$1.09 is for new buyer only" or "to get this discount, you must have 3 different items in your cart".

This is being powered by a local AI model on your own computer (not the cloud). It compares pictures of the product and determines if they are actually the same product. Your account on AliExpress, cart, searches are all local on your computer. Free search.

Which AI model? I've done the creation and testing using MiniCPM-V 4.7, an open free picture understanding model. It is the best one to use and I've done all the results in this post on it. But it doesn't mean you have to use that. You can hook up any Ollama image understanding model. Or any model that runs through some OpenAI-compatible server. Or any other model. The other ones that I've tested were just slightly less careful (numbers below).

Who it's for:

anyone that ordered a 'charger' and got a cable;
repair shops and hobbyists that have to order the same parts every month;
small resellers from AliExpress buying and selling locally. Every dollar they save buying is a profit.

It's like a coffee to you to save $4 on one charger. But it's the difference between a small side business succeeding or not if you save $4 on every one of 30 parts that you have to reorder each month.

Demo

Before search - your choice (with buttons, 33W, black, EU plug) - this is a picture extension will look for. You can choose a different picture if you prefer.

What's happening in the side panel: how many listings it's looked at so far, which ones it's opened, the current lowest price. If you get a captcha on AliExpress, it will switch that tab to the foreground and wait for you to solve it. No cheating.

Code

helgard-orlm / pricehunt

Find the real cheapest AliExpress offer: the variant you want + shipping, checked by a local open-weight vision model

pricehunt — the same thing, cheaper

A browser extension that finds the real cheapest offer for an AliExpress product: the exact variant you want plus shipping to your country. It's checked by an open-weight vision model running on your own machine.

Open a product page, pick the variant with buttons (power, colour, plug…), press Find the same thing cheaper, and watch the agent work in the side panel.

Built for the DEV Hacktoberfest 2026 Open-Source AI Challenge, week 2 ("Side Hustle"). Started 2026-10-14.

Why

Marketplace search shows the lowest price of any variant, and very often that's not the thing in the title:

Listing title	Price shown in search	What that price buys	Real price of the charger
"40W Fast Charger Type C Dual Port PD"	$1.09	a 0.25 m USB-C cable	$2.55
"120W Fast Charger 4 Port USB Type C"	$2.28	a 0.25 m cable	$4.09

…

View on GitHub

How to install:

Get the zip from Releases. Unzip it.
Go to chrome://extensions. Enable Developer mode. Choose Load unpacked and select the folder.
Select your AI model in the extension's settings - either your local Ollama on your computer or any OpenAI-compatible server.

How I Built It

The way I manually locate parts. I walked the extension through this process:

Search by the photo of the one option you chose. When you go to the list of items you'll usually see a collage of all the options first. And a picture showing only the option you want will have a lot more hits.
Look only in the listings that are of the same item, starting with the cheapest.
Search again by the seller's exact title. A lot of things don't actually have names, but people will steal the titles from each other. So if you find one good listing you can find the others.
Sort the results yourself. Do it for free shipping AND for paid. Don't rely on the sorting on the site.
Read every option inside the listing and pick only the one you want. 33W is 33W and make sure you have a plug for your country.
Repeat from the new cheapest offer until a few rounds in a row you fail to find anything cheaper.

A trick that has nothing to do with AI. Sellers don't take photos of their own goods - they all use the same few photos from the factory. So the extension will remember which photos your item started with, and when it finds them on other items it'll know, even if the AI model's not confident because the item is at a different angle. (Only tested it on one family of chargers so far so let's call it a promising idea not a proven fact.)

The AI model. Ran MiniCPM-V (v4.7) on my home PC, with a garden-variety 8 GB gfx card. It's an extension that asks it basic, y/n questions about it: "Is it the same model? same body? same ports? same power print?" Turns out it's very sensitive to phrasing. For example, asking "Is it the same type of product?" allowed 7 out of 11 different chargers through. Asking that strict question above only allowed 1 out of 11 chargers through.

What I measured (all with MiniCPM-V 4.7):

Test	Result
Telling a charger from a cable, a charger+cable set or a car charger, by the option photo (49 photos)	49 of 49 correct
Me vs the extension: a 100 W COB LED chip	a tie at $3.39: it found exactly the offer already in my cart
Me vs the extension: an Essager 33 W charger, starting from $12.43	$8.42; my own best by hand was $8.89

What I haven't tested yet: how good it is at spotting truly deceptive listings, where the listing title says "charger" but the image is of a cable. Those 49 images above were all available, with honest titles. So getting two wins against a human doesn't really set any sort of benchmark. It's two wins.

For the technically curious

Using llama.cpp with mixture of experts weights stored in system RAM (--cpu-moe) so only 4G on the gpu.
Token probabilities for "Yes" are the match percentages (logprobs). If a server does not provide logprobs then it'll default back to just a Yes/No.
We take 256bit diffhash of reference product photos (we remove brand banners first). But we only consider this diffhash matched if model also confirms that it's same type of product.
Real listings testing revealed some bugs: 35/59 discounted cards, the price parser used the crossed-out price. The extension added shipping when AliExpress displays free shipping. And the photo CDN server responds to multiple requests with 403 pages that are cached by browser for 1 year.
By getting a developer log, it is possible to send the full search trace to some small receiver program on my computer. Most of the bugs that were fixed for version 1.5 and 1.6 were the result of analyzing these traces.

Why Does Open Innovation Matter?

Because the marketplace won't build this for you. The "best price" stuff on AliExpress works for AliExpress. For a tool whose sole purpose is to give you a lower price you have to be able to trust it. You want the tool to run on your computer so you can verify and swap out the model.

Three things were enabled by open models.

Privacy. This extension uses your login account to browse. No cookies, cart, history etc are ever uploaded off your computer. I'd only use something like this for local model.
Cost. 1 search = dozens of photo comparisons. If it's a paid cloud AI it would cost money for every search. Here it's just a little bit of electricity.
Choice. Just add an open model and try it yourself. So Gemma 4 12B has 45/49 on photo test. And Gemma 4 31B has 47/49 on photo test. Both fell into cable only pitfall which MiniCPM-V detected. So all the numbers in this thread have the model in which they were measured.

Co-written with Claude (Anthropic) in Claude Code. Claude wrote most of the code and the first draft of this post. I brought the idea, the search method and the key decisions, and tested everything against my own purchases.

Cover: AI-generated illustration. All screenshots are real.

Running the 510 GB DeepSeek-V4.1-Flash on an 8 GB GPU — and three bugs that never raise an error

Helgard — Sat, 03 Oct 2026 12:25:53 +0000

My home machine for local models is modest: an RTX 5060 with 8 GB, a Core Ultra 5 225F, 31 GiB of RAM and a Gen5 NVMe drive used only for model files. DeepSeek-V4.1-Flash is 510 GB on disk. It now runs on that box as a normal chat model in Open WebUI: ~1.6 tokens/s reading experts from disk only, ~2.4 tokens/s with a 16 GB RAM cache. That is slow, and this post is honest about why — but it works, and every speed-up keeps the output token-for-token identical to a plain version that does all the math with DeepSeek's own code.

Code: https://github.com/helgard-orlm/deepseek-v41-flash-8gb

Who did what, up front: I set the goal, chose the model, proposed some of the ideas (predicting the next layer's experts among them) and made the calls. The engine, the server and the bug hunts were done by Claude (Anthropic) in my sessions; the overlap scheme and the one-token short path came from Codex (OpenAI) reviewing the engine. "We" below means the three of us.

This is the second model we run this way. The first, a 133 GB Qwen at 11 tokens/s, is described in the previous post. The comparison between the two is the most useful part of this one.

Bytes per token decide everything

For a mixture-of-experts model, the size on disk barely matters. What matters is how many expert bytes one token pulls in.

DeepSeek-V4.1-Flash: 40 layers, 384 experts per layer, 6 chosen per token, each expert 17.93 MiB in FP4 with scales.

6 experts × 40 layers × 17.93 MiB ≈ 4.2 GiB per token

The Qwen model needs 1.27 GiB per token. Same machine, same disk, same method — 3.3× more bytes, and that alone explains most of the speed gap.

Where the 475 GiB go:

part	size	where it lives
attention (FP8), shared experts, norms, router	~6.7 GiB	GPU
routed experts (15,360 of them)	269 GiB	NVMe → RAM cache → GPU
Engram tables (a per-token memory, FP8)	189 GiB	NVMe, ~48 rows per token, 5 ms
embeddings + output head		RAM, head computed on the CPU in FP32 like the reference

The approach: don't rewrite the model, only move its storage

DeepSeek publishes its reference inference code (model.py + kernel.py, MIT). We use it as is for all the arithmetic and replace only where the weights come from. That has a nice consequence: there is no conversion step. In the original safetensors files each expert's three matrices already sit next to each other (17.7 MB) and its scales are one more block (1.1 MB), so an expert is two pread calls with O_DIRECT straight from the Hugging Face files. A read benchmark showed this already uses 94–97% of what the drive can do, so re-laying-out the data on disk had nothing to give.

We first measured the whole thing on a rented RTX 5090 with VRAM artificially capped at 7.3 GiB, before buying the Gen5 drive. Once the numbers looked worth it, the model moved home.

Three bugs on RTX 50xx that never raise an error

This is the part I'd want to read before trying anything similar on a Blackwell consumer card.

1. TileLang 0.1.8 computes garbage on sm_120, silently. The reference pins this version. On our card the FP4 matrix multiply had cosine 0.0006 to a plain torch computation (i.e. unrelated numbers), the FP8 one returned NaN, and the model happily wrote "athaatha Stone". The reference self-test passed — it checks shapes, not values. TileLang 0.1.9: cosine 0.9996.

2. A race in the reference act_quant kernel. With DeepSeek's scale format (ue8m0, always on in this model) the kernel is built with num_stages = 0. On the RTX 5060, for more than 64 rows, part of the FP8 output comes out NaN — and differently every time: the same input five times gave 367, then up to 2065 NaN values. Generating one token at a time is clean, so only prompts were hit. It wasn't the compiler (CUDA 12.8 did the same). num_stages = 2 gives zero NaN and is bit-exact with a torch implementation at 1, 17, 300 and 1024 rows. That one line is the only change the setup script makes to DeepSeek's code.

3. The sparse attention kernel wants 141 KB of shared memory; consumer Blackwell has ~99 KB. Heads are independent, so we call the same kernel in groups of 16 heads. Small trap inside: .contiguous() on a head slice of a 1-token tensor doesn't change its strides, and the kernel checks strides — clone(memory_format=torch.contiguous_format) does.

The common lesson: on new hardware, "it ran without errors" says nothing. Compare every kernel with a dumb torch version on real weights before trusting a single generated word.

Making it faster: v1 to v4

The rule for every step: the 136 generated tokens of a fixed 4-question check must stay identical to the plain version (v1). Anything that changed a token was rejected or fixed. To be precise about what that proves: the speed-ups changed nothing. It does not prove v1 equals DeepSeek's untouched program run side by side — that program can't load the model on 8 GB, so we never ran it. What backs v1 instead: it calls DeepSeek's own model and kernel code for every computation, the individual kernels were compared with plain torch on real weights (above), the expanded wo_a matches DeepSeek's convert.py bit for bit, and the answers are right (Canberra, a correct TCP/UDP explanation, working merge code).

v1 — plain streaming: 0.89 tokens/s from disk, 1.17 with a 14 GB RAM cache. Read the layer's 6 experts, copy them to the GPU, wait, compute. Disk, bus and compute simply added up.

v2 — overlap (Codex's scheme): 1.28. A separate copy thread with CUDA events instead of a global synchronize. The surprise: issuing all 6 expert reads at once was slower. They share the drive, so they all finish together at the end of the layer and computing can't start any earlier. What worked was a deep queue but experts one after another: at most 2 in flight, each read in 4 parallel pieces. Waiting for the disk went from 0.65 to 0.31 s per token. (The first version of this had a race of its own — a ring buffer overwrote an expert that hadn't reached the GPU yet, and token 29 changed. The token check caught it.)

v3 — a short path for one token: +5.7%. The reference finds each expert's tokens with torch.where(indices == e) — a CPU↔GPU sync, 240 times per token, and each one delays the next copy. For a single token the indices are now moved to the CPU once per layer. Same arithmetic.

v4 — prefetch two experts of the next layer into the gap: 1.64 from disk (+21%), 2.3 with RAM. Apply layer i+1's router to layer i's input (rescaled by the two layers' norm weights) and you get the right top-6 71% of the time. Once layer i's real reads are in flight, read the top 2 guesses that aren't already in RAM. 94% of them get used. Three or four guesses are worse: they start competing with the reads that are actually needed.

One thing we got wrong along the way: I tried to reduce "bytes read in vain" by counting guesses that were already in RAM against the budget. Fewer wasted bytes, lower speed. Reads that land in the gap, while the disk would otherwise idle, are free; the only thing that matters is how many experts arrive on time.

version	disk only	with RAM cache
v1 plain	0.89	1.17
v2 overlap	1.28	1.42–1.67
v3 short path	1.35	1.95
v4 prefetch	1.64	2.28–2.34 (14 GB)
live chat, RAM 16 GB		2.43

A 311-token prompt takes 18 s (27 s in v1). Peak VRAM 7.06 GiB. Follow-up messages don't re-read the conversation: the server snapshots the full model state (30 MiB — KV, the sliding window, compression tails, indexer cache, Engram history) after each answer and processes only the new tail.

What we measured and dropped

Some of these were my ideas, some Claude's; all were measured on the real weights rather than argued about.

Store deltas between experts or layers, they'll compress better. The FP4 codes have 3.895 bits of entropy; the difference to the same expert in the next layer, or to a neighbour, has 3.997 — worse than the original. Even after the best neuron permutation and per-neuron scaling, the remaining difference is 0.999–1.000 of the original, exactly what two random matrices give.
Skip "silent" neurons and don't read their part of the down projection. To keep each expert within 5% error you still need 61–75% of the neurons: ~12% fewer bytes, and the read would have to wait for the first half of the computation. This model's SwiGLU neurons simply don't go quiet the way ReLU² ones do.
Smarter cache policies. A simulation over recorded routes: LRU beat LFU with ageing, per-layer LRU and SLRU. The theoretical optimum is ~15 points higher, but nothing simple gets near it.
More reader threads, splitting reads into smaller pieces — the drive was already near its limit.

Why it stops here

From disk, a token takes about 0.6 s, and just reading its 4.2 GiB at the 8.2 GiB/s the drive delivers on these reads is ~0.5 s of that. There's very little software left between us and that number. The only ways to go faster are fewer bytes (more RAM for the cache — the whole expert set is 269 GiB, so a cache is always partial) or a second drive. With Qwen the same method gives 11 tokens/s because each token needs a third of the bytes. If you're choosing a MoE model to stream on a home PC, look at bytes per token before parameter count.

Try it

You need a fast NVMe drive with ~520 GB free and a Blackwell GPU (the fixes above are for sm_120; other cards may need fewer of them). setup.sh downloads the exact model revision we used if it isn't there, checks that kernel.py is the expected file, and builds a copy of DeepSeek's code with the one-line fix. Before publishing, we cloned the repository fresh on the same machine, ran the setup against the downloaded model and started the server from the clone: it loaded with every parameter accounted for and answered correctly. The engine and server files are byte-identical to what runs at home. The full 510 GB download from scratch was not repeated for this test.

Engine, server, bug hunts and this write-up: Claude (Anthropic). Overlap scheme and the one-token short path: Codex (OpenAI). Goal, model choice, the expert-prediction idea and decisions: me. All numbers are from logs on the machines described above.

Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe

Helgard — Sat, 03 Oct 2026 11:46:52 +0000

I run local models on one home machine: an RTX 5060 with 8 GB, a Core Ultra 5 225F, 31 GiB of RAM and a Gen5 NVMe drive used only for model files. When NVIDIA published Qwen3.8-Flash-Next in NVFP4 (133 GB on disk), the obvious answer was "it doesn't fit". It does now: it runs as a normal chat model in Open WebUI at 9–12 tokens/s, and its output matches the Hugging Face reference implementation.

Code: https://github.com/helgard-orlm/qwen-flash-next-8gb

Who did what, up front: I set the goal, chose the model and made the calls along the way. The engine, kernels and server were written by Claude (Anthropic) in my sessions; the CUDA-graph speed-up was written by Codex (OpenAI) in a parallel session, and Claude then found and fixed a crash in it. "We" below means the three of us.

This post is about the method, the measurements, and what didn't work.

Why this model, of all models

Most of the weights of a mixture-of-experts model are experts, and each token uses only a few of them. So the question is not "how big is the model" but how many expert bytes one token needs.

Qwen3.8-Flash-Next: 48 layers, 512 experts per layer, top-10 routing, and each expert is tiny — three 2560×640 matrices, 2.70 MiB in NVFP4 with scales.

10 experts × 48 layers × 2.70 MiB ≈ 1.27 GiB per token

For comparison, DeepSeek-V4.1-Flash, which we run on the same box with the same trick, needs about 4.2 GiB per token. Same disk, a third of the bytes. (How DeepSeek went — 2.4 tokens/s and three silent bugs on RTX 50xx — is in a separate post, code at https://github.com/helgard-orlm/deepseek-v41-flash-8gb.)

Everything else (embeddings, attention, the 36 Gated DeltaNet layers, router, shared expert, lm_head) is about 7.2 GB in BF16. Storing the big DeltaNet matrices in FP8 with one scale per row brings the resident part to 6.13 GiB of VRAM. The 63 GiB of experts live on the NVMe drive.

The engine

There was no runtime for this architecture that could stream experts, so it's a from-scratch PyTorch + Triton engine. The math was ported from transformers 5.18 (qwen4_exp): DeltaNet, full attention with a block indexer that kicks in beyond ~2048 tokens, four hyper-connection streams, and an n-gram embedding table (PLE, used in layer 1) with 51 billion parameters.

The pieces, in the order they mattered:

1. Repack experts for the disk. In the original files an expert's six tensors are scattered. We rewrote them into one 2,764,800-byte block per expert (675 pages of 4 KiB): gate | up | down | gate_scale | up_scale | down_scale. One expert = one pread with O_DIRECT into page-aligned pinned memory = one copy to the GPU.

2. A RAM cache in front of the disk. LRU over pinned memory (16 GB holds about a quarter of all experts), a pool of reader threads, and two banks of GPU slots so the next layer can load while this one computes.

3. Prefetch into the gap. After a layer's required reads are issued, guess the next layer's experts and read two of them while the GPU is busy (91% of guesses are used). The first version started the prefetch at the same time as the required reads — they shared the disk and everything got slower. It has to go into the gap, not next to the real work.

4. Own kernels. An NVFP4 SwiGLU expert kernel for a single token using Blackwell's hardware cvt.rn.f16x2.e2m1x2 instruction (115 µs per layer, 3.6e-7 from a torch reference), and an FP8 row-scaled matrix-vector kernel. The generic unpack path for the FP8 DeltaNet matrices had been costing ~106 ms per token on its own.

5. Don't read 66 GB per prompt chunk. The first prompt path processed 512 tokens at a time, and each chunk needed nearly every expert, so the whole expert set was read again for every chunk. Running the whole prompt through each layer at once, with experts read in batches of 32, took a 2451-token prompt from 71.6 s to 23.3 s.

6. Read only the PLE rows you need. The n-gram table is a 53.7 GB file; each token needs a handful of rows. Random reads from NVMe: 0.3–0.5 ms per token. (On the HDD it would be a disk seek per row — a second AI acting as reviewer caught that the table was still on the HDD in the first run.)

Where the time actually goes

With every expert already in RAM, a token took 80–88 ms. Splitting it:

39 ms — Python issuing GPU work (the GPU was waiting on Python, not the other way round);
47 ms — moving 1.33 GB of experts over PCIe (measured 28.6 GB/s, PCIe 5.0 ×8);
and the two ran one after the other.

There are 52 CPU↔GPU syncs per token (mostly reading the router's chosen experts). We expected them to be the problem; measured, they are cheap. They only hurt because the expert copy for layer L can't start until layer L's router has answered. That pointed to three changes:

Speculative copies. Run layer L+1's router on layer L's MoE input. It picks the right top-10 65% of the time — enough to start most copies one layer early. (Prefetching top-16 to raise the hit rate moved more bytes over the bus than it saved.)
CUDA graphs for the 36 one-token DeltaNet layers (Codex). One shared memory pool: 36 private pools ran out of memory on an 8 GB card. Graphs are dropped before any prompt ≥1024 tokens.
Pin the decode thread to a P-core. On a hybrid CPU the unpinned thread wandered between P and E cores: 93–108 ms per token, jumping around. Pinned to core 0: a flat 81.6 ms.

Results

configuration	tokens/s
first version, experts from disk, no cache	2.68
disk only + prefetch	4.72–4.75
RAM cache 16 GB + prefetch	8.1
live chat, 1000-token answer, before the last three changes	9.71
live chat, 1000-token answer, after	11.65

Prompt processing: 24 tokens in 4.5 s, 2451 in 23.3 s, 7182 in 98 s. Follow-up messages are fast because the server snapshots the whole recurrent state (DeltaNet matrices + KV + indexer keys) in RAM and only processes the new tail.

How we know it's right

Speed without correctness is a random number generator, so every change was checked against the transformers reference fed with the same quantized weights:

next-token argmax: 32/32, mean |Δlogit| 0.077;
a 2451-token prompt, long enough for the attention indexer to start selecting blocks: 11/11, and the needle-in-a-haystack fact comes back right;
perplexity EN 2.21 / RU 2.22 — plus control runs that must break: with nibbles swapped it's 2016 / 33,269, with experts removed 588 / 5,351. That checks the NVFP4 unpacking independently of the reference.

What didn't work

A fused DeltaNet kernel (one kernel for the recurrence, 4.3 → 0.43 ms per token across 36 layers). 5 of 256 tokens came out different from the reference. Rejected — 4 ms isn't worth a model that says different things.
The 8192-token context didn't actually work, in the old version either: a 7182-token prompt ran out of VRAM in attention. Fixed by sizing prompt pieces so that piece × (position + piece) ≤ 3072².
A crash only after long prompts: CUDA graph capture in the default global mode was invalidated by expert-reader threads calling event.synchronize(). capture_error_mode="thread_local" fixed it.

Try it

The repository has the engine, the server (OpenAI-compatible, streaming, a thinking-mode model id), the repack and reference-check tools, and a setup.sh that prepares a working directory from a Hugging Face snapshot. Before publishing, we ran it the way a stranger would: a fresh git clone of the repo on the same box, setup.sh into an empty directory (15 min 48 s, mostly reading ~120 GB from an HDD; spot check of 204 repacked experts against the originals: 0 differences), then the server from the clone. The repacked experts and the extracted non-expert weights came out byte-identical to the production directory, and three greedy test prompts gave identical answers from both.

You need a Blackwell GPU (the NVFP4 kernel uses the sm_120a e2m1 instruction), a fast NVMe drive with ~130 GB free, and patience for a first-time setup that reads most of the 133 GB.

The bigger point: on a mixture-of-experts model, "does it fit in VRAM" is the wrong question. The right one is bytes per token, and a home PC with a fast drive can serve a lot more of them than it looks.

Engine, kernels, server and this write-up: Claude (Anthropic). CUDA graphs and the speculation/pinning A/B: Codex (OpenAI). Goal, model choice and decisions: me. All numbers are from logs on the machine described above.

An AI hardware advisor that walks every path — and gets its math checked by the rules in Sanity

Helgard — Sat, 03 Oct 2026 05:49:32 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

An agent that answers "what computer do I need to run this AI model?" — including "don't buy, rent" and
"you already have enough". It reads a Sanity dataset of criteria, not a catalog: laws (formulas for weights
memory, KV cache, speed ceiling, own-vs-rent breakeven), machine-checkable rules, 11 solution paths
(keep what you have · upgrade · used · one GPU · multi-GPU · unified memory · MoE experts in system RAM ·
CPU-only small model · smaller model · cloud API · rented GPU), run reports and reference hardware.
New models appear every week, so anything not in the base is looked up on the web by the base's own checklist.

Sanity project: onwa0wvs, public dataset v2 (200 documents), Studio: https://hw-for-ai-lab-v2.sanity.studio
Agent connection: Sanity Context MCP on the full dataset with embeddings enabled (/context/mcp/onwa0wvs/v2); all numbers below are measured through this connection
Second entry: a Knowledge Base (kbW7wsbtJQkl, built from the core 138 documents of the same dataset, refreshed weekly) behind its own Context MCP endpoint in KB mode — created and connected, not measured separately
Demo — try it with your own agent: https://hw-advisor.helgardorlm.tech (MCP https://hw-advisor.helgardorlm.tech/mcp, no login) · Replay of all 27 measured runs: https://helgard-orlm.github.io/sanity-ai-hardware-advisor/
Repo: https://github.com/helgard-orlm/sanity-ai-hardware-advisor

Demo

Live — bring your own agent (no login, no keys): https://hw-advisor.helgardorlm.tech · MCP endpoint https://hw-advisor.helgardorlm.tech/mcp
Replay of all 27 measured runs: https://helgard-orlm.github.io/sanity-ai-hardware-advisor/

Code

https://github.com/helgard-orlm/sanity-ai-hardware-advisor (the public proxy with check_answer is in public_mcp/)

Sanity Project Details

Project ID onwa0wvs, public dataset v2 — 200 documents, 15 types (law, rule, solutionPath, aiModel, gpu, cpu, runReport, offer, cloudOffer, situationTemplate, …)
Query it directly: https://onwa0wvs.api.sanity.io/v2025-01-01/data/query/v2?query=*[_type=="law"]
Studio: https://hw-for-ai-lab-v2.sanity.studio

Try it with your own agent (no keys, no login)

The demo is a public MCP endpoint, not a hosted chatbot: you bring the agent, the base brings the knowledge and the checking.
Add https://hw-advisor.helgardorlm.tech/mcp to ChatGPT (developer mode), Claude, Claude Code, Codex, Cursor or VS Code.
There are one-click buttons for Cursor and VS Code on the page. Or paste one line into any agent that can run commands:
"Read https://hw-advisor.helgardorlm.tech/setup.md and follow it."

Behind it is a ~230-line stdlib proxy. It passes the four Sanity Context MCP tools through unchanged and keeps the token
server-side. It puts today's date and the advisor's work order in front of initial_context. It adds one tool, check_answer,
which is the same validator the app uses: it recomputes every calculation with the law.formula stored in Sanity.
In the first outside test (Codex, its own subscription), check_answer rejected the first draft with 4 errors. The second draft passed:
5 calculations recomputed, 11/11 solution paths walked. A test from ChatGPT (developer mode) also went through to a passing check. Agents that can only browse get the same tools as plain HTTPS links
(/api/context, /api/query?q=…, /api/check), listed in /llms.txt.

Known limit: the check covers numbers, completeness, sources and identity. It does not cover whether a verdict makes sense. In one test an agent said yes to "MoE experts in RAM" for a dense model, and the check passed.

How I Used Sanity

Sanity Context MCP tools the agent calls: initial_context, groq_query, schema_explorer, array_field_reader (plus our check_answer). What it does with the content it retrieves — the structure is used, not just searched:

Paths are data. The agent must give a yes/no/maybe with a number for every solutionPath document. Before this, the same model answered a 321B-MoE question with "4×H200" and never mentioned the one-GPU + large-RAM path.
Rules are the search checklist. For any model/card/CPU that is not in the base, the app requires the fields that the base's rule documents test for that type (e.g. cpu.instructionSets, aiModel.variants.fileGb).
Laws are recomputed by code. The agent writes every calculation with its inputs into a hidden check block; the app evaluates the law.formula from Sanity and sends the answer back if the number is wrong.
The person's situation is stored, not remembered: country, budget, existing PC survive a change of mind.

The same code checks the base itself: every law variable and rule path must be a real schema field
(coverage check), and laws get property tests. That caught the filler model writing an own-vs-rent formula that
added electricity instead of subtracting it and billed rent for 24 h/day — 12.3 vs 110 months on the same inputs.

A real trace (GLM-5.3-Flash, not in the base) — open it in the replay

GROQ over the base → no aiModel document → 8 web searches by the base's own checklist: the model card and config.json (layers, KV heads, head dim), file sizes, and the GPU fields the base's rules test (slots, bandwidth, compute capability).
Laws from Sanity: weights 320 × 4 / 8 = 160 GB (NVFP4), KV for 8k context 22.5 GiB (flagged as rough for hybrid attention).
Validator round 1 sent the answer back: the speed ceiling was written as 19.91 tok/s — code evaluated the law.formula from Sanity with the agent's own inputs and got 199.1 (a 10× slip); plus three entities missing fields that the base's rules check (e.g. variants.fileGb, slots).
Round 2: no errors. All 11 solution paths get a verdict — multi-GPU = yes, MoE-in-RAM / unified memory = maybe, cloud = no (the person ruled it out) — and the build is 3 × RTX PRO 6000 (288 GB) with the numbers above.

Other catches by the same recomputation in the measured run: KV cache 0.949 → 6.75 GiB (A4), monthly electricity cost 0.675 → 2.025 (A5).

Measured, honestly

Blind grader (a different model, sees only the dialogue and a frozen checklist), 9 scenarios × 3 runs,
78 checklist items, same base state for all arms:

version	score	median time
v2: base + instructions	47/78	62 s
v3: + solution paths, validator	53/78	104 s
v3.1 via Sanity Context MCP: + rules-as-checklist, recomputed laws, dated prices	59/78	181 s

What this does not show: that structured retrieval beats plain text for reading. The same 200 documents
pasted as text notes scored the same as v3 (53/78) — at this size the model reads either. Where structure
mattered in our runs is checking: fewer first answers rejected by the validator with structured queries
(8/27 vs 14/27), and the gains of v3.1 are exactly in the scenarios where code recomputes Sanity's laws
(KV for 8 users 7/9 → 9/9, own vs rent 5/12 → 7/12, "don't buy" 4/6 → 6/6, v2 → v3.1). The price: answers take ~3× longer.
An earlier full run of v3.1 scored 62/78 with two bugs in my own checker (nested fields, European number format) that sent
correct answers back for rework; after fixing them the clean run scored 59/78 — with N=3 per scenario, 59 vs 62 is noise.

How it was built

Architect method: a risk catalog (16 classes of agent failure seen in earlier sessions) → acceptance scenarios
written before changes → build → blind grading. The base was filled by a model (GPT Luna) from an empty schema and
two prompts; I never wrote data by hand. An independent judge model watched the session and corrected the method
several times (frozen checklist, snapshot baseline, text control).

Known limits

Answers are slow (minutes) because the validator can send them back up to twice.
Ratings N=3 per scenario; differences of 1–2 points are noise.
"unknown after search" is allowed and can be abused for vague entities.
In the measured run 12 of 27 answers came out in Russian to English questions (an account language setting leaked in); the app now checks the answer's language in code and sends it back (3/3 English on a re-test).

We create a way to unload Qwen2.5 KV cache to RAM.

Helgard — Thu, 11 Jun 2026 20:14:41 +0000

Two separate problems with local LLMs, one mechanism fixes both:

Long context doesn't fit. The KV-cache grows linearly with every token; on 8 GB it dies around 110k.
Memory doesn't survive a restart. Kill the process and the cache is gone — next session re-reads everything from scratch (that's all RAG is: re-reading text, every query).
What the KV-cache actually is: as the model reads, each layer stores a key/value vector per token — this is the model's working state, the thing it attends over to pick the next token. It's not text, it's computed internal state. Normally it lives entirely in VRAM and only exists while the process is alive.

The mechanism:

Keep only the last N tokens' KV in VRAM (the hot window).
Stream everything older to CPU RAM (8-bit), then disk. It's not deleted, just moved somewhere 10× cheaper.
Keep a tiny index in VRAM: one vector per sentence. On a query, that index finds the few relevant chunks and copies only those KV vectors back into VRAM for the attention step.
So VRAM stays flat — it holds the hot window plus a small retrieval budget, regardless of whether total context is 50k or 800k. The cap moves from VRAM to your RAM stick.

The part that makes it memory and not just offload: you write that cold KV to disk and reload it in a different process days later. The model resumes from its own stored state — a few ms of memcpy, no re-reading the source. Two things make the positions line up across the gap: keys are stored already rotary-rotated at their absolute position, and the query is injected at its true position, so relative distances stay exact even between tokens 700k apart.

Result (RTX 5060 8 GB, 4-bit): a planted fact is the top prediction from 120k–500k tokens, top-2 at 800k, VRAM flat ~6 GB — where a normal full cache OOMs at 65k. The live store behind my assistant is 2.5M tokens of past sessions in 467 MB, answered in ~1s on CPU. Miss → LOW CONFIDENCE, not a hallucination.

Why Qwen2.5-7B-1M and not anything else — two independent requirements, most models fail one:

Architecture must keep a growing KV. Mamba/SSM and linear-attention models (Granite-4, RWKV, Qwen3.5's hybrid layers) compress history into one fixed-size state — there's no per-token KV to stream out or pull back. The whole method is inapplicable to them.
The model must actually read at that range. Effective context is ~50–70% of the advertised number (RULER). MiniCPM-1B (128k nominal) goes to garbage by ~120k even with a perfect full cache in VRAM — that's the model's limit, not the cache's. No infrastructure fixes it.
Qwen2.5-7B-1M passes both and fits 8 GB in 4-bit. Substrate alone OOMs at 65k; the model alone can't persist across restarts. The combination is the whole point.

Limits: confidence drops with distance (p 0.98 → 0.21 at 800k); many near-identical facts confuse the cheap index (need full attention there); on messy logs generation can override the retrieved KV with its own prior, so I quote stored text verbatim instead. Research PoC — the core (spilling KV to CPU) has precedent (ArkVale, RetrievalAttention), credited in the README.

Code (MIT): https://github.com/helgard-orlm/living-kv-cache