Run Qwen3.8-Flash-Next at 79 tok/s with Strata and a 5-bit quant on 128 GB of RAM

Two weeks ago I wrote about running Qwen3.8-Flash-Next with EXL3 and TabbyAPI on 40+ GB of RAM. Since then the workstation got 128 GB of RAM, and that changed which setup is fastest. This guide covers the setup we now use every day: Strata serving Unsloth's 5-bit UD-Q5_K_XL GGUF of the same 125B model, with a 512K context, at about 79 tokens per second on one RTX 5090.

Every number below was measured on our own machine on October 15, 2026. Where a number comes from someone else, I name the source.

What changed since the EXL3 guide

The EXL3 guide ran turboderp's 3.05 bpw quant because 64 GB of RAM did not leave room for anything bigger. With 128 GB, two routes opened up:

• TabbyAPI with the 4.05 bpw EXL3 quant and a pinned CPU expert arena. Its server log showed about 68 tokens per second for decoding with a 3 GB VRAM reserve, and about 1,780 tokens per second for prefill at 254K tokens.
• Strata with Unsloth's UD-Q5_K_XL, a 5-bit quant with 8-bit down projections. Same model, more bits per expert, and a runtime built for this one model family.

On the same machine, Strata decoded about as fast as TabbyAPI before tuning and faster after calibration, read long prompts about 1.8-2.4x faster, and ran at 512K context. That is why it is our default now.

The machine

• GPU: RTX 5090, 32 GB of VRAM. It also drives the desktop, which matters later.
• CPU: Ryzen 9 9950X3D (16 cores, 32 threads, AVX-512).
• RAM: 128 GB DDR5 as 4x32 GB, running at 5,600 MT/s. More on why not 6,000 in the pitfalls.
• Disk: one NVMe drive with room for the 158 GB download.
• OS: Arch Linux (Omarchy), NVIDIA driver 615.78, CUDA 13.4.
• Strata at commit 296b59d.

Why Strata

Strata is a C++/CUDA engine plus a Python server for Qwen3.8-Flash-Next and its variants. It reads the model in three tiers: the experts your conversation uses most live in VRAM, the next ones in RAM, and anything that does not fit comes from the SSD. It moves experts between the tiers while it runs, drafts tokens with the model's own MTP layer, and exposes OpenAI- and Anthropic-compatible APIs plus a small web UI with a live monitor.

The project's install docs are written so that a coding agent can follow them (docs/AI_SETUP.md). I still recommend reading them yourself once: the recommended sizes in those docs target 32-96 GB gaming PCs, and the larger Unsloth quants are a separate family in setup.

Pick the quant by RAM, not by bits

Strata keeps every expert in RAM when the experts plus about 16 GB fit. If they do not fit, it reads the rest from the SSD while it answers, and that costs a lot of decode speed. So the rule is simple: take the largest quant whose experts fit in RAM with room to spare.

From Strata's model table, with its own quality measure (how often the quant picks the same top token as Q8_0):

Quant	Download	Experts	RAM for all experts	Same top token as Q8_0
UD-IQ4_XS	94 GB	55 GiB	~72 GB	88.9%
UD-Q4_K_XL	111 GB	72 GiB	~88 GB	93.1%
UD-Q5_K_XL	158 GB	92 GiB	~108 GB	94.6%
UD-Q6_K_XL	169 GB	102 GiB	~118 GB	94.6%
Q8_0	188 GB	120 GiB	~136 GB	100% (reference)

On 128 GB, UD-Q5_K_XL is the sweet spot. UD-Q6_K_XL scores the same 94.6% in Strata's table, with a mean KL divergence of 0.033 against 0.038 for Q5, and needs 10 GB more RAM that a 128 GB machine does not have to spare. Q8_0 does not fit.

Before you download anything, let setup check your PC:

git clone https://github.com/Niko1221/Strata.git
cd Strata
./setup.sh --check

It prints the GPU, driver, RAM, CPU and free disk, and which sizes fit.

Install

This is the setup command we ran, non-interactive, with the model files on the NVMe drive:

./setup.sh --yes --family unsloth --model UD-Q5_K_XL \
  --context 262144 --vision no --gpu 0 \
  --data-dir ~/Models/strata --port 8081 \
  --vram-reserve-mib 3072 --no-start

What the flags do:

• --family unsloth --model UD-Q5_K_XL picks Unsloth's 5-bit files.
• --context 262144 is the model's trained context. You can raise it later.
• --vram-reserve-mib 3072 keeps 3 GB of VRAM free. Our desktop runs on the same card, and when the VRAM was completely full the screen started to flicker.
• --port 8081 because port 8080 is taken by our GLM setup (more on that at the end). The default is 8080.
• --no-start installs without starting the server, so you can start it the way described below.

Setup writes a JSON config (strata-unsloth-ud-q5_k_xl.json) and a start script (run-unsloth-ud-q5_k_xl.sh) in the Strata folder.

If the download is slow

Strata's own downloader reached only about 7 MB/s here, which is around six hours for 158 GB. The faster route: stop setup, fetch the shards with the Hugging Face CLI at the exact revision setup pins, put them in the model folder, and run setup again. It checks every file against its SHA-256 and skips what is already there.

# the revision setup.py pins for Unsloth's repo
grep -n '"unsloth/Qwen3.8-Flash-Next-GGUF"' setup.py

hf download unsloth/Qwen3.8-Flash-Next-GGUF \
  --revision 38bb39ee97821de2c9009abb7e93950eec396e66 \
  --include "UD-Q5_K_XL/*" \
  --local-dir ~/Models/strata/hf-tmp

mv ~/Models/strata/hf-tmp/UD-Q5_K_XL/*.gguf ~/Models/strata/models/unsloth-UD-Q5_K_XL/

Use the revision your own setup.py prints, not the one above, if they differ.

If setup says shard 1 is "short"

The first of the six UD-Q5_K_XL shards holds only metadata and no tensors (10.9 MB). At commit 296b59d, setup's shard check still treated it as a few bytes too short and stopped on a correct download. There is a fix waiting upstream (PR #2505, same idea in #2632; both still open on October 16, 2026). Until one of them is merged, the one-line local patch is to skip shards without tensors in check_shards() in setup.py:

# in check_shards(), before: need = g.data_start + ...
if not g.tensors:
    continue

Check after every git pull whether the fix has landed upstream, so you do not keep a patch you no longer need.

Start it outside your shell's memory limits

Strata pins about 98 GB of RAM with this quant. Two Linux defaults can kill or cripple a process that size:

• A shell or tool runner inside a systemd scope with a memory limit. Our agent's terminal scopes have memory.max of 4 GiB, and a model started from there ends up mostly in swap.
• systemd-oomd. Omarchy enables it for the user's app.slice and kills a cgroup after 20 seconds above 50% memory pressure. A model that pins most of the RAM looks exactly like that during load.

So we start the server in its own slice, without a memory limit:

systemd-run --user --scope --collect --slice=llm.slice \
  --unit="strata-$(date +%s)" -p MemoryMax=infinity \
  ./run-unsloth-ud-q5_k_xl.sh

Stop it with systemctl --user stop 'strata-*.scope'. Avoid a broad pkill -f: the pattern also matches the shell that runs it.

The first start takes a few minutes and the desktop can stall for a moment while the RAM gets pinned. Then check that it is up:

curl -s http://127.0.0.1:8081/health
# {"status": "ok", "max_context": 524288, "model": "...", "loaded": true, ...}

curl -s http://127.0.0.1:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "strata", "messages": [{"role": "user", "content": "Say hello in five words."}],
       "max_tokens": 64, "reasoning_effort": "none"}'

Any OpenAI-compatible client works with base URL http://127.0.0.1:8081/v1, any API key and any model name. Anthropic-compatible clients use /v1/messages. Keep it on 127.0.0.1; if you open it to your network, set --api-key as well.

Go to 512K context

The model is trained for 262,144 tokens. Strata can stretch that with YaRN rope scaling, and setup picks the method for you:

./setup.sh --setup --context 524288

Add --no-start if you run it from a shell with a memory limit (see above), and start the server separately. Setup adds --max-context 524288 --rope-scaling yarn --rope-scale 2 to the engine arguments. If you use a custom chat template (see below), check afterwards that the tokenizer key in the config still points to it. Strata's docs call it experimental and note that needle tests separate scaling methods poorly; real long-document Q&A does. On our machine it found the needle at 520K tokens, and decoding stayed the same speed as at 262K. The KV cache is int8 with 32K tokens resident on the GPU, so the extra context costs RAM, not VRAM.

Calibrate for your PC

Strata has a calibration that measures a few engine settings on your hardware and keeps only changes that are more than 3% faster:

./setup.sh --calibrate --no-start

Without --no-start it starts the server afterwards. Run it with the server stopped, after you set the VRAM reserve, and in the same way as the server (in llm.slice, without a memory limit), because it loads the full model. Strata estimates 15-30 minutes. We called the same calibration function (calibrate_config in setup.py) from a small script, and it finished in about 11 minutes. On our machine it chose:

--pcie-frac 0.35 --spec-min-p 0.70
--adapt-every 1 --adapt-swaps 160 --adapt-decay 0.97

The --adapt-* settings drive the adaptive expert cache: how often it swaps experts between RAM and VRAM, how many per step, and how fast old usage counts fade. --spec-min-p 0.70 makes the MTP drafter stop earlier when it is unsure. The number of CPU workers (4 to 15) made no difference.

Calibration reports its own numbers (86.8 tokens per second at the end here), but those are its internal measurements. I wanted one I control, so I ran an A/B with a restart between each arm, old config and new config, twice each. Back up the config before you calibrate, so you can do the same.

Benchmarks

How we measured: a client-side streaming bench against the OpenAI endpoint. Eight runs per arm, alternating a ~400-word Dutch prose prompt and a coding prompt (a binary search function with tests), greedy decoding, 512 output tokens, median reported. Prefill comes from a needle-in-a-haystack test at the given prompt length. The needle and filler text are fixed per length, so a later run can get a partial prefix-cache hit from an earlier one: read the prefill numbers as approximate. Nothing else heavy ran on the machine.

Setup	Decode (tok/s)	Dutch prose / code	Prefill (tok/s)	Needle	VRAM peak
262K context, default settings	67.3	72.6 / 64.6	4,205 at 259K	found	29.3 GiB
512K context (YaRN 2)	66.8	71.8 / 62.5	3,817 at 520K	found	29.3 GiB
512K, before calibration (A/B, 2 runs)	70.9 / 70.9	77.0 / 65.8 and 76.5 / 65.5	-	-	-
512K, after calibration (A/B, 2 runs)	78.3 / 79.1	82.1 / 74.3 and 83.2 / 75.2	3,196 at 265K	found	29.1 GiB

The first two rows ran earlier that day, without the custom chat template described below; the A/B rows both ran with it. Only rows measured in the same session are a fair comparison, which is why the A/B pairs matter more than the first two rows.

Prefill ranged from 3,196 to 4,205 tokens per second at about 260K tokens, before and after calibration. For TabbyAPI with the 4.05 bpw EXL3 quant, its server log reported about 68 tokens per second for decoding with the same 3 GB reserve, and the same needle test prefilled 254K tokens at about 1,780 tokens per second. Server-log and client-stream decode figures are measured differently, so treat that as close, not one to one.

For comparison, Strata's docs list 57-60 tokens per second for stories and 82 for code with the same quant on an AMD Radeon AI PRO R9700 (32 GB). That is a different card and a different bench, so do not compare them one to one.

Quality checks

Speed only matters if the answers hold up. We ran two checks of our own.

Coding tasks with hidden tests. Five tasks, three trials each, run in a sandbox, reasoning effort medium. Strata with UD-Q5_K_XL passed 15 of 15 (480 of 480 test cases). TabbyAPI with EXL3 4.05 bpw scored the same. These tasks turned out too easy to separate the two quants; the difference between them is speed and context, not this score.

Our own work. A bench with nine tasks from our day job, each with an automatic checker: two Astro helpers in JavaScript, two WordPress PHP snippets run in a PHP sandbox, SEO meta tags as JSON, LocalBusiness JSON-LD, a Search Console tool call, a Search Console quick-win analysis, and a Dutch blog intro. Two rounds per setting, 108 points maximum:

Reasoning effort	Score	Time per round
none	100 / 108	62 s
low	107 / 108	-
medium	108 / 108	221 s
xhigh	90 / 108	577 s

More thinking was not better: xhigh hit the token limit twice and produced a wrong tool call, at 2.6 times the time of medium. Strata's API defaults to high, so we send "reasoning_effort": "medium" from our clients. Sampling did not matter on this bench either: greedy, temperature 0.7 and temperature 1.0 all scored 108 at the same speed. Without a sampling block in its config, Strata decodes greedy.

A custom chat template, if you need one

We write a lot of Dutch, and the model likes to mix English words into Dutch text. A short rule in the system prompt helps. Strata renders chat_template.jinja from the folder that the tokenizer key in its config points to. So you can make a folder with symlinks to the pack's tokenizer files plus your own chat_template.jinja, and point tokenizer there. Leave the files in the pack itself alone, so a setup run or update does not overwrite your changes or trip over them.

Two things to know. Strata passes only enable_thinking and reasoning_effort from chat_template_kwargs to the template, so your own template variables cannot be switched per request: a template change means a config change and a restart. And measure it: in our A/B with restarts, the rule cut English words in 30 Dutch answers from 28 to 13, with no measurable speed difference on our use-case bench (79.8 vs 80.3 tokens per second) and about 350 extra prompt tokens, which the prompt cache absorbs.

Pitfalls we ran into

Free RAM is the real limit. Strata leaves about 4 GB of RAM free on this machine. A browser with many tabs or a second agent process is enough to push part of the engine into zram. When that happened, the server log showed decoding at 40-55 instead of about 80 tokens per second. Check with free -g, swapon --show and the VmSwap line in /proc/<pid>/status of the engine process. Run it on its own: not next to another model runtime, a game or a GPU miner.

Four DIMMs at EXPO speed were not stable. Our four sticks are rated for 6,000 MT/s, and at that speed the machine logged "Bad page state" errors and kernel oopses, and crashed. Worse: most of a model download written in that period was corrupt on disk. We now run the second EXPO profile at 5,600 MT/s, and we check every model file against the SHA-256 published on Hugging Face. Strata's setup does that for its own downloads; do it yourself for anything else.

Restart only after the VRAM is free. After stopping a runtime, the VRAM takes a few seconds to come back. Starting the next one too early fails on a VRAM check, or leaves less VRAM for the expert cache.

Measure with fresh prompts. Strata caches prompt prefixes. Repeating the same long prompt gives absurd prefill numbers. Change the prompt (or at least its start) for every prefill run.

What's next

The same 128 GB also runs GLM-5.3-Flash, a 321B model, through Project Maya, a fork of Strata built for that model. It is a third of the speed of this setup but a much bigger model, and it reads up to 512K tokens without using more VRAM. I wrote a companion article about that setup and its numbers: Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090.

If you are deciding what hardware to buy for running AI locally, Voltti has a practical guide on the trade-offs between local and cloud AI for business use.