Running Qwen3.8-Flash-Next on a 128 GB Mac: The Expert-Pruning Trap, and a Memory-Mapped n-gram Table That Gets You to 240K Tokens

Hello, everyone.

Have you ever wanted to run the smartest model you can on your own Mac? I have. The catch is that the smartest models are also the biggest, and even 128 GB of memory often falls just short. That is exactly when a smaller, pruned build starts to look tempting.

Today's story is about getting Qwen3.8-Flash-Next to run properly on a Mac Studio M4 Max with 128 GB. The expert-pruned build (REAP-288) had lost its Japanese and its general knowledge. The full 4-bit build ran out of memory. In the end, reading the model's 32 GB n-gram table from disk on demand made it fit: no quality loss, prompts up to 240K tokens, and 7.4 minutes to read a prompt that takes Qwen3.8-27B 35 minutes. I then wired it into kiapi, my home inference server, and sent the fix it needed upstream to mlx-vlm.

What is Qwen3.8-Flash-Next?

Qwen3.8-Flash-Next is a model whose weights Alibaba's Qwen team released in August 2026. It combines a 125B-parameter MoE (Mixture of Experts: a model made of many small expert networks, of which only a few are used for each token) with a 51B-parameter n-gram embedding table, and activates about 6B parameters per token. It accepts text, images, and video, with a 262K-token context that can be extended to 1M.

In Qwen's published comparison it beats the dense Qwen3.8-27B on almost every benchmark. The gap is largest on agentic evaluations, for example 58.7 versus 42.2 on DeepSWE 1.1. These are Qwen's numbers; I did not reproduce them.

Three models appear in this article.

Model	What it is	On disk	Role
mlx-community/Qwen3.8-Flash-Next-4bit	4-bit Flash-Next (the full build)	111.5 GB	The one I want
sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit	4-bit build with experts pruned from 512 to 288 per layer	73.5 GB	The smaller candidate
mlx-community/Qwen3.8-27B-4bit	4-bit dense 27B	16.1 GB	The current baseline

REAP runs calibration data through the model, measures how much each expert contributes, and removes the least useful ones. REAP-288 was calibrated only on agentic-coding traffic (about 686K tokens), and its model card reports HumanEval alone (93.9 → 91.5).

Licenses

Item	License
Qwen3.8-Flash-Next (including derivatives such as REAP-288)	Qwen Community License 1.0
Qwen3.8-27B	Apache-2.0
mlx-vlm	MIT

The Qwen Community License 1.0 allows internal use freely, but a commercial Model-as-a-Service or AI work-assistant business needs a separate license from Qwen.

What this article checks

1. How much does the expert-pruned REAP-288 lose on Japanese, long contexts, and agentic work?
2. Does the full 4-bit build run on a 128 GB Mac at all?
3. If not, can it be made to fit without losing quality?
4. What happens when it goes into a real server?

The evaluation is split across two labs:

• 2026/09/24/qwen38-flash-next-reap-eval: REAP-288 vs. the full build vs. 27B
• 2026/09/24/qwen38-flash-next-ple-mmap: the full build again, with its n-gram table memory-mapped

Reproducing it

You need a 128 GB Apple Silicon Mac, mise, uv, and Git. The three models download about 200 GB into the Hugging Face cache.

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .mise/tasks \
  2026/09/24/qwen38-flash-next-reap-eval \
  2026/09/24/qwen38-flash-next-ple-mmap
mise trust . \
  && mise trust 2026/09/24/qwen38-flash-next-reap-eval \
  && mise trust 2026/09/24/qwen38-flash-next-ple-mmap
mise -C 2026/09/24/qwen38-flash-next-reap-eval run

The default run only executes unit tests that load no model. Run the evaluations individually, with nothing else using the GPU.

Command	What it does
mise -C 2026/09/24/qwen38-flash-next-reap-eval run eval dense27b	All suites on Qwen3.8-27B
mise -C 2026/09/24/qwen38-flash-next-reap-eval run eval reap288	All suites on REAP-288
mise -C 2026/09/24/qwen38-flash-next-ple-mmap run prepare	Builds a view of the full build with the memory-map manifest (weights are hard-linked, not copied)
mise -C 2026/09/24/qwen38-flash-next-ple-mmap run eval full4bit-plemmap needle,agent,ja	All suites on the memory-mapped full build

prepare uses hard links, so check the repository out on the same volume as your Hugging Face cache.

The suites

Suite	Contents
32 Japanese items	16 knowledge, 8 instruction following, 6 reading, 2 keigo (honorifics), scored automatically by substring, regex, JSON, and so on
Long-context retrieval	A synthetic Japanese warehouse log hiding three passwords (at 10/50/90% depth) and three decoys; the model must return all three. 32K / 128K / 240K tokens
Agentic work	Four small Python repositories with bugs, fixed using four tools (list, read, write, run tests). Success means held-out tests pass and the test files are untouched

Everything runs at temperature 0, with thinking disabled and a 2,048-token prefill step.

1. REAP-288 lost its knowledge and its Japanese

Model	Japanese (32)	Knowledge	Instruction	Reading	Keigo	Agentic
Flash-Next, full build	32	16/16	8/8	6/6	2/2	4/4
Qwen3.8-27B	30	15/16	7/8	6/6	2/2	4/4
REAP-288	12	1/16	6/8	5/6	0/2	3/4

REAP-288 got one of the 16 knowledge questions right by the automatic check. (Its answer "the lower house" to which house of the Diet considers the budget first was scored wrong because the check wanted the name 衆議院, but it means the same thing, so two is fairer.) Some of its broken answers:

Question	REAP-288's answer
Capital of Mie Prefecture?	"Does not exist"
City where Kenroku-en is?	"Japan"
Author of the poem "Shungyō"?	(in Chinese) "No such poem exists"

This is not only about Japanese. Asked in English for the longest river in Japan, it invented a "Jangwan River". Asked for the capital of Japan, it replied in Chinese that Japan has no capital. Yet it writes a Python function without trouble. The full build answered every one of these correctly in the same runtime, so the cause is the pruning, not the runtime. My guess is that knowledge and Japanese lived in experts that coding traffic alone rated as unimportant.

Even the automatic score of 12 is generous. The three-bullet answer satisfies the format but is written in Chinese, Mount Fuji is described as "the longest-lived Linux distro in the world", and the plain-style (dearu) item came back as a string of broken, repeating particles.

The damage reached code too. The one agentic task it failed asked for a function that turns 1時間30分 ("1 hour 30 minutes") into seconds. REAP-288 rewrote the Japanese unit 時間 as the simplified Chinese 时间, and even changed the examples in the docstring's specification to match. The tests kept failing, and it used up all 24 steps without spotting the one-character difference.

Long-context retrieval, on the other hand, did not degrade. At 32K, 128K, and 240K it returned all three passwords and ignored every decoy. REAP removes only experts; attention and the long-context machinery stay intact, so this is what the architecture predicts.

2. The full 4-bit build ran out of memory

The full build was the best on quality, but its MLX peak was already 111.5 GB after loading.

• Starting the 32K retrieval with default settings stopped with a Metal out-of-memory error (kIOGPUCommandBufferCallbackErrorOutOfMemory)
• A 256-token prefill step got 32K through, then 128K failed the same way
• Agentic work only ran with a 256-token step and a 1 GB prefix cache, and then the cache reused nothing between turns

Raising macOS's GPU wired-memory limit (iogpu.wired_limit_mb) is another option, but it changes a system setting, so I did not try it.

3. With the n-gram table memory-mapped, everything fit

Flash-Next's 51B-parameter n-gram embedding table (PLE) is read only a few rows per token. mlx-vlm's qwen4_exp implementation already has a way to keep that table out of MLX parameters and read just the needed rows from a memory map of the safetensors files (qwen4_exp/ple_storage.py). A comment in the code notes that loading it normally wires about 30 GiB of cold lookup data.

So why was the full build 111.5 GB? Because the mlx-community conversion ships without ple-store.json, the manifest that turns this on. REAP-288 was only 41.5 GB precisely because it does ship one.

mlx-vlm also has the function that builds the manifest, prepare_external_ple_model. It:

• writes ple-store.json with the byte ranges of the PLE tensors inside the original safetensors files,
• adds text_config.ple_storage to config.json,
• hard-links every non-PLE weight file.

Nothing is copied or re-quantized, so the weights are bit-identical, and no extra disk space is used.

	Resident PLE (before)	Memory-mapped PLE
Peak after loading	111.5 GB	79.5 GB
Retrieval at 32K / 128K / 240K	Out of memory	3/3, 3/3, 3/3 (peaks 83.6 / 89.9 / 96.0 GB)
Agentic work, default settings	Does not run	4/4 (prefix cache works across turns)
Japanese (32)	32	32
Decode speed (median over the Japanese suite)	47.1 tok/s	40.2 tok/s

The saving is 32.0 GB, exactly the size of the PLE tensors. Decoding got about 15% slower, but it is still faster than the 27B (34.3 tok/s). Swap did not grow during the run.

4. On long contexts it is nearly 5× faster than the 27B

As the first chart shows, Flash-Next's prefill stays at 530–580 tok/s regardless of length, while the 27B slows from 217 to 152 to 114 tok/s.

Length	Flash-Next (memory-mapped)	Qwen3.8-27B
32K	65.5 s	153.0 s
128K	228.7 s	865.5 s
240K	447.0 s (7.5 min)	2,114.9 s (35.2 min)

These are totals until the answer (about 50 tokens) is finished; nearly all of it is reading the prompt.

I see two reasons, though I have not isolated them. First, the 27B runs all 27B parameters per token while Flash-Next runs about 6B, a 4.5× difference. Second, Flash-Next's Qwen Sparse Attention looks only at the most relevant past tokens, so per-token cost barely grows with context length. Token-by-token decoding, by contrast, differs by only about 1.2×. Flash-Next shines when you hand it a long input.

Wiring it into kiapi

kiapi is my open-source project that turns an Apple Silicon Mac into an API server for LLM agents. Flash-Next is now served there as qwen3.8-flash-next.

• Generation reuses the 27B handler. The chat template and the Hermes/XML tool-call format are the same
• Loading builds the memory-mapped view automatically. Weights are symlinks into the original snapshot; nothing is copied
• Loading raises the open-file limit. The mapped PLE keeps 128 shards open, and macOS's default of 256 fails with OSError: [Errno 24] Too many open files

I also made model required for chat. Once loaded, Flash-Next keeps about 74 GiB resident and does not accept audio or video, while the Omni model does. With inputs and memory that different, an implicit default can silently evict other models or reject media. Instead, Flash-Next gets the short alias vlm.

curl http://HOST:8500/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"vlm","messages":[{"role":"user","content":"What is the capital of Japan? One word."}]}'

One quirk showed up after integration: when tools are offered, it tends to call them even for a plain "Hello" (it asked for the weather). This happened at temperature 0 too, so if you do not want tool calls, say so in the system message.

Sending the image-cache fix upstream

kiapi reuses the cached prefix of a conversation. For the 27B, appending an image mid-conversation already reuses the unchanged history, through my open mlx-vlm PR (#2309). Flash-Next, however, re-read everything whenever an image was added (0 of 1,333 tokens reused in production).

Reading Flash-Next's model code, I found that it inherits the 27B's image processing and position computation; the only thing excluding it was a one-line check for the qwen3_5 model type. I extended it to qwen4_exp and folded the change into the same PR. Flash-Next support should be a high priority for the project, and bundling it seemed more likely to help the PR land.

Action (Flash-Next)	Without cache	With reuse
Append one image to an image conversation	4.77 s	0.50 s
Append two images to an image conversation	4.88 s	0.71 s

Times are to the first token. Both reused the first 2,897 tokens and encoded only the new images. In production kiapi, reuse when adding an image went from 0/1,333 to 1,047/1,333 tokens.

One thing surprised me. Cache correctness is checked by comparing the first-token probability distribution with an uncached run (KL divergence). On the 27B this is below 0.001, but on Flash-Next it came out at 0.104, above the 0.05 threshold I had set for the 27B.

To find out why, I compared uncached runs that differed only in prefill step size.

Comparison	Symmetric KL
Uncached, step 2048 vs. 512	0.016
Uncached, step 2048 vs. 384	0.064
Uncached, step 512 vs. 384	0.096
Uncached step 512 vs. cached image reuse	0.129
Uncached step 256 vs. cached image reuse	0.007

Without any cache at all, Flash-Next's first-token distribution moves this much just from how the computation is chunked. The top token and the generated text were the same in every case, and the cached result sits inside the spread of uncached runs. So the PR's verification script now runs uncached passes at steps 2048, 512, and 256 and requires KL below 0.05 against any one of them. For the 27B every step stays below 0.001, so its effective threshold is unchanged.

What it is good for, and what it is not

The full Flash-Next build suits long documents and agentic work. It reads 240K tokens in 7.5 minutes and solved all four tool-driven repository fixes with default settings. The longer the input, the bigger its lead over the 27B.

REAP-288 is not usable for Japanese or general knowledge. Code generation and long-context retrieval survived, so it may be an option for English-centric coding help. But since it rewrites Japanese inside code into Chinese, it is risky for repositories with Japanese comments or strings.

Memory-wise, Flash-Next takes over the machine. It peaks at 96 GB at 240K tokens, so on a 128 GB Mac it cannot sit alongside other large models. In kiapi I swap it in when needed; loading takes about 12 seconds when the OS page cache is warm.

Limitations and open questions

• One machine (Mac Studio M4 Max, 128 GB), one run per configuration, temperature 0, thinking disabled
• The agentic suite has four tasks; it shows failure modes, not success rates
• Japanese scoring is automatic and format-sensitive. One check (a keigo item that did not anticipate the word 欠席, "absence") was widened after the runs, and every model was rescored from its stored outputs
• The retrieval test uses synthetic data and does not measure reasoning over long contexts
• I did not separate whether memory-mapped PLE reads came from the page cache or the SSD; decoding may be slower with a cold cache, for example right after boot
• Only one REAP build (288 experts, one calibration set) was tested
• Image input was not evaluated beyond the cache checks
• mlx-vlm's per-expert SSD offload (mlx_vlm/moe_offload.py) was not tried

Afterthoughts

The biggest lesson: before hunting for a smaller build, count what is actually using the memory. Of the full build's 111.5 GB, 32 GB was a table that each token reads a few rows from. Reading that table from disk was enough to make it fit, without giving up quality or context length. The mechanism was already in mlx-vlm; all that was missing was one manifest file.

REAP-288's failure honestly shocked me. HumanEval says 91.5%, almost unchanged, yet the capital of Mie Prefecture "does not exist". A model card's benchmark only tells you whether what was calibrated survived. Checking a dozen or so questions from your own use case reveals this immediately, so please do that before picking a pruned build.

Watching a quantized MoE wobble purely from chunking was a fun discovery too. When a number crosses a threshold, measure how much uncached runs disagree with each other before calling it a bug. That habit should carry over to validating caches for other models.

Handing a long document to a model and asking questions, or letting an agent loose on a big repository, now feels realistic on a Mac at home. My Mac Studio suddenly feels like a rather dependable partner.