I ran our own 4-bit build of the official Qwen3.8-Flash-Next weights on one Mac Studio (M5 Max, 128 GB), in our own MLX engine, and it scored 0.6932 on our Gauntlet agentic coding benchmark, tier sb-7.2. That puts it 15th of 51 on the board, between Grok 4.7 and Gemini 3.8 Flash.
Qwen3.8 Flash, Qwen's hosted version on OpenRouter, sits at 0.6846 on the same board. So our best run is 0.0086 ahead as the board stands today. That gap is smaller than the spread between our own two runs. Our other run of the same build scored 0.6606, and the cloud run was graded a month earlier, by an older scorer, at the provider's default reasoning effort. I haven't verified which effort our own run actually got.
We trained a DWQ quantization against Qwen's BF16 weights. We added three engine switches for long-context speed and held a 524,288-token window on one machine. We fixed our grader, and that fix changed our score. We also fixed our tool-call parser, but only after the scored run, so it changed no score. A fix for a cold-prefill slowdown we caused ourselves isn't released yet. We also broke 38 tool calls in one run and mis-graded our best one, and both stories are below.
Our desktop app and the engine that ran this are being prepared for a relaunch as LeanZero Agent. Nothing has shipped under that name yet, so I call them our desktop app and our engine here. Engine releases are numbered lz.NN: lz.27 ran the scored run, and lz.28 is the release after it.
Local vs cloud on the same model: Qwen3.8-Flash-Next on a Mac against OpenRouter
Gauntlet sb-7.2 gives an agent 150 model calls to build a payments console: two Python services, vendor webhooks, a live event stream and a 3D view of the payments, and our scorer then runs the app in a real browser and grades it on 13 tiers of checks.
Qwen3.8 Flash is what Qwen serves, and Qwen3.8-Flash-Next is the open-weights model it's based on. Qwen's card calls the hosted one "the official version based on Qwen3.8-Flash-Next with more production features, e.g., 1M context length by default, official built-in tools". OpenRouter maps its qwen/qwen3.8-flash entry to Qwen/Qwen3.8-Flash-Next, so it's the same model family behind Qwen's own serving stack.
| Run 2, our DWQ build (best) | Run 1, our DWQ build (other) | Qwen3.8 Flash, cloud | |
|---|---|---|---|
| Where it ran | one Mac Studio, M5 Max, 128 GB | same Mac, engine and app version | OpenRouter |
| Date | 3-4 November 2026 | 3 November 2026 | 5 October 2026 |
| Final score | 0.6932 | 0.6606 (not published) | 0.6846 |
| Earned credit before the ceiling | 0.8841 | 0.832 | 0.7126 |
| Admission ceiling | 0.699 | 0.669 | 0.699 |
| Model calls | 138 of 150 | 111 of 150 | 147 |
| Build time | 137 min 57 s | 1 h 32 min | 82 min 11 s |
| Wrap-up nudge | yes | yes | no |
| Inference cost | electricity | electricity | $0.60 |
| Graded by | scorer in app 3.0.154 | scorer in app 3.0.151 | scorer before app 3.0.148 |
I number the runs in the order they ran. Run 2 is the one we published, and run 1 is the other scored run of the same build.
I read the scores and ranks from the live Gauntlet leaderboard on 4 November, and they move. That cloud number is the one from our Qwen3.8 Flash vs Max Prime write-up in October.
Gauntlet's final score is capped. A run earns credit across the tiers, and an admission ceiling caps it whenever certain hard checks fail, most of them in the 3D view. All three of these runs were capped there. Under the cap the final score is min(raw, cap - 0.05 × (1 - raw)), so earned credit counts at a twentieth of its weight. A 0.17 gap in earned credit becomes 0.0086 in the final score, and a single 3D check can swing a run by more than that.
So the 0.0086 isn't proven. Our two scored runs of the same build are 0.0326 apart. That's almost four times the lead. Each one lost points on a different 3D check that caps the score, and nothing about the build changed between them. Our own final number rose from 0.5909 to 0.6932 when we fixed our grader, which I explain further down. With two unrelated noise rows set back, the lead is 0.0078. The cloud run was graded on 5 October by an older scorer, and we didn't keep its files, so nobody can re-grade it. Its run page also notes that the old scorer may have under-read its notifications in two workflow checks. Five failed 3D checks set its cap, and scorer fixes from 2 November changed two of those (the stream and coast checks). If just one of the five passed on a re-score, its cap would rise to 0.709 and its score to about 0.695, above ours.
Our setups differed in two more ways. The first is the wrap-up nudge, which tells the model to start no new work as its call budget runs low: both our runs had it, and the cloud run didn't. Run 2 stopped at 138 of 150 calls, and I haven't checked whether the nudge fired. The second is the reasoning effort. The benchmark asks for medium, but our desktop app only started sending that setting to Qwen models on OpenRouter on 16 October, so the cloud run went out at the provider's default. Our run page says the setting doesn't reach a local model. I've since seen that the app does send medium to our engine. Nobody has verified that the engine applies it, though, so treat our score as not taken at a pinned effort either.
What I'm sure of is narrower. On one 128 GB Mac, with our own quant and engine, the open model now scores where the hosted one does on this task. We measured how wide run-to-run spread is on our other benchmark, Forge 2.0, in our 32-models-built-twice post, and it came to about 0.15 either way per build.
How we built a 4-bit DWQ quant of Qwen3.8-Flash-Next for MLX
Qwen3.8-Flash-Next is big. Its card lists "125B with 6B activated, plus 51B n-gram embedding and 4B MTP". It's a mixture-of-experts model with 512 experts (10 routed plus 1 shared per token), 48 layers and a vision encoder. In BF16 it's far beyond a 128 GB Mac. At 4 bits our build is 100.3 GiB on disk, and it's on Hugging Face under the Qwen Community License the base model carries.
I started from our own earlier 4-bit build. It follows oMLX's oQ4e recipe for how many bits each layer gets (oMLX is Apache-2.0, and our model card credits it). oMLX describes oQ as mixed precision that "measures how much each layer suffers from quantization, then spends extra bits where they help most". Our build kept that bit map and added our own importance-weighted rounding.
DWQ came next. MLX's own version, documented in mlx-lm, "fine-tunes non-quantized parameters (including quantization scales and biases) using the non-quantized model as a teacher". A 4-bit weight is stored as a small integer, plus a scale and an offset for every group of 64 weights. DWQ leaves the integers alone. It nudges only the scales and offsets, by gradient descent, until the quantized model's output looks more like the original's. The original pull request showed it on Llama 3.2 1B, as average accuracy over eight standard tasks: 47.95 for fp16, 46.21 for plain 4-bit and 47.57 with DWQ.
We couldn't run mlx-lm's version as it is, because it needs the full-precision teacher and the student side by side, and 125B parameters in BF16 don't fit next to anything on one Mac. So we wrote a block-wise variant. It streams Qwen's official BF16 weights in one layer at a time and retrains only the routed experts' 4-bit scales and biases, so that each layer's output comes to match the BF16 layer's output on 384 calibration sequences of 1,024 tokens. Everything before the expert block stays frozen. Training took 5 hours 22 minutes, with Adam at a learning rate of 1e-5 for 3 epochs and fp32 master copies, and with layers 0 to 17 fed the student's own input while layers 18 to 47 were fed the teacher's. A per-layer guard kept the old rounding wherever retraining didn't help: 35 of the 48 layers took the new scales and 13 kept the original.
Then we measured it on text it had never seen. Our first number, -13.3%, was wrong. An independent check found that the 64 held-out rows had also been used to pick the training settings, and 21.5% of their positions overlapped training text. So we built a clean test: 64 rows of 1,024 tokens, 65,472 scored tokens. Half is tool-call text (SWE-agent and glaive traces) and half plain text (fineweb-edu and wikitext-2), with zero 32-token overlap with anything used in training.
| Text | KL from BF16, earlier build | KL from BF16, DWQ | Change (95% interval) | Top-1 agreement with BF16 |
|---|---|---|---|---|
| Tool-call text | 0.1122 | 0.0963 | -14.1% (-21.5% to -6.0%) | 92.87% to 93.40% |
| Plain text | 0.0380 | 0.0380 | -0.1% (-2.8% to +2.4%) | 92.20% to 92.38% |
| All | 0.0751 | 0.0671 | -10.6% (-17.0% to -4.1%) | 92.53% to 92.89% |
KL divergence measures how far the quantized model's next-token probabilities sit from the BF16 model's, in nats per token. Lower is closer. I trust it more than perplexity for quants. The Accuracy is Not All You Need paper proposes KL and "flips" because two models can match on accuracy and still disagree a lot underneath.
Without the single row with the biggest gain, the drop is 7.8% overall and 10.8% on tool text, so one row carries part of the gain. And on the tool rows, DWQ's perplexity moved slightly away from the teacher's (3.796 against 3.706 for the earlier build and 3.692 for BF16) while its KL went down. I don't know why yet. The model card says so too.
A byte-level check of all 21 weight shards found 208 scale and bias tensors changed and 0 packed codes changed. Live, the DWQ build accepted 90.6% of the draft tokens from the model's built-in MTP head (454 of 501), against 85% for the earlier build. MTP is one extra layer that guesses the next tokens so the engine can check several in one pass. The more guesses it gets right, the fewer decoding steps it needs.
Running Qwen3.8-Flash-Next on a Mac Studio M5 Max with 128GB
I ran it on a Mac Studio with the M5 Max. Apple only offers 128 GB with the 40-core GPU version of that chip, rated at 614GB/s of memory bandwidth. The weights take 100.3 GiB. So 128 GB is the smallest Mac this build fits on. A 64 or 96 GB Mac can't hold it.
| Setting | Value |
|---|---|
| Model | Mihai-LeanZero/Qwen3.8-Flash-Next-oQ4e-DWQ-LeanZero (4-bit, 100.3 GiB) |
| Engine | our MLX engine, release lz.27, with the three switches below on |
| Context window | 524,288 tokens, YaRN factor 4.0 |
| Thinking | on, effort medium requested (not verified as applied) |
| GPU wired memory limit | 120,832 MB |
| Engine memory peak in the run | 96.99 GiB (one guard sample read 101.1 GiB) |
| Swap | 743.81 MB at the start, 735.81 MB at the end |
| Memory-guard refusals | 0 |
I ran the model inside our desktop app's benchmark view, the same way a cloud model runs there. The app talks to the engine over a local OpenAI-compatible endpoint. Nothing else heavy was meant to run on that Mac during the run, and one thing did anyway (it's in the list of what went wrong).
If you want to try the build, start with the Hugging Face card. A plain mlx-lm install won't load it yet, because Qwen3.8-Flash-Next support is still an open pull request in mlx-lm. The card points to the engine build we use, which already has the tool-call parser fix described below. On an earlier engine release, our earlier 4-bit build decoded at about 48 tokens a second at around 8k tokens on this Mac. I haven't measured the DWQ build at short context.
Qwen3.8-Flash-Next context length: 524,288 tokens on one Mac
Qwen3.8-Flash-Next reads 262,144 tokens natively, and Qwen's card says it's "extensible up to 1,000,000 tokens" with YaRN, which stretches the rotary position encoding, and gives a factor of 4.0 for it. We set a 524,288-token window with factor 4.0.
Before a model gets a benchmark slot, it has to prove the window it claims. For this build that meant a needle test: a 522,913-token prompt with one fact hidden in it, read cold. It found the needle. It also wrote an accurate paragraph about the codebase the prompt held, after 970.8 seconds to the first token, and then decoded at 28.0 tokens a second. Metal memory peaked at 93.0 GiB, and swap stayed flat at 741.56 MB.
Run 2 didn't need all of it. It grew to 315,143 tokens and finished without a single compaction, which is the app summarising old turns to free room.
I'd do one thing differently. Qwen's card says that if your typical context is around 524,288 tokens, "it would be better to set factor as 2.0", and that static YaRN can hurt shorter texts. We ran 4.0. I never tested 2.0, and it's on the list.
Qwen3.8-Flash-Next tokens per second on a Mac: split-K decode, QSA and block-sparse prefill
Once the weights fit, long context on a Mac is mostly a matter of waiting. Three engine switches were on in the scored run to cut that wait, and each exists because of how this model is built.
Qwen3.8-Flash-Next is a hybrid. Its card gives the layout as "12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))". That's three linear-attention layers, which don't grow with context, for every attention layer that does. The attention layer is QSA, Qwen Sparse Attention. Qwen's tech report says it "scores context at micro-block granularity with a compressed lightweight indexer". Each query reads up to 512 blocks (2,048 tokens) instead of the whole history, following the sparse route of DeepSeek's DSA.
Split-K decode. For this model's head size, MLX reads the keys once per query head, and our kernel reads them once per group of heads that share them. To write one token, each attention head reads every key and value in the context. MLX already splits that work into slices for long sequences (its two-pass sdpa_vector_2pass kernel), but its faster path for heads that share keys only covers some head sizes. Flash-Next's attention layers have 24 query heads over 2 key-value heads at a head dimension of 256, which falls outside that path. Our kernel reads each key-value head once for the 12 query heads that share it. It splits the context across GPU threadgroups and merges the partial results at the end, following the idea behind Flash-Decoding. An open mlx-lm pull request does the same for a different model, and ours generalises it.
The QSA pin lift. The sparse-attention code we started from enabled QSA's indexed route for one chip and MLX 0.32.2 only. On our Mac with MLX 0.32.4 it declined ("unqualified MLX version") and fell back to dense attention over the whole cache, the very thing QSA was built to avoid. Lifting the pin lets the sparse route run. We checked that the indexed result matched the gathered one: 0 differing elements in 10 configurations.
Block-sparse prefill. While reading a prompt in, the engine computes QSA attention only over the key blocks the indexer picked. MInference and XAttention do block-sparse prefill on models that weren't trained for it. QSA is trained into this model, so the blocks are the model's own choice.
I took the speed numbers from a live A/B on the same Mac with our earlier 4-bit build of this model, which ran on an earlier engine release, lz.24, on 2 November. I haven't repeated it on lz.27, the engine of the scored run.
| Decode, tokens per second | 8k context | 200k context | 400k context |
|---|---|---|---|
| All switches off | 47.6 | 25.8 | 11.5 |
| QSA route only | 48.3 | 28.5 | 29.9 |
| Split-K only | 48.4 | 32.3 | 16.8 |
| Both | 48.5 | 34.5 | 29.6 |
I'd read the 400k column with care. It's one 256-token sample per arm, the second of two. The first, taken right after the cold read, was much lower on the off and QSA-only arms (3.1 and 2.3) and slightly lower on the other two. The 200k column is the mean of two.
At 8k nothing changes. There's hardly any context to read. At 400k, though, the model writes 2.6 times faster with both switches on. Block-sparse prefill helped less, but it still helped. Adding 6,000 new tokens at 400k of context went from 86 to 120 tokens a second with block-sparse prefill on, and at 200k from 320 to 545. The cold 399,103-token read with the switches off managed 68.5 tokens a second and took 97 minutes.
Each switch also had to give the same answers. We checked it against a float64 reference on real tensors from a 237,904-token context. Block-sparse prefill came in at a relative error of 0.001670, the bf16 rounding floor, with 99.98% of values correctly rounded. The dense path came in at 0.004009, 2.4 times the floor, and block-sparse was closer in all 57 comparisons. Split-K matched the floor on 403 of 403 live samples. The pin lift was the exception. On real single-token steps it was very slightly less accurate than dense (dense was closer in 379 of 384 samples, 3.5% against 1% above the floor). In a synthetic three-token test it was much more accurate. I'd call that inconclusive. We kept it.
Two fixes: the tool-call parser and the grader
Run 2 is the published one. Around it, a grader fix changed the score, and an engine fix came after it and changed nothing on the board.
Run 1 lost 38 tool calls to one extra argument
At message 74 of run 1, the model called the shell tool with a parameter the tool doesn't declare. The schema says timeout_secs; the model wrote timeout:
<parameter=command>
... | head -c 800
</parameter>
<parameter=timeout>
40
Qwen's chat template writes tool calls in an XML style (<tool_call>, <function=...>, <parameter=...>). vLLM and SGLang parse that format with a parser called qwen3_coder. Our engine's parser, under a rule we added in August, only split parameters at names the tool declares, so it folded </parameter>, <parameter=timeout> and 40 into the shell command. Bash ran the real lines and then failed on the junk at the end with exit code 2. My guess is the model took that as its own command failing. It kept the habit. 38 of run 1's 53 shell calls carried it, starting at message 74.
Run 1 scored 0.6606. Its ceiling came from somewhere else. The model drew the hollow currency frame in the 3D view as four thin posts instead of rails, which capped it at 0.669. That was a real model mistake. We kept it as scored. Run 2 used timeout_secs correctly in all 42 of its shell calls and had zero leaks.
We shipped the fix in engine release lz.28. An undeclared parameter now counts as its own key when it starts where the template would start a new parameter, and the engine logs each correction. It doesn't map timeout onto timeout_secs, because guessing what a model meant is how you get the next bug. We replayed recorded calls through it. All 38 broken calls were repaired in all three places the engine parses tool calls, all 268 clean calls stayed byte-identical, and the old release reproduced all 38 failures as a control. The engine never logged the model's raw text, though, so the replay re-rendered the calls from the session, and I'd put the byte-level match at medium confidence.
This fix played no part in the 0.6932. Run 2 ran on lz.27 and didn't need it.
Run 2 scored 0.5909 because of our grader
Run 2 finished at 01:50 with 0.5909. An older third-party build of the same model had 0.6873 on the board at the time, and we only publish a new build of a model when it scores above the one already there, so we'd have published nothing.
So was the model wrong, or the grader? It was the grader. I didn't find that myself; the independent check of that verdict did. One check, s_visible_surface, had capped the score at 0.599. It samples 120 points of the 3D view in a screenshot and needs 114 to show the right colours. It matched 113. Reading the same frame straight from the GPU's framebuffer matched 120 of 120. The model had laid its canvas out at a fractional size, 1209.21875 pixels wide at x = 35.39, because of rem-based padding, so the browser resampled every pixel of a correct image and 7 sample points fell on blended pixels. In a label-free patch of the 3D view, run 1's screenshot showed 17 distinct colours, which is exact flat shading, and run 2's showed 4,419.
We confirmed it on a copy of the build. One CSS rule that snapped the canvas to an integer box took the score from 0.5908 to 0.6933 and the check to 120 of 120. Then we fixed the grader and left the build alone. The new check grades a sample point only where the screenshot pixel draws entirely from texels that show the expected colour, and the text check counts anti-aliased ink. It shipped in app version 3.0.154, and when we re-scored the run in the app on the kept build it came out at 0.6932. Two rows moved because of the fix (s_visible_surface 0 to 1, v_presentation_text 0.75 to 1) and two from ordinary scoring-to-scoring noise. Set those two back and the run scores 0.6924.
I built that fix while staring at this one run, so I wanted it checked independently:
- We re-scored every other Gauntlet run that still had its build files, with both graders. The four others moved by at most 0.0003 (for example 0.6606 to 0.6605 and 0.5873 to 0.5876).
- A triage of all 54 sb-7.2 verdicts found the "insufficient visible text pixels" symptom in 14 of them, but only this run had a false zero on the visible-surface check.
- Three fresh review rounds and 16 of 16 control fixtures passed, including negative ones (covered, unlit and veiled canvases) that must still fail. The old grader fails the fractional-canvas fixture.
- A separate agent, given only the data and none of my conclusions, recomputed the in-app re-score and found it stands: 0.6932, or 0.6924 with the two noise rows set back.
We can't apply the fix to the cloud run. We didn't keep its files. I think that asymmetry, more than anything, is why the 0.0086 can't be called a lead.
A cold-prefill slowdown we caused, and a fix that isn't released yet
Before the runs, I saw that the cold 522k-token proof was slower on lz.27 than on the older engine, lz.24. I logged it as cause unknown. After the runs we found the cause. It was ours. A change in engine release lz.27 sized the RAM budget for the prompt cache to the live conversation. That pushed MLX's allocator cache limit down to its floor, 2.46 GB (3.55 GB in the 522k proof). During a long prompt read the engine then freed and re-allocated a working buffer of 3 to 17 GB on every chunk.
Our fix re-plans the memory budget when a request is admitted. It never lets the allocator cache drop below the working buffer the plan reserves, capped at the room the machine has free. I started the engine by hand with the app's exact launch arguments, and a cold 521,969-token prompt took 880.7 seconds (592.7 tokens a second), against 968.6 seconds for the same build on lz.27. With our earlier build, lz.27 took 1,119.9 seconds and lz.24 took 834.7. Metal peaked at 103.18 GB and swap stayed flat, and a 53-turn session afterwards grew to 320,786 tokens, compacted, and still recalled a test codename 7 times out of 7.
At 200k tokens, one cold request per engine, the fix read 1,040 tokens a second against 694 on lz.28 and 806 on lz.26, while at 16k and 64k tokens it changes nothing that we could measure against the noise between one engine start and the next. 26 of 26 greedy outputs were identical with and without it, because no numerics were touched. I'd put the size of the gain at medium confidence: one request per cell, and at short context results vary by about 30% from one engine start to the next, for reasons we haven't found.
It isn't released yet. An independent review found one more defect (the cache was counted twice when the Mac was busy). That's fixed. The live proof is being re-run on the corrected code before it ships as the next engine release. It played no part in the 0.6932 either.
What went wrong, and what it cost
- The parameter leak broke 38 tool calls of run 1, from a parser rule that had sat in the engine since August.
- The grader capped the scored run at 0.599 until the fix, and a 2 hour 17 minute build would have gone in the bin if the independent check hadn't read the framebuffer.
- Run 3, on the fixed parser, started at 03:08 and was cancelled at 03:34 after 32 calls, before anything was scored. We lost 26 minutes of a Mac kept free for the run.
- A stray search process from another of our own jobs ran for 54 minutes on the Mac kept free for run 2. It used 5.2 GB and shrank the engine's memory ceiling from 108 to 102.4 GB until we killed it. Run 2 had no refusals. I'd call that luck.
- The first DWQ number (-13.3%) came from a test set that had also tuned the training, so I only quote the clean test.
- The DWQ training job was killed once by our own memory guard at layer 19 and restarted.
- The speed A/B ran about 4 hours against a 45-minute budget, mostly because a 400k cold read with everything off takes 97 minutes.
- We deleted the build files of both local runs after publishing, the same mistake that already stops anyone re-grading the cloud run. Only three sb-7.2 builds anywhere still have their files. From now on, a run we compare against keeps its files.
- The scored run tripped our run audit once. The model tried to list the benchmark's build folder, the sandbox refused, and it read nothing. We published the run with that on the record.
What the result says
Earned credit is further apart than the final score: 0.8841 against 0.7126, and a core score of 0.9316 against 0.8425 (the core is the main checks, 88% of earned credit). The cloud run also took a ×0.8857 critical penalty, from a workflow check whose notification parts, according to its own run page, the old scorer may have under-read. Ours was graded by the fixed scorer and theirs couldn't be, so earned credit doesn't settle which is better either.
So I'll stand behind the engine claim. On one Mac Studio, with our own quant and our own engine, the open Qwen3.8-Flash-Next now builds this app about as well as Qwen's hosted Qwen3.8 Flash does. Last week our two-Mac MiMo run finished 0.1265 behind its cloud version on today's board (0.6644 against 0.7909; the post itself, written before a re-score, says 0.6605). This one closed the gap on one machine.
If you have a 128 GB Mac and want to run this model for agent work, check that your engine parses the tool calls the model writes, including the parameters it invents. And before you trust a benchmark score, ours included, find out what capped it.
Next for us: the release of the cold-prefill fix, a run with YaRN factor 2.0, and a third scored run of this build, kept on disk this time. If you've run Qwen3.8-Flash-Next locally and seen different numbers, I'd like to hear about it.
Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.
Top comments (0)