One RTX 5090, 128 GB of RAM, and four open models from 125B to 748B parameters. Over the past weeks I got each of them running on the same workstation, on three different runtimes. I measured all four in one session, one after the other, with the same benchmarks. This article puts the numbers side by side, so you can see what each setup is good for.
Update, October 22, 2026: I measured everything again after updating Project Maya from 1.1.17 to 1.1.21 and retuning Ornith. The first version of this article, measured on October 21, had Ornith at 33.1 tokens per second. It drafted with its built-in MTP head then, not without a draft model as I wrote. Without drafting and with more experts in VRAM, it now does 46.4. Maya 1.1.21 made DeepSeek read long prompts about 3.7 times faster. The tables below replace the old ones.
Update, October 29, 2026: A reader, Mihai, suggested in the comments that Ornith's stray tokens came from sampling, not from the quant. He was right. TabbyAPI without a sampler preset samples at temperature 1.0 with no top-k, top-p or min-p cutoff. With the settings from Ornith's model card, Ornith scored 108 out of 108 on our tasks instead of 83 (107 in a second check with those settings as the server default), with no stray tokens and no loss of speed. See Ornith and sampling below.
Every number below was measured on our own machine on the night of October 21 to 22, 2026, unless I say otherwise.
The four setups
| Model | Parameters | Runtime | Quant | Context |
|---|---|---|---|---|
| Qwen3.8-Flash-Next | 125B, ~6B active | Strata | Unsloth UD-Q5_K_XL GGUF | 512K |
| Ornith-1.5-122B | 125B | TabbyAPI + ExLlamaV3 | Mia-AiLab EXL3 EAQ v1, 4 bpw | 256K |
| GLM-5.3-Flash | 321B, ~18B active | Project Maya | turboderp EXL3 3.05 bpw | 512K |
| DeepSeek-V4.1-Flash-1017 | 748B | Project Maya (maya-ds4) |
Maya-S (experts in 2-3 bits) | 262K |
Parameter counts come from the model cards and the runtimes' READMEs. All four are mixture-of-experts models: only part of the model works on each token, which is what makes them possible on one GPU. The runtimes keep the most-used experts in VRAM and the rest in RAM; Maya with DeepSeek also reads part of the model from the SSD while it generates.
How to set up three of them, step by step:
- Qwen3.8-Flash-Next at 79 tok/s with Strata and a 5-bit quant
- GLM-5.3-Flash (321B) with Project Maya and EXL3
- DeepSeek-V4.1-Flash (748B) with Project Maya
For a machine with less RAM, the oldest article in the series covers Qwen3.8-Flash-Next with EXL3 and TabbyAPI on 40+ GB of RAM.
The machine
- GPU: RTX 5090, 32 GB of VRAM, also driving the desktop. Every runtime is set to leave 3 GiB free for that.
- CPU: Ryzen 9 9950X3D (16 cores, 32 threads).
- RAM: 128 GB DDR5 as 4x32 GB at 5,600 MT/s.
- Disk: one NVMe drive.
- OS: Arch Linux (Omarchy), NVIDIA driver 615.78, CUDA 13.4.
- Versions: Strata
296b59d, Project Maya 1.1.21 (451f9cb), TabbyAPId6e6227with ExLlamaV3 1.6.2.
How we measured
One model at a time, started with its own script, with no other model runtime or GPU miner running. Each runtime started in its own systemd slice without a memory limit, so a shell's limits or systemd-oomd could not get in the way (the setup articles explain why).
- Decode speed: a client-side streaming bench against the OpenAI endpoint. Eight runs, alternating a ~400-word Dutch prose prompt and a coding prompt (binary search with tests), greedy decoding, 512 output tokens, median reported. Time to first token is the median over the same runs.
- Prefill and long context: a needle-in-a-haystack test with prompts of about 31K and about 125K tokens. The needle is a code hidden in Dutch filler text, with a different random seed per run so the prefix cache cannot help. The exact token count depends on each model's tokenizer: the long prompt came to 124.6K tokens for Qwen, 125.6K for DeepSeek, 127.9K for GLM and 131.5K for Ornith. Prefill speed is prompt tokens divided by the time of the whole request, which includes a short answer.
- Drafting: Strata drafts with Qwen's MTP head and Maya with GLM's MTP block. Maya measures DeepSeek's DSpark drafter at the first start and keeps the faster option; on 1.1.21 it measured 23.9 tokens per second without and 23.8 with it, so it kept no drafting. Ornith on TabbyAPI runs without drafting, with 180 of its 256 experts per layer in RAM. In an A/B at 256K context the evening before, its MTP head gave 37.8 tokens per second, the DFlash2 drafter 40.6, and no drafting 45.2. Most of Ornith's experts run on the CPU, so every drafted token the model has to check costs extra reads from RAM.
- Quality on our own work: nine tasks from our day job, each with an automatic checker: two Astro helpers in JavaScript, two WordPress PHP snippets run in a PHP sandbox, SEO meta tags as JSON, LocalBusiness JSON-LD, a Search Console tool call, a Search Console quick-win analysis and a Dutch blog intro. 54 points per round, two rounds, reasoning effort medium, each server's default sampling (for Ornith, see the note under the quality table). The second round ran the models in reverse order. During this run, Node could not start inside the sandbox (a version-manager shim on a read-only file system), so both JavaScript tasks failed for every model. I fixed the sandbox and scored the saved answers again with the same checkers; that changed only those two tasks.
Speed
| Model | Decode (tok/s) | Dutch prose / code | Time to first token | Prefill, ~31K prompt | Prefill, ~125K prompt |
|---|---|---|---|---|---|
| Qwen3.8-Flash-Next, Strata | 78.8 | 81.8 / 76.5 | 0.77 s | 4,073 tok/s | 4,252 tok/s |
| Ornith-1.5-122B, TabbyAPI | 46.4 | 49.8 / 42.2 | 0.42 s | 1,808 tok/s | 1,508 tok/s |
| GLM-5.3-Flash, Maya | 30.1 | 34.8 / 26.4 | 0.78 s | 2,269 tok/s | 2,445 tok/s |
| DeepSeek-V4.1-Flash, Maya | 24.9 | 26.6 / 23.5 | 3.19 s | 1,842 tok/s | 2,570 tok/s |
All four found the needle at both lengths. For a sense of what the prefill means: the long prompt took 29 seconds on Strata, 49 on Maya with DeepSeek, 52 on Maya with GLM and 87 on TabbyAPI with Ornith. On October 21, with Maya 1.1.17, DeepSeek needed 182 seconds for the same length.
Qwen on Strata is in a class of its own: 1.7 times the decode speed of the next one, and the fastest prefill by far. One likely reason is the number of active parameters: about 6B per token for Qwen against about 18B for GLM, according to their model cards, so far fewer weights move through the CPU and over PCIe for every token. Strata's runtime, built for this one model family, plays a part too; we did not measure how much each contributes.
The bench counts tokens from the usage numbers each API returns, not from stream chunks. That mattered for Ornith in the first version: with MTP drafting, TabbyAPI streamed it in chunks of about two tokens.
Quality on our own tasks
| Model | Round 1 | Round 2 | Total | Seconds per round | Tokens written per round |
|---|---|---|---|---|---|
| Qwen3.8-Flash-Next, Strata | 54 / 54 | 54 / 54 | 108 / 108 | 104 | ~8,800 |
| DeepSeek-V4.1-Flash, Maya | 54 / 54 | 54 / 54 | 108 / 108 | 680 | ~15,900 |
| Ornith-1.5-122B, TabbyAPI, model card sampling* | 54 / 54 | 53 / 54 | 107 / 108 | 376 | ~17,300 |
| GLM-5.3-Flash, Maya | 47 / 54 | 51 / 54 | 98 / 108 | 147 | ~3,900 |
* Measured on October 29 with the same config, after a reader's comment (see Ornith and sampling). With TabbyAPI's default sampling on October 22, Ornith scored 37 + 46 = 83 / 108, at about 485 seconds and 22,600 tokens per round.
On October 21 the totals were 108 for Qwen, 108 for DeepSeek, 105 for GLM and 89 for Ornith (default sampling).
What went wrong:
- GLM wrote a WordPress breadcrumb snippet with a PHP syntax error in the first round, and missed the Search Console quick-win analysis in the second (0 of 3 checks), the same task it missed once on October 21.
- Ornith, with TabbyAPI's default sampling on October 22, missed the breadcrumb snippet (it used a PHP constant that does not exist), the JSON-LD task and the Search Console analysis in the first round, and a WordPress shortcode snippet with a PHP syntax error and one check of the Dutch blog intro in the second. Several misses came from stray tokens in the middle of otherwise correct output, such as a Chinese word or the word "Logical" inside a JSON object. That happened in one to four of the nine tasks per round, on both days, with and without MTP drafting. With the sampling from its model card those stray tokens are gone; its one miss on October 29 was a blog intro of 214 words, where the task asks for at most 170.
The time per round says as much as the score. Qwen finishes the nine tasks in under two minutes, GLM in about two and a half. DeepSeek needs 11 minutes, because it writes about 15,900 tokens per round at about 24 tokens per second. Ornith writes even more, about 17,300 tokens, mostly reasoning, and takes about 6 minutes. GLM writes the least, about 3,900 tokens, which is why it is quick despite its slower decode.
Ornith and sampling
Ornith's stray tokens came from sampling. TabbyAPI uses the sampler settings from its config, and our Ornith config had no preset. Without one, TabbyAPI samples at temperature 1.0 with top-k, top-p and min-p switched off, so now and then it picks a token from the far end of the distribution. Ornith's model card asks for temperature 0.6, top-k 20 and top-p 0.95.
On October 29 I ran the same nine tasks again on Ornith, with the same config as above (256K context, 180 experts per layer in RAM, no drafting). I compared three sampler settings, sent with each request, two rounds each, taking turns:
| Sampling | Round 1 | Round 2 | Total | Tasks with Chinese tokens | Decode (tok/s) |
|---|---|---|---|---|---|
| TabbyAPI default: temperature 1.0, no cutoff | 27 / 54 | 34 / 54 | 61 / 108 | 7 of 18 | 47.5 |
| Model card: temperature 0.6, top-k 20, top-p 0.95 | 54 / 54 | 54 / 54 | 108 / 108 | 0 of 18 | 47.2 |
| Temperature 1.0 with min-p 0.05 | 46 / 54 | 54 / 54 | 100 / 108 | 0 of 18 | 47.8 |
A min-p floor alone already removed the Chinese tokens. The model card settings also removed the PHP syntax errors. The default arm scored even lower than on October 22 (83), which fits: with no cutoff, which tasks break is down to chance.
I then made the model card settings the server default, through a sampler preset in TabbyAPI, and ran two more rounds without any sampler settings in the requests, the way our agent sends them: 107 out of 108, no stray tokens, 47.2 tokens per second. The one miss was a blog intro of 214 words where the task asks for at most 170.
So Ornith's earlier 83 was a configuration mistake on our side, not the model or the quant. The quality table above now shows Ornith with the model card settings. With the right sampling, it scores about as high as Qwen and DeepSeek on these tasks, at about 6 to 8 minutes per round.
These tasks reward correct, careful output, not deep reasoning. They do not tell you which model is smartest. They do tell you which ones we can hand our everyday work to without checking every answer.
Memory and VRAM
| Model | VRAM peak (whole card) | VRAM free at peak | RAM in use (whole system) | Engine in swap |
|---|---|---|---|---|
| Qwen3.8-Flash-Next, Strata | 29.0 GiB | 2.9 GiB | ~120 GiB, pinned | ~361 MiB |
| Ornith-1.5-122B, TabbyAPI | 28.5 GiB | 3.3 GiB | ~61 GiB, pinned | ~2.3 GiB |
| GLM-5.3-Flash, Maya | 28.9 GiB | 2.9 GiB | ~117 GiB, pinned | 0 |
| DeepSeek-V4.1-Flash, Maya | 28.6 GiB | 3.2 GiB | ~20 GiB + model files in page cache | ~585 MiB |
The VRAM peak is the highest value nvidia-smi reported for the whole card during the needle tests, so it includes the desktop and two browsers (about 2.2 GiB before any model started). Each runtime keeps its 3 GiB reserve on top of its own use; with the desktop counted, GLM and Qwen left a little less than that free at their peak. Strata and Maya with GLM pin almost all RAM for their expert tiers, so nothing else heavy fits next to them. Ornith's 65.5 GB quant leaves the most room; with more of its experts in VRAM, it now uses about 1 GiB more VRAM than in the first version of this article. DeepSeek's engine works differently: it reads its model files in place, so free counts most of what it uses as page cache and reports about 100 GiB as available. Do not read that as free memory. The engine depends on that cache, and anything you start next to it pushes experts out, which then have to come from the SSD again. In practice it uses the whole machine, like the other two. RAM and swap were read once, right after the needle tests. Ornith ran right after DeepSeek and had about 2.3 GiB in swap at that point; on the previous evening's tuning runs it had none, and its decode speed here (46.4) matched those runs with the same settings (46.4 to 47.2).
Which one we use when
- Qwen3.8-Flash-Next on Strata is our default for coding and agent work. Fastest at everything, full marks in both rounds of our tasks, 512K context.
- GLM-5.3-Flash on Maya is the second opinion: a much bigger model with 512K context, at about 40% of the speed, and the one that writes the least.
- DeepSeek-V4.1-Flash on Maya is the biggest model here, and the slowest to write, with a 3-second wait before the first token. Since Maya 1.1.21 it reads long prompts about as fast as GLM. Good for hard problems where we can wait, not for interactive work.
- Ornith-1.5-122B on TabbyAPI is now the second fastest at writing, with the quickest first token, but the slowest at reading long prompts. It leaves the most RAM free. With the sampling from its model card it scores 107 to 108 out of 108 on our tasks; with TabbyAPI's default sampling it scored the lowest, because of stray tokens in its output.
Running all of this on one desktop is possible because the runtimes move experts between VRAM, RAM and the SSD while you chat. The cost is that only one model fits at a time, and switching takes a stop and a start script.
Caveats
- One machine, one session. Other hardware, other drivers or another day can give other numbers, and all four runtimes move fast: Maya 1.1.17 made DeepSeek's long-prompt reading about eight times faster than 1.1.16, and 1.1.21 made it another 3.7 times faster.
- Two rounds of nine tasks are not many. A single miss can be noise.
- GLM scored 98 here and 105 on October 21, Ornith with default sampling 83 and 89. With two rounds of nine tasks, that difference is within what a few misses can do.
- Each model ran with its runtime's default sampling and reasoning effort medium. For Ornith on TabbyAPI that default was the problem (see Ornith and sampling). Strata samples greedily by default, and Maya uses the sampling from GLM's and DeepSeek's model cards. Check what your runtime does when a request sends no sampler settings.
If you are thinking about running models like these for your business, Voltti has a page about local AI for businesses: what stays on your own hardware, and when a hybrid setup makes more sense.
Top comments (2)
The stray tokens inside otherwise correct output (a Chinese word, "Logical" in the middle of a JSON object) are the most interesting part for me. Since it shows up with and without MTP drafting, I'd guess it's tail sampling rather than the drafting, and you note everything ran on the runtime's default sampling. Have you tried rerunning just Ornith's failing tasks with a tighter setup, like lower temperature or a min_p floor, to see if the strays disappear? If they don't, that points more at the quant or the expert offload path. Either way "read what it writes before we run it" is a fair rule, and the two-round noise caveat is honest.
Thanks, Mihai. You were right. TabbyAPI without a sampler preset runs at temperature 1.0 with no top-k, top-p or min-p cutoff. With the settings from Ornith's model card (0.6, top-k 20, top-p 0.95), Ornith went from 61 to 108 out of 108 on the same tasks, with no stray tokens and the same speed. A min-p floor of 0.05 alone also removed them. I've added the A/B to the article.