After spending the past two weeks redoing all the models around the house, I realized it might make a good topic to chat about. I know that everyone and their brother has their own way to figure out what models they want to run on their hardware, but I figure that my own criteria might help some of the Mac users out there, so I'm tossing it into the mix as well.
Picking which models to even compare
When a new model comes out, the first thing I always do is check what folks are saying: huggingface discussions, reddit comments, etc. Benchmarks are useful, but I want to know what happens when people actually use the thing. Is the tokenizer broken or llama.cpp/mlx implementation bugged? Does it follow instructions? Overthink? Hallucinate a bunch? Discussion comment sections are a treasure trove of info.
After that, I go peek at the model on Artificial Analysis. I know AA isn't everybody's favorite way to judge models, but honestly it has a pretty solid litmus test for whether the model will be good for me or not. In particular, there are a specific few benchmarks there which line up really well with what I need:
- strong context reasoning
- hallucination rate
- how many output tokens the model used to get its scores
That last one is one of the most important. Combined with the "Humanity's Last Exam" score and overall intelligence, it gives me an idea of how much output the model had to produce to actually land on its current spot on the leaderboard. On a Mac, I really care about that.
Remember: our compromise with Macs is getting stupidly large amounts of VRAM at the price of everything being a lot slower than NVidia GPUs, especially as context and token generation sizes increase. So if one model gets a slightly better score by generating dramatically more tokens, you gotta keep that in mind. Really long-winded thinking sessions could make a model almost unusable. You might think "I don't mind waiting for quality", but then suddenly find yourself just going to ChatGPT because you don't feel like waiting 5 minutes for a response.
Qwen3.8 27B is probably the easiest example of this. Benchmark wise it lines up nicely against Opus 4.6, which was top tier at the start of 2026. That's fantastic for a small open source model. But the amount of token generation needed, since it defaults to xhigh reasoning, was so high that realistically I won't be using it that way very often.
Example Comparison
To give you an idea, here are the smaller models I was looking at recently, mostly for my M2 Ultra and M5 Max:
| Model / setting | Size | AA Intelligence | HLE | AA-LCR | Hallucination | Output tokens |
|---|---|---|---|---|---|---|
| Qwen3.8-27B xhigh | 27B dense | 52.0 | 33.9% | 77.3% | ~18% | 160M |
| Qwen3.8-27B Medium | 27B dense | 44 | ~14% | ~76% | ~67% | 75M |
| Qwen3.8-27B Low | 27B dense | 43 | ~14% | ~75% | ~53% | 43M |
| Qwen3.8-27B non-reasoning | 27B dense | 35 | ~12% | ~63% | ~30% | 26M |
| Muse Glimmer High | 30B dense | 35.1 | 22.0% | 80.0% | 81.9% | 48M |
| Gemma 4 31B Reasoning | 30.7B dense | 29.7 | 23.6% | 68.3% | 85.0% | 38M |
| G9v3-39A5B | 39B / 5B active | 34.0 | 17.5% | 62.0% | 13.0% | 91M |
| Qwen3.6-27B Reasoning | 27.8B dense | 37.7 | 23.1% | 73.3% | 49.3% | 140M |
| Ornith-1.5-35B-A3B | ~35B / ~3B active | — | 25.6% | — | — | — |
| Nemotron 3.5 Lightning | 31.6B / 3.6B active | 23.6 | 10.6% | 49.2% | 37.6% | 100M |
| HyperNova 60B 2605 | 58.7B / 4.8B active | 17.8 | 15.1% | 31.7% | — | 140M |
So yea- looking at the chart, Qwen3.8 27B at xhigh gets a really impressive score for its size. But it also took around 160M output tokens to get there. Fortunately, swapping from xhigh to medium took it down from 52 to 44 Intelligence at around 75M tokens, while Low gets a 43 at around 43M.
So Qwen3.8 27b on low is still outpacing the other models, but now coming in at about 43M tokens in total for the suite. For me? That's a happy spot to land. So Low reasoning level wins out.
That doesn't mean I won't use xhigh. If I've got something difficult enough that I actually want the model spending that much compute on it, then I'll go for it. I just have to accept I'll have time to go grab a shower or eat some lunch while it runs.
Gemma 4 31B is also still a model I really like, even if the hallucination number is not quite where I want it to be. Long context is great, output token amount is great, and honestly I love the tone of how it speaks. I particularly like it as a normal chat model for things like recommending wording changes, cleaning up text, and that kind of thing.
Because it's prone to overthink, I generally run Gemma with reasoning off, and just run it through a Wilmer manual Chain of Thought workflow. Forcing it through my own thinking steps has worked pretty well for me so far. I used to do something similar with Qwen3.6, but so far with Qwen3.8 I haven't really felt the need to. I think they finally slayed the overthinking.
Larger models
Then there's the M3 Ultra 512GB.
These are the larger models I was comparing for it, with the 1 rule that I won't go under 4bpw... ideally 4.5 bpw:
| Model / setting | Size | AA Intelligence | HLE | AA-LCR | Hallucination | Output tokens |
|---|---|---|---|---|---|---|
| GLM-5.3-Flash | 320B / 18B active | 57.5 | 39.9% | 78.0% | 27.6% | 150M |
| Qwen3.8-Flash-Next | 180B / 6B active | 55.8 | 38.0% | 77.0% | 45.3% | 200M |
| GLM-5.2 Max | 744B / 40B active | 52.6 | 41.1% | 76.7% | 26.3% | 140M |
| DeepSeek V4 Flash 0731 (Max) | 284B / 13B active | 51.8 | 38.6% | 74.3% | 91.7% | 210M |
| MiniMax-M3 | 428B / 23B active | 45.4 | 39.0% | 80.3% | 18.4% | 89M |
| MiniMax-M2.7 | 230B / 10B active | 38.9 | 29.6% | 75.3% | 35.6% | 85M |
| MiMo-V2.5 | 310B / 15B active | 38.0 | 27.2% | 68.3% | 31.9% | 71M |
| GLM-4.7 Reasoning | 357B / 32B active | 34.5 | 27.4% | 68.0% | 93.0% | 170M |
| Qwen3.5-397B-A17B Reasoning | 397B / 17B active | 34.3 | 29.0% | 72.7% | 88.9% | 88M |
| Step-3.7-Flash | 198B / 11B active | 30.9 | 21.4% | 69.7% | 85.0% | 260M |
| GLM-4.6 Reasoning | 357B / 32B active | 29 | ~14% | ~55% | ~94% | 61M |
If you compare what I wrote above and look at the table, you'll pretty much see a clear winner for my personal tastes right away. MiniMax M3 is an absolute champ here. Sure: GLM-5.3-Flash gets a 57 Intelligence score, while GLM-5.2 gets 52.6 and MiniMax M3 trails behind at 45.4. If I was just sorting by that column, I wouldn't be running MiniMax. But Minimax still gets 39% on HLE, which is right there with GLM-5.3-Flash's 39.9%, and it gets an 80.3% on AA-LCR. Its hallucination rate is also the best at only 18.4%. And then combine that with the fact that it hit the score at 89M output tokens instead of 140-150M, and it pretty much became my go-to model.
Also, I read that a part of the reason for the low hallucination number is apparently because MiniMax M3 is pretty willing to abstain if it doesn't think it knows the answer, which for me is a huge win.
For those curious- I did try GLM-5.3-Flash, and it was definitely impressive. But we're back to the same problem I have with Qwen3.8 xhigh: that great benchmark result is coming with a lot of token generation. It's fast at the start, but it will absolutely start to slow down once the total context + response start encroaching 50-60k context.
MiniMax M3
Right now, MiniMax M3 is pretty much my go-to for the M3 Ultra. It fits into a range where I can actually run it at Q8/around 8bpw instead of trying to quantize something huge down.
In general, for any large model, the happy place for me on total parameters is about 750B or lower because I can stuff at least a 4bpw onto the machine. Anything bigger than 750B and I'm looking at quantizing too heavily to trust it anymore. But this means that I especially love capable models in the 300-400b range, because it's perfect to fit a Q8 in.
Basically, between size, speed and token gen expectations: Minimax lands in exactly the spot I've been looking for.
I ended up testing running it through oMLX using the raw MXFP8 model and through llama.cpp using Bartowski's Q8_0 GGUF. Speed wise, with prompts around 20k tokens, I've been seeing roughly:
| Backend | Model | Prompt processing | Token generation |
|---|---|---|---|
| omlx 0.6.4 | Raw MXFP8 | ~290 t/s | ~14 t/s |
| llama.cpp | Bartowski Q8_0 GGUF | ~230 t/s | ~13 t/s |
omlx wins on speed, but one reason I'm leaning towards llama.cpp is that I was seeing some bits of weirdness from the model that were bothering me. Nothing major; most of the responses were coherent, but I was seeing things like it randomly misspelling "birthday" as "birtday," along with a few other small mistakes which made me start questioning the output. I generally don't see misspellings in modern open weight models, so that's a pretty decent sign something ain't quite right.
Honestly I have no idea why it's doing that. I'm not sure if there's something wrong with omlx, or maybe with the MXFP8 model, or it could be something in the conversion or inference path or how sampling is handled, etc. But I can say I haven't hit the same issues on llama.cpp yet. So- losing some prompt processing speed and around 1 token per second of generation is worth the confidence boost.
Current setup
The way I've got the servers set up right now is basically split by what I want each machine doing.
The M3 Ultra has MiniMax M3, which realistically I think is going to be my main open-source daily driver for a while, unless something truly amazing drops tomorrow. If I'm sitting down and actually want to talk to a larger model, do something complicated, or throw a bunch of context at something, that's generally where I'm sending it.
The M2 Ultra has turned into more of a utility box: running the smaller workhorse models, agents, researchers, and other stuff that I want available all the time. There's pretty much always something hitting it. It also serves the smaller models I use for validation or a second opinion. Right now I have Qwen3.8 27b and Gemma 4 26b a4b both running at BF16, and Gemma 4 31b running at Q8, along with an embeddings model as well.
The M5 Max MacBook Pro is set up with Q8 Qwen3.8 27b and Gemma 4 26b a4b. I use Wilmer's backup endpoint support so that I can route over to the M5 if one of the models on the M2 isn't available, and otherwise I keep basically the same general environment on the laptop so that I can take it with me and still have most of my normal setup when I'm away from home.
Anyhow, that's more or less how I've been choosing models lately. I mostly look for something that fits the hardware at a precision I'm comfortable with, does well with the kinds of large contexts I actually use, doesn't make up too much nonsense, and doesn't need to generate the full LotR trilogy every time I ask it a hard question.
I've been a little slow updating the blog and Wilmer lately because I've been pretty engrossed with work, but I have some Wilmer updates that I want to get out shortly and should hopefully start posting more again soon.
Top comments (0)