Trying MiniMax M3 Without Sparse Attention in llama.cpp

tl;dr: When running MiniMax M3 Q8_0, dropping temp from 1.0 to 0.8 helped a lot with minor hallucinations and oddities, and disabling MSA also seems fairly promising so far, even though llama.cpp warns that its built-in dense fallback may degrade output. And it might; I'm still testing

---

In my last post about picking models as a Mac user, I mentioned that MiniMax M3 had pretty much become my go-to larger model on the M3 Ultra. On paper it lands almost exactly where I want a model to land: it's large enough to be genuinely capable, small enough that I can run the Q8 instead of having to run it heavily quantized, it doesn't generate a stupid amount of tokens, and its long-context benchmarks look great.

With that said... after using it more, something about it bothers me. It's not like the model is incoherent or anything like that; most of the time it actually seems really good. The problem is that every once in a while it seems to get confused by some minor detail that I really wouldn't expect a model in this size/class to get confused by.

This isn't quite the same thing I mentioned in the last post, either: where I was seeing some obvious weirdness with the raw MXFP8 model through oMLX, including things like randomly misspelling "birthday" as "birtday." Moving over to Bartowski's Q8_0 through llama.cpp seemed to get rid of that. This is more behavioral. It'll follow the conversation, reason through the bigger problem, and then occasionally mix up or lose some little detail along the way. Sporadic enough that it's hard to benchmark, but often enough that I started wondering what the heck was going on. And I know that in general LLMs can do that, but it was happening often enough, and on details important enough, that I was taking notice of it.

After digging around a bit, I saw someone online with similar issues who was wondering if MiniMax’s sparse attention might have something to do with it. So obviously I had to try disabling that.

First Test: Turning Off Flash Attention

MiniMax M3 uses MSA (MiniMax Sparse Attention). Looking through llama.cpp's MiniMax M3 implementation as of build b10702, MSA is currently dependent on Flash Attention being enabled:

src/models/minimax-m3.cpp

const bool fa_on       = cparams.flash_attn;
const bool streams_ok  = cparams.n_seq_max == 1 || !cparams.kv_unified;
const bool msa_enabled = fa_on && streams_ok;

So the obvious first test was just:

-fa off

With Flash Attention disabled, msa_enabled becomes false and llama.cpp falls back to its normal dense attention path.

Great. Easy peasy. Except it was sloooooow. With fa off, the model runs about the same speed as GLM 5.2 on my M3 Ultra.

At around 22K context, with Flash Attention on, I was getting roughly 12 tokens/second generation speed. With -fa off, generation dropped down to around 7 tps.

At first I assumed that was the price of switching M3 from sparse to dense attention. It made sense: dense attention means doing more work against the context, so maybe that was just the tradeoff. But then I figured I'd at least test untangling fa and msa, just to see how dense with fa runs.

Separating MSA From Flash Attention

Fortunately, testing this was pretty simple. If I was going to properly untangle them, I'd likely need to check other models for the same pattern and wire them up to a new argument, but since I know I want MSA off and fa on, that makes this a 1 liner update.

I popped over to src/models/minimax-m3.cpp and changed this:

const bool msa_enabled = fa_on && streams_ok;

to this:

const bool msa_enabled = false;

That's it. I then recompiled llama.cpp, dropped the -fa off line, and that left me in this state:

Flash Attention: ON
MiniMax Sparse Attention: OFF

When MSA is disabled, llama.cpp already has a normal attention path sitting there. The code checks whether the current layer should be sparse and otherwise calls the regular build_attn() path, so it was already wired to handle this, they just hadn't exposed it. I'm basically just forcing the model to always take the existing dense fallback.

Interestingly, token generation speed was a lot higher with MSA off and FA on than it was with both MSA and FA on.

At roughly 22K context:

Configuration	Generation Speed
Flash Attention OFF, MSA OFF	~7 t/s
Flash Attention ON, MSA ON	~12 t/s
Flash Attention ON, MSA OFF	~17 t/s

That was a bit surprising, but a pleasant surprise.

Prompt processing seems to be just a little slower with dense attention, so it isn't just free performance all around, but the difference in generation speed more than makes up for it.

The Other Way to Disable It

If you look at the code, you can see that there's technically a way to make llama.cpp disable MSA without modifying anything.

const bool streams_ok = cparams.n_seq_max == 1 || !cparams.kv_unified;
const bool msa_enabled = fa_on && streams_ok;

So if you enable the unified KV cache while also setting parallel higher than 1, streams_ok becomes false.

So something like:

--parallel 2 --kv-unified -fa on

should make llama.cpp fall back to dense attention while leaving Flash Attention enabled. The source actually has a warning specifically for this situation because MSA's block selection depends on KV-cache slot layout, and multiple sequences in a unified cache aren't currently compatible with it.

However, I don't really want to do this, since on a Mac I intentionally run:

--parallel 1

In my experience, Macs don't handle parallel inference well at all, especially on larger models, and I want to guard against parallel requests ever hitting my server so they don't gum it up unexpectedly. Wilmer has a concurrency block, so it shouldn't ever happen, but I generally like making sure of stuff like that.

Anyhow, changing parallelism and the KV-cache layout adds more variables to this little test, so I'd prefer to keep it simple.

Sampler Change As Well

One other thing I threw in that had helped a lot, and this was something I did before the MSA update: MiniMax recommends a temp of 1.0, but I actually started getting FAR better responses at 0.8, so I'm running that now. Even without disabling MSA, that had already improved the hallucinations/oddities quite a bit.

Testing So Far

I've been running M3 with dense attention and lower temp for a few days, and so far the responses feel a little more solid. I haven't noticed the same small-detail confusion that originally made me start poking at this. With that said, the original problems were sporadic, so I can't say for certain that I haven't just been lucky so far.

I'll update as the testing goes. I have been swapping back and forth with old faithful GLM 5.2 UD_Q4_K_M for comparison sake, and I'm eyeballing this new K2 model that came out the other day, so we'll see if I hold on to Minimax or swap away from it.