I run a daily multilingual tech digest that is generated 100% locally on a Mac: an English editorial written by Qwen3.6-35B-A3B, then translated to French, German, Spanish and Japanese. Translation is my weakest link, so I benchmark replacement candidates against production data: LanguageTool issues per 1000 words, number and entity fidelity against the English source, and a nightly latency projection.
Gemma 4 (26B-A4B, the QAT 4-bit community port, mlx-lm 0.31.3 in an isolated venv) looked perfect on paper: Apache 2.0, MoE with 4B active parameters, strong multilingual claims. Two runs later I had zero usable lines. What happened is worth sharing.
Run 1 looked spectacular and was garbage. LanguageTool reported 113 to 175 issues per 1000 words, roughly 20x every other candidate, while fidelity scored a perfect 100 everywhere. That combination should be impossible, and it was the tell: I was not measuring translations. The model had filled the entire token budget with its reasoning channel, which quotes the English source extensively. Perfect fidelity because it was literally the source; absurd error rates because a French dictionary was judging English planning notes. My stripping code split on the channel marker strings, but those markers are special tokens that the detokenizer removes, so the strip could never match. One gate saw through it all along: French diacritic density collapsed to 0.41 per 1000 characters, far under the 16.5 threshold. Mostly English text carries no accents.
Run 2, after fixing everything properly (cutting the thought channel on token ids, adding token 106, the turn delimiter, to the stop list, because generation_config declares eos_token_id [1, 106, 50] but mlx-lm only keeps 1, and using the model card sampler instead of greedy), failed differently. The model closed its turn without ever closing its thought channel. Around 1300 tokens of reasoning at a steady 76 tok/s, then end of turn. The answer never existed. This was with thinking disabled the official way; the chat template defaults to enable_thinking false.
My conclusion is not that Gemma 4 is bad. It is that the local stack around a brand new model is a separate thing from the model, and it needs its own maturity. Three practical takeaways if you are trying Gemma 4 on MLX today: strip reasoning by token ids, never by string matching, because channel markers vanish in detokenization; check your effective stop tokens against generation_config, not against what the library silently keeps; and put a cheap smoke test in front of any long benchmark (mine now checks that output is non empty, that the length ratio to the source is sane, and that language specific character density passes, four calls instead of a 50 minute run).
Meanwhile the boring winner of the bench was Gemma 3 27B, which beat my current Qwen translator on all three measurable languages with zero gate violations, running on a months old stable stack. There is a lesson in that too.
Happy to share the harness details (fidelity gates, LanguageTool setup, the smoke checks) if anyone wants them. And I will retest Gemma 4 the day the stack catches up, the isolated venv is sitting there ready.
Top comments (0)