I benchmarked Gemma 4 as a local translator on Apple Silicon. It never produced a single line, and the autopsy taught me more than the benchmark.

David C. — Mon, 31 Aug 2026 12:05:10 +0000

I run a daily multilingual tech digest that is generated 100% locally on a Mac: an English editorial written by Qwen3.6-35B-A3B, then translated to French, German, Spanish and Japanese. Translation is my weakest link, so I benchmark replacement candidates against production data: LanguageTool issues per 1000 words, number and entity fidelity against the English source, and a nightly latency projection.

Gemma 4 (26B-A4B, the QAT 4-bit community port, mlx-lm 0.31.3 in an isolated venv) looked perfect on paper: Apache 2.0, MoE with 4B active parameters, strong multilingual claims. Two runs later I had zero usable lines. What happened is worth sharing.

Run 1 looked spectacular and was garbage. LanguageTool reported 113 to 175 issues per 1000 words, roughly 20x every other candidate, while fidelity scored a perfect 100 everywhere. That combination should be impossible, and it was the tell: I was not measuring translations. The model had filled the entire token budget with its reasoning channel, which quotes the English source extensively. Perfect fidelity because it was literally the source; absurd error rates because a French dictionary was judging English planning notes. My stripping code split on the channel marker strings, but those markers are special tokens that the detokenizer removes, so the strip could never match. One gate saw through it all along: French diacritic density collapsed to 0.41 per 1000 characters, far under the 16.5 threshold. Mostly English text carries no accents.

Run 2, after fixing everything properly (cutting the thought channel on token ids, adding token 106, the turn delimiter, to the stop list, because generation_config declares eos_token_id [1, 106, 50] but mlx-lm only keeps 1, and using the model card sampler instead of greedy), failed differently. The model closed its turn without ever closing its thought channel. Around 1300 tokens of reasoning at a steady 76 tok/s, then end of turn. The answer never existed. This was with thinking disabled the official way; the chat template defaults to enable_thinking false.

My conclusion is not that Gemma 4 is bad. It is that the local stack around a brand new model is a separate thing from the model, and it needs its own maturity. Three practical takeaways if you are trying Gemma 4 on MLX today: strip reasoning by token ids, never by string matching, because channel markers vanish in detokenization; check your effective stop tokens against generation_config, not against what the library silently keeps; and put a cheap smoke test in front of any long benchmark (mine now checks that output is non empty, that the length ratio to the source is sane, and that language specific character density passes, four calls instead of a 50 minute run).

Meanwhile the boring winner of the bench was Gemma 3 27B, which beat my current Qwen translator on all three measurable languages with zero gate violations, running on a months old stable stack. There is a lesson in that too.

Happy to share the harness details (fidelity gates, LanguageTool setup, the smoke checks) if anyone wants them. And I will retest Gemma 4 the day the stack catches up, the isolated venv is sitting there ready.

My SaaS was silently broken for 4 days because an AI My app was broken for four weeks and I didn't notice. So I got rid of the API.

David C. — Sun, 12 Jul 2026 18:13:18 +0000

On June 14th, the weekly brief feature of my product died. No crash, no alert, nothing in my inbox. It just quietly stopped producing anything new, and the dashboard kept showing the last good brief like everything was fine.

I found out four weeks later.

Some context

I run brevio.news alone. It started as a personal itch: I had hundreds of YouTube videos saved for "later", newsletters I never opened, articles piling up. I wanted something that reads all of it and gives me back a short editorial brief every morning, like a tiny Economist built from my own watchlist. No sponsored content, no recommendation feed.

Today it watches 475+ sources (YouTube, RSS, academic feeds, newsletters) across 7 themes, and publishes a brief every morning at 9:00 in 5 languages. It went through the classic solo builder phases: a CrewAI prototype, then n8n, then a real codebase.

The setup is simple. A Mac Studio M4 Max (64 GB) in my office does all the AI work: Whisper for transcription, Qwen for classification and writing. I started with Ollama and later moved the pipeline to MLX. The Mac pushes results to a small VPS (FastAPI, Postgres, nginx) that serves the dashboard. The Mac generates, the VPS serves.

Everything ran locally except one feature. The weekly brief, a Sunday synthesis of the week for paying users, still called a hosted LLM API.

What actually happened

The weekly job calls a model by its ID. That ID got retired by the provider, and the API started returning 404. The job itself kept "succeeding" every Sunday: the timer fired, the code ran, the error got swallowed somewhere, and no new brief was written. A frozen feature looks exactly like a working feature unless you check the dates.

I'd love to say I caught it through some clever alert. I didn't. I noticed while working on something else.

Two things I had to admit to myself:

A hosted model ID is a dependency with someone else's lifecycle. It can be retired like an old npm package, except there's no lockfile and nothing warns you at build time.

And my monitoring only watched for errors. The failure here wasn't an error, it was an absence. Nothing happened, and "nothing" doesn't trigger alerts unless you design for it.

The fix took one Sunday

The daily editorial already ran on a local model, Qwen3.6-35B-A3B in 4-bit, about 19 GB on disk. So the weekly brief had a proven path to copy. Now it works like this:

Sunday 09:15, on the Mac:
  pull the week's items from the local DB (~700 items, 7 themes)
  generate the brief with Qwen via mlx_lm
  pick the featured video per theme in code (the model never writes URLs)
  convert markdown to HTML in code (the model never writes HTML)
  POST to the VPS, idempotent upsert
  ping a healthcheck that expects a brief to exist by 10:00

The numbers on the M4 Max: model loads in about 6 seconds from cache, generation takes about 28 seconds for ~1,800 tokens from an 8,700 token prompt. Call it 35 seconds for the whole brief. API cost: zero. New dependencies: zero, the model was already on disk for the daily pipeline.

I'm currently the only paying user of this feature (the product has 29 users total), which made the migration comfortable. I could iterate on the prompt against real data and judge the result myself. The local brief turned out as good as the hosted one on the part I care about, connecting trends across themes. Its "what to watch next week" section is actually better, because it sticks to signals from the week instead of speculating.

A few choices I'd defend: the model writes prose, code writes structure. Links are picked by a ranked query and injected afterwards, so a hallucinated URL is impossible, not just unlikely. The sync keeps the last good brief if anything fails, so the dashboard never shows a hole. And there's now a check that fails loudly if the expected brief doesn't exist on Sunday morning. That one would have saved me a month.

Why local, honestly

The zero API bill is nice, but it's not the real reason. The real reason is that a model on my own disk can't be retired, rate-limited or repriced by someone else. For a one-person product, removing a whole class of external failures, the exact class that just bit me, is worth more than being able to switch to the newest hosted model.

The Mac was already paid for and already reads 475+ sources every morning. The weekly brief costs it 35 seconds a week.

If people are interested I'll write up how the daily pipeline works end to end, from Whisper to the 9:00 brief. Brevio is at brevio.news.
The whole thing exists because I believe staying informed shouldn't cost you your mornings.

DEV Community: David C.

I benchmarked Gemma 4 as a local translator on Apple Silicon. It never produced a single line, and the autopsy taught me more than the benchmark.

My SaaS was silently broken for 4 days because an AI My app was broken for four weeks and I didn't notice. So I got rid of the API.