Your local LLM will translate a transcript instead of summarizing it — and other lessons from going 100% offline

#opensource #ai #privacy #showdev

I spent a few weekends building an offline audio/video transcriber, and the most
useful things I learned had nothing to do with transcription. They were about the
boring reality of running AI on someone else's laptop: models that do the wrong task,
RAM numbers that lie, and CPUs that don't have the instructions your binary assumes.

Here are the three that cost me the most time. If you're shipping anything with a
local LLM, at least the first one will probably bite you too.

Why local at all

The trigger was simple: every cloud transcription tool wants you to upload the audio.
Fine for a public podcast. Not fine for a client call, an internal meeting, or an
interview where the other person didn't agree to have their voice sent to a third
party. I wanted the opposite — drop a file, get text, subtitles and a summary, and
have nothing leave the machine.

So the whole thing runs on-device: NVIDIA NeMo Parakeet for speech-to-text
(fast on CPU, 25 European languages in one 0.6B model, word-level timestamps) and a
local Qwen2.5 model via llama.cpp for summaries. That constraint — it has to
run on a normal Windows PC with no GPU and no setup — is where the interesting
problems started.

Lesson 1: "summarize in Italian" makes the model translate, not summarize

This one is worth stealing regardless of what you're building.

Say the audio is English and the user wants the summary in Italian. The obvious prompt
is "Summarize the following transcript in Italian." On a small local model (I started
with a 3B), the output was not a summary — it was a near-complete translation of
the whole transcript. All the length, none of the compression. Useless.

The reason is that you asked the model to do two hard things at once — compress and
change language — and the smaller the model, the more likely it collapses to the
easier, more literal one: translate line by line.

The fix is to stop overloading it. Split into two stages:

Stage 1: summarize in the SOURCE language   (English → English)  ← model's strong suit
Stage 2: translate the short summary         (English → Italian) ← small, safe task

Each step is now something a small model does well: same-language summarization, then
translating a handful of sentences instead of a whole document. Quality jumped
immediately — and even more when I later moved from 3B to 7B.

Two details that mattered:

Gate it. If source and target language are the same, take the exact old single-pass path. The two-stage logic only activates when source != target, so same-language jobs are byte-for-byte unchanged and you can't regress them.
Translate the summary, not the transcript. The whole point is that stage 2 operates on ~10 sentences, not 10 minutes of speech.

General principle: when a small model does the wrong task, check whether you've asked
it to do two things in one shot. Decomposing into single-responsibility calls beats a
cleverer prompt almost every time.

Lesson 2: "16 GB of RAM" is a lie the moment something else is open

My first gate for enabling local summaries checked total system RAM: got 16 GB?
You're good. This is wrong, and it will crash on real machines.

A ~4.7 GB model file needs its weights resident, plus a KV cache, plus compute buffers,
plus whatever the rest of the app and the OS are already using. A user with 32 GB total
but a browser, an IDE and Slack open can easily have less free RAM than someone
with 16 GB and nothing running. Total RAM tells you what the machine could do in a
vacuum; it says nothing about right now.

The right signal is available memory at the moment you're about to load, not the
total the box shipped with. And when it's not enough, the correct behavior is a clear
message — "not enough free RAM, close some apps or use a cloud key" — not an
out-of-memory crash that kills the process with a useless exit code.

It sounds obvious written down. It is not obvious when psutil.virtual_memory().total
is right there and returns a nice big number.

Lesson 3: your prebuilt binary assumes a CPU your users don't have

Local inference on CPU means shipping native code, and native code makes assumptions.
Many prebuilt llama.cpp builds assume AVX instructions. On an older or low-end
processor, that's not a slow path — it's an illegal-instruction crash on startup,
before your app draws a single pixel.

So I compile a baseline build with no AVX assumptions (MinGW, statically linked)
and bundle the runtime next to it. No "please install the Visual C++ redistributable"
step, no "works on my machine" surprises. It's unglamorous packaging work, but for a
desktop app aimed at non-technical users it's the difference between runs and
doesn't. The fanciest model is worthless if the binary segfaults on their hardware.

Takeaways

Decompose tasks for small models. One call = one job. "Summarize and translate" becomes "summarize, then translate."
Gate on available resources, not nominal ones. Free RAM now, not total RAM ever.
Assume nothing about the CPU. Baseline builds and bundled runtimes, or you'll crash on exactly the machines you're trying to reach.

None of this is transcription-specific — it's the tax you pay to move AI from a GPU in
a datacenter to a random laptop. I think that tax is worth paying: keeping people's
audio on their own machine is a feature you can't fake with a privacy policy.

If you want to poke at the thing this came out of, it's a Windows app called Qwibo
— source-available and auditable on GitHub, free for personal use, and very much
alpha (the installer isn't code-signed yet, so Windows SmartScreen will grumble).
I'd genuinely rather hear where it breaks than get a compliment: the "here's the file
that made it fall over" kind of comment is the useful kind.