I built Koe: fully-local, offline voice dictation for Windows (faster-whisper + Ollama)

What I built

Koe is a voice dictation tool for Windows: you tap a key, speak, and clean text is typed into whatever app has focus. The whole pipeline — transcription and cleanup — runs on your own machine. Nothing is ever sent to the cloud.

It's open-source (MIT): https://github.com/dkamehat/Koe

Why

The inspiration is Aqua Voice. The "press a key, talk, get clean text in any app" experience is genuinely great — but I had two constraints:

1. No subscription — I wanted it to be free.
2. Nothing leaves the machine — for some of my work, my voice and on-screen content simply can't go to a third party.

The only way to satisfy both is to run everything locally. So I set the thesis up front: it must run locally, and it should be something anyone can use safely. Every decision flows from that.

How it works

 mic ─► record ─► faster-whisper ─► dictionary ─► refiner ─► clipboard paste ─► app
          ①          ②(CUDA)        proper nouns   local LLM    Unicode-safe       ④
                                                 (context-grounded, streamed)

• STT: faster-whisper (CTranslate2) on CUDA / float16, large-v3-turbo by default. Handles English and Japanese.
• Cleanup ("refiner"): an optional local LLM via Ollama (qwen2.5) removes fillers, adds punctuation, and fixes proper-noun spelling — while preserving your exact wording and never translating. Cloud backends (Claude/OpenAI) exist but are opt-in, bring-your-own-key, and selecting one automatically disables the screen-context grounding so nothing leaves the box.
• Context grounding: it reads the focused window/field locally so it spells code identifiers right (say "get user by id" → types get_user_by_id).
• Self-healing dictionary: correct a term once and it sticks.

Things I learned the hard way

Getting something that works was the easy part. Making it fast and pleasant is where the real lessons were.

1. localhost cost me ~2.2 seconds per request on Windows

It felt like 3–5 seconds from "stop talking" to "text appears," and I assumed the model was slow. But when I actually measured, inference was fine. The culprit: Python's requests resolving http://localhost:11434 on Windows hits an IPv6→IPv4 fallback that cost ~2.2s per request.

Switching to http://127.0.0.1:11434 (plus reusing a requests.Session with trust_env=False) dropped the cleanup step from ~2.6s to ~0.3–0.5s. That was ~90% of my "latency problem." Lesson: on Windows, don't use localhost for a local HTTP service from Python — measure before you optimize.

2. Whisper still hallucinates repetition loops

On silence or noise, Whisper can fall into a loop, emitting the same token over and over (シャッシャッシャッ… × 100). Even with the usual guards (compression_ratio_threshold, temperature fallback, VAD) it occasionally slips through — and then the LLM cleanup step dutifully chews on a thousand repeated tokens, wasting ~10 seconds.

The fix that actually held: a deterministic post-processing guard that collapses any short unit repeated 6+ times in a row down to a single occurrence. Normal speech never does that, so it's safe — and it stops the garbage from ever reaching the user or the LLM.

3. Fixing the clipped first word with "preroll"

The instant you press the key, mic driver start-up latency eats the first syllable ("…ello" instead of "Hello"). The fix: keep an always-on ring buffer running, and when you start a take, prepend the ~0.3s before the keypress. First word intact.

4. Keeping the local LLM from "improving" too much

Left alone, a local LLM loves to paraphrase — and sometimes to translate. The first time my Japanese came back as English, I was surprised. Beyond a hardened "preserve wording, never translate, minimal edits" prompt, I added a deterministic language guard: if the input contains CJK but the output dropped all of it (or vice-versa), the LLM result is discarded and rule-based formatting is used instead. For dictation, predictable beats clever.

5. Packaging CUDA into a standalone .exe

"Anyone can use it" means a build that runs without Python or git. PyInstaller handles the app, but the CUDA DLLs (cuBLAS/cuDNN) are the tricky part. CTranslate2's loader only searches its own package directory and the bundle root — not arbitrary subfolders — so the DLLs have to sit at the root. On top of that, PyInstaller's nvidia hooks were collecting the same DLLs into a second location; dropping those duplicates saved ~650 MB. Final result: a ~1.2 GB zip you can download, unzip, and run.

Try it

# A) Download (no Python needed)
#    Grab Koe-win64-cuda.zip from the Releases page, unzip, run Koe.exe

# B) From source
git clone https://github.com/dkamehat/Koe.git
cd Koe
.\setup.ps1
.\run-admin.bat

Tap Right Ctrl once to start, speak (pause and think freely), tap again to stop → cleaned text lands in the focused app.

What's next

With the foundation in place, I'm prototyping a live meeting interpreter on the same local stack — local English subtitles + Japanese translation for the other side of a Zoom/Meet call. The hard part is WASAPI loopback capture and streaming endpointing. More on that later.

Closing

You can build genuinely useful voice input without giving up "local-only and safe for anyone." A surprising amount of the work was chasing unglamorous culprits — the biggest "slowness" was DNS resolution, not the model. Measure first, then fix.

Repo: https://github.com/dkamehat/Koe — feedback very welcome.