Anatomy of an offline dictation pipeline: where the time actually goes

This is the first post in a series about the parts of offline voice dictation that turned out to be harder than they looked. Most of the series is about bugs: the subtle kind, where every check is green and the user still gets the wrong text. But first, a map.

The project is YazSes. You hold a key, speak, let go, and your words are typed into whatever app has focus: editor, terminal, browser, chat. Speech recognition runs on your own CPU. No account, no API key, and no audio leaves the machine.

The nine stages

When you press the key, audio starts going into a buffer in memory. Nothing is written to disk. When you release it, that burst goes through nine stages:

1. Hotkey: the press and release events (Right Alt on Linux, Right Option on macOS, Right Ctrl on Windows by default).
2. Capture: the audio buffer itself.
3. Silence gate: is this clip loud enough to contain speech at all? If not, drop it. (Post 3 of this series is about what happens when this is set wrong.)
4. Padding: a short silent lead-in is added before decoding, so Whisper doesn't clip your first word.
5. Speech-to-text: faster-whisper on the CPU, quantised to int8. Default model: base.en.
6. Clean-up: strips Whisper artefacts like [BLANK_AUDIO] and stray leading punctuation.
7. Disfluency filter: removes fillers, de-duplicates repeated words, and rolls back self-corrections like "no wait".
8. Command or dictation?: "undo that" should press Ctrl+Z, not type the words "undo that". A regex grammar decides, in well under a millisecond.
9. Inject: type the text into the focused window, by whatever means the OS allows.

Where the time goes

I assumed I'd have to optimise half of these. I measured instead.

On my laptop (an i7-1370P, int8 on CPU, 30 utterances with a median length of 7.3 seconds), speech-to-text with base.en took a median of 1,561 ms. Every other stage combined had a median of 0.29 ms. Speech-to-text is 99.98% of the processing time of a burst.

So the filters, the command grammar and the injector are not worth tuning for speed. The only latency setting that matters is which model you run.

That made the design question simple. Instead of tuning, offer choices:

Engine	Word error rate (LibriSpeech test-clean, 200 utterances)
Parakeet TDT 0.6B v2 (optional)	2.06%
Whisper small.en	2.66%
Moonshine base (optional)	3.17%
Whisper large-v3	3.23%
Whisper base.en (default)	4.01%

Two honest caveats on that table. The confidence intervals overlap, so "Parakeet beats large-v3" is not established by it. And large-v3 loses mostly on insertions (words that weren't spoken), not on recognising what was said. The full method is in docs/benchmarks.md.

The optional engines are not installed by default, because they are big. You turn one on when you want it:

yazses features enable stt-parakeet

If an engine you configured is missing, YazSes logs a warning and falls back to faster-whisper instead of refusing to start. (Post 12 is about the day that fallback hid a real bug.)

The hard part is the last stage

Speech recognition is the expensive stage. It is not the hard one. The hard one is stage 9: getting text into someone else's window.

• macOS: CGEventKeyboardSetUnicodeString.
• Windows: SendInput with KEYEVENTF_UNICODE.
• Linux on X11: xdotool works.
• Linux on Wayland: there is deliberately no general way for one app to type into another. YazSes probes, in order: ydotool (a virtual keyboard through uinput), wtype, the desktop portal's RemoteDesktop interface, and the clipboard as a last resort.

Every one of those has failure modes where the API reports success and the user gets nothing, or gets garbage. Posts 2 and 5 are two of those stories.

Why offline at all?

Partly privacy. What you dictate is often the most private text you write: messages, notes, half-formed ideas. But also reliability: it works on a train, it works when a service is down, and it has no per-minute cost. On a modern laptop CPU, small Whisper models are fast enough that you don't need the cloud for this.

"Nothing leaves the machine" is easy to say. Post 6 is about how the project turns that sentence into a test that fails the build.

The rest of the series

1. Anatomy of an offline dictation pipeline (this post)
2. The tool that said yes and did nothing
3. My silence gate was quieter than my room
4. Whisper typed its own prompt into my document
5. SendInput said success. The app got question marks.
6. Turning "nothing leaves the machine" into a test
7. Testing a speech app with no microphone, network or model
8. A guard is judged by how rarely it fires
9. "No wait, I meant...": when self-correction ate my sentences
10. Voice commands without an LLM
11. Offline meeting transcripts with speaker labels
12. The update button that uninstalled a feature

If you want to try it:

# Linux
bash <(curl -fsSL https://raw.githubusercontent.com/MSKazemi/yazses/main/install.sh)
# macOS (Apple Silicon)
brew tap MSKazemi/yazses && brew trust MSKazemi/yazses && brew install --cask yazses
# Windows
winget install MSKazemi.YazSes
# Any OS with Python 3.11+
pipx install yazses

Linux is the most tested platform. macOS and Windows are labelled preview, and reports from real machines help more than anything else.

---

YazSes is open source under Apache-2.0: github.com/MSKazemi/yazses. The architecture reference, with the measurements above: docs/architecture.md.

I wrote this with help from an AI assistant for drafting and editing; the project, the measurements and the opinions are mine, and I checked every number against the repository.