Chunking Whisper into 1-second pieces makes dictation slower

I'm building Vocs, a free, open-source dictation app for macOS (Tauri, Rust, whisper.cpp). The goal sounds simple: press a key, talk, press it again, and the text is pasted before you can look up. That means the time from stop to paste has to stay flat, whether you spoke for five seconds or five minutes.

Getting there meant unlearning a few things about Whisper. These are the ones that cost me the most time.

1. Small chunks are the slow option

The obvious way to cut latency is to transcribe in small pieces while the person talks. It backfires. Whisper's encoder always works on a 30-second window, and whisper.cpp pads shorter audio up to 30 seconds. A 1-second chunk costs almost as much to encode as a 25-second one, so chunking at 1 second is roughly 10x the total compute of chunking at 10.

What works instead:

• Cut chunks at silence boundaries, not on a timer. I use a small pure-Rust WebRTC VAD (the earshot crate), which is enough for a low-accuracy job like finding pauses.
• Close the first chunk at the first pause after 4 seconds of speech, and cap chunks at 12 seconds.
• Transcribe them in the background while the person is still talking. When they press stop, only the last fragment is left to decode.

2. Shrink the encoder with audio_ctx

whisper_full_params.audio_ctx truncates the encoder's context. The default, 1500, is the full 30 seconds. Scale it to the fragment instead (the same trick whisper.cpp's own stream example uses):

audio_ctx = clamp(ceil(duration_seconds / 30 * 1500) + 128, 256, 1500)

A 1.7-second fragment decodes in 221 ms at audio_ctx 256 and 1783 ms at 1500. That is 8x faster for the same text.

My first version kept the full context for the background chunks, on the theory that their latency was hidden. It wasn't. A full-context decode took 2.3 seconds, and when I pressed stop during one, the final fragment waited behind it: 2.6 seconds to paste. Every chunk uses the shrunk context now.

3. Two settings that were fine alone and a bug together

By default whisper.cpp retries a failed decode at rising temperatures, up to 6 extra passes. A normal 200 ms decode can turn into 1.4 seconds with no warning, so I turned it off (temperature_inc = 0.0). I also set no_timestamps = true, because I don't use word timings and it looked like free speed.

Same utterance, same machine:

no_timestamps	Tail decode	Output
true	340 ms	1019 characters: the sentence repeated 24 times
false	113 ms	41 characters, correct

Timestamp tokens are how the decoder knows when to stop. Without them it keeps re-emitting the sentence until it hits the token limit. The temperature-fallback loop would normally notice the repetition and retry, but I had switched that off too. Each setting made the other dangerous, and the combination that looked fastest on paper shipped a 24x repeated paste. no_timestamps = false is both faster and correct.

4. Assume the decoder will emit garbage

With fallback off, nothing breaks a decoder loop on its own. Real rows from my history: *Scoffs*, *punch*, and a 2.4-second recording returned as 217 words of "the one who is...". The defences are plain code:

• Never send a chunk with no detected speech to the model. Most invented "thanks for watching" text comes from silence.
• Strip sound-effect tags: *...*, [...] and (...).
• Collapse a phrase repeated 3 or more times (a single word, 5 or more). Retry the chunk once, and if it loops again, discard it as non-speech.
• Set no_context = true between chunks, so decoder state never carries across a seam. Carrying it is a common cause of "it repeated the same sentence 40 times".

5. The orange dot is a design constraint

A real pre-roll buffer, so the first syllable is never clipped, needs the microphone open before the key is pressed. On macOS that keeps the orange privacy indicator lit all the time, and most people will not accept that from a background app. So the default opens the mic on the keypress, which takes roughly 20 to 80 ms, and an optional "keep it warm" mode trades the always-lit indicator for never clipping a word.

What I'd take from this

• Measure the tail, not the average. Only the last fragment is on the critical path.
• Benchmark parameter changes on real speech. Two of my "free" speedups were wrong, and one of them only showed up as output, not as a slow decode.
• Treat decoder output as untrusted input, the way you would user input.
• Defaults that look slow can be protecting you.

If you've shipped something on top of whisper.cpp, which setting surprised you most? I'm especially curious how others handle repetition loops without paying for temperature fallback.

Vocs is MIT-licensed, and everything above is in the repo: Github