DEV Community

Krish Verma
Krish Verma

Posted on

How I built streaming voice for my desktop AI companion: first audio in 491 ms, and a queue that reads ahead

Ankita AI is a desktop AI assistant by Krish Verma, and it talks back: when a background job finishes, it reads the result out loud. The problem was the wait before the first word. Buffered text-to-speech fetches the entire MP3, then plays it — so a three-sentence progress report sat silent while the provider finished rendering audio nobody had heard yet. I wanted the first word out while the rest was still arriving. The fix is streaming synthesis: on the live check the first audio chunk arrived in 491 ms, and the player was already speaking while the response was still open.

Prepare one chunk, not the whole file

The core of it is prepareVoiceStreamClip in src/channels/voice-stream-clip.mjs. It does not wait for the audio. It opens the Sarvam streaming request (synthesizeSarvamStream in src/channels/voice.mjs, model bulbul:v4-flash, one POST, subscription-key header, binary MP3 bytes), awaits exactly one chunk from the async iterator, and stops. That first chunk is the seed: a reusable clip descriptor holding the live iterator, its abort controller, and the measured firstChunkMs.

Playback then pulls from that iterator. playMp3Stream pipes chunks into the native player's stdin as they arrive, so the speaker starts on the first chunk instead of the last. When the stream ends, the chunks collected along the way are concatenated into a cached buffer — the whole clip, kept for free replays.

The honesty in the design is the fallback. If the provider answers with an unsupported status or content type before the first audio, the code falls back to the existing buffered Sarvam request and records fallback: 'provider'. But authentication and rate-limit failures stay errors, and a stream that dies halfway never replays the phrase through the buffered path — the user would hear it twice and never know which copy was real.

The queue reads the next sentence while this one plays

Streaming one sentence is only half the latency story. A report is several sentences, and fetching sentence two after sentence one finishes speaking puts the gap back. So the speech queue in src/automation/voice-speech-queue.mjs prepares the next part while the current one plays:

const nextPart = entry.parts[entry.cursor + 1];
if (nextPart) void this.preparePart(nextPart, signal).catch(() => {});
Enter fullscreen mode Exit fullscreen mode

Each part's prepare opens its own stream and waits for its own first chunk, so by the time the player finishes sentence one, sentence two's audio is already waiting. Text is split into sentences first and each chunk is capped at 2,000 characters (SARVAM_TTS_MAX_CHARS), the provider's per-request bound — the same chunker the buffered path already used, so streaming changed the transport, not the text handling.

Repeating a completed report costs nothing: the clip's cached buffer replays and the panel reports mode cached. Interrupting a sentence reopens its stream from the saved descriptor; sentences already delivered keep their cursor, so a resume does not restart the report. Ending voice, interrupting a report, or losing the native player aborts the owned cloud request, and late chunks from a dead session cannot publish delivery state or play. The abort signal is shared end to end — one AbortSignal.any joins the caller's signal and the clip's own controller — so cancellation is a single gesture, not a flag checked in five places.

The panel shows what actually happened

The voice panels in the desktop app display the observed delivery, not a prediction: { mode, fallback, firstChunkMs }. The modes are pending, streaming, buffered, and cached, and only that bounded object crosses IPC — no audio, provider errors, or credentials reach the renderer. If the provider fell back, the panel says so and names the reason. I would rather show "buffered" honestly than claim streaming that never happened.

What I measured

The focused suite covering the provider, player, clip, session, controls, and queue passes 162/162, and the renderer/service boundary checks pass 4/4 with zero added planner requests. The live journey — a synthetic phrase through the real desktop capture and report path, audio routed to an ffmpeg WAV decoder, then transcribed back — came back mode=streaming, firstChunkMs=567, firstConsumedMs=26, 4 chunks, 21,315 bytes in, 42,710 decoded WAV bytes out, transcript exactly "Voice streaming is ready." A re-verification on 10 November repeated it: firstChunkMs=491, firstConsumedMs=32, same chunk and byte counts, same transcript. No person's speech was recorded and nothing played through speakers; the microphone was injected synthetic input.

The limits matter as much as the numbers. Those two first-chunk timings are single samples, not a latency claim — I am not saying streaming is X% faster, because one run cannot prove that. Speech recognition is still the recorded-clip REST path: no live partial transcripts, and the app copy says words appear after a pause rather than pretending otherwise. Physical audio quality, Bluetooth routing, and real acoustic interruptions are untested, because verification used injected audio. The feature adds no dependency: Node streams, fetch, and the installed native audio binaries were enough.

Ankita is open source under MIT at github.com/akyourowngames/A.N.K.I.T.A, and the desktop app lives at ankita-sable.vercel.app. The streaming code is in src/channels/voice.mjs, src/channels/voice-stream-clip.mjs, and src/automation/voice-speech-queue.mjs, with the findings in docs/findings/everyday-voice-streaming.md.

Tell me I'm wrong: is preparing only the first chunk the right unit of work, or should the queue pre-fetch the whole report's audio up front and trade memory for zero mid-report gaps? And where would you draw the line on fallback — is silently dropping to buffered synthesis honest enough when the panel says so, or should a failed stream surface as an error instead?

— Krish Verma, creator of Ankita AI

Top comments (0)