Building Real-Time Dictation from Scratch: Escaping the Offline Trap

Shantanav Kapse — Sun, 09 Aug 2026 19:14:16 +0000

When the requirement first landed on my desk, it sounded simple enough: Build a real-time, cursor-based dictation feature. The user speaks into their browser, and the words appear live at the text cursor so they can draft long documents.

But there was a catch. The domain used highly specialized vocabulary, and the audio was strictly regulated. Sending the data to a slick cloud API was out of the question-audio could never leave our servers.

This is the story of my first deep dive into the trenches of speech-to-text (STT) engineering. Here is what I explored, the brutal obstacles I hit, and the architecture I ultimately shipped.

The Core Choice: Faking It vs. Native Streaming

My first architectural crossroad was choosing the model family. I initially looked at offline attention encoder-decoder models. They are famous for their high accuracy. However, they are trained to transcribe a complete audio file.

To use an offline model for "live" dictation, you have to fake it. You maintain a rolling window of audio (say, the last few seconds), and every 0.5 seconds, you re-transcribe the entire window. To decide what text to actually keep, you wait until two consecutive transcriptions agree on a word (local agreement) and then commit it.

I quickly realized this was a trap. It re-transcribes overlapping audio constantly, melting CPU cycles. Worse, when fed absolute silence, these offline models hallucinated wildly, inventing stock phrases or sign-offs. Because the hallucination was stable across passes, the system would permanently commit cascades of repeated ghost words.

The Pivot: I ditched the offline approach and moved to streaming neural transducers (the RNN-T family).

Transducers are built differently. They emit tokens as the audio arrives and have a built-in notion of "endpointing" (detecting a pause to mark the end of an utterance). They are natively real-time, mathematically stable, and most importantly, they don't hallucinate text out of thin air.

Key Takeaway: If you are building live dictation, force-fitting an offline model fights the use case. A streaming transducer is the right tool for the job.

Nailing the UX: The Two-State Contract

To make the live cursor feel magical rather than jarring, I needed a rock-solid UI contract. A jumping, flickering text cursor is incredibly frustrating for users drafting long documents.

I implemented a two-state output contract where every WebSocket update carries two fields: {committed, buffer}.

Committed: Stable text the client appends permanently (rendered in a normal font weight).
Buffer: The model's current provisional guess (rendered in grey or italics, replacing itself on every update). Because the streaming model handles endpointing naturally, it fires off the final text when the user pauses. The client promotes the buffer to committed, and the stream resets. The stable text never jumps backward, and the provisional tail visibly settles.

Obstacle 1: The Silent WebSocket Killer

Getting the browser to capture audio was straightforward. The browser records microphone audio as Opus-compressed chunks and streams them over a WebSocket every ~250 milliseconds. The server then decodes this into the raw 16 kHz mono PCM the model requires.

Then came the hardest engineering bug of the project.

The dictation would work perfectly for a sentence or two, and then the WebSocket would randomly disconnect. No errors, just a dropped connection.

I traced it to a concurrency and back-pressure failure. Initially, a single task was both reading the decoded audio and running the STT model inline. While the model was busy crunching numbers, nothing was draining the audio decoder's output pipe. The pipe filled up, the decoder stopped pulling input, the upstream socket blocked, and the WebSocket receive loop stalled. The keepalive ping/pong stopped, and the connection died.

The Fix: I decoupled the pipeline into two isolated tasks.

A Reader: Continuously drains the decoder into an in-memory buffer so the pipe never blocks.
A Processor: Consumes that buffer and runs the model at its own pace.

Once decoupled, back-pressure never reached the socket.

Obstacle 2: Formatting and Domain Jargon

Documents need capital letters and periods. Many streaming models emit normalized text (lowercase, no punctuation).

I had two choices: run a secondary punctuation-restoration model on the committed text, or find a model that handles casing natively. I opted for the latter. By checking the model's token vocabulary for mixed-case pieces and punctuation tokens, I bypassed the need for a secondary formatting layer entirely.

Handling specialized vocabulary was trickier. General models stumble on niche jargon. While contextual biasing (word boosting) is great, the lightweight runtime for my streaming model only supported greedy decoding, meaning acoustic biasing was unavailable.

I solved this pragmatically with a Post-ASR Correction Map-a simple dictionary of {misheard -> correct} terms applied to the committed text. It isn't as elegant as acoustic biasing, but it reliably catches repeatable mistakes based on actual user data.

The Deployment Reality: CPU over GPU

You might assume a real-time AI model requires massive GPU power. In reality, the streaming transducer is incredibly efficient.

The Real-Time Factor (RTF) is processing time divided by audio duration. For a ~0.6 billion parameter streaming model, I measured an RTF of roughly 0.09 on a standard laptop CPU. That is 11x real-time headroom.

This led to a counter-intuitive deployment strategy:

Keep it off the GPU: Co-locating this STT model with a large LLM on a single GPU is a bad idea (I already had an LLM deployed on the server, but the CPU was not utilized). The bursty generation of the LLM will stall the latency-sensitive audio stream, causing audible stuttering.
Run on CPU: The ASR runs comfortably on spare CPU cores. I wrapped it in a container with hard CPU and memory limits so it couldn't physically exceed its lane or trigger an Out-Of-Memory (OOM) crash on the server.

Final Thoughts

Building this from scratch taught me that the hardest parts of applied AI often aren't the neural networks themselves-they are the plumbing, the back-pressure, and the UX contracts.

Trade-off	Cloud API	Local Streaming Transducer (My Build)
Data Privacy	Audio leaves premises	Stays entirely on-prem
Cost	Per-minute billing	Free / Compute only
Accuracy / Drift	Vendor-managed, highly tuned	Requires manual correction maps
Latency	Network dependent	Near-instant (RTF < 0.1 on CPU)

Choosing to self-host a streaming model meant giving up the automatic updates and easy jargon-boosting of a cloud API. But for a privacy-regulated environment, it was the only honest trade. We achieved true real-time, zero-cost, private dictation-and the cursor feels just right.

DEV Community: Shantanav Kapse