I spent the last few months building Interview Copilot — a desktop app (Electron) that listens to a live conversation, transcribes it in real time, figures out when a real question has been asked, and streams an answer onto the screen, hands-free. No buttons, no tab-switching — someone finishes a sentence and the answer is already appearing.
It sounds simple. It is not. The whole product is one long fight against latency and against the messiness of real speech. This is the write-up I wish I'd had when I started. It's long — grab a coffee.
The pipeline
Here's the full path a spoken sentence travels before an answer shows up:
system audio (loopback capture)
→ AudioWorklet: voice-activity detection + PCM normalization
→ streaming speech-to-text over WebSocket (partial + final transcripts)
→ a debounce / "is this a complete thought?" gate
→ LLM request (SSE streaming) on a serverless edge function
→ token-by-token markdown render on a transparent overlay
Every arrow is a place where milliseconds and correctness go to die. The counterintuitive lesson up front: the LLM was NOT my bottleneck. The boring, fixed, client-side delays added up to more than the model's time-to-first-token.
1. Capturing audio you don't own
I don't capture a microphone — I capture the other side of the conversation, i.e. the system/loopback audio. In Electron that means getDisplayMedia with audio, piped into an AudioWorklet running on the audio thread. The worklet does two jobs: cheap energy-based voice-activity detection (so I'm not shipping silence over the wire) and PCM normalization/resampling into the exact format the STT expects.
Doing this on the audio thread, not the main thread, matters more than it sounds. Any jank on the main thread while rendering an answer would show up as dropped audio frames — and a dropped frame mid-question means a garbled transcript means a wrong answer. The worklet isolates the hot path.
2. Streaming transcription, and the real hard problem
I use a streaming STT (currently ElevenLabs Scribe v2 Realtime) over a WebSocket. It emits two kinds of results: partials (updated live as you speak, constantly changing) and finals (committed after a pause). You can render partials to show "it's listening," but you can NEVER act on them — they mutate.
The genuinely hard problem isn't transcription accuracy. It's this: when is a thought actually finished? People don't speak in clean sentences. They pause. "Okay… so… let's talk about databases… how would you…" — each of those pauses can produce a separate final transcript. Fire on the first one and you confidently answer a fragment. Wait too long and the whole thing feels dead and useless.
3. The "complete thought?" gate — my single biggest win
What finally worked was a two-path debounce:
- If the accumulated text ends in "?", the question is almost certainly complete → fire fast (~120ms).
- Otherwise the speaker might still be mid-thought → wait longer (~900ms) so a continuation can glue on.
Any new speech before the timer fires resets it. I also drop pure filler — "okay, thanks, got it," "let me grab a coffee" — so it never sticks to the next real question and never burns a model call on nothing.
I tuned those two numbers DOWN over time (from 200/1100ms) and it shaved time off literally every answer. The only reason I could do that safely: I have an end-to-end test that replays nasty "paused speech" transcripts and asserts the questions don't get chopped into pieces. Without that test I'd have been tuning by vibes and shipping regressions.
Design lesson: in a real-time UX, the fixed debounce you picked by feel is probably your single largest latency cost. Measure it, test it, then shrink it.
4. Answer on EVERYTHING, and never lose one
Early on I had a separate "is this technical?" classifier gate. I ripped it out. It kept staying silent on legitimate follow-ups ("and the second letter?" after an acronym question). Now the system answers every completed, non-filler utterance and lets the main model itself decide how to respond. Fewer moving parts, fewer silent failures.
Two invariants keep it sane under fire:
- A single "answering" guard so only one answer is ever in flight (kills duplicates from a hotkey + auto firing together).
- A queue: any question that arrives WHILE an answer is streaming gets queued and handled next, in order. Nothing is dropped, nothing is merged.
5. First-paint render — a few lines, a real feel
The answer streams as markdown, token by token. The trap: re-parsing the whole buffer on every token is O(n²) and janks on long answers, so I batched renders through requestAnimationFrame. Correct — but it made the very first token wait up to a full frame (~16ms), which is exactly the moment the user is staring at an empty bubble waiting for any sign of life. Fix: render the first delta synchronously, batch the rest. Perceived latency is dominated by "time until something appears," not total time.
6. Backend latency levers
The model runs behind serverless edge functions. Three things mattered most:
- Prompt caching with a 1-hour TTL. The system prompt + context block are large and static across a session; caching them means repeat calls skip re-processing a big prefix — lower cost AND lower time-to-first-token. Gotcha: the default 5-minute cache TTL kept expiring during natural pauses, so the 4th question would mysteriously go slow again. Bumping to 1h fixed it.
- SSE streaming end to end, so the first token paints the instant the model emits it.
- Keep bookkeeping off the critical path — usage/analytics logging happens after the stream closes, never before it starts.
7. Resilience — the part nobody demos but everybody hits
A long-lived WebSocket WILL drop mid-session. What saved me:
- Reconnect with backoff, then a slow background retry that never fully gives up (the early version died after N attempts and left the user dead in the water).
- Buffer audio during the reconnect gap (~6s) and flush it in order, so a network blip doesn't eat a sentence.
- Flush any uncommitted partial transcript as a final on unexpected close.
- A generation counter so a stale socket's late "close" event can't clobber the state of the fresh connection. That bug took me an embarrassingly long time to find.
8. The little things that make it feel real
- Tech-term correction: English terms spoken inside another language get mangled by STT ("майкросервіси" → microservices). A post-processing layer plus keyterm biasing fixes the common ones.
- Multilingual: 7 UI languages, and the answer language is independent of the recognition language.
- It's a transparent, always-on-top overlay that stays out of your way and is excluded from screen capture at the OS level.
Takeaways
- Instrument time-to-first-token BEFORE optimizing. My gut said "the model is slow." It was the fastest part.
- In a real-time pipeline, fixed client-side delays (debounce, render frames) can dominate the variable network cost.
- Prompt caching is a latency tool, not just a cost tool.
- Write the replay/E2E test before tuning the magic numbers, or you regress silently.
- Perceived latency ≠ total latency. Optimize the moment the first pixel changes.
Stack: Electron, AudioWorklet, WebSocket streaming STT, serverless edge functions, SSE, vanilla JS on the renderer (zero framework overhead on the hot path).
It's called Interview Copilot. Happy to go deep on any piece — the VAD worklet, the reconnect/generation-counter logic, the caching setup, or how the "complete thought" gate is tuned. And I'm genuinely curious: for a speech-triggered UI, where would you spend the next 100ms of your latency budget?
Interview Copilot — Real-Time AI Co-Pilot for Any Interview
A real-time AI copilot for any interview, technical or not. It hears the interviewer and answers live — invisible on Zoom, Meet and Teams screen share.
theinterviewcopilot.com
Top comments (0)