DEV Community: theinterviewcopilot

Shaving a second off a real-time speech-to-LLM pipeline in Electron

theinterviewcopilot — Fri, 14 Aug 2026 23:18:07 +0000

Every part of a speech→LLM pipeline is fast enough on its own. Put them in a row and you get three seconds, which is far too slow when a human is waiting for you to say something.

I build a desktop overlay that listens to the other side of a video call, transcribes it, and streams an answer. The budget I care about is time from the speaker finishing a sentence to the first token on screen. Here is where that second and a half went, and what actually moved it.

The naive pipeline

mic/loopback → PCM → WebSocket STT → final transcript
→ "is this a question?" classifier → LLM → stream
Roughly 3.2s to first token. Four places to attack, and only two of them turned out to matter.

Stop sending silence The obvious win, and it is not about bandwidth — it is about the STT server's own endpointing. If you stream continuous audio, the server never sees a clean pause and delays its segment commit.

So VAD runs client-side in an AudioWorklet, on the raw signal before normalization:

this._vadThresh = 0.005; // RMS: above → speech
this._hangoverSec = 1.0; // keep sending after level drops
this._prerollMax = 14; // ~600ms @48k of buffered silence
Three parameters, and each one exists because of a specific failure:

Threshold on raw audio. I normalized first, and normalization amplifies room noise into "speech". Detect on the raw signal, normalize afterwards.
Hangover of 1.0s. Cutting the stream the instant RMS drops chops the tail off every sentence. It also has to exceed the server's own silence threshold (0.6s) with margin, or the server never gets the trailing silence it needs to commit the segment.
Pre-roll of ~600ms. A quiet sentence onset sits below the VAD threshold. By the time you detect speech, the first syllable is gone. So keep a rolling buffer of the last 14 chunks and flush it when VAD opens.
That last one is the difference between "what's a database index" and "at's a database index" — and the LLM answers the second one confidently and wrongly.

Let the STT server do endpointing ElevenLabs Scribe v2 Realtime supports commit_strategy: 'vad', so the server commits a segment after a pause instead of waiting for you to ask:

commit_strategy: 'vad',
vad_silence_threshold_secs: '0.6',
Combined with client-side gating, the transcript arrives while the person is still drawing breath.

One sharp edge: keyterm biasing caps at 50. Send 51 and the socket closes with code 1008 and a message you will not see unless you log close reasons. I lost an evening to that.

Reconnects, and the "muted socket" bug Long calls mean dropped sockets. Reconnecting is easy; reconnecting correctly is not.

The failure mode: a reconnect fires, and while it is in flight the user switches language, which triggers another reconnect. Now two sockets exist. The older one still delivers events, the newer one is the real connection, and the transcript interleaves garbage from both. Or worse, the stale one wins and the live one is silently ignored — audio flows, nothing appears, no error anywhere.

The fix is a generation counter:

const myGen = ++this.gen; // this connection
// ...later, in every handler:
if (myGen !== this.gen) return; // a newer connection superseded us
Every async continuation checks whether it is still the current generation. Anything from a superseded socket is dropped on the floor. It is four lines and it removed a whole class of "it just stops working after twenty minutes" reports.

Deleting the classifier (the counterintuitive one) The pipeline had a cheap model deciding "is this a question worth answering?" before spending a call on the expensive one. It cost ~200ms and it seemed obviously correct.

It was the single worst component in the system.

Not because of latency — because of what it got wrong. Real conversation is full of follow-ups that are not questions in isolation:

"What is SOLID?"
— answer —
"And the second letter?"

"And the second letter?" scores as not-a-question. The classifier stayed silent exactly when context made the intent obvious. Precision was fine; the failures were catastrophic and clustered in the most valuable moments.

I deleted it. The main model now sees every completed utterance plus recent context and decides for itself — it has the context the classifier never had. That removed 200ms and fixed the follow-up problem. Two wins from deleting code.

What replaced it is much dumber and works better: a client-side completeness gate.

const AUTO_SILENCE_MS = 900; // no "?" — pause mid-sentence
const AUTO_UTTEREND_MS = 120; // ends with "?" — react almost immediately
If the buffer ends in a question mark, the sentence is over — fire in 120ms. Otherwise wait 900ms in case they are just thinking. No model call, no network hop.

Prompt caching, and why the TTL matters more than the hit rate The system prompt carries a compressed CV card, so it is not small. Anthropic's prompt caching handles that — but the default TTL is 5 minutes, and interviews have pauses longer than 5 minutes. The interviewer talks, the candidate thinks, and the cache quietly expires. You then pay full price on exactly the questions that come after a long pause.

Two blocks, both at a 1-hour TTL:

const CACHE_1H = { type: "ephemeral", ttl: "1h" } as const;
// block 1: static persona — shared across every session and user
// block 2: session data (CV card, role) — stable for one interview
Splitting static from per-session matters: block 1 is identical for everyone, so it is warm before the user's first question. Get greedy and interpolate anything variable into it — the answer language, say — and you shatter one cache entry into sixteen.

Parallelise the gatekeeping Auth, rate limit, quota check, body parse. All independent, all were sequential.

const [, accessRes] = await Promise.all([
enforceRateLimit(db, userId, "ask", 15, 60),
assertAccess(db, userId, "ask"),
assertDailyCap(db, userId),
req.json().then((b) => { body = b; }),
]);
~92ms, for free, with identical guarantees. Unglamorous and the best ratio of effort to result in the whole list.

Pick the model on measurements, not vibes I assumed the frontier model was required. Measured across 10 technical questions:

cost / 500 requests full answer
Sonnet-class $3.54 8.6s
Haiku-class $1.03 5.2s
3.4× cheaper and 3.4 seconds faster. The price is roughly one factual slip every 5–6 questions and slightly rougher prose. For a real-time assistant where a late answer is worth zero, that trade is not close.

The enforcement point matters too: the client asks for a model, but the server owns the allow-list. Updates ship manually, so a client-side default would mean every already-installed copy keeps requesting the expensive model forever.

Where it landed
~1.1s from end-of-sentence to first token, from ~3.2s. The breakdown of what got it there is not what I expected going in:

deleting the classifier: −200ms and a correctness fix
VAD gating + server-side endpointing: −1.2s
parallel gatekeeping: −92ms
cheaper, faster model: −3.4s on full answer
The two biggest wins came from removing a component and from parameters in a 40-line audio worklet. Zero came from the LLM call itself, which is where I spent the first week.

What I would do differently
Log socket close codes from day one. The 1008 keyterm limit was invisible for a day because the close reason was never surfaced.

Treat "it silently stopped" as the default failure. Every quiet path — a superseded socket, an expired cache, a swallowed catch — looks identical to "working" from the outside. The generation counter and an idle watchdog on the stream exist because both failed silently first.

Measure before assuming the model is the bottleneck. It was 200ms of the 3.2s.

The overlay is Interview Copilot — Electron on the client, Supabase Edge Functions on the server so the STT and model keys never reach the desktop app. Happy to go deeper on any part of the pipeline in the comments.

I built Interview Copilot — a real-time desktop overlay that listens, thinks, and answers in under a second. Here's the full engineering story.

theinterviewcopilot — Sun, 19 Jul 2026 18:56:55 +0000

I spent the last few months building Interview Copilot — a desktop app (Electron) that listens to a live conversation, transcribes it in real time, figures out when a real question has been asked, and streams an answer onto the screen, hands-free. No buttons, no tab-switching — someone finishes a sentence and the answer is already appearing.

It sounds simple. It is not. The whole product is one long fight against latency and against the messiness of real speech. This is the write-up I wish I'd had when I started. It's long — grab a coffee.

The pipeline

Here's the full path a spoken sentence travels before an answer shows up:

system audio (loopback capture)
→ AudioWorklet: voice-activity detection + PCM normalization
→ streaming speech-to-text over WebSocket (partial + final transcripts)
→ a debounce / "is this a complete thought?" gate
→ LLM request (SSE streaming) on a serverless edge function
→ token-by-token markdown render on a transparent overlay

Every arrow is a place where milliseconds and correctness go to die. The counterintuitive lesson up front: the LLM was NOT my bottleneck. The boring, fixed, client-side delays added up to more than the model's time-to-first-token.

1. Capturing audio you don't own

I don't capture a microphone — I capture the other side of the conversation, i.e. the system/loopback audio. In Electron that means getDisplayMedia with audio, piped into an AudioWorklet running on the audio thread. The worklet does two jobs: cheap energy-based voice-activity detection (so I'm not shipping silence over the wire) and PCM normalization/resampling into the exact format the STT expects.

Doing this on the audio thread, not the main thread, matters more than it sounds. Any jank on the main thread while rendering an answer would show up as dropped audio frames — and a dropped frame mid-question means a garbled transcript means a wrong answer. The worklet isolates the hot path.

2. Streaming transcription, and the real hard problem

I use a streaming STT (currently ElevenLabs Scribe v2 Realtime) over a WebSocket. It emits two kinds of results: partials (updated live as you speak, constantly changing) and finals (committed after a pause). You can render partials to show "it's listening," but you can NEVER act on them — they mutate.

The genuinely hard problem isn't transcription accuracy. It's this: when is a thought actually finished? People don't speak in clean sentences. They pause. "Okay… so… let's talk about databases… how would you…" — each of those pauses can produce a separate final transcript. Fire on the first one and you confidently answer a fragment. Wait too long and the whole thing feels dead and useless.

3. The "complete thought?" gate — my single biggest win

What finally worked was a two-path debounce:

If the accumulated text ends in "?", the question is almost certainly complete → fire fast (~120ms).
Otherwise the speaker might still be mid-thought → wait longer (~900ms) so a continuation can glue on.

Any new speech before the timer fires resets it. I also drop pure filler — "okay, thanks, got it," "let me grab a coffee" — so it never sticks to the next real question and never burns a model call on nothing.

I tuned those two numbers DOWN over time (from 200/1100ms) and it shaved time off literally every answer. The only reason I could do that safely: I have an end-to-end test that replays nasty "paused speech" transcripts and asserts the questions don't get chopped into pieces. Without that test I'd have been tuning by vibes and shipping regressions.

Design lesson: in a real-time UX, the fixed debounce you picked by feel is probably your single largest latency cost. Measure it, test it, then shrink it.

4. Answer on EVERYTHING, and never lose one

Early on I had a separate "is this technical?" classifier gate. I ripped it out. It kept staying silent on legitimate follow-ups ("and the second letter?" after an acronym question). Now the system answers every completed, non-filler utterance and lets the main model itself decide how to respond. Fewer moving parts, fewer silent failures.

Two invariants keep it sane under fire:

A single "answering" guard so only one answer is ever in flight (kills duplicates from a hotkey + auto firing together).
A queue: any question that arrives WHILE an answer is streaming gets queued and handled next, in order. Nothing is dropped, nothing is merged.

5. First-paint render — a few lines, a real feel

The answer streams as markdown, token by token. The trap: re-parsing the whole buffer on every token is O(n²) and janks on long answers, so I batched renders through requestAnimationFrame. Correct — but it made the very first token wait up to a full frame (~16ms), which is exactly the moment the user is staring at an empty bubble waiting for any sign of life. Fix: render the first delta synchronously, batch the rest. Perceived latency is dominated by "time until something appears," not total time.

6. Backend latency levers

The model runs behind serverless edge functions. Three things mattered most:

Prompt caching with a 1-hour TTL. The system prompt + context block are large and static across a session; caching them means repeat calls skip re-processing a big prefix — lower cost AND lower time-to-first-token. Gotcha: the default 5-minute cache TTL kept expiring during natural pauses, so the 4th question would mysteriously go slow again. Bumping to 1h fixed it.
SSE streaming end to end, so the first token paints the instant the model emits it.
Keep bookkeeping off the critical path — usage/analytics logging happens after the stream closes, never before it starts.

7. Resilience — the part nobody demos but everybody hits

A long-lived WebSocket WILL drop mid-session. What saved me:

Reconnect with backoff, then a slow background retry that never fully gives up (the early version died after N attempts and left the user dead in the water).
Buffer audio during the reconnect gap (~6s) and flush it in order, so a network blip doesn't eat a sentence.
Flush any uncommitted partial transcript as a final on unexpected close.
A generation counter so a stale socket's late "close" event can't clobber the state of the fresh connection. That bug took me an embarrassingly long time to find.

8. The little things that make it feel real

Tech-term correction: English terms spoken inside another language get mangled by STT ("майкросервіси" → microservices). A post-processing layer plus keyterm biasing fixes the common ones.
Multilingual: 7 UI languages, and the answer language is independent of the recognition language.
It's a transparent, always-on-top overlay that stays out of your way and is excluded from screen capture at the OS level.

Takeaways

Instrument time-to-first-token BEFORE optimizing. My gut said "the model is slow." It was the fastest part.
In a real-time pipeline, fixed client-side delays (debounce, render frames) can dominate the variable network cost.
Prompt caching is a latency tool, not just a cost tool.
Write the replay/E2E test before tuning the magic numbers, or you regress silently.
Perceived latency ≠ total latency. Optimize the moment the first pixel changes.

Stack: Electron, AudioWorklet, WebSocket streaming STT, serverless edge functions, SSE, vanilla JS on the renderer (zero framework overhead on the hot path).

It's called Interview Copilot. Happy to go deep on any piece — the VAD worklet, the reconnect/generation-counter logic, the caching setup, or how the "complete thought" gate is tuned. And I'm genuinely curious: for a speech-triggered UI, where would you spend the next 100ms of your latency budget?

Interview Copilot — Real-Time AI Co-Pilot for Any Interview

A real-time AI copilot for any interview, technical or not. It hears the interviewer and answers live — invisible on Zoom, Meet and Teams screen share.

theinterviewcopilot.com