Where the time goes in a live AI call assistant: my code takes 150 ms, the model takes the rest

I'm building a Windows app that listens to both sides of a call, transcribes it and drafts a short answer while the other person is still talking. People use it for job interviews and for calls in a second language. The first thing I wanted to fix was speed: the answer often arrived after the moment had passed.

Before touching the code, I logged every stage of every answer. This post is what 2,314 completed answers from my own logs say about where the seconds go.

The pipeline

1. Speech recognition streams partial text, then a final transcript for each utterance.
2. The client waits briefly for the speaker to finish, then sends a request.
3. The server builds the prompt: the last 30 messages of the conversation (capped at 12,000 characters), plus the user's CV and the job description.
4. The provider streams tokens back, and the server relays them to the UI over Server-Sent Events.

Every answer ends with one log line:

[SSE quick] done via groq final_to_request=5ms prepare=78ms ttft=1188ms total=1329ms chunks=62

• final_to_request: from the final transcript to the request leaving the client
• prepare: server work before the provider call (prompt assembly, context trimming)
• ttft: time to the first token from the provider
• total: time to the last token

Two answers per question

Each question fires two requests in parallel. The quick slot returns a sentence or two you can say straight away. The detail slot returns a fuller answer to read while you're already talking. The slots can use different providers, so the fast answer never waits for the thorough one.

The numbers

Medians for every slot and provider with at least 50 answers:

Slot / provider	n	prepare	first token	total
quick / Groq	638	234 ms	813 ms	937 ms
quick / Groq, second key	60	203 ms	1,351 ms	1,382 ms
quick / Cloudflare Workers AI	131	203 ms	1,109 ms	1,468 ms
quick / Gemini API	341	94 ms	3,063 ms	3,282 ms
quick / Claude Code CLI	52	148 ms	5,047 ms	7,250 ms
detail / Gemini API	96	125 ms	4,054 ms	5,117 ms
detail / Claude Code CLI	185	109 ms	4,203 ms	7,641 ms
detail / Codex CLI	706	172 ms	7,062 ms	10,188 ms

Across all 2,314 answers, prepare has a median of 157 ms and a 90th percentile of 609 ms. Time to first token ranges from 0.8 s to 7 s depending on the provider.

So the part I was about to optimise is roughly 150 ms. Moving the quick slot from Gemini to Groq saves more than two seconds on every answer, and that's a settings change, not a code change.

Why the CLI rows are slow

The Codex and Claude rows go through the vendors' command-line tools, which lets people use a subscription they already pay for instead of an API key. Each call starts a new CLI session, and most of the wait is on the vendor's side: in a separate test, keeping a Codex process warm barely beat starting a cold one. These work for the detail slot. They're too slow for the quick one.

The 1.2 seconds I added on purpose

In the earliest logs, final_to_request was 4 to 6 ms. In most recent rows it's about 1.2 s, and that delay is deliberate.

Speech recognisers sometimes split one sentence into two finals. Deepgram once sent "Can you tell me a little about your experience?" and then, 329 ms later, "about your experience?" as a separate final. The app drew a second bubble and cancelled the answer it had just started.

There are two fixes in place now:

• A tail check: a new final from the same speaker that is the last three or more words of the previous one, arriving within 2.5 s, gets dropped.
• A 1.2 s debounce after each final before asking the model, so a question split by a pause becomes one request.

That costs 1.2 s on every answer in exchange for no duplicate or cancelled answers. I haven't settled whether that's the right trade.

Skipping the long answer for small talk

"Thanks, nice to meet you" doesn't need a 100-word answer. The detail request is skipped when an utterance has no question mark, is 12 words or fewer, opens with a greeting or thanks, and contains no question word. That skips a slow request nobody would read.

Speech recognition comes on top

All of the numbers above start at the final transcript. Recognition adds its own delay before that, about a second with streaming cloud STT in my tests. With Groq in the quick slot, that puts the first words of an answer around three seconds after the speaker stops: one second of recognition, 1.2 s of debounce, a fifth of a second of prep and under a second to the first token. Without the debounce it would be closer to two.

What I took from this

• Log every stage before optimising. Without a ttft field I'd have spent a week on a part that takes 150 ms.
• Put the fastest provider in the slot people wait for, and let the slow, thorough one run in parallel.
• Clean up recogniser output before it reaches the model. A split utterance costs a cancelled request and a second answer.

The usual caveats apply. These are medians from one person's usage on one connection. The models behind each provider changed during the period, and prompt size depends on how much CV and job text is loaded.

The app is Talkaside, a Windows interview assistant and live call translator.

Drafted with AI assistance. The numbers come straight from the app's logs.