I'm building LiveSuggest, a real-time meeting assistant. It listens to your call and shows you written suggestions while you're still talking: a clarifying question to ask, a point you forgot to make, the definition of a term someone just dropped. No bot joins the call. Everything runs from your own browser.
The whole product lives or dies on one number: how long between someone saying something and a useful suggestion appearing on screen. A suggestion that arrives after the conversation has moved on isn't just useless, it's a distraction.
So I assumed the language model would be my enemy. LLMs are the slow, expensive part, right? I instrumented the entire pipeline in production, expecting to spend my life fighting token latency.
I was wrong about where the time goes. Here's what a week of production metrics actually showed me.
The pipeline
Before the numbers, the shape. A suggestion goes through five stages:
- Capture. The browser grabs audio in 100ms chunks and streams them over a WebSocket. (I wrote about the capture side in a previous post.)
- Transcription. A streaming speech-to-text service turns those chunks into text and tells me when a sentence is "finished."
- Gating. I decide whether there's enough new material to be worth a suggestion yet.
- Generation. If yes, I build a prompt and stream a suggestion out of an LLM.
- Delivery. The suggestion streams back over the WebSocket and renders.
I have production metrics on stages 2, 3 and 4. Let me walk them in the order I expected each one to hurt.
Stage 4: the LLM (the part I was scared of)
Real numbers, measured in production over 7 days, streaming enabled:
| Model | TTFT p50 | TTFT p95 | Full suggestion p50 |
|---|---|---|---|
| gpt-4o-mini | 1.17s | 2.40s | 1.96s |
| Claude Haiku 4.5 | 1.20s | 2.43s | 1.91s |
One honesty note on that TTFT: it's time to the first chunk I stream to the client, measured server-side. It does not include the network hop back to the browser or React rendering. More on why that distinction matters at the end.
So the model I was afraid of gives me a first token in about 1.2 seconds at the median. That's not nothing. Hold that number.
A quick word on model choice, because it's the one place the obvious instinct turns out to be right. I tested a bigger, smarter model (gpt-5-mini) on the same prompts. TTFT p50 jumped to 3.74s, p95 to almost 9 seconds. For a real-time feature that's disqualifying. A smarter suggestion that lands four seconds late loses to a decent suggestion that lands now, every time. When your output has an expiry date, you pick the fast model, not the smart one.
(I wanted Gemini 2.5 Flash in this comparison too. I couldn't: it had zero production traffic in my measurement window, so I have no honest numbers for it. I'm not going to invent some.)
Stage 2: transcription (fast median, ugly tail)
Speech-to-text surprised me in a different direction. The median time from "someone starts speaking" to "I have a finalized transcript" is around 75 milliseconds. Genuinely fast.
But the 95th percentile is close to nine seconds. That gap isn't the model being slow, it's a decision I configured. Streaming STT doesn't magically know when a sentence ends. It waits for a short pause of silence (a few hundred milliseconds) before declaring a segment final. If the speaker never pauses (a monologue, an excited customer, someone reading a list out loud), nothing finalizes, so I force a cutoff after a fixed maximum.
endpointing: 0.5, // pause in seconds that ends a segment
maximum_duration_without_endpointing: 10, // hard cutoff if nobody pauses
The lesson: the median told me STT was solved. The tail told me my real latency knob was endpointing, and it's a tradeoff, not a bug to fix. Detecting shorter pauses gives you faster finals but more fragmented, lower-quality transcripts.
Stage 3: gating (the first real surprise)
Here's the stage I hadn't even counted as "latency" until I laid out the budget.
I don't send every finalized sentence to the LLM. If I did, you'd get a suggestion every three seconds and hate the product inside a minute. So before generating anything, I wait until enough has been said to be worth interrupting you.
FIRST_SUGGESTION_WORD_THRESHOLD = 12 // words before the first suggestion
DEFAULT_WORDS_BETWEEN_SUGGESTIONS = 33 // words between suggestions after that
At a normal speaking pace of about 130 words per minute, waiting for 12 words takes roughly 5.5 seconds. Waiting for 33 words takes about 15.
Look at that next to the LLM number. The model I was terrified of contributes about 1.2 seconds. The gate I barely thought about contributes 5 to 15. The single largest component of "time until you see a suggestion" isn't compute at all. It's a product decision about how much context is worth waiting for.
And it should be. This is the part I had backwards. I set out to optimize latency, but most of my latency is a feature. A suggestion fired off half a sentence is noise. The wait is what makes the suggestion good.
Putting the budget together
I don't have a single end-to-end metric (something I'm fixing). But composing the stages, for a first suggestion under normal conditions:
- accumulate about 12 words of speech: ~5.5s
- finalize the transcript (endpointing): ~0.5s
- LLM first token: ~1.2s
- stream the rest: ~1s
That's roughly 8 to 10 seconds from "you start making a point" to "a suggestion appears," climbing past 15 if the speaker never pauses for breath. This is a back-of-the-envelope figure, not a measured one, but the shape is the point: the LLM is maybe 15% of it. The rest is deliberate waiting plus the long tail of endpointing.
If you'd asked me before I measured, I'd have blamed the model for 80% of the latency. It's closer to 15%.
The bottleneck nobody warned me about
Everything above is latency you can feel. But the profiling turned up a second bottleneck that had nothing to do with speed and everything to do with running the same cheap operation ten times a second.
Remember stage 1: audio arrives in 100ms chunks. That's 10 WebSocket events per second, per active session. On every single one of those events, my handler was reading the full session object (about 4.7KB) from Redis. Not once. Four times, from four different places that each independently wanted the session state:
- session validation
- consent check
- quota check
- ownership check
Four reads, times ten chunks a second, times every active session. Over 30 days that came to around 14 million reads and roughly 66GB of Redis bandwidth, which was 132% of my quota. My CPU was fine. The LLM was fine. STT was fine. I was quietly blowing past my Redis limit re-reading state I had loaded milliseconds earlier.
The fix was embarrassingly boring. Load the session once, at validation, and pass the object down instead of re-fetching it:
// before: four independent getContext() reads per chunk
// after: one read, reused everywhere
const session = sessionValidation.session;
// hand `session` to the consent, quota and ownership checks
Four reads became one. That lesson stuck with me harder than any of the latency numbers: in a real-time pipeline running at 10 events per second per user, the thing that bites you is not the glamorous compute. It's the boring per-event I/O you do without thinking, multiplied by a frequency you're not picturing.
What I'd tell myself at the start
Five things, in rough order of how much they surprised me:
- In a real-time LLM feature, the LLM is usually not your latency bottleneck. Measure the whole pipeline before you optimize the part you assume is slow.
- Your largest latency component might be a product decision (how much context to wait for), not a technical cost. Don't "fix" it. Tune it on purpose.
- Watch the tail, not just the median. My STT median was 75ms and my p95 was nine seconds, and the p95 is what people remember.
- Pick the fast model, not the smart one, when the output has an expiry date.
- At 10 events per second per user, redundant state reads dwarf your compute cost. Count your per-event I/O.
And one on measurement itself: know exactly what your metric includes. My TTFT is measured server-side and stops at the first chunk I emit. It does not cover the network trip back to the browser or rendering. That's fine, as long as I never confuse it with what the user actually experiences. The number you don't define precisely is the number that lies to you.
If you're building anything real-time on top of an LLM, I'd like to compare notes. What dominated your latency budget? Drop it in the comments.
I'm building LiveSuggest, a real-time meeting assistant that gives you suggestions during a call without a bot joining it. The audio-capture side is here if you want the earlier chapter.
Top comments (1)
Great instrumentation discipline, and the honesty note about TTFT being server-side measured — excluding the network hop and render — is worth more than the table itself. Most latency posts quietly measure the segment that flatters them.
The finding that isn't really about the LLM at all is the STT tail: 75ms median, ~9s p95, driven by a silence threshold you configured. That's the interesting shape, because it means your p95 isn't a performance problem, it's a product problem — you're paying nine seconds to be sure the sentence ended. We've hit the same thing in a different domain and the only thing that helped was giving up on the binary "finished" signal and generating speculatively on partials, then revising. Expensive, and only worth it because a late-but-correct output was worth less than an early-and-revised one.
Given that your gating stage (3) is where you decide whether to spend, does the gate ever see the tail coming — i.e. can it fire on a partial when the finalization is late?