Two streaming speech-to-text bugs that left no error: binary frames and keyterm budgets

Our desktop app streams microphone and call audio to cloud speech-to-text over WebSockets from a Rust engine. Two of the worst bugs in that layer had the same shape: everything looked healthy, no error was logged, and the output was wrong or empty. This post is about both, and about the testing mistake that made the second one look solved when it wasn't.

1. Gemini Live: connected, streaming, zero transcripts

The symptoms: the socket opened, the setup message was accepted, audio went out at 16 kHz, input levels were fine. Not a single transcript came back, and there was no error anywhere. Deepgram, on the same pipeline, worked.

We checked the API key, the model name, the language, the sample rate and the JSON paths (serverContent.inputTranscription). All of them were correct.

The cause was one match arm. The receive loop handled Message::Text and swallowed everything else:

let payload = match self.ws.read() {
    Ok(Message::Text(text)) => text,
    Ok(_) => continue,          // binary frames silently ignored
    // ...
};

Gemini's Live API (BidiGenerateContent) sends its JSON responses in binary WebSocket frames. Every message with a transcript in it went into Ok(_) => continue. Deepgram worked only because it sends text frames.

The fix reads both frame types as UTF-8 and leaves the rest of the parser alone:

Ok(Message::Text(text)) => text,
Ok(Message::Binary(bytes)) => String::from_utf8_lossy(&bytes).into_owned(),
Ok(Message::Ping(ping)) => {
    let _ = self.ws.send(Message::Pong(ping));
    continue;
}
Ok(Message::Close(frame)) => bail!("Google Gemini closed the session: {}", /* code + reason */),
Ok(_) => continue,

A test now feeds a serverContent.inputTranscription payload through the parser, and the close frame turns into an error with the code and reason instead of a silent stop.

The lesson: Ok(_) => {} in a protocol loop is a place where a whole feature can fail without a trace. If you must ignore frames, log what you ignore at debug level, at least once. When a provider "connects but returns nothing", look at which frame types you actually parse before you start suspecting keys and models.

2. Deepgram keyterms: the limit isn't what it looks like

Deepgram's streaming API accepts keyterm parameters that bias recognition toward domain words. For interview calls in security engineering that matters a lot: acronyms like IDOR, BOLA or SSRF otherwise come out as random English words. We keep a short core list (44 terms) and a larger vocabulary. Terms from the larger list are added only when they appear in the user's CV or job description.

Past some number of terms, Deepgram rejects the connection with HTTP 400. Our first assumption was a URL length limit. It isn't. Measured against the API:

• 85 long, multi-word terms failed at about 1,444 characters of query string;
• 85 short acronyms connected fine at about 1,697 characters;
• a URL of about 2,394 characters made mostly of replace parameters connected too.

The cap is a token budget for keyterms, so what you put in decides how many fit. Acronym-style vocabularies fit roughly 60 to 85 terms.

The engine handles it in two layers:

• Estimate up front. Terms are added in priority order until a conservative character estimate (1,300 characters of &keyterm= parameters) is reached. Characters are only a proxy for tokens, but a cheap and predictable one.
• Halve and retry on 400. If Deepgram still rejects the list, the budget is halved, the query is rebuilt, and the connection is retried. The narrowed budget is stored in an atomic, so the next reconnect in the same run doesn't pay for the failure again.

Err(error) if budget > 0 && keyterms_enabled && is_deepgram_bad_request(&error) => {
    budget = narrowed_keyterm_budget(budget);   // budget / 2
    KEYTERM_BUDGET.store(budget, Ordering::Relaxed);
    keyterm_query = deepgram_keyterm_query(&self.language, budget, &self.call_context);
    warn!("Deepgram rejected the keyterm list with HTTP 400; retrying with {} keyterms", budget);
}

Fitting under the cap turned out to be the smaller problem. Keyterms help a lot when the list is short and specific. On our recordings, an A/B over the same audio found 10 target terms with keyterms and 4 without. But a long list dilutes: with 77 terms the same clip came out as "INDOR BOLA"; with 25 curated terms it came out as "IDOR BOLA". Generic words (REST, IoT, Kubernetes, salt) made things worse, because they pull ordinary speech toward themselves. Deepgram's own guidance is 20 to 50 keyterms. Ours agreed.

3. The testing mistake: batch results don't transfer to streaming

The keyterm work started with Deepgram's batch API, because it's convenient: send a file, get a transcript, compare. In batch, language=multi beat language=ru in every comparison on our Russian-and-English interview recordings.

Over the streaming WebSocket, which is what the app actually uses, the same change lost: the same number of correct terms and three times as many garbage words. Streaming recognition sees short chunks without the context of the whole file. Batch results for the same model don't predict streaming results.

So all STT changes are now measured the way the app runs. A harness streams a recorded WAV over the WebSocket in real time and scores the output by counting correct domain terms and known misrecognitions. It's slower than batch and noisier. It is also the only number that has matched what users hear.

Summary

• Parse every WebSocket frame type your provider might use. Silence is a symptom, not a status.
• When an API rejects "too many" of something, measure what the limit is made of before you encode a guess.
• Shrink-and-retry on a 400 is a cheap safety net, but a short curated list beats a long one on accuracy anyway.
• Test speech recognition through the same streaming path production uses.

I'm building Talkaside, a Windows app for job interviews and calls in another language.

Drafted with AI assistance from this project's code and logs.