Got to attend the Gemini 3.5 Live Translate AMA @ Discord. I summarised some points and want to discuss about it.
Google DeepMind ran an exclusive Discord AMA for Gemini 3.5 Live Translate with Anuda Weerasinghe (Associate PM) and Thor Schaeff (DevRel Engineer). 74 people in the room. I was one of them.
This wasn't a product demo or a launch recap. It was an hour of engineers actually explaining architecture decisions, tradeoffs, and constraints. That's rare. So here's a full technical breakdown of everything worth knowing from it.
First, what even is Gemini 3.5 Live Translate?
The short version: it's a streaming speech-to-speech translation model that supports 70+ languages and generates translated audio in near real-time, a few seconds behind the speaker.
The part people gloss over: it's built on Gemini 3 Pro, not a lightweight translation model. That matters because the quality ceiling is much higher than what you'd get from a dedicated translation pipeline slapped onto a TTS layer.
It rolled out on June 9, 2026 across three surfaces: the Gemini Live API for developers, Google Meet for enterprise (private preview), and Google Translate on Android and iOS for everyone else.
It's not text in the middle.
Most translation systems work in a chain: speech to text, text translated, text back to speech. That chain introduces latency at every step, and each conversion is a place where meaning can degrade.
Gemini 3.5 Live Translate skips the middle entirely. It's audio in, audio out. The model takes streamed audio and generates translated audio directly, without converting to text as an intermediate representation.
The API enforces this at the format level. Both input and output have to be raw 16-bit PCM data. No MP3, no compressed formats. They're deliberately keeping the pipeline lean for latency reasons, and if you're building on it, that's a constraint to know upfront. Your audio capture code needs to output raw PCM, and whatever you're playing back needs to handle it.
If you need text, you can still get it. But it comes from a separate transcription layer, not from the audio model itself. The native audio model only returns audio.
The thinkingLevel parameter is basically a latency knob.
The model has a thinkingLevel parameter that controls how much reasoning it does before responding. At minimal, it skips chain-of-thought entirely and outputs translated speech as fast as possible. Crank it up and you get more deliberate, accurate translations, but you pay for it in latency.
For live conversation, you almost always want minimal. The whole point of real-time translation is that the lag is small enough that conversation still feels natural. Adding deliberate reasoning time defeats that.
What I found interesting is how server events work. One event can contain audio, text, and metadata simultaneously. They arrive bundled, not sequentially. That means your client doesn't have to coordinate waiting for different pieces to come in one after another.
There's also a concept called send_client_content worth understanding. It lets you front-load conversation history at the start of a session. But once the session goes active, that door closes. You can't inject context mid-session. So if you're building something where prior conversation matters, set it up before the first turn starts.
I asked Anuda directly about context degradation.
This was the question I wanted answered. Long-running sessions are where translation systems quietly fall apart, and most product demos never show you an hour-long call.
My question to Anuda: does translation quality degrade over a long session as earlier context is lost? Is there a context window limit on the stream?
His answer: context is retained throughout the session and you shouldn't see quality drops. The live API has a 32k token context window. The native model gets 128k, but the live API variant runs at 32k. Sessions are capped at 10 minutes, but session management APIs let you extend beyond that. Voice consistency is still an area they're actively working on.
The 32k vs 128k split is the thing to note here. If you're building something like a multilingual lecture tool or a long customer support call, you're working within 32k tokens. For most live conversations that's probably fine, but it's worth understanding before you hit an edge case.
The language switching is actually clean.
This was one of the most practically impressive things they talked about. No flags, no session resets, no manual configuration. The model detects language shifts directly from the audio stream and follows along.
If someone switches from Tamil to English mid-sentence, the translation adapts. It works on both incoming and outgoing speech. 70+ languages supported, and the full list includes languages like Bengali, Malayalam, Marathi, Burmese, Basque, and Mongolian, not just the usual high-resource set.
2000+ language pair combinations is the claim. That number comes from the combinatorics of the supported languages, and whether all of those pairs are equally well-evaluated is a fair question to ask. But the mid-conversation switching working without any manual input is genuinely useful, especially for multilingual teams where people naturally shift languages mid-call.
The video input is not what you think.
The API accepts video, but it's not a video call in any conventional sense. Frames are sent as individual JPEGs or PNGs via real-time inputs, and you're capped at one frame per second. It's frame-by-frame visual context, not a video stream.
Think of it less like a webcam feed and more like periodically telling the model what the environment looks like. Useful for grounding the translation in visual context, but if you're expecting the model to track motion or read fast-moving text on screen, that's not what this is.
The architecture they walked through.
The reference implementation uses LiveKit, and the flow is clean enough to be worth understanding.
The organizer publishes audio to a LiveKit room. A TranslationBridge bot joins the room per target language, subscribes to the organizer's audio, and sends it to the Gemini Live API with a translationConfig specifying directional translation and a targetLanguageCode. The model returns translated audio, which the bridge publishes back into the LiveKit room. Attendees then subscribe to a translator-{lang} audio track for whatever language they need.
The nice thing about this design is the separation. Each language gets its own bot. The core call infrastructure doesn't need to know anything about translation. You're just routing audio tracks. It's composable and relatively easy to extend.
The Gemini 3.5 Live Translate Docs:
The AMA was one of the more technically honest ones I've attended. Anuda and Thor explained real constraints, real tradeoffs, and architecture that you can actually implement. The raw PCM requirement and the 10-minute session cap are the two things to design around first. Everything else flows from those.
If you're building on the Gemini Live API, the reference LiveKit architecture is a solid starting point. And if you're just following the space, the audio-to-audio shift without a text intermediate is the architectural detail that actually matters here.
Top comments (0)