Here's a small experiment. Say "user at assembly a-i dot com" out loud and imagine transcribing it with no idea what was just asked. You'd probably write exactly that — a garbled phrase. Now imagine you knew the previous line was "What's your email address?" Suddenly the right transcription is obvious: user@assemblyai.com.
That difference — knowing what was just said — is what "conversation context" gives a speech-to-text model. And it turns out to be one of the highest-leverage accuracy improvements you can make to a voice agent, because it targets the exact words agents get wrong most: emails, names, numbers, and one-word replies.
The problem: models mishear when they can't anticipate
Speech-to-text is, at heart, a prediction problem. The model hears sound and predicts the most likely words. When someone speaks in full, redundant sentences, there's plenty of signal and the prediction is easy. But real conversations are full of short, ambiguous, high-stakes fragments. "Four." "Yes." "S as in Sam." A spelled-out street name. An account number read at speed.
In isolation, those fragments are genuinely hard — "to," "too," and "two" sound identical, and a name the model has never seen has no acoustic anchor. The information that would resolve the ambiguity isn't in the audio. It's in what was said a moment earlier. Without that, even a strong model is guessing.
What conversation context means
Conversation context is the practice of feeding the model both sides of the dialog — what your agent just said and what the user has already said — so it can anticipate the next turn. With the agent's question in memory, the model knows what kind of answer is coming and can sharpen recognition, disambiguate similar-sounding words, and format entities correctly.
The email example is the clearest case. After your agent asks "What's your email address?", a model with that question in context knows an email is coming, so it produces user@assemblyai.com instead of "user at assemblyai dot com." The same effect shows up on "what's your appointment time?" → "7pm," and "can I get your last name?" → a correctly spelled surname. It has the biggest impact precisely where agents are most fragile: short replies and spelled-out entities.
How to use it with Universal-3.5 Pro Realtime
Universal-3.5 Pro Realtime keeps a short, per-session memory of the conversation from two sources — and one of them is automatic.
The user half comes for free. The model automatically carries prior speech-to-text-finalized user turns forward as context. You don't configure anything for this. A parameter called previous_context_n_turns controls how many prior entries are carried (the default is 3, and the range is 0 to 100, where 0 turns carryover off entirely).
The agent half you pass in. The one thing the model can't see on its own is what your agent said, because that audio never reaches the transcription model — it's generated by your text-to-speech layer. You supply it with the agent_context parameter: your agent's most recent spoken reply, up to about 1,500 characters. Set it at connection time to seed your opening greeting, then send an UpdateConfiguration message after each agent reply to keep it current mid-stream.
Put together, the model always has both halves of the dialog when it transcribes the next user turn. Both features are specific to Universal-3.5 Pro; the full reference lives in the context carryover docs.
When it helps most
Conversation context isn't a blanket accuracy boost for every kind of audio — it's a targeted fix for a specific failure mode. It earns its keep whenever the next thing a person says is short, spelled out, or predictable from the question that prompted it. That covers a lot of voice agent ground:
Collecting an email or a physical address. Confirming a phone number, order number, or account ID. Capturing a one-word answer — "yes," "no," a time, a single digit — where there's almost no acoustic signal to work with. Recognizing a name the model has never encountered. In all of these, the agent's own question is the clue that resolves the ambiguity, and passing it in is what lets the model use it.
Practical tips and limits
A few things to keep in mind. Update agent_context after each agent reply, not just once — stale context helps less. Keep it to the most recent reply rather than the whole history; you're seeding the next prediction, not writing a transcript. If you're building a barebones prototype and want to measure the lift, start with carryover off (previous_context_n_turns set to 0), then turn it on and compare — the difference is easiest to see on the entity-heavy turns. And remember this is context, not a command: you're telling the model what the conversation is about, not instructing it how to format output.
If you want to push accuracy further on domain-specific vocabulary, conversation context pairs naturally with keyterms prompting — one gives the model the flow of the dialog, the other gives it the exact terms to expect. Our prompt engineering guide covers how to combine them.
The takeaway
Most accuracy conversations fixate on model size and benchmark WER. But a lot of the errors that actually hurt a voice agent aren't a model-quality problem — they're a context problem. The model heard the audio fine; it just didn't know what to expect. Conversation context closes that gap cheaply, and it's already built into Universal-3.5 Pro Realtime. Give the model both sides of the dialog and the hardest tokens in your conversations get a lot easier.
Frequently asked questions
What is conversation context in voice AI?
Conversation context is the practice of giving a speech-to-text model both sides of a dialog — what the agent just said and what the user has already said — so it can anticipate and more accurately transcribe the next turn. It's especially effective on short replies and spelled-out entities like emails, names, and numbers.
How does conversation context improve transcription accuracy?
By knowing the agent's previous question, the model can predict the type of answer to expect and disambiguate similar-sounding words. For example, after "What's your email address?" the model produces "user@assemblyai.com" instead of "user at assemblyai dot com," because it knows an email address is coming.
What is the difference between agent_context and previous_context_n_turns?
In Universal-3.5 Pro Realtime, the user half of the conversation is carried forward automatically, and previous_context_n_turns controls how many prior turns are kept (default 3). The agent_context parameter is how you supply the agent half — your agent's most recent spoken reply — since that audio never reaches the transcription model.
When should I use conversation context in a voice agent?
Use it whenever the next user response is short, spelled out, or predictable from the agent's question — collecting emails, addresses, phone numbers, account IDs, one-word confirmations, or unfamiliar names. These are the turns where conversation context has the largest accuracy impact.
Does conversation context work for pre-recorded audio too?
Conversation context and automatic turn carryover are documented features of Universal-3.5 Pro Realtime for real-time voice agents. If you're transcribing pre-recorded audio, contextual prompting and keyterms prompting provide similar accuracy benefits; check the current API reference for the exact parameters available on the async endpoint.
Top comments (0)