Here is the note first, because it matters: Everything here comes from the development we ran about a month ago, so newer LLM versions may already behave differently(As you already know the space moves so fasttt..) than what I describe. This is part one of three. I will cover the basics and the terms you need first, then the approaches we tried and why some worked better than others, and finally the automated testing suite that keeps the agent honest. Treat what follows as a snapshot, not a verdict.
Now, what is a voice agent, really??
The textbook answer is: a system that listens to spoken input, passes it through a large language model for understanding, and sends the reply out as synthesized speech, all in real time. Technically correct. Also the kind of sentence that makes you want to close the tab.
Here is the version that actually happened. We built something that picks up when a customer calls our MF company. The caller might want their account statement. Might want to know what a particular fund scheme does. Might want to set up a SIP, which is just a monthly investment plan, and if you have never heard the term, that is fine, neither had most of the people who called us during the trial.
Most of the work is about time: when the agent starts to speak, when it stops, and how fast the first word comes.A chatbot can take three seconds to reply and nobody minds. On a call, two seconds of silence feels long. The caller thinks the line is dead, or starts to speak again. So most of the work in a voice agent is about time: when the agent starts to speak, when it stops, and how fast the first word comes.
How a voice agent is built
Every voice agent has three parts, whatever models you choose.
Caller audio goes through input, brain and output, then back
A multimodal model puts all three parts into one model. A cascade uses a separate service for each part. We tried both.Both works but it which one to choose should be based on your specific use case
Words you will see in this article.
If you are working with voice agents you need to have an understanding on below mentioned words.These are widely used jargons.
A Voice Agent Is A State Machine.
A state machine is a system that is always in one known state. Specific events move it to a different state. A phone call fits this model well. At any moment, the agent greets the caller, finds out what they want, collects a PAN, waits for an OTP or closes the call.
If you put every rule in one large prompt, the LLM forgets some of them. When you split the call into states, each state gets a short prompt and only the tools it needs. The strict rules go into code, where the LLM cannot skip them.
Only one agent speaks at a time. The receptionist stays open for the whole call, and one flow at a time joins it.
Prompt caching and system prompt length
The LLM reads your entire system prompt and the full tool list on every single turn. Every token gets processed, every token gets billed. Ours are long prompts, each flow carries a stack of rules, and in a voice agent, latency is not a nice-to-have, it is the whole game. A slow reply is a dead call.So prompt optimization plays a important role in case of voice agent.
Prompt caching takes most of that pain away. You mark the system prompt and the tool list as cacheable. On the next turn, the provider skips the rework and reads only the new messages. We also mark the last message in the history as a cache point, so the conversation itself gets cached as it grows. The numbers from our traces: 74% of input tokens came from cache on Haiku 4.5, and 83% on Sonnet 4.6. That is a lot of processing you never pay for.
Here is the trap. Each model sets its own minimum size for a cached block. Haiku 4.5 wants 4,096 tokens. Sonnet 4.6 wants 1,024. Our receptionist prompt sat at 1,925 tokens, fine on Sonnet, invisible to Haiku. The moment we switched models, the receptionist stopped caching entirely. No warning. No error. Just a quiet jump in latency and cost that nobody noticed until someone looked.
If you do a model change at any time, open the API response and read the cache numbers. Do not assume what worked yesterday still works today.
Next part of this article will give a detailed on the multimodal approach that we took along with the pros and cons
Top comments (0)