Every conversation about AI voice agents starts in the same place. Which TTS provider, how much latency, whether the intonation sounds natural.
I spent real time on all of that. Fast model, first audio under the threshold where a pause starts to feel wrong, a decent voice. The calls still sounded fake and I could not explain why from the audio alone.
So I stopped listening to them and read the transcripts instead. In text the problem was obvious, and it had nothing to do with the voice.
The code is at github.com/vamshisonnathi/agentic-telephony if you want to skip ahead.
The call that made it clear
The agent was calling a party rental place about chairs.
Agent: Hi. Do you have 20 folding chairs for September 22nd?
Vendor: Who is this?
Agent: Alex.
Vendor: Mhmm.
Agent: Hi. Do you have 20 folding chairs for September 22nd?
Read that last line again. The agent got interrupted, answered the question, then restarted its original sentence from the beginning, word for word.
No human does this. When someone says "Mhmm" you do not reboot, you pick up mid-thought: "Yeah, so do you have them available for the 22nd?"
That exchange is the whole problem in miniature. Transcription was perfect. Voice was fine. Latency was fine. The agent just did not know how conversations work.
Four failures, and what each one needs
I wrote these as explicit prompt rules rather than hoping the model would infer them.
The first is recovery instead of restart. When the agent gets interrupted or hears an acknowledgment token ("Mhmm", "Yeah", "Okay"), it must never repeat its full question. It continues from where the thought was. The rule in my prompt names the failure directly and includes both the bad and the good transcript, because describing the behavior abstractly did nothing. Showing the model the exact output it was producing worked.
Second, pronouns after the first mention. The agent originally said "20 folding chairs and 2 banquet tables for September 22nd" in four consecutive turns. Humans say it once and switch to "them." So the rule is to state the items and the date exactly once, then use "them," "those," "that," or "it" for the rest of the call. It pairs with a hard cap on turn length, which forces the agent to stop monologuing and listen.
Third, say the math out loud. When a vendor quotes "a dollar per chair, five per table," the agent used to just say "okay." That is the moment a human confirms understanding: "Cool, so about 30 bucks total." One extra turn, and it catches a misheard number before it becomes a wrong order. The same rule covers physical reality checks. Are the tables foldable. Can I pick them up Sunday morning. How long will the food take.
Fourth, close the logistics before hanging up. The worst version of this agent would get a yes and end the call. No address, no pickup hours, no confirmation. A successful call that produced nothing actionable. The rule now blocks the hangup until the agent has asked for a text confirmation to the caller ID number.
There is a smaller related rule I did not expect to need: never volunteer your own phone number. The business already sees the caller ID. An agent that recites its number unprompted sounds exactly like an automated system, because that is the only thing that does it.
The pronoun bug I did not see coming
Narrower than the others, but it broke calls badly.
Tell the agent to call "Anna (my brother)" and the model uses she/her for the entire conversation. It pattern matches the name and ignores the parenthetical. The person on the other end hears themselves misgendered for five minutes.
I handle this before the call starts rather than in the prompt. A function parses the request, looks for a relationship word, counts pronoun usage already present in the text, and resolves a locked pronoun set that gets injected into the system instruction as a hard constraint. If nothing resolves cleanly it falls back to they/them instead of guessing.
Relationship words beat name-based inference. That is the entire rule, and it is about forty lines of code.
Sounding human is not the same as pretending to be human
An early version of this had an instruction telling the agent it was a real person and to never admit otherwise. I took it out, and the agent got better, not worse.
Every one of the four rules above is about conversational competence. None of them depend on the other party being deceived. So now the agent says plainly that it is an AI calling on someone's behalf when asked, and announces up front when the call is recorded. The recording line is a config flag that defaults to on, because about a dozen states require consent from everyone on the call and the telephony platforms record by default.
That also turned out to be the honest answer to the TCPA question. The FCC ruled in early 2024 that AI-generated voices in calls count as artificial voices under the statute. A tool that dials arbitrary numbers while denying what it is sits in a worse place than a tool that just says what it is.
Two architecture choices I would repeat
The telephony provider is swappable. Vapi and Retell sit behind one abstract interface and which one runs is an environment variable. I built it that way because I did not want a single platform's uptime deciding whether the thing worked, and it paid off for a different reason. When a conversational problem shows up on both providers, it is my prompt. When it shows up on one, it is theirs. That turns a day of guessing into one test.
Call state lives locally rather than being fetched from the provider when I need it. A thread-safe store accumulates turn-by-turn transcripts as webhooks arrive, which lets live transcripts stream to a dashboard over a WebSocket while the call is still running. Watching a call in text as it happens is the most useful debugging tool in the project. You see the exact turn where it goes sideways.
One thing I got wrong the first time: the webhook server shipped with no auth. The dispatch endpoint places real billable calls and the transcript endpoint returns PII, and the README cheerfully told you to expose the port with ngrok. Both now take a shared secret, and the webhooks verify one too.
The code
Python, FastAPI for the webhook server, Vapi or Retell for telephony. The calling functions are also wrapped as tools so a CrewAI or LangChain agent can place a call, poll it, and end it inside a larger task.
github.com/vamshisonnathi/agentic-telephony
If the prompt rules are the part you care about, they are in protocol.py, with the bad and good transcripts written into the instruction text.
Still unsolved
Nothing here handles being put on hold. Twenty-five or thirty seconds of true silence and provider timeouts start firing, so the agent either hangs up or says something into dead air.
Handoff is the bigger one. There is no mid-call transfer bridge, so when the agent is out of depth the call just ends. A bad transfer is worse than a bad answer, which is why I have not shipped a rough version of it.
If you have built in this space, I want to know what broke for you that was not latency. My guess is that most of the real failures are conversational, and almost nobody writes those up.
Top comments (0)