DEV Community

Hayden Goracke
Hayden Goracke

Posted on

What broke when I put an AI voice agent on real phone lines

I build AI phone agents for home service companies. A voice agent (Retell) answers the call, and n8n workflows do the work behind it: checking a calendar, booking, writing to the CRM. These are the failures that taught me the most, and what fixed each one.

1. The model was bad at "are we open right now?"

The prompt had the business hours and the current time. The model still got the open or closed call wrong often enough to matter.

The fix was to stop asking the model. An n8n step works out the answer and passes it in as a variable, and the prompt just reads the flag. My rule now: if a rule misfires twice, it leaves the prompt and gets computed upstream. Models follow data better than they follow conditional logic.

2. The agent said "you are booked" when nothing was booked

The worst failure on a phone line is a caller who thinks they have an appointment and does not.

Another prompt rule did not hold, so I made it structural. The booking tool now goes through a small n8n proxy that reads the transcript first. No calendar write happens unless the agent actually said a time out loud and the caller replied to it. And the agent is not allowed to say "booked" unless the tool came back with success = true.

3. Dead air during tool calls

The booking tool had a 120 second timeout, and callers were sitting through as much as 27 seconds of silence when the calendar was slow. I cut the timeout to 25 seconds and let the agent talk while the tool runs.

4. Publish succeeded and the phone kept running the old version

A publish call can return success while the phone number is still bound to the previous version of the agent. Now every publish ends the same way: fetch the agent again, check which version the number is actually bound to, then place a real call.

5. My tests passed and production failed

I was testing workflows with payloads I typed by hand. In production the tool arguments arrive nested differently, so my tests were passing against a shape that never shows up on a live call. I now replay real captured webhook bodies and nothing hand-built.

6. One caller, two bookings

A caller rang back an hour later and got a second callback booked next to the first. The booking step now looks for an existing booking from the same number and moves it.

What I would tell someone starting

  • Read the actual tool call data, not the transcript. The transcript is what the model said happened.
  • Anything that has one right answer (time, hours, dates) belongs in code.
  • A publish is not done until a real call lands on the new version.

This post was written with AI assistance and edited by me.

Top comments (0)