Disclosure: I'm Nabeel, founder of VoiceDash, a client portal for Retell agencies. This is an adapted republish of a guide from the VoiceDash blog, rewritten for developers. Nothing below needs VoiceDash; it uses Retell's own features plus a little glue code.
TL;DR: A voice agent that passed every test on launch day can quietly break a month later: a calendar token expires, a webhook starts failing, the business changes its hours, or a provider has a bad afternoon. The agent keeps answering and sounds fine, and books nothing. Monitoring in production means watching four layers (is the line answering, are tools working, is quality holding, is usage normal), wiring a failover before you need it, and routing alerts to one place. Retell now ships most of the primitives; this post maps each layer to the field or feature that covers it.
Why testing is not enough
Pre-launch testing proves the agent worked on the day you shipped it (my checklist for that is in How to Test a Voice AI Agent Before Go-Live). Production is different, because the world around the agent keeps moving:
- Credentials expire. OAuth tokens for calendars and CRMs get revoked or rotated when someone at the client changes a password.
- The business changes without telling you. New hours, a new service, a staff member who left. The prompt is now wrong and nothing errors.
- Integrations fail silently. A workflow throws on a field that changed shape, and the agent still tells the caller "you are all booked."
- Vendors have incidents. You sit on top of Retell, a telephony provider and a model provider.
- Your own edits break things. A prompt change for one branch shifts behavior in another.
None of these stop the phone ringing. That is what makes them dangerous.
Layer 1: is the line answering at all?
The loud failure: calls ring out, drop on connect, or end in an error. Rare, and the one clients never forgive.
Heartbeat call. Once or a few times a day, place an automated test call to each production number from a separate number. Check that it connects, the agent greets within a normal window, and the call ends cleanly. A small scheduled workflow (n8n, a cron job, anything) plus a Twilio number does it. Alert after two consecutive failures, not one.
Volume floor. Every client has a rhythm. Compare today's call count against the same weekday's average and alert when it falls far below during business hours. Zero calls by noon at a busy office is usually a forwarding change on the client side, not an AI problem, but it is still yours to catch.
Disconnection reasons. Every Retell call records a disconnection_reason, available in Call History and on the Get Call API. Retell's disconnection table splits them roughly into three buckets:
expected (call_status = ended)
user_hangup, agent_hangup, call_transfer, voicemail_reached,
inactivity, max_duration_reached, user_requested_dnc ...
not connected (outbound)
dial_busy, dial_no_answer, dial_failed, invalid_destination,
marked_as_spam, network_blocked ...
errors (call_status = error)
concurrency_limit_reached, error_llm_websocket_*, error_asr,
error_no_audio_received, error_retell, error_unknown ...
Log the field for every call and alert on any spike in the error bucket. Two details worth knowing from the docs: dial_failed is also what you see when carriers have flagged your outbound number as spam after many short calls, and the error_llm_websocket_* family only appears when you run a custom LLM over Retell's websocket.
Layer 2: are tools and integrations working?
The quiet failure, and in my experience the most common one. The conversation sounds perfect; the booking, CRM update or follow-up text never happens.
Log every tool call and its result. Whether the agent calls your endpoint directly or goes through a workflow tool, record the request, response, status and latency, keyed by call_id. If tool results are not logged you cannot see this failure at all.
Give every workflow an error branch. Every workflow the agent depends on needs an error path that alerts with the client name, the workflow and the failing step.
Make the agent honest about failure. The prompt should say exactly what to do when a tool returns an error: apologise, take the caller's details, promise a callback, flag the call. It must never confirm a booking that did not go through. A caller who gets a callback is fine; a caller who shows up for an appointment that does not exist is a complaint.
Reconcile against the source of truth. Once a day, compare calls marked booked against bookings that actually landed in the client's calendar or CRM:
// nightly reconcile, per client
const booked = calls.filter(c => c.call_analysis?.custom_analysis_data?.outcome === "booked");
const landed = await calendar.listEventsCreatedBetween(start, end, { source: "voice-agent" });
if (booked.length !== landed.length) {
alert(`${client.name}: ${booked.length} calls say booked, ${landed.length} bookings exist`);
}
The outcome field here is whatever you defined in post-call analysis; the calendar call is your own integration.
Layer 3: is conversation quality holding?
Some failures are not errors. The agent answers, the tools work, and it still handles callers badly because the business changed or an edge case became common.
Post-call analysis as the quality signal. Extract a structured outcome for every call (booked, answered question, transferred, message taken, spam) plus a flag for a frustrated caller or a request for a human. Track the rates per client week over week. A jump in transfers or frustrated callers means the prompt no longer matches reality.
A weekly transcript sample. Numbers tell you something changed; transcripts tell you what. Skim a handful of calls per client each week, weighted toward transfers, very short calls and anything flagged frustrated.
Latency, per component. After a call ends, Get Call returns a latency object with p50, p90, p95, p99, min, max for components such as e2e, llm, tts, asr and knowledge_base (not every field is present on every call). Store e2e.p90 per call and watch its trend. If it creeps up, the breakdown tells you whether it is the model, the voice or retrieval. Note that e2e excludes turns where the agent only ran a tool without speaking, so slow tools need their own timing from your logs.
Layer 4: is usage behaving normally?
Usage anomalies are both a cost problem and a symptom:
- Average duration spiking often means the agent is stuck in a loop, or robocalls are getting long conversations.
- Very short calls climbing can mean callers hang up on a bad greeting or a slow first response.
- Minutes running past the client's bundle is a margin problem you want to see mid-month, not on the invoice.
Set a maximum call duration on every agent (it ends calls with max_duration_reached), then alert when a client's daily minutes run well above normal.
Retell alert rules: the built-in layer
Retell has alert rules that evaluate aggregate metrics on a schedule and send an email or a webhook when a threshold is crossed. Metrics include call_count, call_success_rate, negative_sentiment_rate, custom_function_latency, custom_function_failure_count, transfer_call_failure_count, concurrency_used, total_call_cost and api_error_count. Rules can be absolute ("success rate below 90% over the last hour") or relative to the previous cycle, and can be filtered by agent, environment tag, disconnection reason or post-call extraction field.
Two constraints shape how you use them:
- Rules are dashboard-only. They are created in the dashboard, not through the public API, so if you manage many client workspaces, budget time to set them up per workspace and keep a written list of what exists where.
- They are aggregate, not per call. For "this specific call failed a booking", use your own webhook handler. A sensible split: Retell rules for volume, success rate, function failures and cost; your own code for heartbeats, reconciliation and anything that needs client-specific logic.
Plan the failover before you need it
Monitoring tells you something broke; failover decides how much it costs.
- Backup mode. In Retell, each phone number can have a fallback number. With Backup mode turned on, new inbound calls go straight to that fallback before an agent is selected, which is your kill switch while you investigate. Configure the fallback number at launch; Backup mode cannot be enabled until one exists, and if the fallback cannot be reached the call ends.
- A human transfer path for when the agent is confused, tested as part of every release.
- A written commitment in the client contract: you monitor and respond within a defined window, and upstream vendor outages are excluded from credits. Subscribe to Retell's status page so you know about their incidents as early as they do.
Route alerts so someone actually sees them
The fastest way to kill a monitoring setup is noise. If every short call pings your phone, you will mute the channel within a week and miss the one that mattered.
- One channel, with the client name in every message.
- Only alert on things you would act on: failed heartbeat, workflow error, volume far below normal, error disconnections, a booked count that does not reconcile, a Retell rule firing. Everything else goes into a daily digest.
- Tell the client before they tell you: what happened, when it started, what you did and which calls were affected. Handled like that, an incident builds trust instead of eroding it.
The bottom line
The agent that worked on launch day is not the agent that is answering today, because everything around it changed. Watch the line, the tools, the quality and the usage; put disconnection_reason, latency and your post-call outcome on every call record; let Retell's alert rules cover the aggregates; and set the fallback number before the first incident, not during it.
Drafted with AI assistance and fact-checked against each vendor's docs.
Top comments (0)