DEV Community: Adel Atya

How we test AI phone agents before they take real calls

Adel Atya — Sat, 24 Oct 2026 12:07:14 +0000

An AI phone agent that sounds great in a demo can still fail on the one call that matters: the angry customer, the emergency, the caller who asks something it was never told. You only find out which kind you have by testing it on purpose, before your customers do it for you.

This is how we test every agent before it goes live. The results are public: you can see the latest run on our agent QA console.

Why a few test calls are not enough

Language models are not deterministic. The same question can get a slightly different answer each time. So an agent that handled your three test calls perfectly can still mishandle the fourth. And the calls people test with are usually the easy ones: "I'd like to book an appointment."

Real callers interrupt, change their mind halfway through, refuse to give a phone number, ask if they are talking to a robot, or call at 2am with water coming through the ceiling. Those are the calls worth testing.

Simulated callers

Instead of making test calls by hand, we have a second AI play the caller. Each simulated caller gets:

A persona and a goal, for example "a stressed homeowner with a burst pipe who wants someone today".
Facts it only reveals when asked, such as the address or the unit number, so we can check the agent actually asks.
Behaviours that make it harder: rushing, getting angry, interrupting, speaking in short broken sentences, refusing details.
The outcome that counts as success, for example "an emergency booking, or a transfer to the on-call number".

Our current suite has 38 of these scenarios across three agents: a dental front desk, a plumbing dispatch line and a self-storage receptionist. Every scenario runs three times, because one lucky pass proves little. That is 114 calls per run, and a full run costs about 25 cents.

What we check

Wherever possible, a simple program checks the result, not another AI. Did the agent call the booking tool? Did it save the right name, phone number and time? Did every price it quoted actually exist in the price list? Did it hand over to a human when it should have, and only then?

An AI judge is used only for what code cannot check, such as whether the tone fit the situation or whether a free-text answer was actually true. It gets a written rubric and has to give a one-line reason for every verdict.

The numbers that matter

Containment: the share of calls the agent resolved completely on its own. A caller hanging up early never counts as contained. That is the most common way this number gets inflated.
Correct versus unnecessary transfers: handing over to a person is good when it is needed, and costly when it is not.
Missed handoffs: the scenario needed a human, for example an emergency, and the agent kept going alone. We treat this as a critical failure.
Hallucination rate: calls where the agent stated a price, time or fact that is not in its data.
Consistency: a scenario only passes if it passes all three runs.

What it caught

The first full run caught something no demo would have shown: on the cheaper language model, our plumbing agent missed the required emergency handoff in 5 of 41 calls, including a caller reporting a gas smell. It sounded calm and helpful the whole time. It just did the wrong thing.

That is the point of testing. A gas leak is exactly the call where "it sounded fine" is not good enough.

Testing every change

Every time we change an agent's script or switch the model, the suite runs again and compares the results with the previous baseline. If a scenario that used to pass starts failing, or containment drops, the change is blocked until it is fixed. It works like the automated tests a software team runs before every release.

What this means if you are buying one

Ask whoever builds your agent how they test it. If the answer is "we called it a few times", you are the test. Ask to see the emergency scenarios specifically, and ask what happens when the agent does not know the answer.

You can talk to our plumbing agent and try to break it yourself, or read what an AI receptionist costs before you decide.

Originally published at adelatya.com.

How many calls does a plumber miss, and what does each cost?

Adel Atya — Sat, 24 Oct 2026 07:25:04 +0000

Nobody can tell you how many calls your plumbing business misses, because published numbers range from 14% to nearly half of all calls, depending on what counts as "missed". The useful answer is your own number, and it takes about twenty minutes to get.

Below: what the published data actually says, why it disagrees with itself, and how to turn your own missed calls into a dollar figure.

What do the studies say?

Two large datasets give very different answers, and both are honest about their method.

CallRail, in a January 2025 report built on 1.1 million leads, found home services businesses missed 14% of calls. It counts calls that ended within 15 seconds as "abandoned", not missed, so those are left out.
Invoca, in its 2026 Home Services Benchmarks report, found a person answered only 52% of inbound home services calls. Leave out calls of 15 seconds or less and that rises to 65%.

Both companies sell call-tracking software, so they have a stake in the topic. Neither measures plumbers alone. And you will see a lot of blog posts quoting "62% of calls go unanswered". That figure traces back to a 2016 study of just 85 businesses, so I would not build a decision on it.

The honest summary: somewhere between one call in seven and one call in two does not reach a person at a typical home-services business. Where you sit depends on how many trucks you run, whether the owner picks up while on a job, and what your phone does after hours.

Why does the number vary so much?

A "missed call" is not one thing. Your phone log mixes together:

Rings nobody answered while the crew was on a job.
Calls that went to voicemail and ended there.
Wrong numbers, spam and sales calls that you would never want to answer.
After-hours calls, which are often the most urgent ones.

A tracker that counts all four will show a scary rate. One that filters out short calls will show a gentler one. For pricing purposes, only the real customer calls matter, and only you can sort those.

What does voicemail actually catch?

Less than owners hope. Moneypenny's 2016 small business call report found that 69% of callers who reached voicemail hung up without leaving a message. Moneypenny sells answering services, so treat the exact figure with care, but the direction matches what most owners see in their own inbox: a handful of messages, and a lot of missed rings with no trace.

Even if today's figure is lower than that 2016 number, voicemail is a net with large holes in it. The caller you never hear from has usually already dialled the next plumber on the list.

What is one missed call worth?

Pricing guides disagree because prices differ by city and job. Inch Calculator puts the national average plumber service call at about $300, with a typical range of $150 to $500. Housecall Pro lists emergency dispatch fees of roughly $150 to $350 or more, billed on top of the repair.

Use your own average ticket, not those. Pull your last 50 completed jobs and divide total revenue by 50. That is the value of one booked call. Then adjust for two things:

Not every missed call was a job. Assume a share were spam, wrong numbers or price shoppers.
Emergency calls are worth more than the average, and they are the ones most likely to go to the second plumber, because the caller needs someone now.

A worked example (with made-up numbers on purpose)

These figures are an illustration, not a result from any client. Plug in your own.

You get 40 real customer calls a week.
You miss 20% of them, so 8 calls.
Half of those callers would have booked, so 4 jobs.
Your average ticket is $350.

That is $1,400 a week, or roughly $5,600 a month, going to someone else. Cut the miss rate in half and the figure is still around $2,800. Now change the inputs. At 10 calls a week, a 10% miss rate and a $200 ticket, the loss is about $100 a week, and no phone agent will be worth paying for.

That second case matters. If you are a solo operator with a quiet phone, a missed-call text-back is probably enough.

How to measure your own number this week

Export seven days of call logs from your carrier, your phone app or Google Business Profile.
Mark each unanswered call: real customer, or noise? Call the numbers back to check.
Count after-hours calls separately, evenings and weekends.
Check how many left a voicemail, and how many of those you returned within an hour.
Multiply real missed customers by your booking rate and average ticket.

Do it for two weeks if you can, and include a busy one. Averages hide the Monday after a cold snap.

What actually fixes it?

In order of cost:

Answer faster yourself. Move the phone to whoever is closest to it. Free, and limited by a human's day.
Missed-call text-back. When a call goes unanswered, the caller gets a text within seconds. In our Blue Ridge demo the text goes out in 8 seconds and an AI callback follows in 52. It recovers part of the loss and costs little.
An AI phone agent that answers every call in about a second, takes the job details and books into your calendar. It can handle two calls at once, at 2am as well as 2pm.
A human answering service. It works, but it usually takes messages instead of booking jobs.

An AI agent cannot diagnose a leak, and it should not be left alone with a gas-smell call. Ours is built to hand those to a person on your on-call number, and we test that handoff on purpose before any agent goes live.

Checklist

Export a week of call logs and sort real customers from noise.
Count after-hours calls on their own line.
Work out your average ticket from your last 50 jobs.
Multiply: missed real calls × booking rate × average ticket.
If the monthly number is above a few hundred dollars, look at what an AI receptionist costs and the published pricing.
Call our demo agent and see what a missed-call recovery feels like from the caller's side.

The plumbing and HVAC page shows the specific calls an agent handles in your trade.

Originally published at adelatya.com.

How much does an AI receptionist cost for a small business in 2026?

Adel Atya — Sat, 24 Oct 2026 07:25:03 +0000

Most pages about AI receptionist pricing end with "contact us for a quote". This one doesn't. Here is what an AI phone agent actually costs in 2026, what drives the price up or down, and how to tell whether it pays for itself in your business.

The short answer

For a small business with one location, expect to pay:

A one-off build of roughly $600 to $1,500. This covers writing the agent's script for your business, connecting it to your calendar and CRM, and testing it on real call scenarios.
A monthly fee of roughly $300 to $600. This usually includes a block of call minutes, the phone number, and someone keeping the agent working when your prices or hours change.
Overage of about 10 to 20 cents per minute once you go past the included minutes.

Our own published pricing sits inside that range: a $900 build plus $450 a month with 500 minutes included, then 15 cents a minute. The first three direct clients get the build at $600 in exchange for sharing their before-and-after call numbers. You can see every line on our pricing page.

What you are actually paying for

An AI receptionist is not one product. It is several services stitched together, and each one has a real cost:

The voice platform. Services such as Retell and Vapi handle the phone line, turn the caller's speech into text and the reply back into a natural voice. They charge per minute of conversation.
The language model. The part that decides what to say, using your script and your rules. It is billed per conversation, folded into that per-minute price.
The phone number. A local number costs a few dollars a month. Most businesses keep their existing number and forward it.
The integrations. Booking into your calendar, writing the call into your CRM, texting the caller a confirmation. This is where most of the build time goes.
The upkeep. Prices change, a new service is added, a holiday schedule needs handling. Someone has to update the agent, or it starts giving outdated answers.

When a provider quotes a suspiciously low monthly price, one of these is usually missing: a tiny minute allowance, no integrations, or no one to call when it says something wrong.

How it compares with the alternatives

A full-time receptionist in the US typically costs $35,000 to $45,000 a year before benefits, covers about 40 hours a week, and cannot answer two calls at once.

A part-time receptionist costs less but leaves the evenings and weekends, when many urgent calls arrive, uncovered.

A traditional answering service takes messages. It rarely books the job, and the caller still has to wait for someone to call back. By then many have called the next business on the list.

An AI receptionist answers every call in about a second, day or night, books into your calendar while the caller is still on the line, and texts back anyone who hangs up. It does not replace a great front-desk person. It covers the calls they physically cannot take.

Does it pay for itself?

Do this calculation with your own numbers:

How many calls a week go unanswered? Your phone system or carrier can usually tell you.
What share of those callers would have booked? For urgent trades, it is often high.
What is one booked job worth to you?

If you miss ten calls a week, three of those would have booked, and an average job is worth $300, that is $900 a week walking to a competitor. A $450 monthly fee is recovered by the second booked job.

For plumbers and HVAC firms, where one emergency call can be worth several hundred dollars, the maths usually works within the first month. For businesses where a missed call is rarely urgent, the case is weaker, and you should say so to anyone selling you one.

What to ask before you buy

How many minutes are included, and what does each extra minute cost? Ask for a real number.
What happens when it doesn't know the answer? A good agent says it will check, or transfers the call. A bad one guesses.
Can I hear it handle a real call first? You can talk to one of ours right now.
Who updates it when my prices change? If the answer is "you, in a settings panel", budget your own time for it.
How do you test it before it goes live? We run every agent against dozens of simulated callers before it takes a real call. We wrote up exactly how that testing works.

If you run a plumbing or HVAC business, a dental practice or a restaurant, each of those pages shows the specific calls the agent handles in your trade.

Originally published at adelatya.com.