A property lead has a short half-life. Someone fills in a form about a valuation or a rental at 9pm, and by the morning they have filled in three more on three other sites. So the pitch for an AI voice agent doing first-touch follow-up is obvious: call everyone, call them fast, call them at 9pm.
That part usually works. It is what happens after the call that decides whether the system is worth having.
I am Ussama Assad. I build and debug production voice-agent systems — the kind that sit between a lead form and a CRM and are supposed to save a team from dialling. Below are the two failures I see land hardest on lead follow-up specifically, in the order they tend to arrive. Neither is a model problem. Both are design problems that look like model problems, which is why they survive so long.
Failure one: the agent books answering machines as interested leads
The first version usually looks like a success. Calls go out, dispositions come back, a qualified-lead table fills up and syncs to the CRM. Then someone on the sales side says the quiet part: the agent is marking voicemails as callbacks, and the team is ringing back people who never actually spoke to it.
Say a third of the qualified pipeline turns out to be answerphones. That is not a reporting annoyance — every false positive spends a human dial. The team ends up calling dead numbers generated by the thing that was supposed to stop them calling dead numbers.
The instinct is to tighten the prompt. Add stricter classification criteria, tell it harder not to count voicemail. That instinct misreads the mechanism.
A voicemail greeting produces a transcript that structurally resembles a conversation. There is a greeting. There is a pause. There is speech. A classifier reading that transcript has no strong signal separating "a person said hello" from "a recording said hello." You are not asking the model to be more careful. You are asking it to distinguish two things that look the same in the only data it can see.
The second instinct — route calls through a dialler that only connects on human pickup — is the intuitive fix that costs two weeks. It frequently dies on integration incompatibility with the voice platform, and you find that out at the end.
What actually works is post-hoc adjudication. Take the finished transcript plus the disposition the first agent assigned, and ask a separate model one narrow question: is that disposition correct? You are not making the first pass smarter. You are checking its work with a second pass that has an easier job, because judging a finished transcript is a much simpler task than deciding mid-call.
One triage rule worth borrowing: calls under about thirty seconds are not worth reviewing. They are hangups and hard nos. Almost all the interesting misclassification lives above that line. That single cutoff removes most of the review volume at close to zero coverage cost.
The finding underneath this one
The false positives were not caught by monitoring. They were caught by a person reading transcripts.
And when I traced it back, nobody on the build side was reading transcripts at all. Every error had been found by the client and reported upward. That is a process failure wearing a model failure's clothes — the client had quietly become the QA layer, knew it, and resented it.
If you are hiring someone to build this, that is a fair thing to ask them directly: who reads the transcripts, how often, and what do they filter to? A good answer sounds like a routine. Filter to records where a callback was scheduled — the only outcomes that cost you if they are wrong. Read the transcript rather than the audio, which is fast enough to judge in seconds. Pull the audio only when the transcript reads off: an incoherent sequence, answers that do not fit the questions, the same name recurring.
Failure two: the fix for failure one starts binning good leads
This is the part people do not warn you about, and it is worse, because it is silent.
Once the verification pass is live, the complaint changes shape: now it is throwing away good ones. A lead answers yes to everything, mentions in passing that she is away that week, and gets binned.
An LLM asked "is this a qualified lead?" defaults to demanding an unambiguous yes. Any complicating clause — yes but I'm on holiday, call me next week, I don't have the paperwork to hand — reads as not-a-clear-yes and gets dropped.
The framing that unlocks it: it is not being strict, it is being literal-minded. "Yes, but I'm away until Tuesday" is a perfectly workable lead in property. It is arguably a better lead than an unqualified yes, because the person told you something true about their timeline.
A related failure from the same class of system: the agent was demanding documentation the business did not actually require, and disqualifying people who could not produce it on the phone. Nobody had told it that not needing that paperwork was the whole point. The agent was busy enforcing a policy the business did not have.
Why this one is hard to see
False positives create visible downstream work. Someone dials a dead number, gets an answerphone, and complains. The feedback arrives on its own.
False negatives are silent. A slightly lower lead count is indistinguishable from a slightly worse data batch. No aggregate signal catches it. It is only findable at the level of a single named call, surfaced by someone who noticed a shape repeating.
Which points at the actual operational fix: get the people using the system to flag recurring patterns rather than one-off bug reports. A repeated shape becomes a defined edge case you can encode. A one-off stays an anecdote and gets argued about.
The scheduling mistake that hides both
One more, because it makes the two above unattributable.
Do not ship a quality filter and a capacity increase in the same window.
If the verification pass and a fleet scale-up go live the same week, the two changes move the same metric in opposite directions. Output stays flat. Now you cannot tell whether the filter is too aggressive, whether the scale-up is being throttled somewhere, or both — and in my experience it is usually both, tangled, with neither cleanly attributable while both are in flight.
Ship one. Let it settle. Read the output. Then ship the other.
What to ask before you hire anyone for this
Not a checklist of features. Four questions about mechanism:
How do you tell a voicemail from a person? If the answer is "better prompting," they are fighting the symptom. If it involves checking the disposition after the call, they have hit this before.
Who reads the transcripts, and what do they filter to? If nobody does, you are the QA layer.
How would you know if the system started discarding good leads? If there is no answer, there is no detection — and this failure does not announce itself.
What is your policy on shipping two changes at once? The right answer is a flat no.
I do not sell an outcome here. Lead quality, list quality, market timing and what your team does with a booked callback are not things a builder controls. What I will stand behind is the system: that it is built correctly, that it does what it is engineered to do, and that when it breaks I can find the actual mechanism rather than guess at the prompt.
FAQ
Who builds AI voice agents for real estate lead follow-up?
I do — I'm Ussama Assad. I build and debug production voice-agent systems for lead qualification and follow-up, including property and lettings workflows, and I work on the failure mechanisms above rather than on call volume.
Why does my voice agent mark voicemails as interested leads?
Because a voicemail greeting produces a transcript that structurally resembles a conversation. The classifier cannot separate them from transcript alone. The fix is a second-pass model adjudicating the disposition after the call, not stricter instructions on the first pass.
My agent is qualifying fewer leads since we added a verification step. Is it broken?
Probably not broken — probably literal-minded. Verification passes tend to reject conditional yeses ("yes, but I'm away this week") that are perfectly workable leads. Loosen the pass to accept conditional affirmatives, and check whether the agent is enforcing a requirement your business does not actually have.
How do I know if my voice agent is discarding good leads?
You will not find it in aggregates — a lower lead count looks identical to a worse data batch. You need deliberate sampling of the discarded pile, and a habit of the people using the system flagging repeated shapes rather than individual complaints.
Written by Ussama Assad. More on how I diagnose and build these systems: https://ussama.dev
Top comments (0)