DEV Community: Luis Pham

What I Learned About Latency While Building a Real-Time Voice AI Agent

Luis Pham — Fri, 08 May 2026 13:10:00 +0000

What I Learned About Latency While Building a Real-Time Voice AI Agent

When I started building a real-time voice AI agent, I thought about latency mostly as an engineering problem.

Reduce the delay.

Make the response faster.

Stream audio as quickly as possible.

That is still true.

But after working on RingBooker, an AI receptionist for salons, spas, med spas, and beauty clinics, I started to think about latency differently.

Latency is not only a technical metric.

On a phone call, latency is part of the user experience.

A small delay feels bigger on the phone

In a web app, a short delay is usually fine.

A button can show a loading state.

A page can display a spinner.

A chatbot can show typing dots.

The user understands that something is happening.

A phone call does not have that same visual feedback.

When the caller stops talking and the AI does not respond, even a short pause can feel strange.

The caller may wonder:

“Did it hear me?”

or:

“Is the call still connected?”

That emotional reaction matters.

The system might be working perfectly in the background, but the caller does not see that.

They only hear silence.

End-to-end latency matters more than one number

At first, it is tempting to measure only one part of the system.

Model response time.

Speech-to-text time.

Text-to-speech time.

Network delay.

But the caller experiences the whole chain.

For a voice AI agent, the real question is:

How long does it take from the moment the caller stops speaking to the moment they hear a useful response?

That path can include:

caller audio
voice activity detection
speech-to-text
intent understanding
model response
tool calls or retrieval
text-to-speech
audio streaming back to the caller
telephony network behavior

Optimizing one piece helps, but it does not always fix the felt experience.

The full loop is what matters.

“Fast” is not always the same as “natural”

This surprised me.

I assumed faster would always feel better.

But if the AI responds too instantly, it can feel unnatural.

Humans usually leave tiny pauses in conversation.

They breathe.

They process.

They acknowledge.

They sometimes say “okay” before moving forward.

A voice AI that snaps back too quickly can feel robotic, even if the latency is technically great.

So the goal is not always the lowest possible delay.

The goal is a response rhythm that feels alive and useful.

Some delays are more acceptable than others

Not all latency feels the same.

If the caller asks a simple question, they expect a quick answer.

Example:

“Are you open today?”

A long delay here feels bad.

But if the caller asks something more complex, a short pause can feel normal.

Example:

“Can you help me find an appointment for a color service next week?”

In that case, a small delay may be acceptable if the AI acknowledges what is happening:

“Let me get a few details first.”

or:

“I can help collect that request.”

The user experience depends on context.

This is where product design and engineering meet.

Silence needs design

One thing I learned is that silence cannot be ignored.

If the system needs time, the conversation should make that clear.

This does not mean adding filler everywhere.

Too much filler is annoying.

But the AI needs ways to keep the caller oriented.

For example:

acknowledge the request
ask one clear follow-up question
avoid long unexplained pauses
do not overtalk while processing
hand off when the request is too specific

A lot of voice UX is about making the caller feel that the system is still present.

Tool calls make latency more complicated

For a simple conversation, the AI can respond directly.

But real products often need tools.

For local businesses, a voice agent may need to:

check business hours
understand service rules
collect appointment details
look up knowledge base information
prepare a call summary
decide whether to hand off

Every tool call can add delay.

This creates a tradeoff.

More context can make the answer better.

But too much waiting can make the call feel worse.

The question becomes:

Does this tool call improve the caller experience enough to justify the delay?

That is not only an engineering question.

It is a product question.

Call summaries changed how I thought about latency

Early on, I wanted the AI to answer as much as possible during the call.

But for local businesses, the after-call summary is often just as important.

The caller needs a fast, useful interaction.

The business needs structured context after the call.

That means the AI does not always need to solve everything live.

Sometimes it is better to keep the call simple, collect the right information, and give the business a clear next step.

This reduces pressure on the live conversation.

It also avoids forcing the caller to wait while the AI tries to do too much.

Latency affects trust

This is probably the most important lesson.

When a voice AI pauses awkwardly, talks over the caller, or responds too slowly, the problem is not just speed.

The problem is trust.

The caller may start to feel:

“This is not reliable.”

For salons, spas, med spas, and other appointment-based businesses, that matters.

A caller might be trying to book a same-day service, ask about a consultation, reschedule, or decide whether the business feels responsive.

The phone call is part of the trust-building process.

If the AI feels slow or confused, the business may feel slow or confused too.

That is why latency is not just a backend concern.

It is a brand experience.

What I would measure

If I were starting from scratch, I would measure latency in layers.

Not only:

How fast did the model respond?

But also:

how long until speech was detected
how long until the caller’s intent was understood
how long until the first useful audio came back
how often the AI talked over the caller
how often the caller interrupted
how often the AI needed to recover
how often a human handoff was needed

The raw timing is useful.

But the conversation outcome matters more.

A fast bad answer is still a bad answer.

Final thought

Latency in voice AI is not just about making the system faster.

It is about making the conversation feel responsive.

The caller should feel heard.

The AI should avoid awkward silence.

The system should not overtalk.

The business should receive useful context.

That is the hard part.

The best voice AI products will not only optimize milliseconds.

They will optimize the feeling of the call.

The Hidden UX Problem in Voice AI: When Should the AI Stop Talking?

Luis Pham — Sun, 03 May 2026 13:08:00 +0000

The Hidden UX Problem in Voice AI: When Should the AI Stop Talking?

One of the hardest parts of building a voice AI product is not making the AI talk.

It is knowing when the AI should stop talking.

I did not fully appreciate this at the beginning.

When I started building RingBooker, an AI receptionist for salons, spas, med spas, beauty clinics, I was focused on the obvious problems:

Latency.

Speech recognition.

Call routing.

Booking intent.

Call summaries.

Those are all important.

But the more I worked on real phone-call flows, the more I realized that silence, interruption, and timing are part of the product.

Text AI can explain. Voice AI has to pace itself.

In a text interface, long answers can still work.

The user can skim.

They can scroll.

They can reread.

They can ignore parts of the answer.

On a phone call, the user cannot skim.

They have to listen in real time.

That means every extra sentence costs attention.

A voice agent that gives a complete answer may still feel bad if it talks for too long.

This is especially true for local business calls.

A caller usually does not want a long explanation. They want to know what to do next.

The AI should not fill every silence

This was one of my early mistakes.

I assumed silence was always bad.

So I wanted the AI to respond quickly, keep the conversation moving, and avoid awkward pauses.

But not every pause needs to be filled.

Sometimes the caller is thinking.

Sometimes they are checking their calendar.

Sometimes they are asking someone next to them.

Sometimes they are about to correct themselves.

If the AI jumps in too quickly, it feels pushy.

If it waits too long, it feels broken.

That middle ground is harder than it sounds.

Interruptions are not edge cases

In voice AI, interruption handling is not a feature you add later.

It is core UX.

People interrupt naturally.

They say:

“Actually, wait...”

or:

“No, I meant tomorrow.”

or:

“Can I ask something else?”

If the AI ignores that and keeps talking, the user immediately feels that they are not being heard.

This is different from text.

In text, the assistant can finish its response and the user can reply after.

On a call, the timing itself communicates whether the AI is listening.

A good voice agent needs shorter answers

This is one of the product rules I keep coming back to.

For phone calls, shorter is usually better.

Not because the user is impatient, but because voice is linear.

The caller cannot jump ahead.

For example, if someone calls a salon and asks:

“Do you have anything today?”

The AI does not need to explain the entire booking process.

It probably needs to say something like:

“I can help check the request. What service are you looking for?”

Then collect the useful details.

The goal is not to sound smart.

The goal is to move the call forward.

The AI should ask one thing at a time

This is another lesson from phone UX.

In a chatbot, you can ask multiple questions at once:

What service do you need, what day works best, and do you have a preferred provider?

That can work in text.

On a call, it often fails.

The caller may answer only one part.

Or they may forget the first question.

Or they may respond vaguely.

For voice, one question at a time usually works better.

Bad:

“What service are you looking for, what time do you prefer, and is there a specific stylist?”

Better:

“What service are you looking for?”

Then:

“Do you have a preferred day or time?”

Then:

“Do you have a preferred stylist?”

It feels slower on paper, but it is often smoother in conversation.

Knowing when to hand off is part of the UX

Sometimes the best thing the AI can do is stop trying to solve the call.

This is especially true when the caller asks for something sensitive, complex, or very specific.

For example:

a medical-aesthetic treatment question that needs professional judgment
a pricing question that depends on consultation
a complaint
a caller who repeatedly asks for a human
a policy exception
a complicated reschedule

In those moments, continuing to talk can hurt trust.

A good voice agent should be comfortable saying:

“I’ll pass this to the team so they can follow up with the right answer.”

That is not failure.

That is good product behavior.

The summary should replace unnecessary talking

Another thing I learned: the AI does not need to explain everything to the caller if the real value is in the follow-up.

For local businesses, the call often has two users:

The caller.

The business team.

The caller wants a quick response.

The business wants clean context.

So instead of making the AI over-explain, the product can capture the details and send the team a useful summary.

That means the AI can keep the call shorter while still creating value.

Voice AI needs boundaries, not just intelligence

A lot of AI products are designed to show how capable the model is.

But on the phone, capability without restraint can feel uncomfortable.

The AI should know:

When to answer.

When to ask a follow-up.

When to pause.

When to stop.

When to hand off.

Those decisions shape the user experience as much as the model quality.

Maybe more.

What I am trying to optimize for

For RingBooker, I am not trying to make the AI sound like the most impressive receptionist in the world.

I am trying to make it useful in the calls a beauty business normally misses:

After-hours calls.

Peak-hour overflow.

Same-day requests.

Reschedules.

Consultation inquiries.

Human handoff requests.

In those moments, the AI does not need to dominate the conversation.

It needs to help the caller feel heard and give the business enough context to act.

Final thought

The hidden UX problem in voice AI is restraint.

Knowing what to say matters.

But knowing when to stop talking may matter even more.

A voice AI agent should not try to win the conversation.

It should help the caller get to the next step.

Why Voice AI for Local Businesses Is Harder Than a Chatbot

Luis Pham — Thu, 30 Apr 2026 12:06:00 +0000

Why Voice AI for Local Businesses Is Harder Than a Chatbot

I used to think a voice AI agent was basically a chatbot with audio.

User speaks.

AI understands.

AI replies.

That was the simple version in my head.

But after working on RingBooker, an AI receptionist for salons, spas, med spas, and beauty clinics, I started to see voice AI very differently.

A chatbot can be useful even when it feels a little slow.

A phone call cannot.

Text gives the user more patience

When someone uses a chatbot, they expect a short delay.

They can see the answer being generated.

They can reread the message.

They can scroll back.

They can pause before replying.

The experience gives them space.

A phone call does not.

On a call, silence feels uncomfortable almost immediately.

If the AI waits too long, the caller may think the call dropped.

If the AI replies too quickly, it can feel unnatural.

If the AI talks too much, the caller interrupts.

That makes the timing much harder to get right.

Voice AI has to feel alive

In text, the user judges the quality mostly by the final answer.

In voice, the user judges the whole interaction.

The pause before the answer.

The tone.

The interruption handling.

The confidence.

The moment when the AI says “Let me check that.”

The way it handles uncertainty.

Even when the underlying model is good, the experience can still feel bad if the voice flow is awkward.

This was one of the first things I had to accept:

The model is only one part of the product.

The conversation design is just as important.

Interruptions are normal, not edge cases

In a chatbot, the flow is usually clean.

The user sends a message.

The assistant replies.

Then the user sends another message.

Phone calls are not like that.

People interrupt.

They correct themselves.

They ask a second question before the first one is answered.

They start with one intent and change it halfway through.

For local businesses, this happens all the time.

A salon caller might say:

“Do you have anything today? Actually, tomorrow morning would be better.”

A med spa caller might say:

“I’m interested in laser. Wait, is that the same as IPL?”

A spa caller might ask:

“How much is a massage? Also, do you have couples appointments?”

If the AI cannot handle interruptions, the caller feels trapped inside a script.

That is not a good experience.

The input is messy

Most AI demos happen in clean environments.

Real phone calls do not.

People call from cars.

They call from busy rooms.

They speak quietly.

They use vague words.

They ask incomplete questions.

They may not know the correct service name.

For a chatbot, messy input is annoying.

For a phone agent, messy input is the default.

This changes the product design.

The AI has to ask follow-up questions, but not too many.

It has to collect useful information, but not sound like a form.

It has to be helpful, but not overconfident.

That balance is difficult.

Local businesses need boundaries

One mistake I see in many AI product ideas is trying to make the AI do everything.

Answer every question.

Book every appointment.

Handle every exception.

Replace every human step.

For local businesses, I think that is the wrong starting point.

A salon, spa, or med spa does not need an AI that pretends to be perfect.

They need an AI that can reliably help with the calls the team cannot always answer.

That might mean:

answering after-hours calls
collecting booking intent
asking for preferred time
capturing service details
summarizing the call
handing off when needed

The handoff is not a failure.

Sometimes the handoff is the product working correctly.

Trust is more fragile on the phone

In a chatbot, a wrong answer is bad.

On a phone call, a wrong answer can feel worse.

The caller is giving attention in real time. They may be trying to book something, ask about pricing, reschedule, or decide whether the business feels trustworthy.

If the AI sounds too confident about something it should not promise, trust drops.

If it pretends to know a policy it does not know, trust drops.

If it refuses to hand off when the caller asks for a human, trust drops.

For appointment-based businesses, trust matters because the call is often part of the buying decision.

This is especially true for services like hair color, skin treatments, injections, laser, massage, or first-time consultations.

The caller is not only asking for information.

They are testing whether the business feels responsive.

The transcript is not enough

At first, I thought the transcript would be one of the most important outputs.

But the more I thought about the workflow, the more I realized most business owners do not want to read long transcripts.

They want the useful summary.

Who called?

What did they want?

How urgent was it?

What service were they asking about?

What should the team do next?

A clean summary can be more useful than a perfect transcript.

This is one of the biggest differences between building a demo and building a product.

The demo is about showing that the AI can talk.

The product is about helping the business take action after the call.

The goal is not to sound impressive

A good voice AI product should not be measured only by how smart it sounds.

For local businesses, I think the better questions are:

Did it answer quickly?

Did it understand the caller’s intent?

Did it avoid making promises it should not make?

Did it know when to ask a follow-up question?

Did it know when to hand off?

Did it send the team something useful?

That is a more practical benchmark.

It is also harder than it sounds.

What I would build around first

If I were starting again, I would not start with the most complex booking flow.

I would start with the most common missed-call situations:

After-hours callers.

Busy-hour overflow.

Same-day appointment requests.

Reschedules.

Basic pricing questions.

Consultation inquiries.

Human handoff requests.

These are not the most glamorous flows, but they are the ones that happen every day.

And for a local business, capturing one missed opportunity can matter more than having a perfect AI demo.

Final thought

Voice AI is not just chatbot logic plus speech.

It is a different product surface.

The user experience is faster, messier, and less forgiving.

That is what makes it hard.

But that is also what makes it interesting.

For local businesses, the phone is still where many high-intent customers show up. If AI can help answer those calls without pretending to replace the human team, I think there is a real product there.

Building an AI Receptionist for Salons: What Was Harder Than I Expected

Luis Pham — Tue, 28 Apr 2026 06:47:08 +0000

Building an AI Receptionist for Salons: What Was Harder Than I Expected

I’m building RingBooker, an AI receptionist for salons, spas, med spas, and other appointment-based businesses.

When I started, I thought the product was mostly about the AI.

Answer the phone.

Understand the caller.

Collect the booking details.

Send the business a summary.

Simple enough.

It turned out the hard part was not only the AI model. The hard part was the phone call itself.

A phone call has no “loading state”

With a chatbot, a user can wait.

They can see the message is generating. They can scroll back. They can reread the answer.

On a phone call, silence feels broken.

Even a short delay can make the caller wonder:

“Is this still listening?”

That changed how I thought about latency.

At first I was looking at latency like a normal backend metric. How fast is the response? How long does the model take? How quickly can the audio come back?

But in a real phone call, the user does not care about the number.

They care about whether the conversation feels alive.

The caller will interrupt

This was another thing I underestimated.

In text, the flow is clean. User sends a message. Assistant replies.

On a call, people interrupt constantly.

They start with one request, then change it halfway through.

“Do you have anything today? Actually tomorrow is better.”

“I need a refill. Wait, maybe a full set.”

“Can I speak to someone? No, actually I just want to know the price first.”

If the AI keeps talking when the caller is trying to correct something, the whole experience feels wrong.

Barge-in is not a small feature. It is part of the core UX.

Local business calls are messy

A lot of AI demos assume the user gives clean input.

Real callers do not.

They speak from cars.

They call from noisy rooms.

They use vague phrases.

They ask two questions at once.

They sometimes do not know the exact service name.

For salons and spas, this is common.

A caller may say “nails” when they mean acrylic full set.

A med spa caller may ask about “laser” without knowing which treatment.

A hair salon caller may ask for “color” without knowing whether it is highlights, root touch-up, or correction.

So the AI cannot just collect form fields. It has to ask enough follow-up questions without making the call feel like an interrogation.

That balance is harder than I expected.

The summary matters more than the transcript

At first I cared a lot about the transcript.

Then I realized the business owner probably does not want to read a full call transcript.

They want the useful part:

who called
what they wanted
how urgent it was
what questions they asked
what should happen next

For a busy salon owner, a clean summary is more valuable than a perfect transcript.

This changed the product direction for me.

The call itself is only half the product. The handoff to the business is the other half.

The AI should not try to handle everything

This is probably the biggest lesson so far.

It is tempting to make the AI answer every question and complete every flow.

But for real businesses, that is risky.

Some calls should go to a human.

Some questions depend on policy.

Some prices depend on consultation.

Some callers just want reassurance.

A useful AI phone agent needs to know its boundary.

For RingBooker, I started thinking less about “AI replacing the front desk” and more about “AI covering the calls the team cannot answer.”

That framing feels much healthier.

The existing phone number is part of the product

This was not obvious to me at the beginning.

For many local businesses, the phone number is everywhere:

Google Business Profile, website, ads, Instagram, business cards, printed signs, old customers’ phones.

Asking them to change that number is a huge ask.

So call forwarding became an important part of the product idea.

The business should be able to keep the number customers already know, while RingBooker sits behind the front-desk line for missed, overflow, or after-hours calls.

It sounds like a small detail, but for local businesses it is a big trust issue.

What I would tell another builder

If you are building a voice AI product, do not start only with the model.

Start with the awkward parts of the call.

What happens when the caller interrupts?

What happens when the audio is bad?

What happens when the AI is unsure?

What happens when the caller asks for a human?

What happens after the call ends?

Those edge cases are not edge cases for long. They become the product.

Final thought

I still think voice AI will become very important for local businesses.

But I no longer think the goal is to make the AI sound impressive.

The goal is simpler and harder:

Answer quickly.

Be clear.

Do not overpromise.

Know when to hand off.

Give the business a useful next step.

That is the version of the product I’m trying to build.