Yogapriya N

Posted on Aug 15

Building Anisha: My 10-Day Journey Building a Voice Agent for Local Commerce

#ai #voiceai #python #webdev

From a simple voice assistant to a real-time AI system with memory, tools, phone calls, analytics, and specialist handoffs

For the past 10 days, I have been building a voice agent as part of 10 Days of Voice Agents — VoiceForBharat Edition by Murf AI.

What started as a basic voice conversation gradually became something much bigger: an AI assistant that can remember customers, look up product information, make outbound calls, escalate difficult situations to humans, track call outcomes, and hand conversations to specialist agents.

Meet Anisha — my AI voice assistant for local commerce.

1. The Problem I Wanted to Solve

I chose the Local Commerce track.

Local kirana stores handle many repetitive customer questions every day:

"What is the price of basmati rice?"
"Is this product available?"
"Do you provide home delivery?"
"Can I return this item?"
"Can I speak to someone from the store?"

For many customers, especially when they are busy or not comfortable navigating a complicated application, speaking naturally is easier than typing.

That's where Anisha comes in.

Anisha acts as a voice-based assistant for a local kirana store and helps customers get information through a natural conversation.

The goal wasn't just to make an AI that could talk.

The goal was to make a voice agent that could actually do useful work.

2. Meet Anisha

Anisha is a conversational AI assistant designed for local commerce.

She can help with product prices, stock availability, store-related questions, customer memory, human escalation, and specialist handoffs.

I designed her personality to be:

Friendly
Helpful
Concise
Natural in conversation
Careful about information she cannot verify

The voice is powered by Murf Falcon, which became one of the most important parts of the project because the voice needs to feel natural rather than like a traditional text-to-speech system.

Anisha's interface

The homepage introduces Anisha and provides a simple Start a Call experience.

The interface also highlights some of the main capabilities such as prices and stock, personalization, and human support.

3. How the System Works

The basic voice pipeline looks like this:

              ┌───────────────┐
              │    Customer   │
              │     Voice     │
              └───────┬───────┘
                      │
                      ▼
              ┌───────────────┐
              │    LiveKit    │
              │ Real-time RTC │
              └───────┬───────┘
                      │
                      ▼
              ┌───────────────┐
              │   Deepgram    │
              │      STT      │
              └───────┬───────┘
                      │
                      ▼
              ┌───────────────┐
              │      LLM      │
              │   Reasoning   │
              └───────┬───────┘
                      │
          ┌───────────┼───────────┐
          ▼           ▼           ▼
      Product      Memory      Escalation /
       Tools       / DB         Handoff
          │           │           │
          └───────────┼───────────┘
                      ▼
              ┌───────────────┐
              │  Murf Falcon  │
              │      TTS      │
              └───────┬───────┘
                      │
                      ▼
              ┌───────────────┐
              │    Customer   │
              │     Voice     │
              └───────────────┘

The main components are:

Speech-to-text: Deepgram converts the customer's voice into text.

LLM: The language model understands the request, decides what to do, and generates a response.

Tools: When the agent needs real information, it can call tools instead of guessing.

Memory: Customer information can be stored and retrieved for returning users.

Text-to-speech: Murf Falcon converts the response into natural speech.

Real-time transport: LiveKit handles the real-time audio communication.

The Voice Agent Session

The core voice pipeline is configured as an agent session. For example:

session = AgentSession(
    stt=deepgram.STT(
        model="nova-3",
        endpointing_ms=100,
        no_delay=True,
    ),
    llm=llm_provider,
    tts=murf.TTS(
        voice="Anisha",
        style="Conversational",
        sample_rate=24000,
        streaming=True,
        text_pacing=True,
    ),
    vad=ctx.proc.userdata["vad"],
)

4. Giving the Agent Real Capabilities

One of the biggest lessons I learned was that a voice agent shouldn't simply generate answers.

It should be able to take actions.

Product Lookup

For example, if a customer asks:

"What is the price of basmati rice?"

Anisha should not guess.

Instead, she uses the product lookup tool to retrieve the actual product information.

The Product Lookup Tool

Instead of allowing the LLM to guess product information, I exposed a tool that retrieves the actual data:

@function_tool
async def lookup_product(
    self,
    context: RunContext,
    product_name: str
) -> str:
    """Retrieve real grocery product details."""
    return await self._execute_product_lookup(
        "lookup_product",
        product_name,
        context
    )

The tool connects the conversation to the product data, so Anisha can provide verified information about price and availability instead of making up an answer.

The same applies to stock availability.

This is important because hallucinating a product price in a commerce application can directly mislead a customer.

So I added guardrails telling the agent to use the product lookup tool whenever verified product information is required.

5. Memory for Returning Customers

Another feature I added was customer memory.

Instead of treating every conversation as completely new, Anisha can retrieve information associated with returning callers.

This makes the interaction more personalized.

Memory also introduced an important engineering lesson:

Storing information is easy. Using it correctly in a natural conversation is harder.

The agent needs to retrieve relevant information without making the conversation feel robotic or exposing unnecessary stored information.

6. Human Escalation

AI shouldn't try to solve everything.

Some situations require a human.

For example, if a customer has a complicated payment or order issue, Anisha can recognize that the request needs human support instead of pretending she can resolve it.

The agent can collect the relevant information and ask the customer for permission before passing the issue to human support.

This was an important design principle for me:

A good voice agent should know when it is better to stop being the only agent.

7. Specialist Agent Handoff

I also implemented a specialist handoff for returns and refunds.

For example, if a customer says:

"I bought basmati rice yesterday, but the packet was damaged. I want to return it."

Anisha recognizes that this is a returns/refunds issue.

Instead of continuing to answer as the general commerce assistant, she transfers the conversation to a specialist.

The specialist receives the context of the conversation and continues helping the customer.

This made the system feel much closer to a real support workflow.

Instead of building one giant agent that handles everything, different agents can have different responsibilities.

8. Outbound Phone Calls

The project also includes outbound SIP calling.

This allowed me to move beyond browser-based conversations and experiment with actual phone communication.

This was one of the most exciting parts of the project because it demonstrated how a voice agent can exist beyond a web interface.

The same conversational system can be connected to telephony infrastructure and used for phone-based interactions.

9. Call Analytics Dashboard

Once an agent starts handling calls, another question becomes important:

How are those calls performing?

So I built an operations dashboard to track call outcomes.

The dashboard shows information such as:

Successful calls
Failed calls
Success rate
Total calls
Calls by source
Recent calls
Call outcomes
Call duration
Conversation topics

This turns the project from simply being a voice demo into something that can also be monitored as an application.

10. The Hardest Part: When the Agent Went Silent

Not everything worked on the first attempt.

One of the most frustrating problems happened after I introduced function calling.

I had a strange situation where asking about stock availability could work, but asking for a product price could sometimes result in the agent becoming silent.

At first, it wasn't obvious where the problem was.

Was it the database?

Was the tool being called?

Was the LLM returning the wrong format?

Was the TTS system receiving something it couldn't speak?

I had to trace the complete flow:

User request
    ↓
LLM
    ↓
Tool call
    ↓
Database lookup
    ↓
Tool result
    ↓
LLM response
    ↓
TTS
    ↓
Voice output

The important lesson was that tool output in a voice agent is different from tool output in a normal text application.

A response that is technically valid for an application isn't necessarily a good response for a TTS pipeline.

I changed the tool results into natural spoken text and added sanitization around the TTS flow so that technical fragments such as JSON or other unwanted formatting would not accidentally be spoken to the user.

After that, I tested the complete tool flow again.

This was probably one of my biggest lessons from the challenge:

Debugging a voice agent means debugging the entire conversation pipeline, not just the LLM.

11. What I Learned During the Challenge

Over these 10 days, I learned that building a voice agent involves much more than connecting an LLM to a microphone.

1. Voice needs a complete pipeline

You need speech recognition, reasoning, text-to-speech, and real-time communication working together.

2. Tools make an agent useful

An LLM can understand a question, but tools allow it to access real application data.

3. Guardrails are essential

The agent should know what it can and cannot claim.

4. Voice responses need special design

Text that looks fine on a screen may sound terrible when spoken.

5. Memory changes the experience

A returning user should not always feel like a completely new customer.

6. AI should know its limits

Human escalation is not a failure. In many situations, it is the correct behavior.

7. Specialist agents can make systems easier to manage

Instead of one agent handling every possible situation, specialized agents can handle specific workflows.

12. How You Can Build Your Own Voice Agent

If you want to build a similar system, start with the simplest possible pipeline.

You need four main pieces:

Speech-to-Text
       ↓
      LLM
       ↓
Text-to-Speech
       ↓
Real-Time Transport

For my project, I used:

Deepgram — speech-to-text
LLM — conversation and reasoning
Murf Falcon — text-to-speech
LiveKit — real-time voice transport

Then add capabilities one at a time:

Get basic voice conversation working.
Add a clear system prompt.
Add safety guardrails.
Add one useful tool.
Add database memory.
Add human escalation.
Add telephony.
Add analytics.
Add specialist handoffs.

Don't try to build everything at once.

13. Running the Project

The complete source code is available on GitHub:

Anisha — VoiceForBharat GitHub Repository

Basic setup

Clone the repository:

git clone https://github.com/YogapriyaN2007/voice-for-bharath.git
cd voice-for-bharath

Create and activate the project's virtual environment and install the required dependencies according to the repository instructions.

Then configure your environment variables.

For example:

MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
GOOGLE_API_KEY=your_key_here
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

Never commit real API keys, SIP credentials, phone numbers, caller information, or .env files to GitHub.

Follow the repository's setup instructions to start the agent and frontend, then connect to Anisha and test a conversation.

Try questions such as:

"What is the price of basmati rice?"

"Is basmati rice available?"

"I need help with my order."

"I want to return a damaged product."

14. What I Would Improve Next

This project is a starting point, not the final version.

Some things I would like to improve next are:

Better multilingual support
More robust error recovery
More detailed call analytics
Improved latency monitoring
More specialized agents
Better production-grade authentication
More realistic commerce integrations
Improved UI feedback during live conversations
Better testing for long and complex voice conversations

15. Final Thoughts

When I started this challenge, I thought building a voice agent mainly meant making an AI that could listen and speak.

After 10 days, I understand that a useful voice agent is much more than that.

It needs:

Conversation + tools + memory + guardrails + real-time communication + monitoring + escalation.

The most valuable part of this challenge wasn't just getting Anisha to talk.

It was learning how to make her useful, reliable, and aware of her limitations.

I'm really happy to have completed 10 Days of Voice Agents — VoiceForBharat Edition and to have built Anisha from a basic voice assistant into a more complete local-commerce voice system.

A huge thank you to Murf AI for creating this challenge and making it possible to explore voice AI by actually building something.

Built With

Murf Falcon
LiveKit
Deepgram
LLM
Python
TypeScript
SQLite
SIP / Telephony

Project

Anisha — Local Commerce Voice Agent

Track: Local Commerce

Challenge: 10 Days of Voice Agents — VoiceForBharat Edition

Repository: GitHub — voice-for-bharath

DEV Community