What if you could call a local business, speak naturally in Hindi, Bengali, English, or a mix of them, ask about a product or order, and get help without navigating complicated menus?
That was the idea behind my project for 10 Days of Voice Agents โ VoiceForBharat Edition.
Over the course of the challenge, I evolved a basic real-time voice agent into a multilingual Local Commerce Voice Assistant with live product lookup, consent-based memory, outbound calling, human escalation, specialist-agent handoffs, and a call analytics dashboard.
The project is built using Murf Falcon, LiveKit, Deepgram, an LLM, Python and Next.js.
๐ GitHub: https://github.com/kusalar/agentvoice
๐ฏ The Problem
Local businesses often need to answer repetitive customer questions:
- Is this product available?
- What is the price?
- Where is my order?
- When will my order arrive?
- Can I return this product?
- I have a problem with my order. Can someone help me?
Traditional chat interfaces aren't always convenient, especially for users who are more comfortable speaking than typing.
This is where voice can make the interaction more natural.
My goal was to build an assistant that could provide a more conversational experience while still being useful, reliable and safe.
I chose the local commerce/customer support direction because it allowed me to combine voice AI with real business tools and workflows.
๐ฃ๏ธ Why Voice?
Voice is particularly interesting for the Indian context because conversations don't always happen in a single language.
A customer might say:
"Bhai, mera order kab tak deliver hoga?"
or:
"Product available hai kya? Price bhi bata do."
Or they may switch between Hindi and English during the same conversation.
Instead of forcing users to type formal queries, a voice interface allows them to interact naturally.
My agent is designed to handle English, Hindi, Bengali and code-mixed conversations, while trying to mirror the user's language and conversational style.
"The final Local Commerce Assistant interface: a voice-first shopping experience for discovering local products and interacting with the agent."
๐๏ธ How the Voice Agent Works
The core architecture looks like this:
โโโโโโโโโโโโโโโโโโโโโโโ
โ Customer โ
โ Browser / Phone โ
โโโโโโโโโโโโฌโโโโโโโโโโโ
โ
Voice / SIP / Chat
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโ
โ LiveKit โ
โ Real-time RTC โ
โโโโโโโโโโโโฌโโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโ
โ Deepgram STT โ
โ Speech โ Text โ
โโโโโโโโโโโโฌโโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโ
โ LiveKit Agent โ
โ LLM + Reasoning โ
โ + Tools โ
โโโโโโโโโโโโฌโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโผโโโโโโโโโโโโโโ
โ โ โ
โผ โผ โผ
Product Tool Memory DB Escalation Tool
โ โ โ
โผ โผ โผ
Local Catalogue Caller Memory SQLite Tickets
โ
โผ
Human Escalation
โ
โผ
Returns Specialist
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโ
โ Murf Falcon โ
โ Text โ Speech โ
โโโโโโโโโโโโฌโโโโโโโโโโโ
โ
โผ
๐ Customer
And specialist path:
Main Agent
โ
โ refund / returns issue
โผ
Returns & Refunds Specialist
โ
โผ
Resolution / Escalation
The four fundamental components are:
1. Speech-to-Text
I use Deepgram to convert the user's speech into text.
2. LLM
The LLM understands the request, decides how to respond and determines when it needs to use one of the available tools.
3. Text-to-Speech
I use Murf Falcon to convert the agent's responses back into natural speech.
4. Real-time Transport
LiveKit handles the real-time communication between the user and the voice agent.
๐ฎ๐ณ Indian Voice with Murf Falcon
One of the key parts of the project is the voice itself.
I used Murf Falcon for the TTS layer and experimented with Indian voices including voices such as Pooja, Anisha and Samar.
The goal wasn't simply to make the agent speak.
I wanted the interaction to feel natural enough for an Indian customer-support scenario.
Fast TTS matters in voice applications because long delays between the user finishing their sentence and the agent responding can make the conversation feel unnatural.
๐ง Agent Personality and Safety
A voice agent needs more than an LLM.
I defined a specific personality and behavioral rules for the assistant.
The agent is designed to be:
- Friendly
- Helpful
- Concise
- Professional
- Conversational
But it also has important guardrails.
For example:
It should not invent product prices or availability.
If information needs to be verified, the agent should use the appropriate tool.
Similarly, sensitive actions such as storing customer information or creating an escalation should require user consent.
This helped me move from simply building a chatbot to thinking about how a voice agent should behave in a real customer-facing environment.
๐ Live Product Lookup
One of the most useful features I built was a product lookup tool.
Instead of allowing the LLM to guess product information, the agent can query actual catalogue data.
The project uses Open Food Facts as a live source and also maintains a local catalogue as a fallback.
For example, a customer can ask:
"Do you have organic honey?"
The agent can use the tool to check the product rather than generating a random answer.
If the external service isn't available, the system can fall back to the local catalogue.
This taught me an important lesson:
An AI agent should use tools for facts instead of relying on the LLM to remember business data.
"Live product lookup: the agent retrieves catalogue information such as product name, seller, pack size and price instead of guessing product details."
๐ง Memory with User Consent
I also implemented memory for returning users.
The agent can remember useful information such as:
- Customer name
- Language preference
- Relevant customer information
- Previous interaction information
But I didn't want the system to automatically store everything a user says.
The agent asks for permission before saving information.
The basic workflow is:
Customer shares information
โ
Agent asks for permission
โ
โโโโดโโโ
โ โ
YES NO
โ โ
โผ โผ
Save Don't save
This made memory more privacy-conscious and also made me think about consent as part of the agent's design rather than an afterthought.
"Persistent memory in action: after consent, the assistant remembers the customer's name and previous interaction to provide a more personalized conversation."
๐ Call Analytics Dashboard
A voice application also needs visibility into what is happening during conversations.
During the challenge, I built a call analytics dashboard to monitor and analyze voice interactions.
Instead of only seeing whether the agent works, the dashboard provides a way to understand call activity and conversation performance.
This is important for a real deployment because businesses need to answer questions such as:
- How many calls are happening?
- How are conversations performing?
- What is happening during calls?
- Where are users encountering problems?
- How is the voice agent being used?
Building this dashboard helped me understand that production voice AI isn't only about the conversation itself.
It also needs observability and analytics.
"Call analytics dashboard: tracking call outcomes, success rates, failure categories, channels and recent conversation activity."
๐ Outbound Calling
The project also goes beyond incoming browser-based conversations.
I implemented outbound calling using LiveKit SIP.
This can be useful for scenarios such as:
- Customer follow-ups
- Order notifications
- Restock notifications
- Callback requests
- Proactive customer communication
This was an interesting part of the project because it introduced telephony into the voice-agent architecture.
"Outbound Calling: Call from team to customer for support."
๐จโ๐ผ Human Escalation
AI shouldn't try to solve every problem.
For difficult cases such as:
- Refund disputes
- Return problems
- Damaged products
- Missing orders
- Serious delivery issues
the assistant can escalate the issue to human support.
Before creating an escalation, the agent asks the customer for consent.
The escalation can contain relevant information such as the issue description and what has already been checked.
The customer can then receive a reference ID for the support request.
Sensitive information such as passwords, OTPs and payment credentials should never be collected or exposed.
"Human-in-the-loop escalation dashboard: support tickets created by the voice agent are tracked through open, in-progress and resolved states."
๐ Specialist Agent Handoff
I also implemented a specialist Returns Agent.
Instead of forcing one agent to handle every possible scenario, certain conversations can be handed to a specialist.
For example:
Main Agent
โ
Return / Refund?
โ
โผ
Returns Specialist
โ
โผ
Specialist Agent
This introduced me to a more scalable approach to voice agents:
multiple specialized agents working together instead of one enormous agent trying to do everything.
"Specialist-agent handoff: the main commerce assistant recognizes a returns/refund issue and transfers the conversation to a dedicated Returns & Refunds Specialist."
๐ The Difficult Parts
The hardest part wasn't simply getting the agent to speak.
The difficult part was making the system behave reliably.
One of the biggest problems was preventing the LLM from confidently producing information that wasn't actually verified.
For example, if someone asked:
"Is this product available?"
the LLM could theoretically generate a plausible answer even if it had no actual catalogue information.
I solved this by introducing dedicated tools for business data.
The agent uses the product lookup tool when it needs product information instead of relying on its own knowledge.
I also added fallback behavior when external product data isn't available.
Another challenge was coordinating multiple components:
Frontend
โ
LiveKit
โ
STT
โ
LLM
โ
Tools
โ
TTS
โ
Frontend
A problem in any one of these layers can affect the entire conversation.
Working through these issues taught me that a voice agent is really a system of connected components, not just an LLM with a voice.
๐ ๏ธ How to Run the Project
If you want to experiment with the project yourself, the complete source code is available on GitHub:
๐ https://github.com/kusalar/agentvoice
Requirements
You'll need:
- Python 3.10+
- Node.js 18+
uvpnpm- LiveKit
- Murf API credentials
- Deepgram API credentials
- LLM API credentials
Clone the repository:
git clone https://github.com/kusalar/agentvoice.git
cd agentvoice
Install the backend dependencies:
cd backend
uv sync
Install the required model/files:
uv run python src/agent.py download-files
Then install the frontend dependencies:
cd ../frontend
pnpm install
๐ Environment Variables
Never put API keys directly into the source code.
Create the appropriate environment file and add your credentials there.
For example:
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_llm_api_key
Use your actual credentials locally, but never commit them to GitHub or publish them in screenshots.
โถ๏ธ Running the Agent
Start the LiveKit development server:
livekit-server --dev
Run the backend agent:
cd backend
uv run python src/agent.py dev
Start the frontend:
cd frontend
pnpm dev
Then open:
http://localhost:3000
Allow microphone access and start a conversation with the agent.
๐งช Testing the Agent
A simple test conversation could be:
User:
"Do you have organic honey?"
Agent:
Checks the product tool.
User:
"How much is it?"
Agent:
Returns the verified catalogue information.
You can also test:
- Switching between Hindi and English
- Bengali conversations
- Code-mixed sentences
- Product availability
- Memory consent
- Return/refund escalation
- Specialist-agent handoff
- Outbound calling
For testing screenshots and demos, use dummy customer information rather than real personal data.
๐ธ Evidence From My Build
The project evolved throughout the 10-day challenge, and I documented different capabilities as I built them.
The most useful evidence includes:
- The final voice-agent interface
- A real conversation with the agent
- Product lookup in action
- The call analytics dashboard
- Memory/consent flow
- Human escalation
- Specialist-agent handoff
- Outbound calling
I don't need to include every screenshot from every day.
Instead, I want the screenshots to show the progression from a simple voice agent into a more complete voice system.
๐ What I Learned
The biggest lesson from this challenge is that building a voice agent isn't just about making AI talk.
A useful voice agent needs:
Voice + Reasoning + Tools + Memory + Safety + Real-time communication + Observability + Human handoff
The voice is what the user experiences.
But the tools, data validation, consent, fallback behavior, analytics and escalation logic are what make the system useful.
Over these 10 days, I learned how these different pieces can come together to create something much closer to a real-world voice application.
๐ฎ What's Next?
There is still a lot I would like to improve:
- More Indian languages
- Better production telephony
- More business integrations
- More advanced call analytics
- Better observability
- More specialist agents
- Production deployment
- Automated voice-agent evaluation
But this challenge gave me a strong foundation for exploring all of these areas.
Final Thoughts
What started as a basic voice agent became a multilingual local-commerce assistant capable of using tools, remembering users with consent, making calls, escalating problems, handing conversations to specialists and analyzing calls.
The most important thing I learned is:
A good voice agent shouldn't just sound intelligent. It should be useful, accurate, safe and measurable.
I'm grateful to Murf AI for organizing 10 Days of Voice Agents โ VoiceForBharat Edition and giving me the opportunity to build and experiment with voice AI over 10 days.
๐ Project: https://github.com/kusalar/agentvoice
Built with:
Murf Falcon + LiveKit + Deepgram + LLM + Python + Next.js
Top comments (0)