Building a Multilingual Voice Agent for Local Commerce with Murf Falcon

KUSAL SARKAR — Sat, 15 Aug 2026 10:59:30 +0000

What if you could call a local business, speak naturally in Hindi, Bengali, English, or a mix of them, ask about a product or order, and get help without navigating complicated menus?

That was the idea behind my project for 10 Days of Voice Agents — VoiceForBharat Edition.

Over the course of the challenge, I evolved a basic real-time voice agent into a multilingual Local Commerce Voice Assistant with live product lookup, consent-based memory, outbound calling, human escalation, specialist-agent handoffs, and a call analytics dashboard.

The project is built using Murf Falcon, LiveKit, Deepgram, an LLM, Python and Next.js.

🔗 GitHub: https://github.com/kusalar/agentvoice

🎯 The Problem

Local businesses often need to answer repetitive customer questions:

Is this product available?
What is the price?
Where is my order?
When will my order arrive?
Can I return this product?
I have a problem with my order. Can someone help me?

Traditional chat interfaces aren't always convenient, especially for users who are more comfortable speaking than typing.

This is where voice can make the interaction more natural.

My goal was to build an assistant that could provide a more conversational experience while still being useful, reliable and safe.

I chose the local commerce/customer support direction because it allowed me to combine voice AI with real business tools and workflows.

🗣️ Why Voice?

Voice is particularly interesting for the Indian context because conversations don't always happen in a single language.

A customer might say:

"Bhai, mera order kab tak deliver hoga?"

or:

"Product available hai kya? Price bhi bata do."

Or they may switch between Hindi and English during the same conversation.

Instead of forcing users to type formal queries, a voice interface allows them to interact naturally.

My agent is designed to handle English, Hindi, Bengali and code-mixed conversations, while trying to mirror the user's language and conversational style.

"The final Local Commerce Assistant interface: a voice-first shopping experience for discovering local products and interacting with the agent."

🏗️ How the Voice Agent Works

The core architecture looks like this:

                 ┌─────────────────────┐
                 │      Customer       │
                 │    Browser / Phone  │
                 └──────────┬──────────┘
                            │
                    Voice / SIP / Chat
                            │
                            ▼
                 ┌─────────────────────┐
                 │      LiveKit        │
                 │   Real-time RTC     │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │    Deepgram STT     │
                 │    Speech → Text    │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │   LiveKit Agent     │
                 │   LLM + Reasoning   │
                 │      + Tools        │
                 └──────────┬──────────┘
                            │
              ┌─────────────┼─────────────┐
              │             │             │
              ▼             ▼             ▼
        Product Tool    Memory DB    Escalation Tool
              │             │             │
              ▼             ▼             ▼
       Local Catalogue  Caller Memory  SQLite Tickets
                                          │
                                          ▼
                                   Human Escalation
                                          │
                                          ▼
                                Returns Specialist
                                          │
                                          ▼
                                 ┌─────────────────────┐
                                 │    Murf Falcon      │
                                 │    Text → Speech    │
                                 └──────────┬──────────┘
                                            │
                                            ▼
                                       🔊 Customer

And specialist path: 

Main Agent
    │
    │ refund / returns issue
    ▼
Returns & Refunds Specialist
    │
    ▼
Resolution / Escalation

The four fundamental components are:

1. Speech-to-Text

I use Deepgram to convert the user's speech into text.

2. LLM

The LLM understands the request, decides how to respond and determines when it needs to use one of the available tools.

3. Text-to-Speech

I use Murf Falcon to convert the agent's responses back into natural speech.

4. Real-time Transport

LiveKit handles the real-time communication between the user and the voice agent.

🇮🇳 Indian Voice with Murf Falcon

One of the key parts of the project is the voice itself.

I used Murf Falcon for the TTS layer and experimented with Indian voices including voices such as Pooja, Anisha and Samar.

The goal wasn't simply to make the agent speak.

I wanted the interaction to feel natural enough for an Indian customer-support scenario.

Fast TTS matters in voice applications because long delays between the user finishing their sentence and the agent responding can make the conversation feel unnatural.

🧠 Agent Personality and Safety

A voice agent needs more than an LLM.

I defined a specific personality and behavioral rules for the assistant.

The agent is designed to be:

Friendly
Helpful
Concise
Professional
Conversational

But it also has important guardrails.

For example:

It should not invent product prices or availability.

If information needs to be verified, the agent should use the appropriate tool.

Similarly, sensitive actions such as storing customer information or creating an escalation should require user consent.

This helped me move from simply building a chatbot to thinking about how a voice agent should behave in a real customer-facing environment.

🛒 Live Product Lookup

One of the most useful features I built was a product lookup tool.

Instead of allowing the LLM to guess product information, the agent can query actual catalogue data.

The project uses Open Food Facts as a live source and also maintains a local catalogue as a fallback.

For example, a customer can ask:

"Do you have organic honey?"

The agent can use the tool to check the product rather than generating a random answer.

If the external service isn't available, the system can fall back to the local catalogue.

This taught me an important lesson:

An AI agent should use tools for facts instead of relying on the LLM to remember business data.

"Live product lookup: the agent retrieves catalogue information such as product name, seller, pack size and price instead of guessing product details."

🧠 Memory with User Consent

I also implemented memory for returning users.

The agent can remember useful information such as:

Customer name
Language preference
Relevant customer information
Previous interaction information

But I didn't want the system to automatically store everything a user says.

The agent asks for permission before saving information.

The basic workflow is:

Customer shares information
          ↓
Agent asks for permission
          ↓
       ┌──┴──┐
       │     │
      YES    NO
       │     │
       ▼     ▼
     Save   Don't save

This made memory more privacy-conscious and also made me think about consent as part of the agent's design rather than an afterthought.

"Persistent memory in action: after consent, the assistant remembers the customer's name and previous interaction to provide a more personalized conversation."

📊 Call Analytics Dashboard

A voice application also needs visibility into what is happening during conversations.

During the challenge, I built a call analytics dashboard to monitor and analyze voice interactions.

Instead of only seeing whether the agent works, the dashboard provides a way to understand call activity and conversation performance.

This is important for a real deployment because businesses need to answer questions such as:

How many calls are happening?
How are conversations performing?
What is happening during calls?
Where are users encountering problems?
How is the voice agent being used?

Building this dashboard helped me understand that production voice AI isn't only about the conversation itself.

It also needs observability and analytics.

"Call analytics dashboard: tracking call outcomes, success rates, failure categories, channels and recent conversation activity."

📞 Outbound Calling

The project also goes beyond incoming browser-based conversations.

I implemented outbound calling using LiveKit SIP.

This can be useful for scenarios such as:

Customer follow-ups
Order notifications
Restock notifications
Callback requests
Proactive customer communication

This was an interesting part of the project because it introduced telephony into the voice-agent architecture.

"Outbound Calling: Call from team to customer for support."

👨‍💼 Human Escalation

AI shouldn't try to solve every problem.

For difficult cases such as:

Refund disputes
Return problems
Damaged products
Missing orders
Serious delivery issues

the assistant can escalate the issue to human support.

Before creating an escalation, the agent asks the customer for consent.

The escalation can contain relevant information such as the issue description and what has already been checked.

The customer can then receive a reference ID for the support request.

Sensitive information such as passwords, OTPs and payment credentials should never be collected or exposed.

"Human-in-the-loop escalation dashboard: support tickets created by the voice agent are tracked through open, in-progress and resolved states."

🔄 Specialist Agent Handoff

I also implemented a specialist Returns Agent.

Instead of forcing one agent to handle every possible scenario, certain conversations can be handed to a specialist.

For example:

                 Main Agent
                     │
             Return / Refund?
                     │
                     ▼
             Returns Specialist
                     │
                     ▼
              Specialist Agent

This introduced me to a more scalable approach to voice agents:

multiple specialized agents working together instead of one enormous agent trying to do everything.

"Specialist-agent handoff: the main commerce assistant recognizes a returns/refund issue and transfers the conversation to a dedicated Returns & Refunds Specialist."

😅 The Difficult Parts

The hardest part wasn't simply getting the agent to speak.

The difficult part was making the system behave reliably.

One of the biggest problems was preventing the LLM from confidently producing information that wasn't actually verified.

For example, if someone asked:

"Is this product available?"

the LLM could theoretically generate a plausible answer even if it had no actual catalogue information.

I solved this by introducing dedicated tools for business data.

The agent uses the product lookup tool when it needs product information instead of relying on its own knowledge.

I also added fallback behavior when external product data isn't available.

Another challenge was coordinating multiple components:

Frontend
   ↓
LiveKit
   ↓
STT
   ↓
LLM
   ↓
Tools
   ↓
TTS
   ↓
Frontend

A problem in any one of these layers can affect the entire conversation.

Working through these issues taught me that a voice agent is really a system of connected components, not just an LLM with a voice.

🛠️ How to Run the Project

If you want to experiment with the project yourself, the complete source code is available on GitHub:

🔗 https://github.com/kusalar/agentvoice

Requirements

You'll need:

Python 3.10+
Node.js 18+
uv
pnpm
LiveKit
Murf API credentials
Deepgram API credentials
LLM API credentials

Clone the repository:

git clone https://github.com/kusalar/agentvoice.git
cd agentvoice

Install the backend dependencies:

cd backend
uv sync

Install the required model/files:

uv run python src/agent.py download-files

Then install the frontend dependencies:

cd ../frontend
pnpm install

🔐 Environment Variables

Never put API keys directly into the source code.

Create the appropriate environment file and add your credentials there.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_llm_api_key

Use your actual credentials locally, but never commit them to GitHub or publish them in screenshots.

▶️ Running the Agent

Start the LiveKit development server:

livekit-server --dev

Run the backend agent:

cd backend
uv run python src/agent.py dev

Start the frontend:

cd frontend
pnpm dev

Then open:

http://localhost:3000

Allow microphone access and start a conversation with the agent.

🧪 Testing the Agent

A simple test conversation could be:

User:
"Do you have organic honey?"

Agent:
Checks the product tool.

User:
"How much is it?"

Agent:
Returns the verified catalogue information.

You can also test:

Switching between Hindi and English
Bengali conversations
Code-mixed sentences
Product availability
Memory consent
Return/refund escalation
Specialist-agent handoff
Outbound calling

For testing screenshots and demos, use dummy customer information rather than real personal data.

📸 Evidence From My Build

The project evolved throughout the 10-day challenge, and I documented different capabilities as I built them.

The most useful evidence includes:

The final voice-agent interface
A real conversation with the agent
Product lookup in action
The call analytics dashboard
Memory/consent flow
Human escalation
Specialist-agent handoff
Outbound calling

I don't need to include every screenshot from every day.

Instead, I want the screenshots to show the progression from a simple voice agent into a more complete voice system.

🚀 What I Learned

The biggest lesson from this challenge is that building a voice agent isn't just about making AI talk.

A useful voice agent needs:

Voice + Reasoning + Tools + Memory + Safety + Real-time communication + Observability + Human handoff

The voice is what the user experiences.

But the tools, data validation, consent, fallback behavior, analytics and escalation logic are what make the system useful.

Over these 10 days, I learned how these different pieces can come together to create something much closer to a real-world voice application.

🔮 What's Next?

There is still a lot I would like to improve:

More Indian languages
Better production telephony
More business integrations
More advanced call analytics
Better observability
More specialist agents
Production deployment
Automated voice-agent evaluation

But this challenge gave me a strong foundation for exploring all of these areas.

Final Thoughts

What started as a basic voice agent became a multilingual local-commerce assistant capable of using tools, remembering users with consent, making calls, escalating problems, handing conversations to specialists and analyzing calls.

The most important thing I learned is:

A good voice agent shouldn't just sound intelligent. It should be useful, accurate, safe and measurable.

I'm grateful to Murf AI for organizing 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build and experiment with voice AI over 10 days.

🔗 Project: https://github.com/kusalar/agentvoice

Built with:

Murf Falcon + LiveKit + Deepgram + LLM + Python + Next.js

VoiceForBharat #MurfAI #VoiceAI #GenerativeAI #AI #BuildInPublic

DEV Community: KUSAL SARKAR

Building a Multilingual Voice Agent for Local Commerce with Murf Falcon

🎯 The Problem

🗣️ Why Voice?

"The final Local Commerce Assistant interface: a voice-first shopping experience for discovering local products and interacting with the agent."

🏗️ How the Voice Agent Works

1. Speech-to-Text

2. LLM

3. Text-to-Speech

4. Real-time Transport

🇮🇳 Indian Voice with Murf Falcon

🧠 Agent Personality and Safety

🛒 Live Product Lookup

🧠 Memory with User Consent

📊 Call Analytics Dashboard

📞 Outbound Calling

👨‍💼 Human Escalation

"Human-in-the-loop escalation dashboard: support tickets created by the voice agent are tracked through open, in-progress and resolved states."

🔄 Specialist Agent Handoff

"Specialist-agent handoff: the main commerce assistant recognizes a returns/refund issue and transfers the conversation to a dedicated Returns & Refunds Specialist."

😅 The Difficult Parts

🛠️ How to Run the Project

Requirements

🔐 Environment Variables

▶️ Running the Agent

🧪 Testing the Agent

📸 Evidence From My Build

🚀 What I Learned

🔮 What's Next?

Final Thoughts

VoiceForBharat #MurfAI #VoiceAI #GenerativeAI #AI #BuildInPublic