*10 Days of Voice Agents — VoiceForBharat Edition
*
What if a small business owner could simply speak to an AI assistant in Hindi or a mix of Indian languages and get useful business information instantly?
That question led me to build Vyapar AI, an AI-powered voice assistant designed around the needs of Indian businesses.
Over the last 10 days, I went from building a basic conversational voice agent to creating a system that can remember users, use tools, make outbound calls, understand when human help is needed, track conversations, and hand off conversations to specialist agents.
This wasn't a smooth journey. I faced API rate limits, networking problems, unclear voice output, telephony configuration issues, and several integration challenges.
But those problems became some of the most valuable parts of the project.
🚀 What is Vyapar AI?
Vyapar AI is an Indian-focused AI voice assistant designed to help business users interact with information and services through natural voice conversations.
The goal is simple:
Make business assistance as easy as talking to another person.
Instead of opening dashboards, searching through menus, or typing complicated queries, a user can simply speak.
For example:
User:
"आज गेहूं के आटे का भाव क्या है?"
Vyapar AI:
"आज के अपडेट के अनुसार गेहूं का आटा 45 रुपये प्रति किलो है।"
The idea is especially useful for users who are more comfortable speaking than typing, particularly in Indian-language and code-mixed conversations
🎯 Why Voice?
Traditional applications expect users to:
- Open an application.
- Find the correct section
- Search for information
- Read the result
- Take action
Voice can reduce that friction.
With a voice interface, the interaction becomes:
Speak → Understand → Process → Respond
For business users working on the move, managing shops, handling customers, or working with operational information, this can be much more natural.
And for India, voice has another important advantage:
Language.
People don't always communicate in formal English.
They may use:
- Hindi
- English
- Hinglish
That's why I wanted Vyapar AI to feel like a practical Indian business assistant rather than simply another English-speaking chatbot.
*🏗️ What I Built During the 10 Days
*
The project evolved step by step.
Day 1–2: The Foundation
I started with the basic voice-agent architecture and defined:
- Personality
- Objectives
- Conversation behavior
- Safety guardrails
- Response style
The goal was to make Vyapar AI helpful but also predictable.
🇮🇳 Indian Voice with Murf Falcon
One of the most important parts of the project was the voice itself.
I used Murf Falcon, a fast TTS API, to give Vyapar AI a natural Indian voice.
The voice is not just an output layer.
For a voice agent, the voice is part of the user experience.
A robotic or unnatural voice can make even a technically powerful agent difficult to use.
Murf Falcon helped me create a more natural conversational experience for Indian users.
🧠 Speech-to-Text + LLM + Text-to-Speech
The core voice pipeline looks like this:
USER
│
▼
🎤 Microphone Input
│
▼
Speech-to-Text
(Deepgram)
│
▼
LLM
(Reasoning Layer)
│
┌───────┴────────┐
│ │
▼ ▼
Tools Memory
│ │
└───────┬────────┘
▼
Response Generation
│
▼
Text-to-Speech
(Murf Falcon)
│
▼
🔊 Voice
│
▼
USER
Real-time communication is handled through LiveKit.
So the major components are:
- Speech-to-Text: Deepgram
- LLM: reasoning and conversation
- Text-to-Speech: Murf Falcon
- Real-time transport: LiveKit
- Tools: business data and actions
- Memory: returning-user context
🧩 Giving the Agent a Personality
One of the first lessons I learned was that a voice agent needs more than an API connection.
It needs a clear role.
I defined Vyapar AI's:
- Personality
- Objectives
- Conversation style
- Language behavior
- Safety boundaries
- Tool usage rules
- Escalation rules
I also configured it to respond naturally in Hindi when appropriate instead of defaulting to English.
This made a huge difference to the overall experience.
*💾 Giving Vyapar AI Memory
*
A useful assistant should not behave as if every conversation is the first conversation.
I added memory so the agent can retain useful information about returning users.
The basic concept is:
User
│
▼
Conversation
│
▼
Important information
│
▼
Memory Storage
│
▼
Future Conversation
│
▼
Personalized Response
This transforms the agent from a simple chatbot into something closer to an assistant.
🛠️ Giving the Agent Tools
An LLM alone doesn't automatically know the latest business information.
So instead of asking the model to invent an answer, I connected it to tools.
For example:
User asks for business information
↓
Agent understands intent
↓
Agent calls appropriate tool
↓
Tool retrieves/produces data
↓
Agent explains result
↓
Murf converts response to speech
One example from my testing was a product-price lookup.
The agent could call a tool such as:
lookup_product_inventory()
and then communicate the result naturally.
This was an important lesson:
A good AI agent isn't just an LLM. It's an LLM connected to the right tools.
📞 Outbound Calls
I also explored outbound calling.
The idea was to allow Vyapar AI to initiate a call rather than only waiting for a user to connect.
This introduced a completely different set of challenges.
The architecture becomes:
Vyapar AI
│
▼
Telephony
│
▼
Phone/SIP connection
│
▼
User
I experimented with telephony integrations and faced configuration issues involving SIP, phone numbers, trunks, and connection settings.
This was one of the most technically challenging parts of the project.
🧑💼 Knowing When to Ask for Human Help
An AI agent shouldn't try to solve everything.
Sometimes the correct answer is:
"Let me connect you with a human."
I added a human-escalation mechanism so the agent can recognize situations where human assistance is appropriate.
For example:
User request
↓
Can AI safely handle it?
│
┌─┴─┐
YES NO
│ │
▼ ▼
Respond Request Human Help
This is important for real-world AI systems because reliability isn't only about answering questions.
Sometimes reliability means knowing when not to answer.
📊 Call Analytics
Another important part of the project was understanding what happened during conversations.
A voice system becomes much more useful when we can analyze:
Number of calls
Conversation outcomes
User requests
Escalations
Agent performance
Call duration
Successful interactions
This led me toward a call analytics dashboard where conversation outcomes can be visualized.
For a production system, analytics would be essential for improving the agent over time.
🤖 Specialist Agent Handoff
One of my final major features was specialist-agent handoff.
A single agent shouldn't have to be an expert in everything.
Instead:
Main Agent
│
Understands User Intent
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
General Sales Support
Agent Agent Agent
The main agent can handle normal conversations and transfer the conversation when a specialist is more appropriate.
This creates a more scalable architecture.
😅 The Difficult Parts
The most valuable lessons came from things that didn't work.
- LLM Rate Limits
During development, I encountered rate-limit problems while using an LLM API.
At one point, the available token limit was nearly exhausted.
The result was frustrating:
User speaks
↓
STT works
↓
LLM request
↓
❌ Rate Limit
↓
No response
This taught me that building an AI application isn't just about writing prompts.
You also have to think about:
Token consumption
Rate limits
Fallback models
Error handling
Request frequency
Production quotas
- Telephony and SIP Problems
Outbound calling was another difficult area.
I encountered issues involving:
SIP configuration
Trunk configuration
Caller identity
Connection errors
Network/DNS issues
Some errors were caused by configuration rather than the agent code itself.
This taught me to separate problems into layers:
Application
↓
AI / LLM
↓
Voice
↓
Real-time transport
↓
Telephony
↓
Network
When something fails, identifying the correct layer makes debugging much easier.
- Voice Quality
At one point, the voice sounded slightly unclear or disturbing.
Initially, it was tempting to assume that the TTS system itself was the problem.
But voice quality can depend on multiple factors:
Network latency
Audio streaming
STT latency
TTS generation
Browser/device audio
Packet loss
Real-time transport
This taught me that a voice agent needs to be tested under real conversational conditions, not just with a single successful response.
- Getting the Language Right
Another challenge was making the agent consistently respond in the desired Indian language.
A voice agent may have:
Indian TTS
+
Indian user
but still produce an English response if the system instructions aren't clear.
I therefore had to carefully define language behavior and conversation rules.
The result was a much more natural experience for Hindi and code-mixed interactions.
🔐 Security Lesson
One rule I learned during the challenge is extremely important:
Never publish API keys or private user information in a public repository.
API keys should be stored in environment variables.
For example:
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
LLM_API_KEY=your_key_here
The actual .env file should never be committed.
Instead, provide something like:
.env.example
containing placeholders.
Also never publish:
API keys
Phone numbers
Caller information
Personal user data
Private logs
Authentication tokens
🚀 How to Build Your Own Voice Agent
If you want to build a similar project, start with four core components.
- Speech-to-Text
Converts the user's speech into text.
Example:
🎤 "आज की बिक्री कितनी है?"
↓
"आज की बिक्री कितनी है?"
I used Deepgram for speech recognition.
- LLM
The LLM understands the user's intent and decides what to do.
It can:
Answer questions
Call tools
Use memory
Ask follow-up questions
Escalate to humans
Hand off to specialists
- Text-to-Speech
The response from the LLM needs to become audio.
I used Murf Falcon for this layer.
LLM Response
↓
Murf Falcon
↓
🔊 Natural Voice
- Real-Time Transport
The system needs to move audio between the user and the agent in real time.
I used LiveKit for the real-time communication layer.
⚙️ Basic Project Setup
My development environment used Python and a LiveKit-based voice-agent structure.
A simplified workflow looks like:
git clone
cd vyapar-ai
uv sync
Create your environment file:
cp .env.example .env
Then add your API keys:
MURF_API_KEY=...
DEEPGRAM_API_KEY=...
LLM_API_KEY=...
Never commit .env.
Then start the agent using the project's development command, for example:
uv run python src/agent.py dev
The exact command may vary depending on your project configuration.
🧪 Testing the Agent
Don't test only with:
"Hello"
Use realistic scenarios.
Test 1 — Normal Query
User:
"आज गेहूं के आटे का भाव क्या है?"
Expected:
The agent calls the appropriate business-data tool and provides the result.
Test 2 — Memory
User:
"My name is Rahul."
Later:
User:
"What is my name?"
Expected:
The agent retrieves the remembered information.
Test 3 — Human Escalation
User:
"I need help with something that requires a human."
Expected:
The agent recognizes that human intervention is appropriate.
Test 4 — Specialist Handoff
User:
"I need detailed help regarding a specialized business issue."
Expected:
The main agent transfers the conversation to the appropriate specialist.
Testing these scenarios helped me identify problems that wouldn't appear during basic testing.
🏛️ Final Architecture
The overall architecture of Vyapar AI can be represented like this:
┌──────────────┐
│ USER │
└──────┬───────┘
│
Voice / Audio
│
▼
┌─────────────────┐
│ LiveKit │
│ Real-Time Layer │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Deepgram STT │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Main Agent │
│ LLM │
└───┬────┬────┬───┘
│ │ │
┌────────────┘ │ └────────────┐
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌───────────┐
│ Memory │ │ Tools │ │ Specialist│
└────────┘ └────────┘ │ Agent │
└───────────┘
│
▼
┌──────────────┐
│ Murf Falcon │
│ TTS │
└──────┬───────┘
│
▼
🔊 USER
Around this core system, I also built capabilities for:
Outbound calls
Human escalation
Call outcomes
Analytics
Personalization
Indian-language conversations
📈 What I Learned
The biggest lesson from these 10 days is that building a voice agent is much more than connecting an LLM to a microphone.
A production-quality voice agent needs:
Voice + Intelligence + Tools + Memory + Guardrails + Real-Time Infrastructure + Observability
I also learned that failures are part of the development process.
Rate limits taught me about scalability.
SIP problems taught me about infrastructure.
Voice-quality problems taught me about real-time systems.
Language issues taught me about prompt design.
Specialist handoffs taught me about agent architecture.
And human escalation taught me that a good AI system should know its limits.
🇮🇳 Why I Built It for Bharat
India has an enormous opportunity for voice-first AI.
People shouldn't need to become experts at using software to benefit from AI.
They should be able to simply speak.
That is the vision behind Vyapar AI:
Making AI-powered business assistance more natural, accessible, and voice-first for Indian users.
There is still a lot more I want to build—better regional-language support, stronger business integrations, improved analytics, more reliable telephony, and production-grade scaling.
But this 10-day challenge gave me the foundation.
🔗 Project & Demo
GitHub Repository:https://github.com/SUDIPTO-2005/Murf-AI
LinkedIn:www.linkedin.com/in/sudipto-bhadra
Before publishing the repository, make sure all API keys, phone numbers, caller data, tokens, and private credentials have been removed.
🙌 Final Thoughts
Ten days ago, I started with the goal of building a voice agent.
Today, Vyapar AI can do much more than simply talk.
It can:
🎙️ Understand spoken conversations
🇮🇳 Support Indian-language interactions
🧠 Remember users
🛠️ Use tools
📞 Make outbound calls
🧑💼 Escalate to humans
📊 Track call outcomes
🤖 Hand conversations to specialist agents
🔊 Respond using Murf Falcon
The most important thing I gained from this challenge wasn't just a working project.
It was an understanding of how voice AI systems are actually built, debugged, tested, and improved.
Thank you to Murf AI for creating the 10 Days of Voice Agents — VoiceForBharat Edition challenge.
This is only the beginning of Vyapar AI. 🚀
Built with:
Murf Falcon • LiveKit • Deepgram • LLMs • Python
Top comments (0)