Building Vyapar AI: My 10-Day Journey Building an Indian Voice Agent with Murf Falcon

Sudipto Bhadra — Sat, 15 Aug 2026 05:08:12 +0000

*10 Days of Voice Agents — VoiceForBharat Edition
*
What if a small business owner could simply speak to an AI assistant in Hindi or a mix of Indian languages and get useful business information instantly?

That question led me to build Vyapar AI, an AI-powered voice assistant designed around the needs of Indian businesses.

Over the last 10 days, I went from building a basic conversational voice agent to creating a system that can remember users, use tools, make outbound calls, understand when human help is needed, track conversations, and hand off conversations to specialist agents.

This wasn't a smooth journey. I faced API rate limits, networking problems, unclear voice output, telephony configuration issues, and several integration challenges.

But those problems became some of the most valuable parts of the project.

🚀 What is Vyapar AI?

Vyapar AI is an Indian-focused AI voice assistant designed to help business users interact with information and services through natural voice conversations.

The goal is simple:

Make business assistance as easy as talking to another person.

Instead of opening dashboards, searching through menus, or typing complicated queries, a user can simply speak.

For example:

User:
"आज गेहूं के आटे का भाव क्या है?"

Vyapar AI:
"आज के अपडेट के अनुसार गेहूं का आटा 45 रुपये प्रति किलो है।"

The idea is especially useful for users who are more comfortable speaking than typing, particularly in Indian-language and code-mixed conversations

🎯 Why Voice?

Traditional applications expect users to:

Open an application.
Find the correct section
Search for information
Read the result
Take action

Voice can reduce that friction.
With a voice interface, the interaction becomes:

Speak → Understand → Process → Respond

For business users working on the move, managing shops, handling customers, or working with operational information, this can be much more natural.

And for India, voice has another important advantage:

Language.

People don't always communicate in formal English.

They may use:

Hindi
English
Hinglish

That's why I wanted Vyapar AI to feel like a practical Indian business assistant rather than simply another English-speaking chatbot.

*🏗️ What I Built During the 10 Days
*
The project evolved step by step.

Day 1–2: The Foundation

I started with the basic voice-agent architecture and defined:

Personality
Objectives
Conversation behavior
Safety guardrails
Response style

The goal was to make Vyapar AI helpful but also predictable.

🇮🇳 Indian Voice with Murf Falcon

One of the most important parts of the project was the voice itself.

I used Murf Falcon, a fast TTS API, to give Vyapar AI a natural Indian voice.

The voice is not just an output layer.

For a voice agent, the voice is part of the user experience.

A robotic or unnatural voice can make even a technically powerful agent difficult to use.

Murf Falcon helped me create a more natural conversational experience for Indian users.

🧠 Speech-to-Text + LLM + Text-to-Speech

The core voice pipeline looks like this:

          USER
            │
            ▼
    🎤 Microphone Input
            │
            ▼
    Speech-to-Text
      (Deepgram)
            │
            ▼
         LLM
    (Reasoning Layer)
            │
    ┌───────┴────────┐
    │                │
    ▼                ▼
 Tools           Memory
    │                │
    └───────┬────────┘
            ▼
    Response Generation
            │
            ▼
    Text-to-Speech
    (Murf Falcon)
            │
            ▼
         🔊 Voice
            │
            ▼
          USER

Real-time communication is handled through LiveKit.

So the major components are:

Speech-to-Text: Deepgram
LLM: reasoning and conversation
Text-to-Speech: Murf Falcon
Real-time transport: LiveKit
Tools: business data and actions
Memory: returning-user context

🧩 Giving the Agent a Personality

One of the first lessons I learned was that a voice agent needs more than an API connection.

It needs a clear role.

I defined Vyapar AI's:

Personality
Objectives
Conversation style
Language behavior
Safety boundaries
Tool usage rules
Escalation rules

I also configured it to respond naturally in Hindi when appropriate instead of defaulting to English.

This made a huge difference to the overall experience.

*💾 Giving Vyapar AI Memory
*
A useful assistant should not behave as if every conversation is the first conversation.

I added memory so the agent can retain useful information about returning users.

The basic concept is:

User
│
▼
Conversation
│
▼
Important information
│
▼
Memory Storage
│
▼
Future Conversation
│
▼
Personalized Response

This transforms the agent from a simple chatbot into something closer to an assistant.

🛠️ Giving the Agent Tools

An LLM alone doesn't automatically know the latest business information.

So instead of asking the model to invent an answer, I connected it to tools.

For example:

User asks for business information
↓
Agent understands intent
↓
Agent calls appropriate tool
↓
Tool retrieves/produces data
↓
Agent explains result
↓
Murf converts response to speech

One example from my testing was a product-price lookup.

The agent could call a tool such as:

lookup_product_inventory()

and then communicate the result naturally.

This was an important lesson:

A good AI agent isn't just an LLM. It's an LLM connected to the right tools.

📞 Outbound Calls

I also explored outbound calling.

The idea was to allow Vyapar AI to initiate a call rather than only waiting for a user to connect.

This introduced a completely different set of challenges.

The architecture becomes:

        Vyapar AI
           │
           ▼
       Telephony
           │
           ▼
  Phone/SIP connection
          │
          ▼
         User

I experimented with telephony integrations and faced configuration issues involving SIP, phone numbers, trunks, and connection settings.

This was one of the most technically challenging parts of the project.

🧑‍💼 Knowing When to Ask for Human Help

An AI agent shouldn't try to solve everything.

Sometimes the correct answer is:

"Let me connect you with a human."

I added a human-escalation mechanism so the agent can recognize situations where human assistance is appropriate.

For example:

            User request
                 ↓
        Can AI safely handle it?
                 │
               ┌─┴─┐
              YES  NO
               │    │
               ▼    ▼
           Respond Request Human Help

This is important for real-world AI systems because reliability isn't only about answering questions.

Sometimes reliability means knowing when not to answer.

📊 Call Analytics

Another important part of the project was understanding what happened during conversations.

A voice system becomes much more useful when we can analyze:

Number of calls
Conversation outcomes
User requests
Escalations
Agent performance
Call duration
Successful interactions

This led me toward a call analytics dashboard where conversation outcomes can be visualized.

For a production system, analytics would be essential for improving the agent over time.

🤖 Specialist Agent Handoff

One of my final major features was specialist-agent handoff.

A single agent shouldn't have to be an expert in everything.

Instead:

                Main Agent
                    │
          Understands User Intent
                    │
      ┌─────────────┼─────────────┐
      │             │             │
      ▼             ▼             ▼
   General       Sales         Support
    Agent        Agent          Agent

The main agent can handle normal conversations and transfer the conversation when a specialist is more appropriate.

This creates a more scalable architecture.

😅 The Difficult Parts

The most valuable lessons came from things that didn't work.

LLM Rate Limits

During development, I encountered rate-limit problems while using an LLM API.

At one point, the available token limit was nearly exhausted.

The result was frustrating:

      User speaks
        ↓
      STT works
        ↓
     LLM request
        ↓
    ❌ Rate Limit
        ↓
     No response

This taught me that building an AI application isn't just about writing prompts.

You also have to think about:

Token consumption
Rate limits
Fallback models
Error handling
Request frequency
Production quotas

Telephony and SIP Problems

Outbound calling was another difficult area.

I encountered issues involving:

SIP configuration
Trunk configuration
Caller identity
Connection errors
Network/DNS issues

Some errors were caused by configuration rather than the agent code itself.

This taught me to separate problems into layers:

     Application
         ↓
     AI / LLM
         ↓
       Voice
         ↓
   Real-time transport
         ↓
     Telephony
         ↓
      Network

When something fails, identifying the correct layer makes debugging much easier.

Voice Quality

At one point, the voice sounded slightly unclear or disturbing.

Initially, it was tempting to assume that the TTS system itself was the problem.

But voice quality can depend on multiple factors:

Network latency
Audio streaming
STT latency
TTS generation
Browser/device audio
Packet loss
Real-time transport

This taught me that a voice agent needs to be tested under real conversational conditions, not just with a single successful response.

Getting the Language Right

Another challenge was making the agent consistently respond in the desired Indian language.

A voice agent may have:

Indian TTS
+
Indian user

but still produce an English response if the system instructions aren't clear.

I therefore had to carefully define language behavior and conversation rules.

The result was a much more natural experience for Hindi and code-mixed interactions.

🔐 Security Lesson

One rule I learned during the challenge is extremely important:

Never publish API keys or private user information in a public repository.

API keys should be stored in environment variables.

For example:
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
LLM_API_KEY=your_key_here

The actual .env file should never be committed.

Instead, provide something like:

.env.example

containing placeholders.

Also never publish:

API keys
Phone numbers
Caller information
Personal user data
Private logs
Authentication tokens

🚀 How to Build Your Own Voice Agent

If you want to build a similar project, start with four core components.

Speech-to-Text

Converts the user's speech into text.

Example:

🎤 "आज की बिक्री कितनी है?"
↓
"आज की बिक्री कितनी है?"

I used Deepgram for speech recognition.

The LLM understands the user's intent and decides what to do.

It can:

Answer questions
Call tools
Use memory
Ask follow-up questions
Escalate to humans
Hand off to specialists

Text-to-Speech

The response from the LLM needs to become audio.

I used Murf Falcon for this layer.

LLM Response
↓
Murf Falcon
↓
🔊 Natural Voice

Real-Time Transport

The system needs to move audio between the user and the agent in real time.

I used LiveKit for the real-time communication layer.

⚙️ Basic Project Setup

My development environment used Python and a LiveKit-based voice-agent structure.

A simplified workflow looks like:

git clone

cd vyapar-ai

uv sync

Create your environment file:

cp .env.example .env

Then add your API keys:

MURF_API_KEY=...
DEEPGRAM_API_KEY=...
LLM_API_KEY=...

Never commit .env.

Then start the agent using the project's development command, for example:

uv run python src/agent.py dev

The exact command may vary depending on your project configuration.

🧪 Testing the Agent

Don't test only with:

"Hello"

Use realistic scenarios.

Test 1 — Normal Query

User:
"आज गेहूं के आटे का भाव क्या है?"

Expected:

The agent calls the appropriate business-data tool and provides the result.

Test 2 — Memory

User:
"My name is Rahul."

Later:

User:
"What is my name?"

Expected:

The agent retrieves the remembered information.

Test 3 — Human Escalation

User:
"I need help with something that requires a human."

Expected:

The agent recognizes that human intervention is appropriate.

Test 4 — Specialist Handoff

User:
"I need detailed help regarding a specialized business issue."

Expected:

The main agent transfers the conversation to the appropriate specialist.

Testing these scenarios helped me identify problems that wouldn't appear during basic testing.

🏛️ Final Architecture

The overall architecture of Vyapar AI can be represented like this:

                     ┌──────────────┐
                     │     USER     │
                     └──────┬───────┘
                            │
                     Voice / Audio
                            │
                            ▼
                   ┌─────────────────┐
                   │     LiveKit     │
                   │ Real-Time Layer │
                   └────────┬────────┘
                            │
                            ▼
                   ┌─────────────────┐
                   │   Deepgram STT  │
                   └────────┬────────┘
                            │
                            ▼
                   ┌─────────────────┐
                   │   Main Agent    │
                   │      LLM        │
                   └───┬────┬────┬───┘
                       │    │    │
          ┌────────────┘    │    └────────────┐
          ▼                 ▼                 ▼
      ┌────────┐       ┌────────┐       ┌───────────┐
      │ Memory │       │ Tools  │       │ Specialist│
      └────────┘       └────────┘       │  Agent    │
                                         └───────────┘
                       │
                       ▼
                ┌──────────────┐
                │ Murf Falcon  │
                │     TTS      │
                └──────┬───────┘
                       │
                       ▼
                     🔊 USER

Around this core system, I also built capabilities for:

Outbound calls
Human escalation
Call outcomes
Analytics
Personalization
Indian-language conversations
📈 What I Learned

The biggest lesson from these 10 days is that building a voice agent is much more than connecting an LLM to a microphone.

A production-quality voice agent needs:

Voice + Intelligence + Tools + Memory + Guardrails + Real-Time Infrastructure + Observability

I also learned that failures are part of the development process.

Rate limits taught me about scalability.

SIP problems taught me about infrastructure.

Voice-quality problems taught me about real-time systems.

Language issues taught me about prompt design.

Specialist handoffs taught me about agent architecture.

And human escalation taught me that a good AI system should know its limits.

🇮🇳 Why I Built It for Bharat

India has an enormous opportunity for voice-first AI.

People shouldn't need to become experts at using software to benefit from AI.

They should be able to simply speak.

That is the vision behind Vyapar AI:

Making AI-powered business assistance more natural, accessible, and voice-first for Indian users.

There is still a lot more I want to build—better regional-language support, stronger business integrations, improved analytics, more reliable telephony, and production-grade scaling.

But this 10-day challenge gave me the foundation.

🔗 Project & Demo

GitHub Repository:https://github.com/SUDIPTO-2005/Murf-AI

LinkedIn:www.linkedin.com/in/sudipto-bhadra

Before publishing the repository, make sure all API keys, phone numbers, caller data, tokens, and private credentials have been removed.

🙌 Final Thoughts

Ten days ago, I started with the goal of building a voice agent.

Today, Vyapar AI can do much more than simply talk.

It can:

🎙️ Understand spoken conversations
🇮🇳 Support Indian-language interactions
🧠 Remember users
🛠️ Use tools
📞 Make outbound calls
🧑‍💼 Escalate to humans
📊 Track call outcomes
🤖 Hand conversations to specialist agents
🔊 Respond using Murf Falcon

The most important thing I gained from this challenge wasn't just a working project.

It was an understanding of how voice AI systems are actually built, debugged, tested, and improved.

Thank you to Murf AI for creating the 10 Days of Voice Agents — VoiceForBharat Edition challenge.

This is only the beginning of Vyapar AI. 🚀

Built with:
Murf Falcon • LiveKit • Deepgram • LLMs • Python

VoiceForBharat #MurfAI #VoiceAI #GenerativeAI #AI #ArtificialIntelligence #IndianAI #VoiceAgents #Python #LiveKit

DEV Community: Sudipto Bhadra

Building Vyapar AI: My 10-Day Journey Building an Indian Voice Agent with Murf Falcon

VoiceForBharat #MurfAI #VoiceAI #GenerativeAI #AI #ArtificialIntelligence #IndianAI #VoiceAgents #Python #LiveKit