Building Vyapaar Voice: From a Simple Voice Assistant to a Multi-Agent AI System for India's Small Businesses

Reeba Sebastian — Sat, 15 Aug 2026 18:32:09 +0000

Walk through any local market in India and you'll find artisans, street vendors, and small business owners juggling customers, inventory, pricing, and orders—often all at the same time.

Many of them still rely on notebooks, spreadsheets, or even memory to manage their business. While there are plenty of digital tools available today, most of them assume that users are comfortable navigating complex interfaces, typing long forms, or learning new software.

That made me wonder:

What if managing a business was as simple as having a conversation?

When I joined Murf AI's 10 Days of Voice Agents – #VoiceForBharat Edition, I wanted to build something practical rather than another AI demo. Instead of creating ten separate projects over ten days, I decided to keep improving one product every day.

That's how Vyapaar Voice was born.

It started as a simple voice assistant and gradually evolved into a multi-agent AI system capable of remembering sellers, managing products, retrieving inventory, making outbound calls, escalating complex issues to human support, tracking analytics, and even collaborating with specialist AI agents.

The Problem I Wanted to Solve

India has millions of small businesses—artisans, street vendors, local shop owners, and MSMEs.

Most of them don't need another complicated dashboard.

They need something that understands how they already work.

Many business owners are constantly moving, serving customers, or creating products. Typing into an app isn't always convenient. Voice, however, is natural.

My goal was to build a voice-first assistant that lets sellers manage their business simply by speaking.

Instead of clicking through menus, they could say things like:

"Add a new product."
"Update the price of my clay diyas."
"How many jute bags do I have left?"
"A customer wants a refund."

The assistant would understand the request and help them complete the task conversationally.

Meet Vyapaar Voice

Vyapaar Voice is an AI-powered voice assistant built specifically for Indian small businesses.

The system is designed to make everyday business operations easier through natural voice conversations.

Over the course of the challenge, it grew into a system capable of:

Registering new sellers and remembering them across sessions.
Managing product catalogues and prices.
Looking up inventory in real time.
Calculating customer order totals.
Making outbound reminder calls.
Escalating complex issues to human support.
Tracking real call analytics.
Handing conversations to specialist AI agents whenever required.

Rather than trying to make one giant AI that knows everything, I focused on creating an assistant that knows when to help, when to ask for more information, and when to hand the conversation to someone better suited for the task.

How the System Works

Under the hood, the project combines several components working together in real time.

Features I'm Most Proud Of

Rather than listing everything I built day by day, these are the features that I think define the project.

Natural Voice Conversations:
The project uses LiveKit for real-time communication, Gemini as the reasoning engine, and Murf Falcon for text-to-speech.

Using an Indian English voice made conversations feel much more natural for the target audience.

Persistent Memory:
Returning sellers don't have to introduce themselves every time.

With the seller's permission, the assistant remembers useful business information such as:

Business name
Preferred language
Frequently sold products
Previous interactions

This allows future conversations to feel much more personal without storing unnecessary sensitive information.

Inventory & Business Tools:
One of the biggest improvements was moving from prompt-only responses to real backend tools.

The assistant can:

Check product inventory.
Calculate order totals.
Retrieve business information from SQLite.
Update products and pricing.

These aren't hardcoded responses—they're backed by real function calls.

Outbound Calls:
The assistant isn't limited to waiting for sellers to contact it.

It can proactively make outbound calls to remind sellers about important business events.

Human Escalation:
One lesson I learned while building this project is that AI shouldn't try to solve every problem.

Payment disputes and complex return requests are better handled by people.

Instead of pretending to know the answer, the assistant creates a structured escalation request with the seller's permission and provides a reference ID for future follow-up.

Call Analytics:
Another feature I really enjoyed building was the analytics dashboard.

Instead of using demo data, it records actual conversations and tracks:

Total calls
Successful calls
Failed calls
Success rate
Recent call history
Failure breakdown

It made the project feel much more like a real product than a prototype.

Multi-Agent Collaboration:
The final evolution of the project was introducing multiple AI agents.

Instead of expecting one assistant to know everything, I split responsibilities between:

Anisha — the main business assistant

Karan — the Returns & Refunds Specialist

When a seller has a return or refund issue, Anisha hands the conversation over to Karan without requiring the seller to explain everything again.

If the topic changes, Karan hands the conversation back to Anisha.

It creates a much more natural support experience.

The Challenges I Didn't Expect

Not everything worked the first time.

One of the hardest parts was implementing multi-agent handoffs.

Initially, the specialist agent would announce that it was taking over but wouldn't actually receive the conversation context. This resulted in awkward pauses, repeated introductions, and the seller having to explain the issue all over again.

After digging deeper into LiveKit's agent handoff mechanisms, I redesigned the workflow so both agents shared the same conversation context and the transfer felt much more seamless.

Another challenge was the analytics dashboard. My first implementation looked correct but was powered by mock data. I rebuilt it to record real conversations in SQLite so the dashboard reflected actual business outcomes rather than placeholder values.

These issues taught me that building reliable AI systems isn't just about prompts—it's about architecture, state management, and thoughtful user experience.

If You Want to Build Something Similar

If you're interested in building your own voice agent, here's a simple starting point:

Clone the Murf LiveKit Starter project.
Configure your API keys using environment variables (.env)—never hardcode or publish them.
Set up your speech-to-text, LLM, and text-to-speech providers.
Add function tools to connect your agent with real business logic instead of relying only on prompts.
Test conversations frequently and iterate based on real interactions.

One of the biggest lessons I learned is to build incrementally. Adding one well-integrated feature at a time made the project far easier to debug and improve.

💻 GitHub Repository: https://github.com/reebaseb/murf-livekit-starter

What's Next?

Although the challenge is complete, I think Vyapaar Voice still has plenty of room to grow.

Some ideas I'd love to explore include:

WhatsApp integration
OCR-based invoice scanning
Regional language support
Customer CRM
Inventory forecasting
ERP integration
Multi-store support
Voice biometrics for seller authentication

Final Thoughts

These ten days taught me far more than how to build a voice agent.

They taught me how to think about AI as a product.

From designing conversations and guardrails to building memory, analytics, human escalation, and multi-agent collaboration, every feature reinforced the importance of creating systems that are not only intelligent but also practical, trustworthy, and genuinely useful.

I'm grateful to Murf AI for organizing the 1*0 Days of Voice Agents – #VoiceForBharat Edition*. It pushed me to build something much bigger than I originally imagined.

And while this challenge may be over, I feel like this is just the beginning for Vyapaar Voice.

DEV Community: Reeba Sebastian