From a simple voice agent to a multilingual AI assistant with memory, tools, outbound calls, human escalation, analytics, and specialist handoffs.
Introduction
What if understanding a bill was as simple as asking a friend?
For many users, invoices, GST charges, convenience fees, payment details, and other commerce-related information can be confusing. Reading through a long bill and figuring out what each charge means isn't always easy.
That's the problem I wanted to address with BillBhasha AI — a voice-first AI assistant that helps users understand bills, GST, invoices, fees, and payment-related information through natural conversations.
I built BillBhasha AI as part of 10 Days of Voice Agents — VoiceForBharat Edition, using Murf Falcon for text-to-speech.
The interesting part wasn't just making an AI that could talk.
The real challenge was making it behave like a useful assistant.
Over ten days, I gradually added personality, safety, memory, tools, outbound calling, human escalation, analytics, and specialist-agent handoffs.
💡 What is BillBhasha AI?
BillBhasha AI is a multilingual voice assistant designed for commerce-related queries.
A user can simply speak to the agent and ask questions such as:
"Mere bill mein GST kitna laga hai?"
"Convenience fee kya hoti hai?"
"Can you explain this charge?"
"Mujhe ye bill Hindi mein samjha do."
Instead of forcing users to navigate complicated interfaces or read technical terminology, BillBhasha focuses on simple, conversational explanations.
It can also handle code-mixed conversations, such as Hindi + English, which is common in everyday communication.
🎯 Why Voice?
I chose a voice-first approach because not every user wants to type a detailed query or understand a complicated dashboard.
Voice makes the interaction more natural:
User speaks → AI understands → AI processes → AI responds
This becomes especially useful when the user wants a quick explanation rather than searching through documentation.
For BillBhasha, the goal was not just:
"Build a chatbot that speaks."
It was:
"Build a voice assistant that can actually help."
🏗️ How BillBhasha AI Works
At a high level, the system follows this flow:
User
↓
Microphone / Phone Call
↓
Speech-to-Text
↓
LLM / Agent Logic
↓
┌───────────────┐
│ Memory │
│ Tools │
│ Guardrails │
│ Escalation │
│ Specialist │
└───────────────┘
↓
Text-to-Speech
↓
Murf Falcon
↓
User
The agent uses real-time voice infrastructure to receive and process speech, while the LLM handles reasoning and conversation.
Murf Falcon generates the agent's voice response.
🚀 What I Built During the Challenge
The project evolved significantly throughout the challenge.
1. 🎙️ Voice Conversation
The first milestone was getting BillBhasha to actually talk.
The user could start a conversation, speak naturally, and receive a spoken response.
This became the foundation for everything that followed.
2. 🧠 Personality & Guardrails
An AI assistant needs to know not only what it can do, but also what it should not do.
I defined:
- Its identity
- Its main objectives
- Its communication style
- Language behaviour
- Things it must refuse
- Situations requiring human help
For example, BillBhasha should not invent a GST amount or confidently claim information that it doesn't have.
It should explain uncertainty instead of hallucinating an answer.
🌐 3. Multilingual & Code-Mixed Conversations
One important part of BillBhasha is language flexibility.
A user might say:
"Mere bill mein GST kitna hai?"
and then switch to:
"Can you explain the convenience fee?"
The agent should be able to understand this naturally and respond in an appropriate language/register.
I also configured the agent to respect the correct native script when generating multilingual text.
🎨 4. Personalised Voice-Agent Frontend
I didn't want the interface to look like a generic chatbot.
The frontend was redesigned around the voice experience and clearly communicates different agent states:
Ready → Connecting → Listening → Speaking → Call Ended
This makes it immediately clear to the user whether the agent is waiting, listening, or responding.
Microphone permission errors are also handled with clear feedback rather than leaving the user wondering why the call isn't starting.
🧠 5. Giving BillBhasha Memory
One of the biggest improvements was adding memory.
Initially:
Call ends → Everything is forgotten.
After adding persistent storage:
Call ends → User information can be stored → Next call can retrieve it.
For example, during one conversation a user may say:
"I prefer speaking in Hindi."
After receiving permission, that preference can be saved.
During the next conversation, BillBhasha can recognise the returning user and continue more naturally.
Importantly, the agent asks for permission before saving information.
This makes memory an intentional feature rather than silent data collection.
🔧 6. Adding Tools
An AI model shouldn't have to guess information that can be obtained through a function or external data source.
I added tool-based functionality so the agent can retrieve or calculate relevant commerce information when needed.
The important design principle was:
If reliable data is available through a tool, use the tool instead of inventing an answer.
I also added a failure path so that if a data source is unavailable, the agent can communicate the issue instead of silently failing or making up information.
📞 7. Outbound Voice Calls
Next, I moved beyond browser-based conversations.
BillBhasha was extended to support outbound calling.
This changes the interaction completely.
When an agent calls someone, the user didn't necessarily request the conversation.
So the opening needs to be transparent:
- Who is calling?
- Why are they calling?
- What can the user do if they don't want to continue?
This taught me that outbound voice agents need a different conversational design from inbound agents.
👩💻 8. Human Escalation
Another important lesson was that an AI agent shouldn't try to solve everything.
For certain situations, BillBhasha can decide that a human should take over.
The flow is:
User Request
↓
AI Understands Problem
↓
Does AI Need Human Help?
↓
Yes
↓
Ask User Permission
↓
Create Escalation Request
↓
Provide Reference ID
The agent doesn't simply say:
"I'll contact a human."
It creates a structured escalation request containing only useful information.
Privacy is also important here.
The system should never send sensitive information such as:
- OTPs
- PINs
- Passwords
- Account numbers
to the human escalation flow.
📊 9. Call Analytics Dashboard
Once an agent starts handling multiple conversations, another question appears:
How well is it actually performing?
So I added a call analytics dashboard.
The dashboard tracks key metrics such as:
- Total Calls
- Successful Calls
- Failed Calls
The data is based on actual calls rather than hardcoded numbers.
I also redesigned the UI to make the dashboard more visually engaging and easier to understand.
This helped turn BillBhasha from simply a voice demo into something closer to a complete voice-agent product.
🤝 10. Specialist Agent Handoff
One of the final improvements was introducing a specialist agent.
The main agent doesn't need to be an expert at everything.
For example:
User
↓
BillBhasha AI
↓
Does this require specialist help?
↓
Yes
↓
Specialist Agent
↓
Continues the conversation
The main agent informs the user before transferring the conversation.
The specialist then receives the context and continues from there instead of asking the user to repeat everything.
This made the architecture more modular and showed me how multiple specialised agents can work together.
🧩 The Technology Behind It
The project combines several technologies and services to create the complete experience.
Core components
- Frontend: HTML / CSS / JavaScript and voice-agent UI
- Voice orchestration: LiveKit
- LLM: Google Gemini
- Speech-to-Text: Deepgram
- Text-to-Speech: Murf Falcon
- Database: SQLite / persistent storage
- Telephony: Twilio / outbound calling setup
- Tools: Function-based integrations
- Analytics: Call data + dashboard
- Specialist agents: Agent handoff architecture
The most important thing I learned is that a voice agent isn't a single model.
It's a system of multiple components working together in real time.
⚙️ Challenges I Faced
Building the project wasn't always straightforward.
1. Multilingual voice behaviour
One challenge was getting the agent to understand Hindi/code-mixed speech while maintaining the appropriate language and voice behaviour.
This required careful configuration of speech recognition, language handling, and prompting.
2. Memory shouldn't become automatic data collection
Adding memory sounds simple:
"Just save the user's information."
But that isn't enough.
The agent needs to ask:
"Would you like me to remember this?"
Only after confirmation should the information be saved.
This changed the way I thought about AI memory.
3. Voice responses need different prompting
A response that looks good on a screen doesn't always sound good when spoken.
Long explanations, complicated formatting, and overly technical sentences can feel unnatural in a voice conversation.
I learned to keep responses:
- Short
- Clear
- Conversational
- Easy to understand when heard rather than read
4. APIs can fail
Real-world voice agents depend on multiple external services.
A tool may fail.
An API may timeout.
A call may disconnect.
A microphone permission may be denied.
The agent therefore needs a failure path, not just a success path.
That was one of the biggest lessons from this challenge.
🔐 Privacy & Safety
Because BillBhasha deals with commerce and payment-related conversations, privacy cannot be an afterthought.
The agent is designed not to request or expose sensitive information such as:
OTP, PIN, passwords, or account numbers.
Memory also requires user consent.
Similarly, human escalation requires permission before sharing information with another person.
The goal is to make the assistant useful without making it careless.
📂 Project Repository
The complete project is available on GitHub:
BillBhasha AI
https://github.com/Tanya-garg10/BillBhasha-AI
The repository contains the implementation and setup required to explore the project.
Important: API keys and other secrets should always be stored in environment variables and should never be committed to GitHub.
🛠️ How to Start Building Your Own Voice Agent
If you're starting your own voice-agent project, think about it as these building blocks:
Step 1 — Speech-to-Text
Convert the user's voice into text.
Step 2 — LLM
Send the text to your language model and decide what the agent should do.
Step 3 — Tools & Memory
Give the agent access to reliable data and persistent information when necessary.
Step 4 — Text-to-Speech
Convert the response back into natural speech.
Step 5 — Real-Time Transport
Connect everything through a real-time voice layer.
Step 6 — Guardrails
Define what the agent can and cannot do.
Step 7 — Test Failure Cases
Don't test only:
"Hello, how are you?"
Also test:
- API failures
- Silence
- Wrong requests
- Language switching
- Permission denial
- Human escalation
- Call disconnects
That's where the real engineering begins.
🌱 What I Would Build Next
BillBhasha is still a work in progress.
Some improvements I'd like to explore next are:
- Better bill/invoice document understanding
- OCR-based bill analysis
- More Indian language support
- Better GST explanation workflows
- More advanced analytics
- Improved specialist-agent routing
- Stronger privacy controls
- Real-world user testing
- Better accessibility for first-time technology users
The long-term goal would be to make BillBhasha useful beyond a hackathon or challenge demo.
🎓 What I Learned
The biggest takeaway from these 10 days is simple:
Building a voice agent is much more than connecting an LLM to a microphone.
A useful voice agent needs:
Voice + Intelligence + Memory + Tools + Safety + Human Support + Observability
The challenge pushed me to think about the complete user experience rather than just the AI model.
I learned how small things — such as a clear greeting, a permission request, a fallback message, or a handoff explanation — can make a huge difference in how trustworthy an AI system feels.
🚀 Final Thoughts
BillBhasha AI started as a simple idea:
"What if users could simply ask an AI to explain their bills?"
Over the course of 10 Days of Voice Agents — VoiceForBharat Edition, that idea evolved into a much more complete voice-agent system.
It can now:
✅ Hold natural voice conversations
✅ Handle code-mixed language
✅ Follow safety guardrails
✅ Remember users with permission
✅ Use tools for useful information
✅ Make outbound calls
✅ Escalate problems to humans
✅ Track call outcomes
✅ Provide analytics
✅ Hand conversations to specialist agents
And throughout the journey, Murf Falcon powered the voice experience.
This challenge wasn't just about building an AI that can speak.
It was about learning how to build an AI system that can listen, understand, remember, act, and know when it shouldn't act alone.
10 Days. One voice agent. A lot of learning. 🎙️🚀
Project: BillBhasha AI
Track: Local Commerce
Challenge: 10 Days of Voice Agents — VoiceForBharat Edition
TTS: Murf Falcon
Top comments (0)