Introduction
Building BankMitra: My 10-Day Voice AI Journey
Over the past 10 days, I built BankMitra, a voice-based financial services assistant as part of Murf AI's 10 Days of Voice Agents — VoiceForBharat Edition.
The idea was simple: make financial and government-scheme information easier to access through a natural voice conversation. Instead of navigating complicated websites or trying to understand lengthy forms, users can simply talk to BankMitra and ask their questions in English, Hindi, or Hinglish.
BankMitra is powered by LiveKit for real-time voice communication, an LLM for reasoning, Deepgram for speech-to-text, and Murf Falcon for text-to-speech.
But the project became much more than just a chatbot that could talk. Over the challenge, I gradually added memory, real data and tools, outbound phone calls, human escalation, call analytics, and finally a Government Scheme Specialist that BankMitra can hand conversations over to when a query requires more specialized assistance.
What started as a basic voice agent turned into a complete voice-based customer-support system—and along the way, I learned that building a reliable voice agent involves a lot more than simply connecting an LLM to a TTS API.
The Problem I Wanted to Solve
Accessing financial information can be confusing, especially when users have to navigate multiple websites, understand eligibility rules, or deal with lengthy documentation.
For BankMitra, I focused on the Financial Services track. The goal was to create a voice-first assistant that could help users with common banking questions and guide them toward relevant government schemes and financial services through a simple conversation.
Voice felt like the right interface because users don't need to learn how to navigate another application. They can simply ask what they need in their own language or in Hinglish and continue the conversation naturally.
I also wanted BankMitra to go beyond answering questions. A useful financial assistant should know when it can handle something itself, when it needs additional information or tools, and when it should involve a human or a specialist instead of pretending it knows everything.
That became the core idea behind the project: a voice agent that knows not only how to answer, but also when to use a tool, ask for help, or hand the conversation to another agent.
What BankMitra Can Do
Over the 10-day challenge, I gradually expanded BankMitra from a basic conversational agent into a more complete voice-based financial assistant.
Some of its key capabilities include:
- Natural voice conversations using Murf Falcon for fast text-to-speech.
- English, Hindi, and Hinglish support, with Hindi responses delivered in Devanagari script.
- Government-scheme assistance, including eligibility-related guidance and document requirements.
- User memory, allowing BankMitra to remember relevant information across conversations.
- Outbound calls through SIP/Linphone, allowing the agent to proactively reach a user.
- Human escalation for situations such as suspected fraud or problems that require human intervention, with requests sent to Discord.
- Call analytics, tracking real browser and SIP calls, their outcomes, channels, and other useful metrics through a dashboard.
- Multi-agent handoffs, where BankMitra can transfer government-scheme queries to a dedicated Government Scheme Specialist and continue the conversation with the existing context. The main idea was to make BankMitra more than a voice chatbot. Each feature was added to make the system more useful, reliable, and closer to how an actual customer-support system would work.
How the System Works
BankMitra follows a real-time voice pipeline where each component has a specific role.
> User → Speech-to-Text → LLM → Tools / Agents → Text-to-Speech → User
- LiveKit handles the real-time audio communication and connects the user with the voice agent.
- Deepgram converts the user's speech into text so the system can understand the request.
- The LLM acts as the reasoning layer. It decides how to respond, whether a tool is needed, or whether the conversation should be handed to another agent.
- Tools allow BankMitra to perform specific tasks, such as checking scheme eligibility, retrieving required documents, looking up user information, or creating a human-escalation request.
- Murf Falcon converts the generated response back into natural speech.
- SQLite stores persistent information such as user memory, escalation requests, and call analytics.
- The Government Scheme Specialist is a separate agent that can take over when the user's primary intent is related to government schemes.
This architecture allowed me to keep the system modular. Instead of putting every responsibility into one large prompt, different components handle speech, reasoning, data, specialized tasks, and real-time communication separately.
The Most Important Features I Built
Instead of building everything at once, I added features progressively throughout the challenge.
🧠 Memory
BankMitra can remember relevant user information and use it in future conversations, making interactions more personalized.
🛠️ Tools & Real Data
The agent can use tools for tasks such as checking government-scheme eligibility and retrieving required documents instead of relying entirely on the LLM's knowledge.
📞 Outbound Calling
BankMitra can initiate outbound calls using SIP/Linphone, allowing it to proactively reach users instead of only waiting for them to start a conversation.
🧑💼 Human Escalation
When the agent encounters situations such as suspected fraud that require human intervention, it asks for the user's permission and creates an escalation request with a reference ID. The request is also sent to a Discord channel for the human support side.
📊 Call Analytics
I built a dashboard that tracks real browser and SIP calls, including total calls, successful and failed calls, success rate, duration, channel, and escalation status.
🤝 Specialist Agent Handoff
The final major feature was multi-agent handoff. BankMitra can recognize when the user's primary intent is related to government schemes and transfer the conversation to a dedicated Government Scheme Specialist while maintaining the existing context.
Together, these features turned BankMitra from a simple conversational voice bot into a more complete voice-based financial support system.
The Difficult Parts and How I Solved Them
The most challenging part of the project was not building the first working conversation. It was making the agent reliable across different real-world situations.
One major issue came during outbound calling. The agent could receive the call, but sometimes it would not speak first. I also encountered a LiveKit ChatMessage validation error because of a mismatch in the expected message-content format. Fixing these issues taught me that real-time voice systems require careful handling of the underlying framework APIs, not just prompt engineering.
Another challenge was call outcome tracking. Initially, the agent marked some useful conversations as failed because success was tied too narrowly to specific tools. I changed the success logic to consider whether the user's actual task had been completed rather than simply whether a particular tool was called.
The most interesting challenge was the multi-agent handoff. Initially, BankMitra would announce that it was transferring the user but the specialist would not actually take over. I had to fix the handoff flow and explicitly preserve the conversation context so that the specialist could continue without asking the user to repeat their question.
These problems made one thing very clear to me: building a voice agent is an engineering problem, not just an LLM prompting problem. Audio transport, speech recognition, turn detection, state management, tool execution, context, and failure handling all have to work together reliably.
How You Can Build and Run BankMitra
The project is built around a LiveKit voice-agent backend and a Next.js frontend. To build a similar system, the basic setup looks like this:
- Clone the repository and install the required dependencies.
- Create a .env.local file for your API credentials and configuration.
- Add the required keys for services such as LiveKit, Deepgram, Gemini, and Murf. These credentials should remain in environment variables and should never be committed to GitHub.
- Start the LiveKit agent backend.
- Start the frontend and open the BankMitra web interface.
- Connect to the agent and test a normal conversation.
- For outbound calling, configure the SIP/Linphone setup and trigger the outbound-call script.
- For analytics, start the analytics API and open the dashboard to view real call data.
The basic voice pipeline is:
> Microphone → LiveKit → Deepgram STT → LLM → Tools/Agents → Murf Falcon TTS → Speaker
The important part is keeping API credentials separate from the source code and using real service connections rather than hardcoding test data.
What I Would Improve Next
BankMitra is functional, but there is still a lot of room for improvement before taking it to production.
I would focus next on better response latency and interruption handling, especially during longer conversations. I would also add more specialist agents for areas such as cards, loans, and fraud instead of relying on a single specialist.
Another improvement would be stronger security and privacy controls, particularly for real financial users and production telephony. The current system is designed as a challenge project, so production deployment would require additional authentication, monitoring, data protection, and compliance measures.
I would also like to improve the analytics system with more detailed failure categorization, latency tracking, and long-term conversation-quality metrics.
Most importantly, I would continue testing BankMitra with a much wider range of real-world conversations. Voice agents can behave very differently depending on accents, background noise, interruptions, languages, and ambiguous user requests.
What I Learned
The biggest lesson from this challenge was that building a voice agent is much more than connecting an LLM to speech-to-text and text-to-speech.
I learned how important real-time systems are in voice AI. Small issues with turn detection, interruptions, audio buffering, or latency can completely change the user experience.
I also learned that good prompting alone isn't enough. Tools, memory, error handling, state management, telephony, analytics, and multi-agent orchestration all need to be designed together.
Most importantly, the challenge taught me to treat failures as part of the development process. Many of the best improvements to BankMitra came from things that initially didn't work—from SIP errors and broken outbound greetings to handoff and language-routing issues.
After ten days, I have a much better understanding of what it takes to move from a simple voice demo to a more reliable and practical voice AI system.
Conclusion
Building BankMitra over these 10 days gave me a practical understanding of what it takes to build a real-time voice AI system.
What started as a basic voice agent evolved into a system with memory, tools, multilingual conversations, outbound calling, human escalation, call analytics, and multi-agent handoffs.
The biggest takeaway for me is that reliable voice AI requires much more than a good LLM. The entire system—from audio transport and speech recognition to reasoning, tools, state management, and error handling—has to work together.
I’m grateful to Murf AI for creating the 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build, break, debug, and improve a voice agent over these ten days.
BankMitra is just the beginning. 🚀
🔗 Project Links
GitHub: Repository URL
Top comments (0)