Building FinSafe AI: My 10-Day Voice Agent Journey 🚀

🎙️ Building FinSafe AI: My 10-Day Voice Agent Journey 🚀

10 Days. One Voice Agent. Countless lessons.

Over the last 10 days, I took my FinSafe AI Voice Assistant from a basic voice conversation to a much more capable AI system that can talk, remember, use tools, make calls, involve humans, and hand conversations to specialist agents. 🤖🎙️

This journey was part of the Murf AI 10 Days of Voice Agents — VoiceForBharat Edition.

And honestly, I learned that building a voice agent is not just about making an AI talk.

It's about making it useful, reliable, safe, and natural. 💡

💰 WHAT IS FInSAFE AI?

FinSafe AI is a voice-first assistant designed for the Financial Services track.

🎯 Goal: Make financial information easier to access through natural voice conversations.

Instead of typing questions, users can simply speak with the assistant and receive a voice response.

The idea is simple:

🗣️ Speak naturally → AI understands → AI takes action → AI responds

🏗️ HOW DOES FInSAFE AI WORK?

The basic architecture looks like this:

👤 User
↓
🎤 Speech-to-Text
↓
🧠 LLM
↓
🛠️ Tools / 🧠 Memory / 🤖 Specialist Agents
↓
🔊 Text-to-Speech
↓
👤 User

The real-time communication layer connects the different parts of the voice experience.

I used LiveKit for the real-time voice-agent infrastructure and Murf Falcon for text-to-speech. 🎙️⚡

🚀 WHAT I BUILT DURING THESE 10 DAYS

🎙️ 1. A VOICE AGENT WITH A CLEAR ROLE

I started by creating the FinSafe AI Voice Assistant with a clearly defined financial-services role.

Instead of making it a general-purpose chatbot, I gave it:

🧠 A clear personality
🎯 Defined objectives
🛡️ Safety guardrails
📋 Specific responsibilities

This helped make conversations more focused and predictable.

🌐 2. MULTILINGUAL VOICE CONVERSATIONS

One of the features I worked on was multilingual communication.

FinSafe can communicate in languages such as:

🇬🇧 English
🇮🇳 Hindi
ગુજરાતી Gujarati
🌎 Other supported languages

I also worked on making the agent respond according to the language the user is speaking.

This was an important lesson:

🌐 A multilingual voice agent shouldn't just understand another language — it should also respond naturally in that language.

💻 3. A PERSONALIZED FRONTEND

I improved the frontend experience so users can understand what the voice agent is doing.

The interface clearly shows different agent states:

🟢 Ready
🔄 Connecting
🎤 Listening
🔊 Speaking
🔴 Call Ended

This makes the voice experience easier to understand because users can see whether the agent is listening, responding, connecting, or finished.

🧠 4. PERSISTENT MEMORY

One of my biggest improvements was giving FinSafe long-term memory.

Before this, the agent could forget everything when a conversation ended.

I added SQLite-based persistent memory so approved caller information can be stored and retrieved in future conversations. 🗄️

The memory can include:

👤 User ID
📝 Name
🌐 Language preference
💰 Relevant financial facts
🕐 Last interaction

For example:

📞 First Conversation

User → My name is Ramesh.

Agent → Can I remember your name for future conversations?

User → Yes.

    ↓

📞 Second Conversation

Agent → Welcome back, Ramesh! 👋

🔐 Privacy was also important.

The agent asks for consent before saving information.

For financial services, sensitive information such as account numbers or government ID numbers should not be stored.

🔎 5. GIVING MY AGENT REAL-WORLD TOOLS

A voice agent becomes much more useful when it can use tools instead of relying only on the model's existing knowledge.

For FinSafe, I implemented a financial scheme eligibility lookup. 💰

The flow looks like:

🗣️ User asks a question
↓
🧠 Agent decides a tool is needed
↓
🔎 Eligibility tool
↓
📊 Data is processed
↓
🔊 Agent explains the result

I also added handling for situations where the data source is unavailable.

🚫 The agent should never invent an answer just because a tool failed.

Instead, it should clearly explain that the information couldn't be retrieved.

📞 6. OUTBOUND VOICE CALLS

I also moved beyond a browser-only experience and worked with outbound voice calls. 📱🎙️

This showed me how voice agents can proactively communicate with users instead of always waiting for someone to start a conversation.

👤 7. HUMAN ESCALATION

AI shouldn't try to solve every problem.

Sometimes the right answer is:

🤝 "Let me connect you with a human."

I added a human escalation path for situations where human assistance may be more appropriate.

This is especially important in financial services, where some situations require additional support.

📊 8. CALL TRACKING & ANALYTICS

I also worked on tracking information about voice interactions and call outcomes. 📈

Analytics can help answer questions like:

📞 How many conversations happened?
🎯 What was the outcome?
👤 Which interactions required additional help?
📈 How is the agent being used?

This turns the project from a simple voice demo into something that can be monitored and improved.

🤖 9. SPECIALIST AGENT HANDOFF

One of my final improvements was adding a specialist agent.

For FinSafe, I created a:

🏛️ Government Scheme Specialist

Its job is focused specifically on government-scheme-related questions.

The architecture looks like:

         🤖 FinSafe Main Agent
                 │
      Is specialist help needed?
            /          \
          No            Yes
          │              │
          ▼              ▼
    💬 Main Answer    🔄 Handoff
                          │
                          ▼
                🏛️ Scheme Specialist
                          │
                          ▼
                     👤 User

Before transferring the conversation, the main agent clearly explains what's happening:

🗣️ "I'll connect you with our government scheme specialist who can help you with this."

The specialist then continues the conversation using the user's existing context.

💡 Biggest lesson:

A good AI agent doesn't need to be an expert at everything. It needs to know when to involve the right specialist.

🧩 THE CHALLENGES I FACED

The project definitely didn't work perfectly on the first attempt. 😅

And honestly, that's where some of the best learning happened.

🌐 Multilingual Response Issues

Initially, the agent could understand multiple languages, but sometimes the response language wasn't what I expected.

I had to improve the language instructions so the agent follows the user's current language more consistently.

🔌 LiveKit Connection Issues

During testing, I also encountered LiveKit connectivity problems such as:

WinError 1231
ConnectionReset 10054

These issues taught me that voice applications depend not only on application code, but also on stable real-time network connections.

🧠 Memory Challenges

Another challenge was making sure the agent could actually recognize a returning caller instead of simply remembering a name from the conversation.

The system uses a user/caller identity to retrieve saved information from the database.

🛠️ WANT TO BUILD YOUR OWN VOICE AGENT?

Here is the basic architecture you need:

🎤 Speech-to-Text
↓
🧠 Large Language Model
↓
🛠️ Tools / Database / Agents
↓
🔊 Text-to-Speech
↓
⚡ Real-Time Voice Transport

You need:

🎤 Speech-to-Text — understand the user
🧠 LLM — process the conversation
🔊 Text-to-Speech — generate the response
⚡ Real-time transport — handle the live conversation
🛠️ Tools — fetch external information
🗄️ Database — store approved memory

⚙️ SETTING UP THE PROJECT

Clone the repository:

git clone --> https://github.com/Dodiyayash/murf-livekit-starter?utm_source=chatgpt.com
cd murf-livekit-starter

Create your Python environment and install the dependencies according to the repository instructions.

Then configure your environment variables:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key

⚠️ Never publish your real API keys.

Use environment variables and keep secret files out of your public repository.

🔐 KEEP YOUR DATA SAFE

Never publish:

❌ API keys
❌ API secrets
❌ Phone numbers
❌ Caller information
❌ Passwords
❌ OTPs
❌ Bank account numbers
❌ Government ID numbers
❌ Private database information

Security should be part of the design from the beginning. 🔐

💻 EXPLORE THE CODE

The project is available on GitHub:

https://github.com/Dodiyayash/murf-livekit-starter

You can explore the project and the different branches created during the challenge. 🚀

💡 WHAT I LEARNED

The biggest lesson from these 10 days:

A useful voice agent is much more than an LLM connected to a microphone.

A strong voice agent needs:

🗣️ Conversation
🧠 Context
🛠️ Tools
💾 Memory
🛡️ Safety
📊 Reliability
🤖 Specialization

Memory makes conversations personal.

Tools allow the agent to access useful information.

Guardrails keep the agent within its intended role.

Human escalation provides a safety net.

Specialist handoffs allow the system to handle more complex requests.

And a good frontend helps users understand what's happening.

🚀 MY 10-DAY JOURNEY

🎙️ Voice Agent
↓
🛡️ Role & Guardrails
↓
🌐 Multilingual Support
↓
💻 Personalized Frontend
↓
🧠 Persistent Memory
↓
🔎 Financial Tools
↓
📞 Outbound Calls
↓
👤 Human Escalation
↓
📊 Call Analytics
↓
🤖 Specialist Handoff

What started as a simple voice assistant became a much more complete conversational AI system. 🚀

❤️ FINAL THOUGHTS

This 10 Days of Voice Agents challenge has been an incredible hands-on learning experience.

I learned that building voice AI isn't just about making something that speaks.

It's about creating something that can:

Listen → Understand → Remember → Act → Escalate → Respond 🎙️🧠⚡

I'm excited to continue improving FinSafe AI and explore what I can build next.

A huge thanks to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build with Murf Falcon. 🎧🇮🇳

🚀 One journey. 10 days. One voice agent. And a lot of learning.

Keep building. Keep experimenting. Keep learning. 💻🔥

10DaysOfVoiceAgents #VoiceForBharat #MurfAI #MurfFalcon #VoiceAI #AIVoiceAgent #GenerativeAI #ConversationalAI #FinTech #BuildInPublic #AI #VoiceTechnology