I used to be the person who would see an AI challenge and think:
“This looks interesting... but what if I don't know enough to do it?”
So I would learn something first, wait until I felt “ready”, and then maybe build something later.
Spoiler: later rarely came.😭
That changed when I joined the 10 Days of Voice Agents — VoiceForBharat Edition challenge by Murf AI.
For 10 days, I worked on a voice agent called CashCompass, a voice-based financial mentor built for the Financial Services track.
And honestly, the biggest thing I built wasn't just the agent.
It was the confidence to actually build.
Before this challenge, I had never seriously participated in a challenge like this. I was always a little afraid of getting stuck, breaking something, or realizing halfway through that I didn't understand what I was doing.
And yes, I got stuck.
A lot.
There were errors, confusing logs, configuration problems, things that worked but didn't behave the way I expected, and at least a few moments where I stared at my screen wondering whether the code was personally attacking me.
But that was exactly what made this challenge useful.
I wasn't just learning about AI agents anymore.
I was actually building one.
1. The Problem: Financial Decisions Can Be Confusing
I chose the Financial Services track because financial decisions are something almost everyone eventually has to deal with.
Someone gets their first salary.
They want to start saving.
They don't know whether to keep money in a savings account, invest it, look into a government scheme, or simply figure out where their money is disappearing every month.
The information exists.
The problem is that finding it, understanding it, and knowing what applies to you can be overwhelming.
That's where I wanted CashCompass to help.
CashCompass is a voice financial mentor designed for everyday financial conversations, especially for young professionals and people who may find speaking more natural than navigating complicated forms or dashboards.
Instead of:
“Open this website → find this section → read five paragraphs → search another website → get confused.”
the idea is:
“Just talk to it.”
And this is where voice becomes interesting.
A voice interface feels much closer to having an actual conversation with someone.
You can ask:
“I just started earning. How should I start managing my money?”
You don't need to know the exact terminology.
You don't need to formulate a perfect prompt.
You just speak.
2. Meet CashCompass
The goal wasn't to make a chatbot that simply speaks its answers aloud.
I wanted to understand what makes an agent actually agentic.
So throughout the challenge, CashCompass evolved from a basic voice interaction into a system with multiple capabilities.
It can:
- Understand spoken input using Deepgram STT
- Reason using Gemini
- Respond using Murf Falcon
- Maintain a defined personality and objectives
- Follow financial safety guardrails
- Handle multilingual and code-mixed conversations
- Remember approved user information
- Use tools to fetch useful financial information
- Make outbound calls
- Escalate conversations when human help is needed
- Track call outcomes
- Hand conversations to specialist agents
That list looks simple when written like this.
Building each piece was a different story.
3. So... How Does a Voice Agent Actually Work?
Before this challenge, I understood the individual technologies.
Speech-to-text.
LLMs.
Text-to-speech.
APIs.
Databases.
But understanding the pieces individually is very different from making them work together in real time.
The basic architecture of CashCompass looks like this:
┌─────────────────┐
│ User │
│ Voice / Phone │
└────────┬────────┘
│
▼
┌─────────────────┐
│ LiveKit │
│ Real-time audio │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Deepgram STT │
│ Speech → Text │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Gemini │
│ Agent Brain │
└───────┬─────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌─────────┐ ┌──────────┐ ┌──────────┐
│ Memory │ │ Tools │ │ Handoff │
└─────────┘ └──────────┘ └──────────┘
│
▼
┌─────────────────┐
│ Murf Falcon │
│ Text → Speech │
└────────┬────────┘
│
▼
User
In simple terms:
User speaks → speech becomes text → the agent decides what to do → tools/memory may be used → response becomes speech → user hears it.
And all of this has to happen fast enough that it feels like a conversation rather than a very slow walkie-talkie.
That's where LiveKit became important.
4. The Voice Stack
My core stack included:
Speech-to-Text — Deepgram
Deepgram handled the conversion from the user's speech into text.
This gave the LLM something it could actually process.
LLM — Gemini
Gemini was responsible for reasoning, deciding how to respond, following instructions, and interacting with tools.
Text-to-Speech — Murf Falcon
Murf Falcon handled the voice output.
One of the fun parts of this challenge was making the agent sound more natural and appropriate for an Indian audience rather than just making it technically functional.
Real-time transport — LiveKit
LiveKit handled the real-time voice communication layer.
And this was one of the things I didn't fully appreciate before actually building a voice agent:
A voice agent isn't just an LLM with a microphone attached.
You are dealing with audio streams, turn detection, latency, interruptions, transport, speech recognition, response generation, and speech synthesis at the same time.
Basically, the LLM is only one piece of the puzzle.
5. Giving the Agent a Personality and Guardrails
One of the first things I learned was that putting an LLM behind a microphone doesn't automatically give you a useful agent.
You have to define:
- What the agent is supposed to do
- What it should not do
- How it should communicate
- What information it can remember
- When it should ask for help
- When it should stop and hand the conversation to someone else
For CashCompass, I created a defined personality and objectives around being a helpful financial mentor.
But financial conversations also require boundaries.
The agent shouldn't casually behave like:
“I know everything about finance. Do exactly what I say.”
That's exactly the kind of confidence an AI agent should not have.
The goal was to keep the agent useful while avoiding unsupported or overly authoritative financial recommendations.
This was one of my first practical lessons about AI guardrails:
The model being capable doesn't mean you should let it do everything it is capable of doing.
6. Making the Agent Remember Users
One feature I particularly enjoyed building was memory.
A returning user shouldn't necessarily have to start from zero every time.
For example, the agent could remember approved information such as:
- The user's name
- A financial goal
- A preferred language
- Previously explored schemes
But there was an important rule:
Memory should not silently become a personal-data dumping ground.
So, I implemented consent-based memory.
The agent can identify useful information, ask the user whether it should be remembered, and save it only after explicit approval.
This was another part of the challenge where the difference between “I know what memory is” and “I have actually implemented memory” became very obvious.
Before this challenge, I had read about memory in agent systems.
After building it, I started thinking about questions like:
What exactly should be stored?
When should it be stored?
Who decides?
What happens if the user doesn't consent?
That's a much more useful understanding.
7. Giving the Agent Tools
An LLM by itself doesn't magically know the latest structured information you want it to retrieve.
That's where tools come in.
I added a financial scheme finder tool so the agent could search for relevant government financial schemes based on a user's situation.
This changed the way I thought about agents.
There is a big difference between:
“Here is an LLM. Ask it questions.”
and
“Here is an agent that can reason and take actions using external tools.”
The second one starts to feel much more like an actual system.
I also separated tool-related logic from the main agent code instead of putting everything into one giant file.
That sounds like a small engineering decision.
It wasn't.
When the project started growing, I realized very quickly that dumping everything into agent.py is a great way to make future-you regret your decisions.
Future-you is not your friend.
Future-you is going to be pissed about all the shortcuts you took today.😂
8. Outbound Calls
Then things got more interesting.
The challenge also introduced outbound calling.
Instead of waiting for the user to initiate the conversation, the system could initiate a call.
This forced me to think beyond:
“Does the agent answer when I talk to it?”
and toward:
“How does a voice agent actually interact with a phone system?”
I explored LiveKit SIP integration and outbound calling.
And this was also one of the places where reality politely reminded me that tutorials are easier than systems.😭
I ran into SIP configuration issues, including an invalid From header problem during outbound dialing.
That experience was useful because it taught me something that documentation doesn't emphasize enough:
A working AI component doesn't mean the entire system works.
Your STT can work.
Your LLM can work.
Your TTS can work.
And the call can still fail.
That is real engineering.
The part of the challenge I can never forget😭😭
9. Human Escalation and Specialist Handoffs
Another thing I wanted CashCompass to understand was that an AI agent doesn't always need to be the final destination.
Sometimes the right answer is:
“I can't responsibly handle this. Let me connect you to someone who can.”
That led me to explore human escalation and agent handoffs.
I also implemented the idea of specialist agents so that a conversation could be transferred rather than forcing one agent to handle every possible type of request.
This is something I found particularly interesting about agentic AI.
The goal isn't necessarily to build one giant super-agent that does everything.
Sometimes a better architecture is:
one agent → another specialist → human when necessary.
It is much closer to how real organizations work.
10. The Analytics Side
I also built a call analytics layer to track what was happening during calls.
This included things like call outcomes and failure/success information that could be surfaced through a dashboard.
And yes, this gave me another important lesson.
At one point, my dashboard was showing a failed call count that didn't match what I expected.
That forced me to stop thinking:
“The dashboard is working, so the data must be correct.”
A dashboard is only as trustworthy as the logic feeding it.
So I had to inspect the actual event flow and call-status handling rather than blindly trusting what the UI displayed.
That is a lesson I think applies far beyond voice agents:
Don't debug the screen first. Debug the data underneath it!!
11. The Hardest Part Wasn't the Code
The hardest part of this challenge wasn't one particular API.
It was dealing with the fact that something was always unfinished.
There was always another thing to fix.
A package behaving differently than expected.
A configuration issue.
A dependency error.
A call that didn't connect.
A state that wasn't updating properly.
A piece of logic that worked in one situation but broke in another.
And honestly, that's probably the part I needed most.
Because before this challenge, I had a habit of thinking:
“I should learn more first and then build.”
But the challenge flipped that around.
I built first.
Then I learned because I had a reason to learn.
That was a massive difference.
12. What I Learned About Agentic AI
Before the challenge, I was learning about agentic AI mostly from the theory side.
I knew about things like:
- LLMs
- tools
- memory
- RAG
- agents
- prompts
- function calling
- guardrails
But there is a huge gap between understanding those terms and actually building a system around them.
This challenge helped me connect the dots.
For example:
Memory
Before:
“Agents can have memory.”
After:
“I need to decide what information is worth remembering, get consent, store it, retrieve it, and make sure the retrieval happens at the right time.”
Tools
Before:
“Agents can call tools.”
After:
“I need to define the tool, its inputs, its output, when the agent should use it, and what happens when the tool fails.”
Guardrails
Before:
“Guardrails keep AI safe.”
After:
“Guardrails have to be designed into the actual behavior of the system.”
Handoffs
Before:
“Agents can transfer conversations.”
After:
“I have to decide when a handoff should happen, what context needs to move with it, and which agent should take over.”
That's the difference between knowing about something and knowing how to build with it.
And now I completely understand why people keep saying:
Learning skills practically teaches you more than just learning them theoretically.
Because theory tells you that a bicycle has two wheels.
Building one teaches you why you're falling over.
13. How You Can Build Your Own Voice Agent
You don't need to build the entire CashCompass system on day one.
Start with the smallest possible version.
You need four core pieces:
1. Speech-to-Text
Something like Deepgram converts the user's voice into text.
2. LLM
A model such as Gemini receives the text and generates the response.
3. Text-to-Speech
Murf Falcon converts the response back into speech.
4. Real-Time Transport
A framework such as LiveKit handles the real-time audio connection between the user and the agent.
Conceptually:
Microphone
↓
Speech-to-Text
↓
LLM
↓
Text-to-Speech
↓
Speaker
Then add the other pieces one at a time:
┌── Memory
│
User → Agent → Tools
│
├── Handoff
│
└── Human escalation
Don't start with ten features.
Get one voice conversation working first.
Then add memory.
Then tools.
Then more advanced capabilities.
That progression makes debugging dramatically easier.
14. Running the Project
The project is built around a Python voice-agent environment and LiveKit.
A typical setup looks like this:
git clone YOUR_REPOSITORY_URL
cd YOUR_PROJECT
python -m venv .venv
Activate the virtual environment and install the project dependencies:
pip install -r requirements.txt
Then create your environment file:
.env
Keep API keys there rather than writing them directly in your source code.
For example:
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_google_key
MURF_API_KEY=your_murf_key
Never commit **.env**** to GitHub.**
Add it to .gitignore:
.env
Then start your LiveKit environment and run the agent using the project's entry point.
The exact command depends on your project configuration, so check the repository README before running it.
Once the agent is running, connect from the frontend/LiveKit client and start speaking.
Start with something simple:
“Hi CashCompass, I just started earning. How should I begin managing my money?”
Then test the things that make your agent different:
“Can you remember my financial goal?”
“What schemes might be relevant to me?”
“Can I talk to a specialist?”
The important part isn't just getting the agent to speak.
Test whether it behaves correctly.
15. What I Would Improve Next
This challenge gave me a working foundation, but there is still a lot I would improve.
For example:
Better multilingual conversations
I want the agent to handle Indian languages and code-mixed speech even more naturally.
Better evaluation
Instead of manually deciding that the agent “seems to work”, I'd like automated evaluations for:
- response quality
- tool selection
- guardrail adherence
- handoff accuracy
- memory behavior
Better telephony reliability
Outbound calls need more robust handling of different SIP and telephony scenarios.
Better analytics
I'd like more detailed metrics around latency, call duration, failures, successful outcomes, and where users drop off.
More specialized agents
Instead of one general financial mentor, the system could eventually route conversations to specialist agents for budgeting, government schemes, investments, insurance, and other areas.
16. The Biggest Thing I Took Away
The biggest lesson from these 10 days wasn't:
“I learned how to build a voice agent.”
It was:
I learned that I can build things before I feel completely ready.
I avoided challenges before because I was afraid of not knowing enough.
Which, looking back, was a pretty bad strategy.
Because you don't become confident and then start building.
A lot of the time, you start building and then become confident because you survived the problems.
This challenge gave me practical experience with things I had previously only been learning about:
agentic AI, memory, tools, guardrails, real-time voice systems, multilingual interaction, outbound calls, human escalation, analytics, and agent handoffs.
And I'm leaving the challenge with something I didn't have when I started:
the confidence to participate in more challenges instead of watching other people build from the sidelines.
I definitely still have a lot to learn.
But now I have a much better idea of what learning actually looks like.
It looks less like:
“Let me finish five more tutorials before I start.”
and more like:
“Let me build it, break it, figure out why it broke, and try again.”
Apparently, that is also a very efficient way to become friends with error messages.
17. Final Thoughts
CashCompass, started as an idea for a voice financial mentor.
Over 10 days, it became much more than a basic conversational demo.
I got hands-on experience building a real-time voice pipeline, adding memory, connecting tools, designing guardrails, experimenting with outbound calls, implementing escalation and handoffs, and tracking call behavior.
More importantly, I finally experienced the difference between studying a technology and actually building with it.
And that's probably the biggest reason I enjoyed this challenge.
I didn't know everything when I started.
I still don't.
But I don't think that's a requirement anymore.
You just need to be willing to start building.
Links
GitHub Repository:
https://github.com/advaitashub/murf-livekit-starter/tree/original-murf-state
Technologies Used
- LiveKit
- Deepgram
- Gemini
- Murf Falcon
- Python
- SQLite / database layer
- SIP / telephony
- Real-time voice processing
Built during 10 Days of Voice Agents — VoiceForBharat Edition by Murf AI.
Top comments (0)