Building an AI voice agent sounded simple at first.
Give it a voice. Connect it to an LLM. Let users talk to it.
But as I progressed through the 10 Days of Voice Agents – Voice for Bharat Edition, I realized that a useful voice agent needs much more than the ability to listen and speak.
It needs to understand the user.
It needs to remember context.
It needs to know when to use external tools.
It should know when to continue helping and when a human should take over.
And most importantly, it should be able to measure whether it is actually helping the user achieve something.
Over the last 10 days, I built a voice-first Learning & Literacy agent focused on helping learners practice spoken English in a more natural and supportive way.
This is the story of what I built, the problems I faced, and what I learned while building it.
Why I Chose the Learning & Literacy Track
Many learners understand English but hesitate when it comes to speaking.
Sometimes the problem is not a lack of knowledge.
It is the fear of making mistakes.
People may worry about pronunciation, grammar, or simply saying something wrong in front of others.
I wanted to build something that gives learners a space to practice without feeling judged.
That became the main idea behind my project.
A voice-based learning companion that can:
Talk with learners naturally
Help them practice spoken English
Give exercises
Evaluate responses
Provide supportive feedback
Remember previous interactions
Support Hindi-English code-mixed conversations
Reach out for practice sessions
Know when a human teacher should step in
Track whether learning sessions are successful
Hand off specialised requests to another agent when needed
Instead of building just another chatbot with a voice attached to it, I wanted to explore what happens when a voice agent starts behaving more like a complete learning system.
Why Voice?
For spoken-English practice, typing is not enough.
A learner needs to actually speak.
Voice makes the interaction more natural.
The learner can respond as they would in a normal conversation instead of typing carefully constructed sentences.
It also makes the experience more accessible.
A learner can simply speak:
"Mujhe English practice karni hai."
And continue the conversation naturally.
This was especially important for my project because I wanted to support Hindi-English code-mixed conversations.
In India, many people naturally switch between languages while speaking.
Forcing users to speak in only one language can make an interaction feel unnatural.
So one of my goals was to make the agent respond in a similar register when appropriate.
The Core Idea
The final project can be viewed as a learning journey.
Learner speaks
↓
Voice agent understands the request
↓
Learner context and memory are considered
↓
Personalized learning interaction
↓
Exercise is provided
↓
Learner responds
↓
Answer is evaluated
↓
Supportive feedback
↓
Learning outcome is recorded
↓
If needed → Human help
↓
If specialist knowledge is needed → Agent handoff
The goal was not to make every feature independent.
I wanted the different parts of the project to work together.
Building the Voice Agent
The first step was creating the basic voice interaction.
A typical voice agent involves several components working together:
User speaks
↓
Speech-to-Text
↓
LLM / Agent Logic
↓
Memory + Tools + Agent Decisions
↓
Text Response
↓
Text-to-Speech
↓
User hears the response
For the voice output, I used Murf Falcon, the fast text-to-speech API.
The voice layer was important because the project was built around real conversations rather than text-based interactions.
But getting an agent to speak is only the beginning.
The next challenge was deciding how it should behave.
Giving the Agent a Clear Personality and Guardrails
For a Learning & Literacy agent, how the agent responds is just as important as what it responds with.
A learner can make mistakes repeatedly.
That is part of learning.
So one of the important guardrails I implemented was:
The agent should never shame a learner for giving a wrong answer.
Instead of saying:
"That's wrong."
The agent should respond in a supportive way.
For example:
"Good attempt. Let's look at this together."
The goal was to make the interaction feel encouraging.
Another important rule was that the agent should not make inappropriate assumptions about learners.
It should focus on helping them practice and improve without making claims beyond its role.
These instructions became the foundation for the rest of the project.
Adding Memory and Personalization
One of the biggest improvements came when I added persistent memory.
A normal conversation can feel disconnected when the agent forgets everything after the session ends.
If a learner practiced yesterday and comes back today, the experience should not always start from zero.
I integrated a database to store useful learner context.
This allowed the agent to remember information such as:
Learner identity
Preferences
Relevant learning context
Previous interactions
The frontend was also adapted to reflect learner information and session state.
This changed the experience from:
"Hello, how can I help you?"
every single time,
to something more contextual and personalized.
Memory made the agent feel less like a one-time interaction and more like a continuing learning companion.
Teaching the Agent to Use Tools
An AI agent cannot rely only on what the language model already knows.
Sometimes it needs to fetch information or perform a specific action.
For my Learning & Literacy track, I added tools that support the learning flow.
The agent can:
Fetch or select the next exercise
Evaluate a learner's answer
Use the result to provide feedback
One lesson I learned here was that creating the function itself is not enough.
The model needs to understand when it should use that function.
A tool can work perfectly in the backend and still fail to be useful if the description is unclear.
The tool description became an important part of the system.
The agent needed to know:
When to fetch an exercise
When to evaluate an answer
When not to call a tool unnecessarily
This was one of the first moments where I understood that building AI agents is not only about writing functions.
It is also about designing the decisions around those functions.
Adding Outbound Practice Calls
Most voice agents wait for the user to start the interaction.
For Day 6, I explored the opposite approach.
What if the agent could proactively reach out to the learner for practice?
For a learning agent, the use case was simple:
A learner chooses a practice time, and the agent can initiate a practice session.
This introduced a new challenge.
Outbound conversations need to begin differently from inbound conversations.
The user did not initiate the interaction.
So the agent needs to clearly communicate:
Who is calling
Why it is calling
How the user can stop future calls
While building this feature, I also faced practical challenges with telephony services, trial limitations, and account requirements.
I initially explored different options before moving toward a SIP-based setup using Linphone for testing.
This was one of the more challenging parts of the project because the idea was simple, but getting the communication setup working reliably required experimenting with different approaches.
It reminded me that real-world AI products are not only about AI.
Integration and infrastructure can sometimes take more time than the model logic itself.
Teaching the Agent When to Ask for Human Help
This was one of my favourite parts of the project.
An AI agent should not try to solve every situation by itself.
For my Learning & Literacy agent, I defined two situations where human support may be needed:
The learner is genuinely frustrated or upset.
The learner explicitly asks to speak with a teacher or human.
However, I also needed to make sure that the agent did not escalate unnecessarily.
A wrong answer is not a reason to involve a human.
Making mistakes is part of learning.
So the system needed to distinguish between:
"I got this answer wrong."
and
"I'm really frustrated. I don't understand this anymore. Can I talk to a teacher?"
When human help is needed, the agent does not immediately share information.
It first asks for permission.
The flow looks like this:
Learner needs help
↓
Agent detects the situation
↓
Agent explains what information can be shared
↓
Learner gives permission
↓
Human-help request is created
↓
Reference ID is generated
↓
Teacher or human can review the request
The request contains only useful information, such as:
Who needs help
What the learner is struggling with
What the agent has already tried
Urgency
Preferred language or follow-up method
The agent does not need to send the entire conversation.
This feature made the project feel more realistic.
Sometimes the best thing an AI system can do is recognize that another person is better suited to help.
Measuring Whether Learning Actually Happened
Building an agent is one thing.
Knowing whether it is useful is another.
For the Call Analytics Dashboard, I had to answer a simple question:
What does a successful call mean for my agent?
I decided that a successful learning session means:
The learner completes at least one exercise and receives feedback.
A failed session does not necessarily mean the system crashed.
It simply means the learning objective was not completed.
For example:
The learner ends the session early.
The learner does not complete an exercise.
The conversation stops before feedback is given.
Every session is recorded with an outcome.
The dashboard tracks:
Total Calls
Successful Calls
Failed Calls
The important part was that these numbers come from actual agent sessions.
They are not hardcoded values.
This changed the way I thought about AI agents.
It is easy to measure activity.
For example:
How many conversations happened?
But activity does not automatically mean success.
For a learning agent, a better question is:
Did the learner actually complete something useful?
That became the purpose of the analytics system.
Moving from One Agent to Multiple Agents
By Day 9, the main agent could already handle several tasks.
But one agent should not try to become an expert at everything.
For the Learning & Literacy track, I created a separate:
Maths Practice Specialist
The main Learning & Literacy agent continues to handle things such as:
Spoken English practice
English exercises
Grammar support
General learning conversations
But when the learner asks for maths help, the conversation is handed off to the Maths Practice Specialist.
For example:
Learner:
"I don't understand percentages."
The main agent responds:
"I'll connect you to our Maths Practice Specialist."
The specialist then continues the conversation.
The important part is that the learner does not have to repeat the entire problem.
Relevant context is passed during the handoff.
The flow looks like this:
Learner asks for help
↓
Main Learning Agent
↓
Does this require Maths expertise?
↓
YES
↓
Agent announces handoff
↓
Maths Practice Specialist
↓
Continues with existing context
The specialist can:
Explain concepts step by step
Give practice questions
Provide hints
Evaluate answers
Give supportive feedback
The same language style can also continue after the handoff.
For example:
"Mujhe fractions samajh nahi aa raha."
The specialist can continue naturally in a Hindi-English code-mixed style.
This was my first deeper experience with a multi-agent workflow.
The biggest lesson was simple:
Instead of making one agent responsible for everything, it can be better to give different agents clear and focused responsibilities.
The Overall Architecture
The final system can be represented like this:
┌───────────────────┐
│ Learner │
│ Voice / Browser │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Speech-to-Text │
└─────────┬─────────┘
│
▼
┌─────────────────────────────┐
│ Main Learning & Literacy │
│ Agent │
│ │
│ • English Practice │
│ • Guardrails │
│ • Memory │
│ • Learning Flow │
└───────┬──────────┬──────────┘
│ │
│ │
┌──────▼───┐ ┌──▼────────────────┐
│ Learning │ │ Maths Practice │
│ Tools │ │ Specialist Agent │
└──────┬───┘ └───────────────────┘
│
▼
┌─────────────────┐
│ Memory / SQLite │
└────────┬────────┘
│
┌────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Human Help Call Analytics Outbound
Escalation Dashboard Calling
│
▼
┌─────────────┐
│ Murf Falcon │
│ TTS │
└─────────────┘
The Difficult Parts I Faced
Not everything worked perfectly on the first attempt.
And honestly, that was probably one of the biggest learning experiences from this challenge.
- Telephony Setup
I initially explored different telephony services for outbound calling.
Some required upgrades before allowing access to certain features.
Others had trial limitations or different onboarding requirements.
I had to experiment with different approaches instead of assuming that the first service would work perfectly.
Eventually, I explored a SIP-based setup using Linphone for the outbound calling workflow.
The lesson was:
Always have an alternative approach when a third-party integration becomes a blocker.
- Tool Calling
Creating a function was relatively straightforward.
Making sure the agent calls it at the correct time was harder.
If the tool description is too vague, the agent might:
Call it unnecessarily
Never call it
Call it at the wrong point in the conversation
I learned that tool descriptions are part of the agent's reasoning interface.
The description needs to clearly explain:
What the tool does
When it should be used
When it should not be used
- Human Escalation
The difficult part was defining when the agent should stop.
A learner making mistakes does not mean the system failed.
The agent should continue encouraging and teaching.
But genuine frustration or an explicit request for a teacher is different.
Finding this boundary helped me understand the importance of defining clear conditions for AI systems.
- Defining Success
At first, it would have been easy to mark every completed conversation as successful.
But that would not tell me much.
A learner could talk for five minutes and still not complete a single exercise.
So I defined success around the actual learning outcome.
The learner needs to:
Complete at least one exercise.
Have the answer evaluated.
Receive feedback.
Only then is the session marked as successful.
This made the analytics more meaningful.
What I Learned About Building Voice Agents
After working on this project for 10 days, I learned that a voice agent is much more than:
Speech in → AI → Speech out.
A useful agent needs a complete system around it.
Here are some of my biggest takeaways.
- Voice is only one layer
The voice may be the first thing the user notices.
But memory, tools, guardrails, decisions, and context determine whether the interaction is actually useful.
- Memory changes the experience
Remembering previous interactions makes the agent feel more continuous.
The conversation no longer needs to restart from zero every time.
- Tools need good descriptions
A working function is not enough.
The agent needs clear instructions about when and why it should use it.
- AI should know its limits
Human escalation was an important reminder that an AI agent does not need to solve everything.
Knowing when to ask for help can be just as important as answering correctly.
- Measure outcomes, not just conversations
Counting calls is easy.
Understanding whether users achieved something meaningful is more valuable.
- One agent does not need to do everything
Specialist agents can have smaller, clearer responsibilities.
The main agent can focus on coordinating the experience and hand off specific tasks when needed.
How to Build and Run This Project
GitHub Repository
Repository: https://github.com/dibyendu-coder/murf-livekit-starter
Prerequisites
1.Python (v3.9 or higher)
2.Node.js (v18 or higher) and npm
3.LiveKit Cloud / Server credentials or local LiveKit Server instance
4.API Keys:
LiveKit API Key & Secret
Murf AI API Key
LLM Provider API Key (e.g., OpenAI / Gemini API key used in your agent backend)
Installation
Obtain the Project Files
Clone the repository or download the project files into a local folder.Configure Environment Variables (.env.local)
Create a .env.local file inside the murf-livekit-starter root (or backend/frontend folders as required by your setup) containing:
env
LiveKit Configuration
LIVEKIT_URL=wss://your-livekit-domain.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
Next.js Public URL
NEXT_PUBLIC_LIVEKIT_URL=wss://your-livekit-domain.livekit.cloud
Murf & AI Keys
MURF_API_KEY=your_murf_api_key
OPENAI_API_KEY=your_openai_api_key # (or relevant LLM key)
- Backend Setup Open a terminal and navigate to the backend folder:
bash
cd murf-livekit-starter/backend
Create and activate a Virtual Environment:
Windows (PowerShell):
powershell
python -m venv .venv
..venv\Scripts\Activate.ps1
macOS / Linux:
bash
python3 -m venv .venv
source .venv/bin/activate
Install Dependencies:
bash
pip install -r requirements.txt
Start the Agent Backend Worker:
bash
python agent.py dev
- Frontend Setup Open a second terminal window and navigate to the frontend folder:
bash
cd murf-livekit-starter/frontend
Install Node Dependencies:
bash
npm install
Start the Next.js Development Server:
bash
npm run dev
- Launch the Application Open your browser and go to http://localhost:3000. Click Connect / Start Call to initiate a voice session with your LiveKit + Murf AI voice agent.
Environment Variables
LiveKit Configuration
LIVEKIT_URL=wss://your-livekit-domain.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
Next.js Public URL
NEXT_PUBLIC_LIVEKIT_URL=wss://your-livekit-domain.livekit.cloud
Murf & AI Keys
MURF_API_KEY=your_murf_api_key
OPENAI_API_KEY=your_openai_api_key or GEMINI_API_KEY=your_gemini_api_key
Also make sure .env is included in .gitignore.
How to Run the Project
Activate the livekit server from the root folder :
.\livekit-server.exe --dev
Activate the backend : uv run python src/agent.py dev
Activate the frontend : pnpm dev
What I Built in 10 Days
Looking back, the project started as a simple idea:
Build a voice agent for learning.
By the end of the challenge, the project had grown into a system that could:
Communicate through voice
Follow learning and safety guardrails
Support Hindi-English code-mixed conversations
Remember returning learners
Personalize interactions
Fetch and provide learning exercises
Evaluate answers
Give supportive feedback
Make outbound practice calls
Ask for permission before escalating to a human
Create structured human-help requests
Track successful and failed learning sessions
Display analytics from real session data
Hand conversations to a specialist agent
The project changed significantly from the original idea.
And that is probably the biggest thing I enjoyed about this challenge.
Every day added a new question.
What should the agent remember?
When should it use a tool?
How should it contact the user?
When should it stop?
How do we know whether it is actually helping?
When should another agent take over?
Answering those questions slowly turned a simple voice assistant into something that feels much closer to a real product.
Final Thoughts
Ten days ago, I started with the goal of building a voice agent.
I thought the biggest challenge would be getting the AI to listen and respond.
It was not.
The bigger challenge was deciding how the agent should behave once the conversation starts.
What should it remember?
What should it do with that information?
When should it use a tool?
When should it continue helping?
When should it ask for human support?
And how do we know if the interaction was actually successful?
By the end of this journey, I realized that building an AI agent is not just about making it more capable.
It is also about giving it boundaries.
Knowing what it should do.
Knowing what it should not do.
And knowing when it should step aside.
This project is still a work in progress, but the last 10 days gave me a much better understanding of how voice agents can move beyond simple conversations.
I started by building a voice agent.
I ended up building the foundation of a voice-first learning companion that can remember learners, personalize practice, use tools, proactively initiate conversations, involve humans when needed, measure outcomes, and work with specialist agents.
And this is only the beginning.
Built as part of the 10 Days of Voice Agents – Voice for Bharat Edition using Murf Falcon.
Top comments (0)