Introduction
As part of Murf AI's 10 Days of Voice Agents — VoiceForBharat Edition, I built an AI Voice Learning Assistant under the Learning & Literacy track.
The goal was to create a voice-first learning experience where learners can interact naturally with an AI assistant, practice learning exercises, receive personalized assistance, request human help, and even be connected to a specialized Maths agent.
Instead of building an AI that simply answers questions, I wanted to build a complete voice learning system with memory, tools, analytics, human escalation, and multi-agent handoff.
The Problem
Many digital learning platforms depend heavily on typing and reading. This can make interaction less natural, especially for learners who are more comfortable communicating through speech.
I wanted to explore how real-time voice AI could make learning more interactive and conversational.
The project focuses on creating a voice-based learning assistant that can:
- Understand a learner's speech
- Respond naturally using voice
- Remember useful learner information
- Conduct learning exercises
- Provide feedback
- Escalate to human assistance when needed
- Hand off specialized questions to another AI agent
- Track learning session outcomes
Technology Stack
| Technology | Purpose |
|---|---|
| Python | Backend and AI agent |
| LiveKit | Real-time voice communication |
| Deepgram Nova-3 | Speech-to-Text |
| Google Gemini | LLM / reasoning |
| Murf Falcon | Text-to-Speech |
| Silero VAD | Voice activity detection |
| Next.js | Frontend |
| React + TypeScript | User interface |
| Tailwind CSS | Styling |
| SQLite | Learner memory |
Architecture
The voice interaction follows this pipeline:
User Speech
↓
LiveKit
↓
Deepgram Nova-3
↓
Google Gemini
↓
Agent Tools / Memory / Specialist
↓
Murf Falcon
↓
User hears response
My 10-Day Journey
Day 1 — Make the Agent Talk 🎙️
Built the basic real-time voice agent using LiveKit, Deepgram, Google Gemini, and Murf Falcon.
Day 2 — Personality & Guardrails 🛡️
Defined the agent's personality, behavior, role, and safety boundaries.
Day 3 — Frontend Interface 💻
Customized the frontend and added visual feedback for the voice interaction.
Day 4 — Multilingual Interaction 🌐
Worked on language-aware conversations and Indian language/code-mixed interaction.
Day 5 — Tools & Memory 🧠
Added persistent learner memory and interactive learning exercises.
Day 6 — Outbound Calling 📞
Experimented with connecting the voice agent to phone calls using Twilio and LiveKit.
Day 7 — Human Escalation 🆘
Added the ability to request human/teacher assistance and generate an escalation ID.
Day 8 — Analytics Dashboard 📊
Built a dashboard showing total, successful, and failed learning sessions.
Day 9 — Specialist Handoff 🧮
Implemented handoff from the main learning assistant to a dedicated Maths Specialist Agent.
Day 10 — Share the Journey 🚀
Documented the complete project, architecture, challenges, solutions, and lessons learned.
✨ Key Features
🧠 Memory
The AI Voice Learning Assistant can remember useful information shared by the learner during conversations. The memory system uses SQLite to store learner-related information and preferences, allowing the assistant to provide a more personalized learning experience. The stored information can also be updated when the learner's preferences change.
📚 Learning Exercises
The assistant provides interactive learning exercises through voice. Learners can answer questions verbally and receive immediate feedback from the AI. Exercises are stored in exercises.json, and learner responses are evaluated to determine whether they are correct or incorrect. The exercise results are also used to track the outcome of learning sessions.
🆘 Human Escalation
When a learner needs help from a human teacher or the AI is unable to appropriately handle a request, the learner can request human assistance. The system creates a unique escalation ID for the request, which can be used to track the learner's need for human support through the dashboard.
📊 Analytics Dashboard
The analytics dashboard provides an overview of voice learning sessions by displaying Total Calls, Successful Calls, and Failed Calls. Session outcomes are recorded based on the learner's interaction with the learning exercises, providing useful insights into the effectiveness of learning sessions.
🧮 Maths Specialist Agent
The system includes a dedicated Maths Practice Specialist Agent for mathematics-related learning requests. When the learner asks for help with a maths problem, the main learning assistant can hand off the conversation to the specialist agent. Relevant conversation context is passed during the handoff so that the learner does not need to repeat the entire problem.
🎙️ Voice Interaction
The application provides real-time voice interaction, allowing learners to communicate naturally through speech. Deepgram Nova-3 is used for speech-to-text, Google Gemini handles AI reasoning and conversation, and Murf Falcon provides text-to-speech. LiveKit enables real-time voice communication, while Silero VAD and LiveKit Turn Detector help detect speech and conversation turns.
🛡️ Guardrails
The assistant follows defined instructions and safety boundaries to provide a focused learning experience. It stays within its educational role, asks for clarification when requests are unclear, avoids exposing sensitive information or API credentials, and uses human escalation when AI assistance is not sufficient. Guardrails also help the assistant decide when to use learning tools and when to hand off mathematics-related requests to the specialist agent.
Challenges I Faced
Building the project was not completely straightforward.
1. Persistent Memory
Initially, the agent remembered a learner's first preference but did not correctly update it when the learner changed their preference. I had to improve the memory workflow so that information could be retrieved and updated correctly.
2. Analytics Outcome Detection
One challenge was correctly distinguishing successful and failed learning sessions. I improved the session outcome logic so that the actual exercise evaluation determines the final analytics result.
3. Human Escalation
Initially, generating an escalation ID did not immediately provide the expected dashboard visibility. I worked on connecting the escalation request with the frontend dashboard.
4. Specialist Handoff
The main challenge was making the transition between agents feel natural while preserving enough conversation context so the learner would not need to repeat the problem.
What I Learned
This challenge helped me understand that building a voice agent is much more than connecting an LLM to a text-to-speech API.
I learned how to work with:
- Real-time voice communication
- Speech-to-Text and Text-to-Speech pipelines
- Agent tools
- Persistent memory
- Session state
- Analytics
- Human escalation
- Multi-agent systems
- Frontend and backend integration
- API key and environment-variable security
Future Improvements
Some improvements I would like to make include:
- Supporting more specialized learning agents
- Adding more learning subjects
- Improving multilingual support
- Providing richer learner progress analytics
- Adding teacher accounts and dashboards
- Improving personalization using long-term learner history
- Deploying the complete system for real-world learners
Conclusion
The 10 Days of Voice Agents challenge gave me the opportunity to move from building a simple conversational voice agent to designing a more complete AI learning system.
The biggest lesson I learned is that a useful voice agent needs more than a good voice. It needs memory, tools, reliable state management, safety boundaries, analytics, and the ability to involve humans or specialized agents when necessary.
This project helped me understand how real-time voice AI can be applied to learning and how technologies such as LiveKit, Deepgram, Google Gemini, and Murf Falcon can work together to create a natural voice-first experience.
Project Links
Github Repository -https://github.com/NariboyinaRajaVaishnavi7/murf-livekit-starter
Top comments (0)