Building a Voice-First Learning Companion — My 10 Days of Voice Agents Journey

Roshan Sankhat — Sat, 15 Aug 2026 05:21:09 +0000

Over the last ten days, I took part in 10 Days of Voice Agents — VoiceForBharat Edition, building a voice agent from scratch using Murf Falcon, the fastest TTS API. Here's the story of what I built, the problems I ran into, and how you can build something similar.

The problem and the users

School-level students often need help across general learning and literacy topics but don't always have someone available to talk to instantly. Typing out doubts is also a real barrier — especially for younger students or those less comfortable with text.

I built a voice-first learning companion for the Learning & Literacy track that lets students just ask their question out loud, the way they'd ask a teacher in person, instead of typing it out.

What the voice agent does

The agent helps school-level students with learning and literacy topics across subjects. For maths specifically, it hands the conversation off to a dedicated Maths Practice Specialist agent that can walk students through problems step by step. When the agent hits something it can't confidently handle, it escalates to a human via email rather than guessing — so students always have a reliable path to help.

How the system works

Audio flows from the student through speech-to-text, into the main LLM agent, and back out through text-to-speech — all in real time. The main agent can branch out to tools, a specialist agent, or human escalation whenever the conversation needs it.

(See the architecture diagram shared alongside this post.)

The most important features
Indian voice powered by Murf Falcon — the agent speaks in a Murf voice built for India, making it feel natural and familiar to Indian students.
Personality, objectives & guardrails — clear objectives (help with learning, never shame wrong answers, never diagnose learning disabilities) and a defined escalation script.
Code-mixed language support — handles students who mix Hindi and English mid-conversation, replying in the same register.
Stateful frontend — clearly shows Ready, Connecting, Listening, Speaking, and Call Ended states so students always know what's happening.
Memory for returning students — saves each learner's level, topics covered, and recurring mistakes, and greets them by name on their next call, continuing where they left off.
Tools for real learning tasks — fetches the next exercise based on level and scores spoken answers, with a graceful fallback if data is unavailable.
Outbound practice calls — calls the student at a time they picked for a daily practice session.
Human escalation — when a learner is upset or needs a teacher, the agent creates an escalation request and emails a human with a summary (not the full transcript).
Call analytics dashboard — tracks total, successful, and failed calls using real data, without exposing transcripts or private info.
Specialist handoff — routes maths-related queries to a dedicated Maths Practice Specialist agent, which continues the conversation without the student repeating themselves.
Challenges and how I overcame them

One of the trickiest parts of the challenge was getting outbound calls working. Since the agent had to initiate contact on its own instead of waiting for a user to connect, testing wasn't as simple as opening a browser tab — I used Linphone, a SIP client, to actually receive and test these outbound calls, which took some setup to get right.

The bigger, recurring problem across all nine days was API token limits. Thoroughly testing a voice agent — across multiple features and conversation paths — burns through tokens fast, and I kept running into rate limits mid-testing. To keep moving without losing momentum, I ended up using API keys across almost all the major LLM providers, switching between them whenever one ran low. Fortunately, the switch was smooth with no compatibility issues between providers.

How readers can build and run it

If you want to build something similar, here's how the pieces fit together and how to get started.

Main components:

Speech-to-text (STT) — converts the student's spoken input into text
LLM — processes the text, decides what to say, and calls tools (fetch exercise, escalate to human, hand off to specialist) when needed
Text-to-speech (TTS) — Murf Falcon converts the agent's reply into natural Indian-voice speech
Real-time transport — LiveKit handles the live audio connection between the student and the agent

Setup & running:

Clone the repo: git clone https://github.com/RoshanSankhat/murf-livekit-starter.git
Follow the setup and run instructions in the repo's README
Add your API keys to a .env.local file — never commit this file or expose your keys publicly
Run the project locally and connect via the frontend to start a conversation

Testing a conversation: Open the frontend, click to connect, and talk to the agent once it's in the "Listening" state. Watch the state indicators (Connecting → Listening → Speaking) to follow the flow.

You can inspect the full code, including the human escalation and specialist handoff logic, in the repo.

What I would improve next
Reduce dependency on switching between multiple LLM providers by optimizing token usage during testing
Expand outbound call testing to more real-world scenarios beyond Linphone
Add more subject-specific specialist agents beyond maths
Links to the code and demos
Repository: murf-livekit-starter (fork)
Day 1 – First voice connection: https://lnkd.in/p/dKeQzP8P
Day 2 – Persona, objectives & guardrails: https://lnkd.in/p/dp_QBgzx
Day 3 – Frontend with agent states: https://lnkd.in/p/db_dGTpp
Day 4 – Memory for returning students: https://lnkd.in/p/dxxRzjaT
Day 5 – Tools for fetching exercises: https://lnkd.in/p/dUY4w4VS
Day 6 – Outbound practice calls: https://lnkd.in/p/gJWHjZTq
Day 7 – Human help escalation: https://lnkd.in/p/gwFzKAbt
Day 8 – Call analytics dashboard: https://lnkd.in/p/dxgTD4Sz
Day 9 – Maths specialist handoff: https://lnkd.in/p/dgrmVqzB

Built using Murf Falcon — the fastest TTS API — as part of 10 Days of Voice Agents — VoiceForBharat Edition. #VoiceForBharat

DEV Community: Roshan Sankhat

Building a Voice-First Learning Companion — My 10 Days of Voice Agents Journey