I Built a Voice AI Tutor in 10 Days: What I Learned Building Palo for Voice for Bharat

Building a voice agent sounds simple at first.

You give an AI a microphone, connect an LLM, add text-to-speech, and let it talk.

That was more or less what I thought when I started the 10 Days of Voice Agents; Voice for Bharat challenge by Murf AI.

Ten days later, I had something quite different.

I had built Palo, a voice-first AI learning assistant designed around the Learning & Literacy track. It could have conversations with students, use tools, remember information, fetch practice questions, evaluate answers, make outbound calls, escalate problems to humans, and eventually hand mathematics questions over to a dedicated specialist.

Getting there was not clean.

There were API limits, model changes, context problems, SIP issues, broken Git states, latency problems, voice configuration issues, and several moments where something that looked like it should work simply did not.

But that was probably the most useful part of the challenge. This is what I built and what I learned along the way.

The Idea: A Tutor You Can Talk To

The project started with a relatively simple idea.

I wanted to build a learning assistant that did not require a student to sit in front of a text box and type perfectly formulated questions.

Instead:

• You talk to Palo. Palo talks back.
• The focus was accessibility and natural interaction rather than building another conventional chatbot interface.
• Palo is designed to help with subjects such as mathematics, physics, chemistry, biology, history, general knowledge, and other educational topics.
• The goal was not to make Palo an all-knowing teacher.
• The goal was to make it feel like a patient learning companion.

The final interface also went through several iterations.
I initially had a much more conventional voice-agent interface. Eventually, I moved toward a cleaner education-focused design with a more intentional visual identity.
The interface became secondary to the actual voice interaction.
That was an important lesson in itself.

Why Malayalam Matters

One of the decisions I made early was to support Malayalam, not just English.

That matters because voice AI becomes much more useful when people can interact with it in the language they are actually comfortable speaking.

For Kerala, this is especially relevant. A student might understand English well enough to read a textbook but still prefer asking a question naturally in Malayalam.

For younger students, parents, or people who are less comfortable with English-first technology, forcing everything through English adds unnecessary friction.

With Malayalam voice support, Palo can handle the interaction more naturally:

"എനിക്ക് ഈ കണക്ക് എങ്ങനെ ചെയ്യണമെന്ന് മനസ്സിലാകുന്നില്ല."

Instead of requiring the learner to translate their question into English first, the agent can respond in Malayalam and continue the conversation.

This is where I think voice agents have an advantage over traditional interfaces. You don't need to learn how to use the technology before you can use it. You just need to speak.

For Kerala, this could make an AI tutor more accessible in schools, homes, and especially in situations where parents or students are more comfortable communicating in Malayalam.

Murf Falcon: Making the Voice Feel Natural

For a voice agent, latency matters, but the voice itself matters just as much. I used Murf Falcon, Murf's fastest TTS API, for Palo's speech generation.

The goal wasn't simply to make the model speak. It was to make the interaction feel fast enough and natural enough that you could actually have a conversation with it.

For the main tutor, I use Nimisha.
For the Maths Practice Specialist, I use Madhavan, a male Malayalam-capable voice.

That became particularly important when I added Malayalam support. A Malayalam-speaking learner shouldn't have to switch to English just because the AI's voice experience was designed around English.

With Malayalam speech and a Malayalam-capable voice, the system can be much more relevant to learners in Kerala.

This is one of the things I found interesting while building the project: localisation isn't only about translating text. In a voice agent, language, speech recognition, voice selection, latency, and conversation design all become part of the user experience.

Then near the conclusion, I'd bring Kerala back into the bigger idea:

What This Could Mean for Kerala

Kerala has a strong education culture, but that doesn't mean every learner interacts with technology in the same way.

A Malayalam-first or Malayalam-friendly voice tutor could be useful for students who are more comfortable asking questions in their native language, parents helping their children study, and learners who find traditional English-first interfaces intimidating.

And because the interaction is voice-based, there is no need to navigate a complicated interface, type a question, or figure out exactly how to phrase a prompt.

You can simply ask.

That's ultimately what I wanted to explore with Palo: can voice AI make learning feel more accessible by adapting to the learner, rather than forcing the learner to adapt to the technology?

Building it over these ten days didn't give me a final answer.
But it gave me a working experiment to start finding one.

Day 1: Getting a Voice Agent to Speak

The first step was simply getting the pipeline working. The basic architecture looked like this:

Student → Speech-to-Text → LLM → Text-to-Speech → Student

I used:

• LiveKit Agents for real-time voice orchestration
• Deepgram for speech recognition
• Murf Falcon for text-to-speech
• An LLM for the conversational intelligence

The first milestone was surprisingly satisfying.

• Palo could hear me.
• Palo could think.
• Palo could speak.

It was a very basic agent, but it established the foundation for everything that came later.

Day 2: Turning It Into a Tutor

A voice model by itself is not a tutor. The next step was giving Palo an actual identity and behavioural rules. I added a structured system prompt covering:

• educational behaviour
• conversational style
• safety boundaries
• mathematical speech formatting
• language handling
• appropriate refusals
• how Palo should respond to incorrect answers

One thing I learned very quickly is that voice agents require different prompting from normal text chatbots. A response that looks fine on a screen can sound terrible when spoken. For example, mathematical notation such as:

x² + 5x

is perfectly understandable visually.
For speech, Palo needs to say:

"x squared plus five x"

That meant the agent had to be explicitly instructed to avoid raw mathematical notation, LaTeX, and other symbols when speaking. I also started working on English and Malayalam support.

SYSTEM_PROMPT = "No Math/LaTeX Symbols:
Speak all mathematical expressions in plain English.

Write "x squared" NOT "x^2".

Write "the derivative of x with respect to y" NOT "d/dy(x)".

Write "plus", "minus", "divided by", "times", "integral of".
Never use raw symbols like +, -, *, /, ^, or LaTeX commands."

Day 3: Personalising Palo's Frontend

Once Palo could talk and follow its role, the next step was to make the experience feel like an actual learning product rather than a generic voice-agent demo.

The goal for Day 3 was to build a simple frontend around Palo that clearly communicates what the agent is doing at every stage of a conversation.

The frontend needed to handle five states:

• Ready — Palo is waiting for the learner to start.
• Connecting — Palo is joining the voice session.
• Listening — Palo is listening to the learner.
• Speaking — Palo is responding.
• Call ended — The conversation has finished, with an option to start again.

I also made the interface clearly indicate who is speaking through visual feedback, so the learner does not have to guess whether Palo is listening or responding.

Day 4: Giving Palo Memory

The next step was making Palo remember things.
A conversational agent becomes much more useful when every interaction does not start from zero. I introduced an SQLite-based memory layer that could store information about the learner and their learning progress. This opened up possibilities such as remembering:

• student information
• language preference
• learning level
• topics covered
• previous mistakes
• other relevant learning progress

The important distinction here was that memory was not just about storing conversation history. It was about storing useful learning state.

That changed how I thought about voice agents. A voice agent does not have to be just a conversational interface. It can become an interface to an underlying stateful system.

Day 5: Giving Palo Tools

This was one of the first days where Palo started feeling like an actual agent rather than a chatbot. I gave Palo the ability to call external tools. For the Learning & Literacy track, this meant fetching practice exercises instead of simply asking the LLM to invent one every time.
The flow became:

Student asks for practice → Palo calls a tool → exercise is retrieved → Palo asks the question → student answers → Palo evaluates the response

This introduced a completely different class of problems.

• APIs fail.
• Rate limits happen.
• Models change.
• Context gets larger.

And if the tool fails, the agent still needs to respond naturally.
I also ran into Groq model and token-limit issues during testing. Repeated voice conversations accumulated enough context to hit model limits, and some model endpoints I tried were no longer available. So I changed models and adjusted the architecture rather than abandoning the feature. This was probably one of the more important lessons of the entire challenge.

The hardest part of an AI application is often not the AI. It is everything surrounding it.

Day 6: Taking Palo Outside the Browser

Until this point, the agent primarily existed inside a web application. Then came telephony. I wanted Palo to be able to make an outbound call and conduct a short practice session. The architecture became:

Palo → LiveKit → SIP trunk → phone

This was considerably more complicated than browser-based voice. There are more moving pieces:

• SIP
• telephony providers
• rooms
• participants
• call lifecycle
• dispatch
• audio transport

I eventually got outbound calling working through my SIP setup. But this was also where I encountered one of the most frustrating problems of the challenge.

• The phone would ring.
• The call would appear to connect.
• And then it would immediately become a missed call.
• The problem was not Palo's intelligence.
• It was the lifecycle of the SIP session.

That was a good reminder that voice AI is not just an LLM problem.
It is also a real-time system problem.

Day 7: Human Escalation

A useful AI assistant should also know when it should stop trying to solve something itself. I added a human-help escalation mechanism.

If a learner explicitly asks for a teacher or human assistance, Palo can offer to create a help request. But there is an important safety rule: Palo cannot create the request without the learner's permission.

Before creating one, Palo explains what information it intends to share and asks for consent. The request can include information such as:

• what the learner needs help with
• what happened
• what Palo already tried
• urgency
• language
• preferred follow-up method

A reference ID is then generated for the request. This was less flashy than adding another AI feature, but I think it is one of the more important parts of the project. A good assistant is not one that tries to solve everything. Sometimes the correct action is:

"A human should take this from here."

Day 8: Measuring Whether the Call Actually Worked

At this point, I wanted more than a working conversation. I wanted to know what actually happened during a learning session. So I added call analytics. Palo now tracks information such as:

• channel
• exercise subject
• difficulty
• question
• student answer
• expected answer
• whether the answer was correct
• failure reason

For simple exercises, the system normalizes the student's spoken answer and compares it with the expected answer. This means a completed call can be recorded as a meaningful learning outcome rather than simply:

"The AI talked to someone."

That distinction matters. If you are building an educational AI, measuring conversation length is much less useful than knowing whether the student actually completed the learning activity.

Day 9: Introducing a Maths Specialist

Day 9 was where the architecture became more interesting.
Instead of asking one model to handle everything, I introduced a dedicated Maths Practice Specialist. and Palo remains the general tutor. When the student asks for focused mathematics help, such as:

• solving an equation
• algebra practice
• calculus
• quadratic equations
• step-by-step mathematical reasoning

Palo can hand the conversation to the specialist. The important part is that the student does not have to repeat the question. The conversation context is passed to the specialist.

The specialist also uses a stronger reasoning model through Groq:

• Palo: Qwen 3.6 27B
• Maths Specialist: GPT-OSS 120B

This allowed me to separate the responsibilities of the two agents. Palo is responsible for general tutoring and conversation.
The maths specialist is responsible for focused mathematical reasoning. That architecture is much closer to how I now think production voice agents should be designed.

You do not necessarily need one giant model that knows everything.
Sometimes you need a good router and several focused specialists.

Giving the Specialist Its Own Voice

This was another small feature that made a surprisingly large difference.

**- The main Palo voice is Nimisha.

• For the mathematics specialist, I switched to Madhavan, a male Malayalam-capable voice.**

So when the specialist takes over, it does not just feel like the same model changing its system prompt. It feels more like being connected to another tutor.

# main vc
ENGLISH_VOICE = "Nimisha"

# ml-in palo voice
MALAYALAM_VOICE = "Nimisha"

# voice api for math spc
MATH_SPECIALIST_VOICE = "Madhavan"

The voice is powered by Murf Falcon, Murf's fastest TTS API. For a voice agent, latency matters enormously. If the system thinks for several seconds and then starts generating a long response, the interaction immediately feels unnatural.

Falcon's low-latency design helped keep the spoken interaction responsive. And this challenge made something very clear to me:

• TTS is not just the final layer of an AI application. It is part of the user experience.
• The voice, pacing, pronunciation and latency all influence whether the system actually feels conversational.

The Architecture

After ten days, Palo evolved into something considerably more complex than the original prototype.

At a high level:

                Student
                   |
                   v
            Speech / Voice
                   |
                   v
             Deepgram STT
                   |
                   v
             LiveKit Agent
                   |
                   v
            Palo - General Tutor
              /            \
             /              \
            v                v
   Practice Tools      Maths Specialist
            |                |
            v                v
      SQLite / Data     Groq Reasoning
                             |
                             v
                       Murf Falcon TTS
                             |
                             v
                          Student

Alongside that main flow are additional systems for:

• persistent memory
• exercise retrieval
• answer evaluation
• human escalation
• call analytics
• SIP telephony

The implementation remained intentionally simple in one important respect:

the Maths Specialist lives inside the same agent.py architecture rather than being split into a separate service.
That kept the project manageable while still allowing the agent to have specialist behaviour.

The Technology Stack

The final stack looked roughly like this:

• LiveKit Agents : LiveKit handles the real-time voice-agent infrastructure and communication layer.
• Deepgram Nova-3 : Used for speech-to-text.
• Groq : Used for the LLM layer.The general tutor runs on Qwen 3.6 27B, while the mathematics specialist uses GPT-OSS 120B.
• Murf Falcon : Used for text-to-speech. The main tutor uses Nimisha, while the mathematics specialist uses Madhavan. Falcon was particularly important because voice interaction is extremely sensitive to latency.
• SQLite : Used for local memory, escalation data, and call analytics.
• Python : The backend and agent logic are implemented in Python.
• Next.js : Used for the frontend voice interface. The frontend eventually evolved into a more focused education-oriented design rather than looking like a generic AI dashboard.

What Went Wrong

This might actually be the most useful part of the project.
Because almost everything went wrong at some point.

• There were model limits.
• There were unavailable model endpoints.
• There were context-size problems.
• There were TTS configuration issues.
• There were Malayalam pronunciation problems.
• There were Git problems.
• There were frontend layout problems.
• There were SIP calls that rang but immediately became missed calls.
• There were situations where the code looked correct but the real-time system behaved differently.
• And there were several moments where I had to stop adding features and figure out why something that worked yesterday had suddenly stopped working.

One of the biggest lessons was that building voice AI is not:

LLM + microphone = voice agent.

It is closer to:

real-time networking + STT + LLM + tools + memory + TTS + state management + error handling + telephony + UX.

The LLM is only one component.

Another Lesson: Faster Models Are Not Always Better

During the challenge, I experimented with different models and ran into token and rate limits.

At one point, repeatedly testing the voice loop caused the available token capacity to become a practical bottleneck. I switched models and model families several times before finding a setup that behaved reliably enough.
The interesting part was that model quality was not the only consideration.

For voice agents, you have to balance:

• reasoning quality
• token limits
• latency
• reliability
• cost
• context size
• tool-calling behaviour

A theoretically stronger model is not automatically the better choice if it makes the entire voice interaction painfully slow.

What I Learned About Voice UX

The biggest difference between text AI and voice AI is that silence is visible in voice.

In a text chatbot, waiting two seconds is barely noticeable.
In a conversation, two seconds can feel like the person stopped responding.

That makes latency a UX feature. So does response length.

I eventually constrained spoken responses heavily.
Instead of producing a paragraph, Palo should generally give one or two concise spoken ideas and continue the conversation. That makes the interaction feel much more natural.

The challenge also made me pay much more attention to things that are easy to ignore in text applications:

• pronunciation
• sentence length
• interruptions
• pacing
• speech recognition errors
• mathematical notation
• language switching
• TTS latency
• Why I Chose Voice for Learning

There are already countless educational chatbots.
I did not want Palo to simply be another text box with an LLM behind it.
Voice changes the interaction.
A student can ask:

"Can you explain this again?"

• without having to formulate a perfect written question.
• They can answer a practice question naturally.
• They can interrupt.
• They can ask follow-up questions.
• And eventually, the same system can move beyond a browser and into a phone call.
• That is where I think voice agents become particularly interesting for education.
• The interface becomes almost invisible.
• You just talk.

What I Would Build Next

Palo is still a prototype.

There are plenty of things I would improve.

1. Better multilingual support: The current implementation focuses primarily on English and Malayalam. A real Bharat-focused learning assistant should eventually support more Indian languages natively.

2. Better student modelling: The memory layer currently stores useful learning information, but a production system could build a much richer learner profile.

For example:

• weak concepts
• recurring mistakes
• preferred explanation styles
• learning pace
• topic mastery
• Better evaluation

Simple exact-answer comparison works for basic exercises.
It is not enough for open-ended mathematics or descriptive answers.
A future version should evaluate reasoning and partial correctness more carefully.

1. Better telephony infrastructure: The SIP work taught me that outbound voice calls introduce a completely different set of engineering problems. I would like to make that layer more robust and production-ready.

2. Better specialist routing: The maths specialist is only the beginning.

The same architecture could eventually support:

Palo
|
+-- Maths Specialist
|
+-- Science Specialist
|
+-- Language Specialist
|
+-- Exam Practice Specialist
|
+-- Human Teacher

The general tutor becomes the conversational front door, while specialists handle focused tasks.

Ten Days Later

When I started the challenge, I thought I was mainly going to learn how to build a voice chatbot.

• I ended up learning something broader.
• I learned about real-time systems.
• I learned how much latency changes conversational UX.
• I learned that tool calling introduces an entirely different class of failure modes.
• I learned that memory is useful only when you know what should actually be remembered.
• I learned that telephony is much harder than browser-based voice.
• I learned that guardrails matter just as much as intelligence.

And I learned that an AI agent becomes much more interesting when it can decide which capability should handle a problem, instead of trying to do everything itself.

Most importantly, I learned that making something work once is very different from making it reliable.

Palo is nowhere near a finished educational product.

But after ten days, it can listen, speak, remember, use tools, conduct learning sessions, evaluate answers, make phone calls, escalate to humans, and bring in a dedicated mathematics specialist.

That is a much more interesting starting point than the chatbot I had on Day 1.

Final Architecture:

• Palo — General Tutor: Qwen 3.6 27B through Groq, Murf Falcon + Nimisha
• Maths Practice Specialist: GPT-OSS 120B through Groq, Murf Falcon + Madhavan
• Speech-to-Text: Deepgram Nova-3
• Real-Time Voice Infrastructure: LiveKit Agents
• Memory and Analytics: SQLite
• Backend: Python
• Frontend: Next.js

Final Thoughts

The most valuable thing about the 10 Days of Voice Agents challenge was not ending up with a finished product.

• It was being forced to build continuously.
• Every day introduced another layer.
• A voice.
• Then a personality.
• Then tools.
• Then memory.
• Then telephony.
• Then analytics.
• Then specialist agents.
• And with every new layer came another engineering problem to solve.

That is probably the biggest takeaway I am carrying forward:

Building an AI agent is not about making a model talk.
It is about building everything around the model that allows it to be useful.

Palo started as a voice.

After ten days, it became a system.
And I think that is where voice AI starts getting genuinely interesting.

Built during the 10 Days of Voice Agents — Voice for Bharat challenge by Murf AI.

Stack: LiveKit Agents · Deepgram Nova-3 · Groq · Murf Falcon · Python · Next.js · SQLite

Project: https://github.com/Thush-ar/murfVoiceForBharat[](url)

VoiceForBharat #VoiceAI #MurfAI #LiveKit #AI #EdTech #BuildInPublic #LearningAndLiteracy