From a Voice That Talks to a Voice Agent That Knows What to Do

Nakkina Sai Venkata Teja — Sat, 15 Aug 2026 07:58:49 +0000

What if an AI could teach a child the way a patient parent does?

That was the question I started with.

Not Can I build an AI that talks?

That part is becoming easier.

The harder question was:

Can I build a voice agent that actually understands its role, remembers the right things, knows when to use a tool, knows when to ask a human for help, and knows when another agent is better suited to handle the conversation?

Over the last ten days, I built BoloBuddy around that question.

At the center of BoloBuddy is Chinnu 🐥 a voice-first language learning companion designed to help young children learn Telugu, Hindi, and English naturally through conversation.

The journey started with a simple voice pipeline.

It ended with a system that could remember individual children, use external tools, make outbound phone calls, escalate to a human, measure whether a learning session succeeded, and hand conversations to a pronunciation specialist.

And somewhere along the way, I realized something important:

Building a voice agent isn't really about giving an AI a voice. It's about teaching it when and how to act.

The problem I wanted to solve

Think about how a young child learns their first words.

A parent points at a ball.

“Ball.”

Then again.

“Ball.”

The child tries.

Maybe they get it right.

Maybe they don't.

The parent doesn't immediately turn the interaction into a test. They repeat the word, encourage the attempt, and try again.

That kind of patient, repetitive interaction is difficult to reproduce consistently.

Parents are busy. Families may not always have someone available to practice with a child. And in multilingual environments, children naturally move between languages rather than staying inside the boundaries of a textbook.

So I wanted to build a voice-first companion that could make language practice feel like a conversation rather than a lesson.

That became BoloBuddy.

The idea was simple:

A child talks to Chinnu. Chinnu listens, introduces simple words, encourages attempts, remembers progress, and adapts future conversations around what the child has already learned.

Voice was important because children shouldn't have to read menus, type answers, or navigate a complicated interface to interact with their learning companion.

They should be able to simply talk.

It started with four pieces

The first version of the system was much simpler.

At its core was a real-time voice pipeline:

Child → Speech-to-Text → LLM → Text-to-Speech → Child

The components were:

Deepgram Nova-3 for multilingual speech-to-text
Google Gemini as the LLM
Murf Falcon for text-to-speech
LiveKit Agents for the real-time voice pipeline and transport
Next.js + Tailwind CSS for the frontend

Murf Falcon became the voice of Chinnu, providing an Indian voice that helped make the experience feel more natural for the target users.

But getting the pipeline to work was only the beginning.

A voice that can respond isn't necessarily a useful voice agent.

So I started asking what Chinnu would actually need.

First, Chinnu needed an identity

The first major realization was that the personality of a voice agent matters.

I didn't want Chinnu to behave like a generic chatbot that answers every question.

Chinnu needed a specific role.

It should behave like a caring language-learning companion:

Introduce one word at a time
Encourage every attempt
Correct gently
Keep conversations natural
Support Telugu, Hindi, and English
Refuse requests outside its role

More importantly, I had to define what Chinnu should never do.

It should never shame a child for a wrong answer.

It should never compare children.

It should never diagnose developmental conditions.

And it shouldn't pretend to know things outside its role.

That was one of my first lessons in building AI agents:

A good system prompt isn't just a description of what an agent can do. It is also a boundary around what it should not do.

Then the voice needed a place to live

A voice agent can technically work with nothing more than a microphone and a response.

But for a child, the surrounding experience matters.

So I built a complete interface around Chinnu.

The frontend included:

Animated word bubbles
A hand-crafted Chinnu character
Language badges
A clear “Talk to Chinnu” call-to-action
Five visible agent states: Ready → Connecting → Listening → Speaking → Call Ended
A “Who is speaking?” indicator
A friendly microphone permission experience
A child-oriented visual design using playful animations and Nunito typography

This changed the experience from:

“Here is an AI voice.”

to:

“Here is Chinnu, your learning buddy.”

That distinction matters when the user is a child.

The moment Chinnu became personal: memory

A child learning a language shouldn't have to start from zero every time they return.

That led to the next question:

What should Chinnu remember?

I integrated MongoDB to give each child persistent learning memory.

Every child gets their own learning identity, and the system can store information such as:

Name
Words learned
Previous mistakes
Interaction history

But I didn't want to simply dump all of this information into the system prompt.

Instead, Chinnu interacts with memory through function calling.

When memory is needed, the agent can retrieve it.

When new learning information needs to be saved, it can update the relevant data.

I also added consent-based memory saving.

Chinnu asks before remembering new learning information, and saving is blocked when permission hasn't been granted.

There is also strict child-level memory isolation so an agent tool call cannot simply select another child's data.

That produced a much more meaningful learning loop:

Child struggles with a word → Chinnu remembers → conversation ends → child returns → Chinnu retrieves the memory → future practice continues from there.

The goal wasn't to make Chinnu remember a child's name.

The goal was to make every conversation useful to the next conversation.

Then I gave Chinnu access to the outside world

An LLM has a lot of knowledge.

But sometimes the right answer isn't something it should generate from its internal knowledge.

Sometimes it should look something up.

So I integrated the Free Dictionary API using a function tool.

The flow became:

Gemini decides a lookup is needed → Dictionary tool → External API → Gemini → Child-friendly explanation → Murf Falcon

This taught me an important distinction between an LLM and an agent.

An LLM can generate an answer.

An agent can decide:

“I need to use a tool before I answer.”

I also had to think about what happens when the tool fails.

During testing, I said “OK” and Chinnu treated it as a word to look up.

The dictionary didn't return a useful result.

Instead of inventing a definition, Chinnu followed the failure path and responded naturally by suggesting another word.

That small failure was actually one of my favorite moments of the challenge.

Because production systems don't only need a successful path.

They need a sensible answer when things go wrong.

Then Chinnu got a phone

Until this point, Chinnu waited for someone to open the application.

Day 6 changed that.

I connected LiveKit SIP with Twilio and gave Chinnu the ability to make outbound phone calls.

The architecture became:

Chinnu → LiveKit → SIP → Twilio → Phone

And the conversation travelled back through the voice pipeline:

Phone → Twilio → LiveKit → Deepgram → Gemini → Murf Falcon → Phone

This was the first time the agent stopped being limited to a browser.

It could initiate a real telephone conversation.

But telephony introduced a completely different layer of complexity.

The AI wasn't directly “calling a phone.”

There was a bridge between the voice agent and the traditional telephone network.

Understanding that bridge was one of the most valuable technical lessons of the challenge.

I also had to teach Chinnu when to stop

One of the most important changes wasn't adding another capability.

It was adding a boundary.

A good agent shouldn't try to solve every problem itself.

For BoloBuddy, I identified two situations where Chinnu should ask for human help:

When a child becomes frustrated
When a child explicitly asks for a teacher

But escalation couldn't simply be:

Problem → Human

There needed to be a consent step.

The flow became:

Problem → Explain → Ask Permission → Escalate → Notify Parent

Chinnu asks what information will be shared and gets permission before creating the escalation request.

The system creates a short, useful request rather than simply storing the entire conversation, connects it to the correct child's identity, and surfaces an alert in the parent dashboard.

And if the child says no?

Nothing gets created.

That made me think differently about AI autonomy.

Knowing what an agent can do is only half the design problem. Knowing when it should not act alone is equally important.

Then came the question: how do I know if Chinnu is actually helping?

An AI system can have a perfect connection, a long conversation, and a very natural voice—and still fail at its actual purpose.

For a language-learning agent, I needed a measurable definition of success.

So I defined one:

A session is successful when the child successfully learns at least two unique words.

I then built session-level tracking around that objective.

The system:

Tracks successfully learned words
Prevents duplicate words from being counted multiple times
Determines whether a completed call succeeded or failed
Stores call outcomes and learning data in MongoDB
Displays Total Calls, Successful Calls, and Failed Calls in an analytics dashboard

This changed the way I thought about agent evaluation.

For BoloBuddy, success isn't:

“The child talked to the AI for ten minutes.”

It isn't even:

“The voice connection worked.”

The meaningful question is:

“Did the child achieve the learning objective?”

That shift—from capability to measurable outcome—is something I want to carry into every future agent I build.

Finally, Chinnu learned that it doesn't have to be the smartest agent in the room

The final major step was introducing a second agent.

I created a Pronunciation Specialist whose job is focused specifically on pronunciation help.

Instead of forcing Chinnu to handle every pronunciation problem, it can hand the conversation over when specialized help is needed.

The flow became:

Child → Chinnu → Identify Pronunciation Need → Handoff → Pronunciation Specialist → Continue Conversation

The important part wasn't simply creating another agent.

It was defining a clear responsibility for that agent.

The conversation context is preserved, so the child doesn't have to explain the problem again.

And Chinnu doesn't hand off normal conversations unnecessarily.

That gave me another important lesson:

Multi-agent systems aren't about adding more agents. They're about giving each agent a clear responsibility.

What the final system looks like

After ten days, BoloBuddy had evolved considerably from the first voice prototype.

At a high level:

                     ┌──────────────────┐
                     │      Child       │
                     └────────┬─────────┘
                              │
                     Voice / Phone
                              │
                     ┌────────▼─────────┐
                     │     LiveKit      │
                     │ Real-time Voice  │
                     └────────┬─────────┘
                              │
                ┌─────────────┼─────────────┐
                │             │             │
                ▼             ▼             ▼
           Deepgram        Gemini      Murf Falcon
              STT            LLM            TTS
                              │
                ┌─────────────┼──────────────┐
                │             │              │
                ▼             ▼              ▼
            MongoDB      Dictionary API   Escalation
             Memory          Tool            Tool
                │                            │
                │                            ▼
                │                     Parent Dashboard
                │
                ▼
          Learning Context

                     Gemini
                       │
                       ▼
              Pronunciation Need?
                       │
                     Yes
                       ▼
            Pronunciation Specialist

The browser experience, telephony experience, memory, tools, escalation, analytics, and specialist handoff all build around the same core voice-agent pipeline.

What I would tell someone building their first voice agent

If you're starting from scratch, you don't need to build everything I built.

Start with the smallest possible loop:

Speech → STT → LLM → TTS → Speech

Then add capabilities one at a time.

The starter repository I used provides a production-oriented foundation around LiveKit Agents and Murf Falcon.

Clone the starter

git clone https://github.com/murf-ai/murf-livekit-starter.git
cd murf-livekit-starter

The project is split into a backend Python voice agent and a Next.js frontend.

Install the prerequisites

You'll need:

1. Python 3.10+
2. uv
3. Node.js 18+
4. pnpm
5. A LiveKit project

For Windows, uv can be installed with:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

And pnpm:

npm install -g pnpm

Configure your API keys

Create .env.local files in the backend and frontend using the provided .env.example files.

The backend needs credentials such as:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

These should be stored as environment variables.

Never hard-code API keys into your source code or commit them to GitHub.

Install the backend

cd backend
uv sync
uv run python src/agent.py download-files

Install the frontend

cd frontend
pnpm install
6. Run the application

You can run everything together using the provided startup script, or run the services separately.

For the separate-terminal approach:

livekit-server --dev

Then:

cd backend
uv run python src/agent.py dev

And in another terminal:

cd frontend
pnpm dev

Open:

`http://localhost:3000`

Click Start talking, allow microphone access, and speak.

You now have the basic voice-agent loop running.

From there, the interesting part begins.

Where the behavior actually lives

One of the simplest ways to customize the starter is to change the system prompt.

In the starter project, the system prompt lives in:

backend/src/agent.py

The SYSTEM_PROMPT defines the agent's role and behavior.

Change that prompt and you can turn the same voice infrastructure into something completely different—a customer support agent, language tutor, receptionist, or another specialized assistant.

That separation is powerful:

The infrastructure provides the voice-agent capabilities. The prompt defines what the agent is supposed to be.

Then tools, memory, telephony, and specialist agents extend what it can actually do.

What was actually difficult?

The most difficult part of this challenge wasn't writing one massive piece of code.

It was understanding how all the pieces fit together.

Every new capability introduced another system boundary.

Memory meant thinking about authentication, child identity, database access, consent, and isolation.

Tools meant thinking about when the LLM should call an external service and what happens when that service fails.

Telephony meant understanding the bridge between LiveKit, SIP, Twilio, and the phone network.

Human escalation meant thinking about consent and what information should actually be shared.

Analytics meant deciding what “success” means before measuring it.

Multi-agent handoffs meant deciding which agent should own which responsibility.

The biggest lesson was therefore not a particular API or framework.

It was this:

A reliable voice agent is a system of decisions, not just a conversational model.

What I would build next

BoloBuddy is still a starting point.

If I continue developing it, I would want to explore richer learning analytics, more sophisticated personalization, stronger pronunciation evaluation, and deeper parent-facing insights.

But I wouldn't want to add features simply because they are technically possible.

The original question should remain the filter:

Does this make the child's learning experience better?

That is the standard I want to use for the next version.

From a voice demo to an agentic system

Ten days ago, I started with something simple:

An AI that could hear a child and talk back.

Today, Chinnu can:

Speak using an Indian voice powered by Murf Falcon
Handle Telugu, Hindi, English, and code-mixed conversations
Follow a defined personality and safety boundaries
Remember child-specific learning information
Retrieve and update memory through tools
Use an external dictionary API
Handle tool failures gracefully
Make outbound phone calls through LiveKit SIP and Twilio
Ask for human help with consent
Notify parents about escalations
Measure learning outcomes through call analytics
Hand pronunciation conversations to a specialist agent

The progression wasn't:

Voice → more voice features.

It was:

Voice → personality → memory → tools → telephony → human escalation → measurement → specialization.

And that changed how I think about voice AI.

The most interesting voice agents aren't necessarily the ones that can talk the most.

They're the ones that understand:

what to do, when to do it, what not to do, when to ask for help, and what outcome they're actually responsible for.

That's what I wanted BoloBuddy to become.

And that's what these ten days taught me about building agents.

Explore the project

Source code:

N-SAI-VENKATA-TEJA / murf-livekit-starter

Voice Agent Starter — Powered by Murf Falcon

Build a production voice AI agent in 5 minutes. Powered by the fastest TTS on the market - swap the system prompt to build anything from customer support to language tutors.

Why Murf Falcon

55ms model latency - fastest production TTS
130ms time-to-first-audio across 10+ global regions
$0.01/1000 characters - up to 10x cheaper than alternatives
150+ voices across 35+ languages
99.38% pronunciation accuracy

Architecture

flowchart LR
    A[🎙️ User speaks] -->|audio| B[Deepgram STT]
    B -->|text| C[LLM]
    C -->|response text| D[Murf Falcon TTS]
    D -->|audio| E[LiveKit]
    E -->|stream| F[🔊 User hears]

    style A fill:#444441,stroke:#888780,color:#fff
    style B fill:#185FA5,stroke:#85B7EB,color:#fff
    style C fill:#534AB7,stroke:#AFA9EC,color:#fff
    style D fill:#0F6E56,stroke:#5DCAA5,color:#fff
    style E fill:#D85A30,stroke:#F0997B,color:#fff
    style F fill:#444441,stroke:#888780,color:#fff

Quickstart

Prerequisites

Python 3.10+

uv - fast Python package manager

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Node.js 18+
pnpm…

View on GitHub

Built with:
Murf Falcon · LiveKit Agents · Deepgram Nova-3 · Google Gemini · MongoDB · Next.js · Tailwind CSS · Twilio

Challenge:
10 Days of Voice Agents — VoiceForBharat Edition by Murf AI

DEV Community: Nakkina Sai Venkata Teja

From a Voice That Talks to a Voice Agent That Knows What to Do

The problem I wanted to solve

What should Chinnu remember?

What the final system looks like

What I would tell someone building their first voice agent

What was actually difficult?

Explore the project

N-SAI-VENKATA-TEJA / murf-livekit-starter

Voice Agent Starter — Powered by Murf Falcon

Why Murf Falcon

Architecture

Quickstart

Prerequisites