Naman

Posted on Aug 15

Building Shiksha: My 10-Day Voice Agent Journey with Murf Falcon

#agents #ai #buildinpublic #devchallenge

For the last 10 days, I have been building a voice agent called Shiksha as part of the 10 Days of Voice Agents — VoiceForBharat Edition challenge by Murf AI.

My original idea was simple:

Build a voice agent that can help students learn through natural conversation.

Over the challenge, that idea grew into a complete voice-based learning system with memory, tools, human escalation, call analytics, and a specialist agent.

What is Shiksha?

Shiksha is a voice-based learning partner for students.

Instead of typing questions and reading answers, a student can simply talk to Shiksha.

A student can:

Ask learning questions
Take quizzes
Continue learning with their saved profile
Get help when they are stuck
Practice mathematics
Get transferred to a Maths Specialist when needed

The main goal was to make the experience feel more like a conversation than a traditional chatbot.

Tech Stack

Component	Technology
Real-time voice	LiveKit
Speech-to-Text	Deepgram
LLM	Gemini
Text-to-Speech	Murf Falcon
Backend	Python
Memory	SQLite
Call analytics	Flask + SQLite
External data	Open Trivia Database

The voice experience is powered by Murf Falcon, which was one of the main parts of the challenge.

How Shiksha Works

At a high level, the system looks like this:

                    STUDENT
                       │
                       ▼
                  LiveKit
              Real-time Audio
                       │
                       ▼
                  Deepgram
                Speech-to-Text
                       │
                       ▼
                   Gemini
                Agent Reasoning
                       │
          ┌────────────┼─────────────┐
          │            │             │
          ▼            ▼             ▼
       Memory         Tools       Handoff
       SQLite       Quiz API    Maths Specialist
          │            │             │
          └────────────┴─────────────┘
                       │
                       ▼
                  Murf Falcon
                 Text-to-Speech
                       │
                       ▼
                    STUDENT

This was the basic architecture that I built and expanded throughout the challenge.

What I Built

1. Student Memory

One of the first things I added was a simple memory system using SQLite.

Shiksha can store:

Student name
Current learning level
Topics covered
Last interaction

This means the agent can use information from previous conversations instead of starting from zero every time.

2. Real Tool Calling

For quizzes, I didn't want the agent to always generate questions from memory.

Instead, I added a tool that can fetch a question from an online source.

Example

Student:
"Give me a quiz question."

        ↓

Shiksha
        ↓

fetch_educational_quiz()
        ↓

Open Trivia Database
        ↓

Question returned
        ↓

Shiksha speaks the question

I also added a local fallback question.

So if the live source is unavailable, the agent can still continue the conversation instead of staying silent.

3. Guardrails

The agent has clear instructions about how it should behave.

Some of the important rules are:

Speak only in English for this version of the project
Keep responses short and speech-friendly
Use the quiz tool instead of inventing quiz questions
Don't expose technical errors to the student
Use fallback data when a live source fails
Don't mark an exercise as complete unless the student actually answers

These rules made the agent more predictable.

4. Human Escalation

Another feature I added was a human escalation flow.

Sometimes the agent should know when not to solve everything itself.

For example, if a student is clearly stuck or asks for a teacher, Shiksha can ask for permission and create a human-help request.

Flow

Student needs help
        ↓
Shiksha asks for permission
        ↓
Student agrees
        ↓
Escalation created
        ↓
Human can follow up

I also kept the escalation information limited so that unnecessary private conversation data is not sent.

5. Call Analytics Dashboard

For Day 8, I wanted to know whether the conversations were actually achieving the goal.

I built a small dashboard using Flask + SQLite.

Dashboard metrics

Metric	Meaning
Total Calls	Total calls made by the agent
Successful Calls	Calls where the learning exercise was completed
Failed Calls	Calls where the success condition was not reached
Success Rate	Successful calls as a percentage of total calls
Recent Calls	Recent call outcomes and details

For this project, I defined:

Successful call = the student completes a learning exercise by answering a quiz question.

The dashboard reads data directly from the agent's SQLite database, so the numbers are not hardcoded.

6. Maths Practice Specialist

Day 9 was one of my favorite parts of the challenge.

I didn't want one agent to pretend to be an expert at everything.

So I created a dedicated Maths Practice Specialist.

Main Agent vs Specialist

Shiksha	Maths Specialist
General learning partner	Maths-focused assistant
Student memory	Maths practice
General questions	Maths questions
Quiz tools	Evaluate answers
Human escalation	Focused maths guidance

The idea is simple:

Student
   │
   ▼
Shiksha
   │
   ├── General question
   │      ↓
   │   Shiksha answers
   │
   └── Maths practice
          ↓
   Maths Specialist

The student should not have to explain their request again after the handoff.

One of the Hardest Parts

The biggest technical challenge for me was the specialist handoff.

The main agent was correctly understanding that the student wanted maths help, but the transition to the specialist did not always work smoothly.

At different points, I had to look at:

Agent session state
Conversation context
Tool execution
Specialist initialization
Voice configuration
Handoff behavior

This taught me something important:

A voice agent is not only about the model response. The real-time orchestration around the model is equally important.

I also had to deal with API failures and incomplete calls.

Those cases forced me to think about what happens when things go wrong, not just when everything works.

What I Learned

The biggest lesson from these 10 days is that a useful voice agent needs more than an LLM and a voice.

You need to think about the complete flow:

Speech
   ↓
Understanding
   ↓
Reasoning
   ↓
Tools / Memory
   ↓
Decision
   ↓
Response
   ↓
Voice
   ↓
Outcome

And because the interaction is voice-first, small problems become much more noticeable.

For example:

Long answers sound worse when spoken
Delayed responses feel awkward
A failed API call can make the agent feel broken
A bad handoff can interrupt the whole conversation

These were some of the most useful lessons from the project.

How to Build Your Own Voice Agent

You don't need a huge system to start.

At the basic level, you need four main components:

Component	Purpose
Speech-to-Text	Understand what the user says
LLM	Decide how the agent should respond
Text-to-Speech	Turn text into spoken audio
Real-time transport	Connect the user and the agent

For my project, I used:

Deepgram for Speech-to-Text
Gemini for the LLM
Murf Falcon for Text-to-Speech
LiveKit for real-time communication
Python for agent logic
SQLite for memory and analytics

Environment Variables

Keep your API keys in a local .env.local file.

Example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

Never commit real API keys to GitHub.

Add .env.local to .gitignore before publishing your repository.

Testing the Agent

Once the environment is configured, start the backend and connect the browser client to your LiveKit room.

Then test the agent with simple conversations.

Example 1 — Normal question

"What is photosynthesis?"

Shiksha handles it directly.

Example 2 — Quiz

"Give me a quiz question."

The quiz tool is triggered.

Example 3 — Maths specialist

"I want to practice maths."

Shiksha routes the conversation to the Maths Specialist.

Example 4 — Human support

"I want to talk to a teacher."

The human escalation flow can be triggered after the required permission step.

Evidence From the Build

Throughout the challenge I recorded and captured different parts of the system, including:

Voice agent conversations
Tool calling
Student memory
Human escalation
Call analytics dashboard
Specialist handoff
Architecture and workflow

These demos helped me verify that the features were actually working rather than just existing in the code.

What I Would Build Next

The project is working, but there is still a lot more I want to improve.

Next ideas

More specialist agents
Better learning recommendations
More realistic learning exercises
Better student progress tracking
More detailed analytics
More robust multi-agent routing
Better handoff recovery
More real-world testing with different conversations

Final Thoughts

I started this challenge with a simple goal:

Build a voice agent that can help students learn.

After 10 days, Shiksha became much more than a basic voice chatbot.

It can now:

Talk with students
Remember student information
Use external tools
Handle API failures
Ask humans for help
Track call outcomes
Analyze real calls
Hand maths conversations to a specialist

The biggest thing I learned is that building a voice agent is really about building the whole system around the conversation.

The model is only one part of it.

The memory, tools, guardrails, real-time communication, analytics, error handling, and handoffs are what make the experience actually useful.

DEV Community