Building Nexa AI: My 10-Day Journey Building a Voice Agent with LiveKit and Murf Falcon

Introduction

What if students could simply talk to an AI instead of typing every question?

That question became the starting point for Nexa AI, a voice assistant I built during the 10 Days of Voice Agents — VoiceForBharat Edition challenge.

Over 10 days, I went from working with a basic voice agent to building a more complete system that can understand conversations, follow safety guardrails, use tools, remember returning users, make outbound calls, escalate situations to humans, track call outcomes, and hand coding-related conversations to a specialist agent.

I chose the Education, Learning & Literacy track because I wanted to build something that could genuinely be useful for students.

This challenge taught me that a voice agent is much more than an LLM connected to a microphone. A useful voice agent requires speech recognition, reasoning, text-to-speech, real-time communication, tools, safety, state management, and a good user experience.

This article shares what I built, the challenges I faced, and what I learned throughout the journey.

1. The Problem I Wanted to Solve

Students regularly need help with programming, debugging, DSA, study planning, productivity, and technical concepts.

Although chat-based AI is useful, typing every question is not always the most natural way to interact with an assistant.

Voice makes the experience more conversational.

Instead of typing a programming problem, a student can explain it naturally. The agent can listen, understand the request, process it, and respond using a voice.

That was the idea behind Nexa AI: a voice-first learning assistant that students can talk to naturally.

2. Meet Nexa AI

Nexa AI is an AI voice assistant focused on education and learning.

It is designed to help students with:

• Programming questions
• Python, Java and JavaScript concepts
• DSA and debugging
• Study planning
• Productivity
• Technical learning
• General academic guidance

I also wanted *Nexa AI * to feel more natural for Indian users, so the conversational experience supports code-mixed conversations such as Hinglish.

For text-to-speech, I used Murf Falcon, which provided the voice layer for Nexa AI and helped make responses feel faster and more natural.

3. How the System Works

A voice agent is made up of several important components.

The basic flow of Nexa AI looks like this:

                ┌─────────────────┐
                │      User       │
                │  Voice / Text   │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │     LiveKit     │
                │ Real-time Audio │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Speech-to-Text  │
                │      (STT)      │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │   AI Agent /    │
                │      LLM        │
                └───────┬─────────┘
                        │
            ┌───────────┼────────────┐
            │           │            │
            ▼           ▼            ▼
         Tools      Memory      Specialist
                                  Agent
            │           │            │
            └───────────┼────────────┘
                        │
                        ▼
                ┌─────────────────┐
                │ Text-to-Speech  │
                │   Murf Falcon   │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │      User       │
                │ Voice Response  │
                └─────────────────┘

The user speaks to Nexa AI. Speech is converted into text, the AI agent processes the request, and the response is converted back into speech.

LiveKit handles the real-time communication layer, while the AI agent manages the conversation and decision-making.

This architecture also makes it possible to add additional capabilities such as tools, memory, phone calls, human escalation, analytics, and specialist agents.

4. The Important Features I Built

Natural Voice Interaction

The main purpose of Nexa AI is voice-based interaction.

Instead of relying only on text, users can have a spoken conversation with the agent.

The voice layer is powered by Murf Falcon, which became an important part of the overall experience.

Personality, Objectives and Guardrails

A useful AI agent needs clear instructions.

I created a system prompt that defines Nexa AI's:

• Personality
• Objectives
• Conversation behavior
• Language rules
• Safety boundaries

I also added guardrails for situations involving unsafe or inappropriate requests.

For example, the agent should not provide medical diagnoses or prescriptions, request sensitive information such as passwords or OTPs, or assist with harmful activities.

This was an important lesson for me: an agent should not only be intelligent; it should also know its boundaries.

Indian and Code-Mixed Conversations

Nexa AI was designed with Indian users in mind.

The agent can handle conversational language and code-mixed interactions such as Hinglish.

The language behavior was designed so that the assistant follows the language used by the user instead of forcing every conversation into formal English.

This makes the interaction feel more natural for students who regularly switch between English and Hindi while discussing technical topics.

Memory for Returning Users

Another important capability was memory.

A voice assistant becomes more useful when it can maintain relevant context instead of treating every interaction as completely new.

I explored how memory can be used to improve continuity for returning users while keeping the system's safety boundaries in place.

Tools and Agent Capabilities

Nexa AI was not designed to be only a conversational chatbot.

The agent can use tools and structured actions when required.

This allows the system to move beyond simply generating text and actually perform useful operations.

Outbound Phone Calls

One of the more challenging features I implemented was outbound calling.

I connected the voice agent to a phone-call workflow so that ** Nexa AI** could initiate an outbound call and interact with a person through the phone.

This helped me understand that a voice agent can exist beyond a browser interface.

It can become a real-time communication system.

Human Escalation

AI should not always try to solve everything itself.

I implemented a human escalation flow where Nexa AI can hand a situation over when human assistance is needed.

The escalation process can capture useful information such as:

• Reason for escalation
• Problem description
• What the agent already tried
• Urgency
• Caller language
• Preferred follow-up method

This prevents the AI from repeatedly attempting to solve a problem when human intervention is more appropriate.

Call Analytics Dashboard

I also built a call analytics system to understand what was happening with calls.

The backend stores call-related information in a local SQLite database, and an analytics API exposes the results to the dashboard.

The dashboard tracks metrics such as:

• Total calls
• Successful calls
• Failed calls

This was useful because building a voice agent isn't only about making a conversation work. You also need ways to understand how the system performs.

Specialist Agent Handoff

One of my final features was a Coding Specialist Agent.

Instead of forcing the main assistant to handle every programming-related problem itself, Nexa AI can transfer coding-related conversations to a specialist agent.

The coding specialist focuses on areas such as:

• Python
• Java
• JavaScript
• DSA
• Debugging

This introduced me to the idea of multi-agent systems, where different agents can have different responsibilities instead of putting everything into one huge prompt.

5. The Challenges I Faced

The project definitely did not work perfectly on the first attempt.

One of the biggest challenges was dealing with API and model issues.

During development, I encountered errors such as 429 Too Many Requests and quota-related problems with the Gemini API.

At first, this was confusing because the application code itself could look correct while the request was still failing.

I had to investigate the logs, understand that the issue was related to API limits rather than simply a Python bug, and work around the problem by checking configuration, credentials, model availability, and project quotas.

This taught me an important debugging lesson:

Don't assume every error is caused by your code.

Sometimes the problem is the API, quota, configuration, permissions, network, or service itself.

6. Another Challenge: Building a Real Voice System

Getting a basic agent to respond is one thing.

Building a system with multiple components is another.

I had to work with:

• LiveKit
• Python
• Voice AI plugins
• Speech-to-text
• LLM integration
• Murf TTS
• Phone communication
• Frontend/backend communication
• SQLite analytics
• Agent handoffs

Small configuration problems could break the complete voice pipeline.

I learned to rely heavily on logs and incremental testing.

Instead of changing everything at once, I tested individual components and gradually connected them together.

7. How You Can Build Your Own Voice Agent

If you want to build a project similar to Nexa AI, start with the basic architecture.

You need four major pieces:

1. Speech-to-Text

The STT component converts the user's voice into text that the AI can understand.

2. Large Language Model

The LLM processes the user's request and decides what the agent should say or do.

3. Text-to-Speech

The TTS system converts the AI's response back into natural speech.

For Nexa AI, I used Murf Falcon for this part.

4. Real-Time Transport

You also need a system capable of handling real-time audio communication.

I used LiveKit for the real-time voice infrastructure.

Together, the basic flow becomes:

Voice Input
↓
Speech-to-Text
↓
LLM / Agent
↓
Tools / Memory / Handoff
↓
Text-to-Speech
↓
Voice Output

Start with this simple pipeline before adding advanced features.

8. Setting Up the Project

First, clone the project repository and move into the backend directory.

bash
git clone

cd Nexa-AI/backend

Create and activate a virtual environment.

bash
python -m venv .venv

On Windows:

powershell
.venv\Scripts\Activate.ps1

Install the required dependencies according to the project's configuration.

Then configure your environment variables.

Keep API keys inside an environment file such as:

text
.env.local

Do not hard-code API keys inside Python files.

Your environment file should contain the required credentials for services such as your LLM provider, LiveKit, speech services, and Murf.

Also make sure your environment file is included in .gitignore.

Never publish:

• API keys
• Phone numbers
• OTPs
• Passwords
• Caller information
• Private user data

9. Running and Testing the Agent

After configuring the environment, start the backend agent using the project's development command.

For example:

bash
uv run python src/agent.py dev

If the frontend is included, start it separately using the project's frontend development command.

Then open the application and connect to the voice agent.

Start with simple conversations such as:

text
"Hi Nexa, help me create a study plan."

Then test technical questions:

text
"Can you explain binary search?"

You can also test a debugging request:

text
"I'm getting an error in my Python code. Can you help me debug it?"

For advanced testing, verify the specialist handoff, escalation flow, analytics, and phone-call functionality separately.

Testing each feature independently makes debugging much easier.

10. Evidence From My Build

Throughout the challenge, I captured different parts of the project to show how Nexa AI evolved.

Useful evidence includes:

• Nexa AI frontend
• Live voice conversation
• Outbound call
• Human escalation
• Call analytics dashboard
• Coding specialist handoff
• GitHub repository

Evidence From My Build

1 — Nexa AI Voice Interface

2 — Live Voice Conversation

3 — Voice Response

4 — Call Analytics Dashboard

Screenshots are useful because they show the difference between simply describing a feature and actually building it.

1. What I Would Improve Next

Although Nexa AI has many capabilities, there is still a lot I would like to improve.

Some of my next goals would be:

• Better long-term memory
• More reliable multilingual conversations
• Improved latency
• More detailed analytics
• Better error recovery
• More specialist agents
• Better phone-call workflows
• Production-grade deployment
• More robust authentication and privacy controls
• Better user personalization

I would also like to evaluate the agent using measurable metrics such as response latency, call success rate, task completion rate, and user satisfaction.

1. What I Learned From the Challenge

The biggest lesson from these 10 days was that building AI applications is an iterative process.

Things will break.

APIs will return unexpected errors.

Dependencies will conflict.

Configuration will be wrong.

And sometimes the problem will be somewhere completely different from where you initially expect it to be.

The important part is learning how to debug, test, read logs, break a problem into smaller pieces, and keep improving the system.

I also learned that a good voice agent needs more than a powerful model.

It needs:

Good conversation design + safety + tools + reliable infrastructure + observability + a useful purpose.

The specialist handoff and human escalation features especially changed how I think about AI agents. Instead of trying to make one agent do everything, we can design systems where different components have clearly defined responsibilities.

13. Final Thoughts

When I started the 10 Days of Voice Agents — VoiceForBharat Edition, my goal was simply to learn how voice agents work.

By the end, I had built Nexa AI, a voice-based learning assistant with real-time conversations, Murf Falcon TTS, guardrails, memory, tools, outbound calling, human escalation, analytics, and specialist-agent handoffs.

The project is far from perfect, but that is actually one of the most valuable things I learned.

A project doesn't have to start perfect.

Start with a simple voice pipeline.

Make it work.

Understand each component.

Then keep adding capabilities one at a time.

That's how Nexa AI evolved during these 10 days.

14. Links

Source code

GitHub:
https://github.com/riyadhiman852/Nexa-AI-

Challenge

10 Days of Voice Agents — VoiceForBharat Edition

Conclusion

Building Nexa AI over these 10 days gave me hands-on experience with voice AI, real-time communication, LLMs, text-to-speech, tools, phone calls, analytics, human escalation, and multi-agent systems.

More importantly, it taught me how to take an idea and continuously turn it into a more complete working system.

This was not just 10 days of building a voice agent.

It was 10 days of learning how to build, debug, experiment, and improve with AI.

Nexa AI is just the beginning. 🚀