My 9-Day Journey Building a Real-Time AI Voice Agent | VoiceForBharat

Helly Dungrani — Sat, 15 Aug 2026 14:18:53 +0000

10 Days of AI Voice Agents — VoiceForBharat Edition

Over the past 9 days, I built and improved a real-time AI voice agent as part of the 10 Days of AI Voice Agents — VoiceForBharat Edition challenge.

The journey showed me that building a voice agent is much more than connecting Speech-to-Text and Text-to-Speech.

A useful voice agent needs memory, tools, guardrails, human handoff, analytics, and reliable real-time communication.

For Text-to-Speech, I used Murf Falcon, Murf's fast TTS API.

🚀 What I Built

My voice agent can:

🎙️ Have real-time voice conversations
🧠 Remember useful user information
🛡️ Follow conversational guardrails
🔧 Use external tools
📞 Make outbound calls
👨‍💼 Request human assistance
📊 Track call analytics and outcomes
🔄 Support human/specialist handoff workflows

🏗️ How It Works

                     USER
                       │
                       ▼
                🎤 Voice Input
                       │
                       ▼
                 Deepgram STT
                       │
                       ▼
                🤖 AI AGENT
                /     |      \
               /      |       \
          Memory    Tools   Guardrails
               \      |       /
                \     |      /
                       ▼
                     LLM
                       │
                       ▼
                Murf Falcon
                       │
                       ▼
                🔊 Voice Output
                       │
                       ▼
                      USER

             ⚡ LiveKit
      Real-time communication.

🗓️ My 9-Day Journey
Day 1 — Voice Agent
I started by building a basic voice agent that could listen to the user and respond through voice.
Day 2 — Personality & Guardrails
I gave the agent a defined role and behavior while adding guardrails to keep conversations within the intended scope.
Day 3 — Frontend
I built the frontend interface and worked with different conversation states such as connecting, listening, speaking, and call ended.
Day 4 — Memory
I added memory so the agent could retain useful information about users and provide more context-aware responses.
Day 5 — Tools
I added tool calling so the agent could use external information instead of relying only on the language model.
Day 6 — Outbound Calls
I worked on outbound calling and telephony integration, allowing the agent to initiate calls.
Day 7 — Human Handoff
I implemented a human-handoff workflow so the agent could recognize situations where human assistance was required.
Day 8 — Call Analytics
I built an analytics dashboard to monitor total calls, call duration, completed calls, human handoffs, and call outcomes.
Day 9 — Specialist Handoff
I worked with specialist/handoff workflows so conversations could be passed to the appropriate specialist when needed.

🛠️ Technologies Used
Murf Falcon — Text-to-Speech
Deepgram — Speech-to-Text
LiveKit — Real-time communication
LLM — Reasoning and responses
Python — Backend agent
Next.js — Frontend and dashboard
SQLite — Call analytics

🧠 From Voice Bot to Voice Agent
One of my biggest learnings was understanding the difference between a simple voice bot and an AI agent.
A basic voice bot can:
User → Speech → AI → Speech → User
But an agent can:
User
↓
Understand
↓
Reason
↓
Remember
↓
Use Tools
↓
Take Action
↓
Ask for Human Help
This makes the system much closer to a real AI assistant.

🛠️ Challenges I Faced
The project wasn't completely smooth.
I faced issues with:
LiveKit real-time connectivity
API and environment configuration
TTS/voice connections
SIP and outbound calling
Analytics API requests
Debugging these issues helped me understand how different parts of a real-time AI system work together.
I also learned the importance of keeping API credentials inside environment variables instead of exposing them in source code.

GitHub: https://github.com/Helly033/murf-ai
Linkedin: https://www.linkedin.com/in/helly-dungrani-718871393?utm_source=share_via&utm_content=profile&utm_medium=member_android

💡 What I Learned
This challenge taught me that building an AI voice agent involves much more than AI models.
It combines:
AI + APIs + Voice + Real-Time Communication + Databases + Frontend + Debugging + Security
Most importantly, I learned that a good AI agent should not only generate answers.
It should be able to understand, decide, act, remember, and know when to ask for help.

🙌 Final Thoughts
The 10 Days of AI Voice Agents — VoiceForBharat Edition was a great hands-on experience.
Building this project with Murf Falcon gave me practical experience with real-time AI voice technology, while LiveKit helped me understand the infrastructure behind interactive voice applications.
I'm grateful to the Murf AI team for creating this challenge and giving me the opportunity to learn by building.
A good AI agent is not just something that talks. It is a system that can understand, reason, use tools, remember context, take action, and know when to ask a human for help.

VoiceForBharat #MurfAI #MurfFalcon #VoiceAI #AgenticAI

DEV Community: Helly Dungrani

My 9-Day Journey Building a Real-Time AI Voice Agent | VoiceForBharat

10 Days of AI Voice Agents — VoiceForBharat Edition

🚀 What I Built

🏗️ How It Works

VoiceForBharat #MurfAI #MurfFalcon #VoiceAI #AgenticAI