Zikra

Posted on Aug 15

From a Voice Agent to a Voice Assistant for Bharat: My 10-Day VoiceForBharat Journey

#voiceforbharat #murfai #ai #programming

Over the last 10 days, I built a voice agent as part of 10 Days of Voice Agents — VoiceForBharat Edition.

What started as a simple voice conversation gradually became a much more complete system — one that can remember users, use tools, make outbound calls, escalate to a human, track call outcomes, and hand conversations over to a specialist agent.

The project was built using LiveKit and Murf Falcon, with the goal of making government-scheme and financial assistance information easier to access through natural voice conversations.

1. The Problem I Wanted to Solve

Government schemes can be extremely useful, but discovering the right scheme, understanding eligibility, and figuring out what to do next can be confusing.

For many users, especially those who are more comfortable speaking than typing, a voice interface can make this interaction much simpler.

Instead of navigating through multiple pages and forms, a user can simply talk to an assistant and ask questions such as:

What government schemes are available for me?
Am I eligible for a particular scheme?
What information do I need?
Can someone help me with this?

This became the core idea behind my VoiceForBharat project: use voice as a more natural interface for accessing useful government and financial information.

2. What I Built

The final agent is more than a basic question-and-answer bot.

It can:

Have natural voice conversations
Use an Indian voice powered by Murf Falcon
Handle Indian languages and code-mixed conversations
Follow a defined personality and safety guardrails
Remember information about returning users
Use tools to retrieve or calculate useful information
Make outbound phone calls
Escalate conversations when a human is needed
Track call outcomes
Hand conversations from the main agent to a specialist agent

The idea was to make the agent feel less like a demo chatbot and more like an actual voice assistant.

3. How the System Works

At a high level, the voice pipeline looks like this:

User → Speech-to-Text → LiveKit Agent → LLM → Tools / Memory → Murf Falcon TTS → User

The user speaks into the interface. Their speech is converted into text, processed by the agent and LLM, and the response is converted back into speech using Murf Falcon.

LiveKit handles the real-time voice communication, while the agent logic controls the conversation, tools, memory, escalation, and handoffs.

The system also has additional paths for things like human escalation, outbound calls, and specialist-agent handoffs.

4. Building the Agent Step by Step

Voice and Personality

The first step was getting the basic conversation working.

I defined what the agent should do, how it should communicate, and what it should avoid doing. This was important because a useful voice agent needs more than an LLM — it needs clear instructions and boundaries.

I also wanted the experience to feel natural for Indian users, including conversations where users switch between English and Hindi.

Murf Falcon provided the voice layer, making the responses feel much more conversational than a purely text-based interaction.

Memory

One of the next important steps was giving the agent memory.

Instead of treating every conversation as completely new, the system can store relevant user information and use it when the user returns.

This introduced an important lesson: memory is not just about storing information; it is also about correctly identifying the user and retrieving the right information.

While testing this, I ran into issues where user identification and stored information did not behave exactly as expected. Debugging those issues helped me understand how important consistent user IDs and clean memory logic are in a conversational system.

Tools

The agent also uses tools instead of relying entirely on the LLM's internal knowledge.

This is important for practical applications because tools allow an agent to work with structured information and perform specific actions.

The general pattern became:

User asks → Agent understands intent → Appropriate tool is used → Result is explained naturally through voice.

Outbound Calls

Another major part of the project was adding outbound calling.

This allowed the agent to move beyond a browser-based demo and interact through an actual phone-call workflow.

Getting this working required connecting the voice agent with the telephony flow and testing the complete call lifecycle.

It also introduced new considerations around caller information, user identification, and making sure private information never gets exposed publicly.

5. Human Escalation

A voice assistant should not try to solve everything itself.

One of the features I added was the ability to escalate a conversation when the user needs human assistance.

For example, if a user reaches a point where an automated response is not enough, the agent can move toward a human-support path instead of continuing to guess.

This was one of the most interesting parts of the project because it changes the role of the AI.

The goal isn't:

AI should answer everything.

The goal is:

AI should know when it can help and when it should involve a human.

6. Specialist Agent Handoff

I also implemented a specialist-agent workflow.

Instead of having one giant agent responsible for every possible task, the main agent can hand a conversation to a more specialized agent when appropriate.

This creates a more scalable architecture.

For example:

Main Agent → Understands User → Determines Need → Specialist Agent → Handles Specific Task

This is much closer to how a real conversational system could be structured as it grows.

7. Call Analytics

Another feature I added was call analytics.

The system records useful information about calls so that conversations can be analyzed after they happen.

This makes it possible to look beyond:

“Did the agent answer?”

and start asking:

Did the call succeed?
What happened during the conversation?
Was a human needed?
Which path did the call take?
What was the final outcome?

This was especially useful because it showed me that building a voice agent isn't only about real-time interaction. Observability matters just as much.

8. The Challenges

The project definitely did not work perfectly on the first attempt.

One of the biggest challenges was debugging the interaction between different parts of the system.

For example, while working on memory, I encountered situations where the agent appeared to work correctly but retrieved the wrong stored information because the user identity being used by different parts of the system wasn't consistent.

I also faced issues while setting up and testing outbound calling and the handoff flow.

These problems taught me an important lesson:

A voice agent is a system, not just a prompt.

When something goes wrong, the problem may be in the prompt, memory layer, tool, user identification, transport, telephony, or agent routing.

Breaking the system into smaller components and testing each path separately made debugging much easier.

I also learned the importance of testing more than the happy path.

A good test isn't only:

“Ask the agent a question and see if it responds.”

It should also include:

“What happens when the user needs a human?”

“What happens when the user returns?”

“What happens when the agent needs a specialist?”

“What happens when a call ends?”

9. How You Can Build Your Own Voice Agent

If you want to build a similar system, the basic architecture doesn't need to be complicated at first.

You need four major components:

1. Speech-to-Text

Converts the user's voice into text so the agent can understand the conversation.

2. LLM

Processes the user's request and decides what response or action is appropriate.

3. Text-to-Speech

Converts the agent's response back into natural speech.

In my project, I used Murf Falcon for this layer.

4. Real-Time Transport

Handles the live audio connection between the user and the agent.

I used LiveKit for the real-time voice infrastructure.

Once the basic pipeline works, you can gradually add:

Memory → Tools → Telephony → Human Escalation → Analytics → Specialist Agents

That is a much better approach than trying to build everything at once.

--> Escalations Dashboard-

10. Running the Project

The project is available here:

GitHub: https://github.com/ZikraRahman/murf-livekit-starter

The general setup process is:

Clone the repository.
Install the project dependencies.
Create your local environment configuration.
Add your API credentials to the local environment file.
Start the backend/agent.
Start the frontend if required.
Connect to the voice agent.
Test a normal conversation.
Test the escalation and specialist-agent paths.

Never commit your .env file or API keys to GitHub.

Phone numbers, caller information, user identifiers, and other private data should also never be included in a public repository.

11. What I Would Improve Next

There is still a lot I would like to improve.

Some of the next things I would explore are:

Better multilingual support
More government-scheme integrations
More robust memory and user identification
Better call analytics
Improved error handling
More specialist agents
Better monitoring and observability
More extensive real-world testing
Improved accessibility for users with limited digital literacy

The most exciting part is that the architecture can keep growing without requiring the entire system to be rebuilt from scratch.

12. What I Learned

The biggest thing I learned during these 10 days is that building a voice agent is very different from simply connecting an LLM to a microphone.

A useful voice agent needs:

Voice + Intelligence + Tools + Memory + Guardrails + Real-Time Communication + Observability

And, most importantly, it needs to be designed around the user.

I also learned that debugging is a major part of building these systems. Things like user IDs, tool responses, call states, handoffs, and telephony flows can all create unexpected problems.

But every failure made the architecture a little clearer.

What started as a simple conversational agent ended up becoming a much more complete voice system with memory, tools, outbound calling, human escalation, analytics, and specialist handoffs.

13. Final Links