agnidiptasarkar123-dev

Posted on Aug 15

Building Arthashathi: My 10-Day Journey Building a Voice Agent for Bharat

#ai #startup #agents #llm

From a Simple Voice Assistant to a Multi-Agent Financial Guide

Over the last 10 days, I took part in 10 Days of Voice Agents — VoiceForBharat Edition, a challenge that pushed me to go beyond simply making an AI chatbot speak.

My project is Arthashathi, a voice-first financial guide designed for Indian users.

The idea was simple:

What if accessing financial information could be as easy as having a conversation?

Instead of navigating complicated websites, searching through government portals, or trying to understand financial terminology, a user can simply speak to Arthashathi and ask for help.

Over these 10 days, Arthashathi evolved from a basic voice agent into a system with memory, tools, safety guardrails, government-scheme assistance, human escalation, outbound calls, analytics, and specialist-agent handoffs.

This post is a summary of what I built, what went wrong, what I learned, and how someone else could start building a similar voice agent.

1. The Problem I Wanted to Solve

India has a huge amount of financial and government-scheme information available online.

The problem is not always the lack of information.

The problem is accessibility.

A user may have questions such as:

“Which government schemes am I eligible for?”
“What benefits does this scheme provide?”
“Is this UPI message a scam?”
“How can I stay safe from banking fraud?”
“Can someone explain this financial information to me simply?”

For someone who is not comfortable navigating websites or typing long queries, a voice conversation can be much more natural.

That is why I chose to build a voice-first financial assistant rather than another text-only chatbot.

2. Meet Arthashathi

Arthashathi is an AI-powered voice financial guide designed primarily for Indian users.

The main agent can help users with:

Government schemes
Banking safety
Fraud awareness
UPI safety
General financial guidance
Scheme eligibility
Human escalation when necessary

The system is also designed to handle Indian languages and code-mixed conversations, allowing users to communicate more naturally instead of being forced to use formal English.

The voice experience is powered by Murf Falcon, with real-time communication handled using LiveKit Agents.

3. How Arthashathi Works

At a high level, the system follows a real-time voice pipeline:

             ┌──────────────────┐
             │       USER       │
             │  Speaks naturally│
             └────────┬─────────┘
                      │
                      ▼
             ┌──────────────────┐
             │   LiveKit        │
             │ Real-time Audio  │
             └────────┬─────────┘
                      │
                      ▼
             ┌──────────────────┐
             │ Speech-to-Text   │
             │      (STT)       │
             └────────┬─────────┘
                      │
                      ▼
             ┌──────────────────┐
             │      LLM         │
             │ Reasoning +      │
             │ Agent Decisions  │
             └────────┬─────────┘
                      │
          ┌───────────┼────────────┐
          ▼           ▼            ▼
       Memory       Tools      Escalation
          │           │            │
          └───────────┼────────────┘
                      │
                      ▼
             ┌──────────────────┐
             │ Specialist Agent │
             │ when required    │
             └────────┬─────────┘
                      │
                      ▼
             ┌──────────────────┐
             │ Text-to-Speech   │
             │   Murf Falcon    │
             └────────┬─────────┘
                      │
                      ▼
                   USER

The important idea is that the LLM is not working alone.

It is part of a larger system involving real-time audio transport, speech recognition, tools, memory, safety logic, and voice synthesis.

4. The Main Features I Built

🎙️ 1. Real-Time Voice Conversation

The core of Arthashathi is a real-time voice interaction.

The user speaks naturally, the system understands the request, generates a response, and speaks back.

The voice layer uses Murf Falcon, allowing the project to focus on making the interaction feel conversational rather than robotic.

🧠 2. Memory for Returning Users

One of the goals was to make Arthashathi more than a stateless chatbot.

The system can maintain relevant user context so that returning users do not necessarily have to repeat everything from the beginning.

This also introduced an important engineering challenge: correctly identifying the user across calls and ensuring that the correct memory was associated with the correct conversation.

🌐 3. Indian Languages and Code-Mixing

Indian conversations are rarely limited to perfectly formal English.

A user might naturally say something like:

“Mujhe batao ei scheme-ta-r eligibility ki?”

Arthashathi was designed with this reality in mind.

The agent can handle Indian-language and code-mixed interactions while keeping important financial terminology understandable.

🛡️ 4. Safety Guardrails

Financial assistance is a domain where an AI agent must know its limits.

Arthashathi includes safety-oriented instructions around sensitive information and banking security.

The system is designed not to request or expose sensitive credentials such as:

OTPs
PINs
CVVs
Passwords
Banking credentials

The goal is not just to make the agent helpful, but to make it responsible.

🏛️ 5. Government Scheme Assistance

Government schemes became one of the core use cases of Arthashathi.

I created a dedicated scheme dataset containing information such as:

Scheme name
Benefits
Eligibility criteria
Relevant information dates

The agent can use this information to help users understand schemes and determine whether a particular scheme may be relevant to them.

📞 6. Outbound Calls

The project also explored the idea of proactive voice assistance.

Instead of waiting for the user to call the agent, Arthashathi can be used for outbound reminders, such as informing eligible users about important scheme deadlines.

This introduced additional considerations around consent, opt-out handling, and responsible calling.

🚨 7. Human Escalation

An AI should not pretend it can solve every problem.

For situations where human assistance is more appropriate, Arthashathi can escalate the conversation instead of attempting to answer beyond its capabilities.

This was an important design principle throughout the project:

Knowing when not to answer is also part of being a good AI assistant.

📊 8. Call Analytics

I also added call-level analytics to understand what is happening during conversations.

This allows the system to track useful information about calls and outcomes and provides a foundation for improving the agent based on real usage.

5. The Specialist Agent

One of the most interesting parts of Day 9 was implementing agent handoff.

The main agent should not try to be an expert at everything.

So I created a dedicated:

Government Scheme Specialist

The architecture becomes:

                USER
                  │
                  ▼
          ┌───────────────┐
          │  Arthashathi  │
          │  Main Agent   │
          └───────┬───────┘
                  │
        General request?
             /         \
           YES          NO
            │            │
            ▼            ▼
         Answer     Specialist Handoff
                         │
                         ▼
                ┌─────────────────┐
                │ Government      │
                │ Scheme          │
                │ Specialist      │
                └────────┬────────┘
                         │
                         ▼
                       USER

The main agent first tells the user that it is connecting them to the specialist.

The specialist then continues the conversation with the relevant context rather than forcing the user to repeat the entire problem.

This was one of the most valuable lessons from the challenge:

A good multi-agent system is not about having more agents. It is about giving each agent a clear responsibility.

6. The Problems I Faced

This project definitely did not work perfectly on the first attempt.

In fact, some of the most useful things I learned came from the failures.

Problem 1: “Agent did not join the room”

At one point, the frontend could connect to the application but the agent would not reliably join the LiveKit room.

After debugging the system, one major issue turned out to be an agent-name mismatch.

The frontend was requesting one agent name while the backend worker was registering under another name.

Because the LiveKit dispatch system relies on the correct agent identity, the worker was not receiving the expected job.

Fixing the naming mismatch and cleaning up duplicate workers resolved the dispatch problem.

Lesson:

When debugging real-time agent systems, don’t immediately assume the LLM or TTS is broken.

Check the complete chain:

Frontend
   ↓
Token
   ↓
LiveKit Room
   ↓
Agent Dispatch
   ↓
Worker
   ↓
Agent Session

7. The Latency Problem

Another major challenge was latency.

Initially, I tried making the system extremely aggressive about responding quickly.

I reduced endpointing and silence thresholds so that the agent would react faster.

That improved raw response speed — but introduced a new problem.

The agent started interpreting normal pauses and breaths as the end of the user’s sentence.

The result?

The agent could interrupt too early, respond to incomplete sentences, or produce fragmented speech.

I learned an important lesson:

The lowest possible latency does not necessarily produce the best voice experience.

A good voice agent needs a balance between:

STT endpointing
VAD silence detection
Interruption handling
LLM generation
TTS buffering
Natural conversational pauses

Eventually, I moved toward a more balanced configuration instead of blindly minimizing every delay.

8. The Specialist Voice / Handoff Challenge

Another interesting issue appeared during the specialist handoff.

The specialist had its own Murf voice configuration, but changing the active agent did not automatically guarantee that the running audio pipeline would immediately use the specialist’s TTS configuration.

This forced me to inspect how the LiveKit AgentSession manages its active TTS provider and how the handoff interacts with the running session.

That experience taught me something important:

Changing an agent’s logic and changing the active audio pipeline are not always the same operation.

The handoff architecture therefore required careful testing of both the conversational state and the voice pipeline.

9. What I Learned

The biggest lesson from these 10 days is that building voice AI is much more than connecting:

Speech → LLM → Voice

A production-oriented voice agent involves many moving pieces:

Speech Recognition
       +
LLM Reasoning
       +
     Tools
       +
     Memory
       +
     Safety
       +
Real-Time Transport
       +
      TTS
       +
  Agent Routing
       +
    Analytics
       +
Failure Handling

A problem in any one of these layers can affect the entire user experience.

I also learned that debugging a voice agent requires thinking about the whole system, not just individual files.

10. How You Can Build Your Own Voice Agent

If you’re starting your own voice-agent project, I strongly recommend starting small.

You don’t need memory, specialist agents, outbound calls, analytics, and ten tools on day one.

Start with:

User speaks
     ↓
    STT
     ↓
    LLM
     ↓
    TTS
     ↓
User hears response

Once that works reliably, add capabilities one at a time.

Core Components

Speech-to-Text

Converts the user’s audio into text that the LLM can understand.

LLM

Handles reasoning, conversation, instructions, routing, and tool decisions.

Text-to-Speech

Converts the generated response back into natural speech.

Real-Time Transport

A framework such as LiveKit handles the real-time communication between the user and the agent.

11. Running the Project

For a local development environment, the backend can be started with:

cd backend
source .venv/bin/activate
python src/agent.py dev

Then start the frontend separately:

cd frontend
npm run dev

Open the local frontend in your browser and start a voice session.

I strongly recommend testing the basic conversation first before testing specialist handoffs or outbound calling.

12. Protect Your API Keys

Never put API keys directly into your source code.

Use environment variables instead:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key

And make sure your environment files are excluded from Git.

Never publish:

API keys
API secrets
Private phone numbers
Caller information
Authentication credentials
Private database contents

A public GitHub repository should contain the code, not your secrets.

13. My Build Journey — Day by Day

I also recorded my progress throughout the challenge.

These demos show how Arthashathi evolved from the initial voice agent into the final multi-feature system.

Together, these demos show the progression of the project much better than a list of features alone.

14. What I Would Build Next

Arthashathi is still a work in progress.

Some of the things I would like to explore next are:

More specialist agents
Smarter automatic agent routing
Better multilingual voice handling
More robust failure recovery
Better latency monitoring
More detailed analytics
Production deployment
Larger-scale real-world testing
Better personalization while preserving privacy

I would also like to test the system with real users and learn where the current conversation design breaks down.

15. Source Code

The project is publicly available on GitHub:

Arthashathi — GitHub

https://github.com/agnidiptasarkar123-dev/murf-livekit-starter

The repository contains the implementation and project structure used throughout the challenge.

16. Final Thoughts

When I started this challenge, my goal was simply to build a voice agent.

Ten days later, I ended up building something much more complex:

A voice-first financial assistant with memory, tools, safety guardrails, government-scheme assistance, outbound calling, human escalation, analytics, and specialist-agent handoffs.

But the most valuable part of the challenge wasn’t the number of features.

It was learning how all these components interact.

I learned how important real-time systems are, how small configuration changes can completely change a voice experience, why agent boundaries matter, and why debugging the entire pipeline is more important than debugging one component in isolation.

Most importantly, I learned that building voice AI is an iterative process.

Things break.

Connections fail.

Agents interrupt at the wrong time.

Latency appears where you don’t expect it.

Handoffs don’t always behave the way you imagined.

And that’s exactly where the learning happens.

I’m grateful to have completed 10 Days of Voice Agents — VoiceForBharat Edition, and I’m excited to keep building on Arthashathi.

Thank you for reading! 🚀

If you’re also building a voice agent, I’d love to hear what you’re working on.

Built With

Murf Falcon(The Fastest TTS API) — Text-to-Speech
LiveKit Agents — Real-time voice infrastructure
Python
LLM-based agent orchestration
SQLite
Web frontend
Multi-agent handoffs

Project

🔗 GitHub: https://github.com/agnidiptasarkar123-dev/murf-livekit-starter

🎙️ Challenge: 10 Days of Voice Agents — VoiceForBharat Edition

VoiceForBharat #VoiceAI #ConversationalAI #AI #VoiceAgents #LiveKit #MurfAI #GenerativeAI #Python #ArtificialIntelligence