ArthSakhi: Building a Voice Agent That Knows When to Speak, Act, and Ask for Help

Takshak Singh Rajawat — Sat, 15 Aug 2026 18:03:15 +0000

A 10-day journey building a responsible financial-literacy voice agent for Bharat with LiveKit, Gemini, Deepgram, and Murf Falcon.
Project: ArthSakhi (अर्थसखी)
Track: Financial Services
Built for: The Murf AI 10 Days of Voice Agents - VoiceForBharat Edition

The idea
Many financial services are technically available but still difficult to access.
The information may be hidden behind complicated websites, long forms, unfamiliar terminology, or language barriers. For someone who is more comfortable speaking than typing, a voice interface can make the first step much easier.
That was the idea behind ArthSakhi: a voice-based financial-literacy assistant that helps users understand government schemes, ask basic eligibility questions, learn about required documents, and reach appropriate support when the situation is beyond the agent's role.
ArthSakhi is not a bank, government department, financial adviser, or approval authority. It provides guidance, uses verified tools where available, explains limitations, and directs users to official or human support when necessary.
What ArthSakhi does
ArthSakhi can:
Explain basic financial-literacy concepts.
Guide users through supported government-scheme questions.
Ask non-sensitive questions for basic eligibility checks.
Explain document requirements.
Remember consent-related preferences.
Make controlled outbound reminder calls.
Respect opt-out phrases such as "stop" and "do not call again."
Create consent-based human-support requests.
Track successful and unsuccessful call outcomes.
Hand scheme-specific questions to a dedicated specialist agent.

The goal was not to make one agent answer everything. The goal was to make the system useful without making it careless.
Why voice matters
Voice is especially useful when users:
Are more comfortable speaking than typing.
Prefer Hindi, English, or Hinglish conversations.
Find long forms and text-heavy websites difficult.
Need information explained in simpler language.
Want to ask a question without knowing the exact technical terms.

A user can say:
"Mujhe is scheme ke liye kaunse documents chahiye?"
Instead of searching through multiple pages, the agent can understand the intent and guide the conversation step by step.
System architecture
The voice path looks like this:

In simple terms:
The caller speaks through a browser or SIP endpoint.
LiveKit carries the real-time audio.
Deepgram converts speech into text.
Gemini interprets the request and decides what should happen.
Python tools perform structured actions.
SQLite stores safe operational data.
Murf Falcon converts the response into natural speech.
LiveKit sends the response back to the caller.

The core components
ComponentRole in ArthSakhiLiveKitReal-time browser voice and SIP transportDeepgramSpeech-to-textGeminiReasoning, intent detection, and tool selectionMurf FalconNatural text-to-speechLinphoneControlled SIP endpoint for testingPythonAgent logic and tool implementationSQLiteConsent, escalation, and call-outcome storageDashboardDisplays real call metrics
Murf Falcon was particularly important because a voice agent is experienced through its responses, not just its backend logic. Fast and natural speech helps the interaction feel more conversational.
The most important design decisions

Clear boundaries before clever responses ArthSakhi has a defined role. It can explain and guide, but it should not pretend to be an official decision-maker. The agent must not claim that: A scheme application has been approved. A transaction has been blocked. A refund has been processed. A fraud complaint has been resolved. A caller is definitely eligible when only a basic check was completed.

The final decision belongs to the relevant government department, bank, or authorized institution.

Privacy by default The system avoids requesting or displaying: OTPs. PINs. Passwords. CVVs. Full account numbers. Card numbers. Aadhaar or PAN numbers. Raw audio. Full conversation transcripts.

The agent uses only the minimum safe information required for a flow.

Consent-aware memory Memory should not mean storing everything a user says. ArthSakhi remembers only information that is useful for a future interaction and permitted by the user. Consent is required before saving caller preferences or using them for reminders. This was an important shift in my thinking: useful memory is selective, explainable, and controllable.
Tools instead of unsupported generation The agent uses tools for actions that should be structured and traceable: Scheme eligibility checks. Document guidance. Human-support request creation. Opt-out checks. Duplicate-call prevention. Call-outcome recording.

A tool gives the agent a defined action and a defined result instead of asking the model to improvise everything.
Outbound calls with consent
For controlled outbound testing, I connected LiveKit SIP with Linphone.
Outbound calls require more responsibility because the user did not initiate the conversation. ArthSakhi begins by explaining:
Who is calling.
Why the call is being made.
How the caller can stop future reminders.

It recognizes phrases such as:
stop
do not call again
unsubscribe
The system also handles:
Unanswered calls.
SIP connection failures.
Data-source failures.
Duplicate-call prevention.
No automatic retries.

The purpose is not simply to make an outbound call. It is to make the call respectfully.
Human escalation
Some requests should move from AI guidance to human support.
ArthSakhi can create a support request for cases such as:
Suspected fraud.
Unauthorized transactions.
Disputed charges.
Account-specific banking issues.
Situations requiring an official decision.

Before creating the request, it explains what will be shared and asks for explicit consent.
A safe request may include:
Issue: Caller reported a suspicious banking message.
Checked: Caller was advised not to share OTPs or PINs.
Urgency: High.
Language: Hinglish.
Follow-up: Phone.
Reference ID: ASH-2026-XXXXXXXX.
It must not include account numbers, OTPs, PINs, passwords, or government IDs.
Measuring whether a call worked
A connected call is not automatically a successful call.
For ArthSakhi, a successful call means one of two things happened:
The agent completed a scheme eligibility flow and communicated the result.
The agent created a consented human-support request and gave the caller a reference ID.

If the caller disconnects before reaching either outcome, the call is recorded as failed. In this context, "failed" means the defined user goal was not completed; it does not necessarily mean the software crashed.
The dashboard displays:
Total calls.
Successful calls.
Failed calls.

These values are calculated from actual browser and SIP call records stored in SQLite rather than being hardcoded.
Specialist handoff
For scheme-specific questions, ArthSakhi hands the conversation to a separate Government Scheme Eligibility Specialist.
The main agent handles:
"What does financial literacy mean?"
The specialist handles:
"Am I eligible for PMJDY, and what documents do I need?"
Before transferring, ArthSakhi says:
"I'll connect you to our government-scheme eligibility specialist for more focused guidance."
The specialist receives:
The user's latest question.
A short conversation summary.
Preferred language.
Scheme name, if known.
Safe, non-sensitive context.

The caller does not need to repeat the full problem.
A simplified handoff looks like this:
@function_tool
async def transfer_to_scheme_specialist(
self,
context: RunContext,
) -> tuple[GovernmentSchemeEligibilitySpecialist, str]:
specialist = GovernmentSchemeEligibilitySpecialist(
chat_ctx=self.chat_ctx.copy(
exclude_instructions=True,
)
)
return (
specialist,
"I'll connect you to our government-scheme eligibility specialist "
"for more focused guidance.",
)
The specialist has its own instructions and limits. It does not handle fraud, account-specific banking complaints, or sensitive payment information.
A difficult part: integration is where the complexity appears
The hardest part was not making the agent speak once. It was coordinating all the pieces around a real conversation:
Browser audio.
SIP audio.
Speech recognition.
LLM decisions.
Tool calls.
Text-to-speech.
Consent storage.
Call cleanup.
Dashboard writes.
Agent handoffs.

Each part can fail independently.
Some issues I had to work through included:
SIP connections failing.
Calls ending before the intended outcome.
Data sources becoming unavailable.
Duplicate outbound calls.
Unclear or denied consent.
Cleanup logic running more than once.
Dashboard metrics pointing to the wrong SQLite database.
LiveKit APIs differing between installed versions.
OpenCode and local tooling requiring platform-specific setup.

The most useful lesson was to inspect the installed SDK instead of guessing its API. For example:
uv run python -c "import inspect; from livekit.agents import Agent; print(inspect.signature(Agent))"
This kind of inspection is faster and safer than repeatedly writing code against an API that may not exist in the installed version.
Another lesson was to define success before implementing analytics. Without a clear success condition, "successful call" becomes a vague technical status instead of a user outcome.
Build your own voice agent
You do not need to build every feature at once. Start with one complete voice loop:
User speaks
→ Speech-to-text
→ LLM response
→ Text-to-speech
→ User hears the response
Then add tools, memory, safeguards, and analytics one layer at a time.

Clone the project git clone YOUR_PUBLIC_REPOSITORY_URL cd murf-livekit-starter Replace the placeholder with your actual public repository URL.
Install dependencies This project uses Python and a frontend. Follow the commands in the repository's package files and existing documentation. Typical commands are: uv sync npm install Run these from the correct project directories.
Add secrets locally Create a local environment file in the location expected by the project, for example: backend/.env Use placeholder structure such as: LIVEKIT_URL=your_livekit_url LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_api_secret DEEPGRAM_API_KEY=your_deepgram_api_key GOOGLE_API_KEY=your_gemini_api_key MURF_API_KEY=your_murf_api_key Never publish: API keys. SIP credentials. Phone numbers. Caller information. Database files containing private data.

Add environment files and local databases to .gitignore:
.env
.env.*
*.sqlite3
*.sqlite3-shm
*.sqlite3-wal
The exact variable names should match the project's code. Do not copy these names blindly if your implementation uses different ones.

Start the services Use the commands already defined in the repository. A typical setup may include: uv run python backend/src/agent.py dev And, in the frontend directory: npm run dev Open the local URL printed by the frontend terminal.
Test the conversation Start with: What does financial literacy mean? Then try: Am I eligible for PMJDY, and what documents do I need? Finally, test safety behavior: I received a suspicious banking message. Verify that: General questions stay with the main agent. Scheme questions reach the specialist. Fraud questions follow the human-escalation flow. Sensitive information is never requested. The dashboard records the appropriate call outcome.

Troubleshooting
The browser cannot access the microphone
Check:
Browser microphone permission.
The correct local URL.
Whether another application is using the microphone.
Whether the frontend is connected to the running backend.

SIP or Linphone does not connect
Check:
LiveKit URL and credentials.
SIP participant configuration.
Linphone account details.
Whether the SIP endpoint is registered.
Backend logs for connection or authentication errors.

Never publish SIP credentials in a README or screenshot.
The dashboard shows zero calls
The dashboard may be reading a different SQLite file from the agent.
Check:
The database path printed by the dashboard.
The database path used by the agent.
Whether the call_outcomes table exists.
Whether the agent writes a record when the call ends.

Specialist handoff fails
LiveKit Agents APIs can differ between installed versions. Inspect the installed API rather than copying a method name from another version:
uv run python -c "import inspect; from livekit.agents import Agent; print(inspect.signature(Agent))"
Also check the installed package version and keep the handoff implementation compatible with that version.
A port is already in use
Find the process using the port or start the service on another local port. Keep the frontend and backend configuration consistent.
What I would improve next
ArthSakhi is a challenge project and local demonstration, not a production financial service.
Before real-world deployment, I would improve:
Verified, regularly updated scheme data.
Stronger multilingual evaluation.
Regional-language voice testing.
Authentication for the support dashboard.
Secure production database storage.
Monitoring and audit logs.
Better noisy-audio and interruption handling.
More SIP test coverage.
Human-support status tracking.
Formal privacy and security review.

What the challenge changed for me
At the beginning, I thought building a voice agent mainly meant connecting speech-to-text, an LLM, and text-to-speech.
By the end, I understood that the difficult and valuable work sits around the conversation:
What is the agent allowed to do?
What should it remember?
When should it ask for consent?
When should it use a tool?
When should it stop?
When should a human take over?
How do we know whether the user actually reached the intended outcome?

ArthSakhi started as a financial-literacy voice assistant. Over ten days, it became a small but complete voice-agent system with memory, tools, outbound calling, escalation, analytics, and specialist handoffs.
The central lesson was simple:
A responsible voice agent is not the one that answers everything.
It is the one that knows what to say, what not to ask, and when to bring in help.
Links
Repository:
LinkedIn post:
Challenge: 10 Days of Voice Agents - VoiceForBharat Edition

DEV Community: Takshak Singh Rajawat

ArthSakhi: Building a Voice Agent That Knows When to Speak, Act, and Ask for Help