From Zero to Hindi Voice Agent in 10 Days: Schemes, Memory, and Outbound Calls"

Tanmaya sree Chirra — Sat, 15 Aug 2026 18:37:54 +0000

Building Ashley: A Hindi Voice Agent for Rural Financial Access

The Problem and the Users

India has over 500 million Jan Dhan account holders — yet millions of first-time banking users in rural areas don't know what schemes they qualify for, how UPI works, or even how to
open a zero-balance account. They can't navigate government portals. They don't read English. And they're often afraid of being cheated.

A chatbot doesn't help them. A voice agent does.

Ashley is a Hindi/Hinglish voice agent built for exactly these users — someone who speaks to them like a helpful neighbour, in their own language, for free, 24/7. Built for the #
VoiceForBharat challenge under the Financial Services track.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

What Ashley Does

• Speaks and understands Hindi and Hinglish
• Checks eligibility for 5 government schemes: Jan Dhan, PM Kisan, Mudra Yojana, PMJJBY, PMSBY
• Remembers returning users (with their consent)
• Escalates fraud cases to human agents with a reference ID
• Places outbound Twilio reminder calls for scheme deadlines
• Hands off to Priya, a specialist agent, for deep scheme queries
• Shows a live call analytics dashboard and escalations dashboard

How the System Works

Browser mic → Web Speech API (STT)
→ Flask /chat → LLM (OpenRouter) → Tool calls
→ Murf Falcon API (TTS) → Audio URL → Browser plays

The frontend handles speech recognition via the Web Speech API. The text goes to a Flask backend, which runs it through an LLM with tool-calling enabled. The LLM can call tools like
check_eligibility, save_user, create_escalation, or handoff_to_scheme_specialist. The response text is sent to Murf Falcon for TTS, and the audio URL is played back in the browser.

The Most Important Features

1. Indian Voice with Personality — Murf Falcon (hi-IN-shweta)

Ashley uses Murf Falcon GEN2 with the hi-IN-shweta voice — warm, natural, and unmistakably Indian. The system prompt enforces a strict personality: warm, unhurried, under 2 sentences
per response, mirrors the user's language exactly.

python
MURF_VOICE_ID = "hi-IN-shweta"

def murf_tts(text: str) -> str:
r = requests.post(
"https://api.murf.ai/v1/speech/generate",
headers={"api-key": MURF_API_KEY, "Content-Type": "application/json"},
json={"voiceId": MURF_VOICE_ID, "text": text, "format": "MP3", "modelVersion": "GEN2"},
)
r.raise_for_status()
return r.json()["audioFile"]

2. Safety Guardrails

Ashley will never ask for OTPs, PINs, or account numbers. If someone tries, she says exactly:
│ "Main aapka OTP ya PIN kabhi nahi maangunga. Koi bhi yeh maange toh fraud ho sakta hai — turant call kaatein."

3. Scheme Eligibility Tool

A local dataset of 5 schemes with eligibility rules. The LLM calls check_eligibility as soon as it has one relevant data point — age, farmer status, business ownership — without
waiting to collect everything first.

python
def check_eligibility(answers: dict) -> dict:
eligible = []
for name, scheme in SCHEMES.items():
try:
if scheme"rules":
eligible.append({
"scheme": name,
"description": scheme["description"],
"documents": scheme["documents"],
"apply_at": scheme["apply_at"],
})
except (TypeError, ValueError):
continue
return {"eligible_schemes": eligible, "data_as_of": "August 2025 (local dataset)"}

4. Consent-Gated Memory

Ashley asks before saving anything:
│ "Kya main yeh yaad rakh sakta hoon aapke liye?"

Only name, language preference, and scheme facts are stored — never account numbers or Aadhaar.

5. Human Escalation + Outbound Calls

Fraud reports and blocked account cases get escalated with a reference ID. Users can also request a Twilio outbound reminder call to their phone for scheme deadlines.

6. Specialist Handoff

When a user needs detailed scheme guidance, Ashley hands off to Priya — a specialist agent with a different system prompt — without making the user repeat themselves.

Challenges and How I Overcame Them

The Overlapping Audio Problem

The biggest bug: when a user typed a message before clicking "Start Call", two audio responses would play simultaneously — the greeting and the reply — completely overlapping.

Root cause: speakAgent() was creating a new Audio() object every time without stopping the previous one. Also, the chat input depended on an active voice session — if /start hadn't
been called, /chat had no session history and silently failed.

Fix: Made speakAgent() return a Promise and always call stopAudio() first. Created a single handleUserMessage(text, source) function that both voice and chat feed into. If no session
exists when a chat message arrives, it auto-starts one, awaits the greeting, then sends the message — in sequence, never in parallel.

javascript
function stopAudio() {
if (agentAudio) {
agentAudio.onended = null;
agentAudio.onerror = null;
agentAudio.pause();
agentAudio = null;
}
isSpeaking = false;
}

function speakAgent(reply, audioUrl) {
return new Promise((resolve) => {
stopAudio(); // always kill previous before starting new
// ...
});
}

Free LLM Rate Limits

OpenRouter's free tier has a 50 requests/day cap. Hit it mid-demo. Solution: keep a fallback model ready (nvidia/nemotron-3-super-120b-a12b:free) and consider adding $5 credits for
recording days.

How to Build and Run It

Components You Need

Component	What it does	Tool used
STT	Converts speech to text	Web Speech API (browser)
LLM	Understands and responds	OpenRouter
TTS	Converts text to speech	Murf Falcon API
Transport	Connects everything	Flask + fetch

Setup

bash
git clone https://github.com/your-username/ashley
cd ashley
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

API Keys

Create a .env file — never commit this:

MURF_API_KEY=your_key
OPENROUTER_API_KEY=your_key
TWILIO_ACCOUNT_SID=your_sid
TWILIO_AUTH_TOKEN=your_token
TWILIO_FROM_NUMBER=+1xxxxxxxxxx
PUBLIC_BASE_URL=https://your-ngrok-url.ngrok.io

Run

bash
python server.py

open http://localhost:5000

For outbound calls, run ngrok in a separate terminal:
bash
ngrok http 5000

Then update PUBLIC_BASE_URL in .env with the ngrok URL.

Test a Conversation

Open http://localhost:5000
Enter your name or phone number
Click "Baat Shuru Karein" or just type in the chat box
Ask: "Main kisan hoon, mujhe kya mil sakta hai?"
Ashley will ask your age and land size, then show eligible schemes

What I'd Improve Next

• Replace Web Speech API with Deepgram or AssemblyAI for better Hindi accuracy
• Add support for regional languages: Tamil, Bengali, Marathi
• Move from SQLite to PostgreSQL for production
• Add a proper job queue so outbound calls don't block the server
• Stream TTS audio instead of waiting for the full clip

Links

• 🔗 GitHub: https://github.com/25wh1a6678-art/voice_agent
• 📊 Analytics: http://localhost:5000/dashboard
• 🆘 Escalations: http://localhost:5000/escalations

DEV Community: Tanmaya sree Chirra