Building Ashley: A Hindi Voice Agent for Rural Financial Access
The Problem and the Users
India has over 500 million Jan Dhan account holders — yet millions of first-time banking users in rural areas don't know what schemes they qualify for, how UPI works, or even how to
open a zero-balance account. They can't navigate government portals. They don't read English. And they're often afraid of being cheated.
A chatbot doesn't help them. A voice agent does.
Ashley is a Hindi/Hinglish voice agent built for exactly these users — someone who speaks to them like a helpful neighbour, in their own language, for free, 24/7. Built for the #
VoiceForBharat challenge under the Financial Services track.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
What Ashley Does
• Speaks and understands Hindi and Hinglish
• Checks eligibility for 5 government schemes: Jan Dhan, PM Kisan, Mudra Yojana, PMJJBY, PMSBY
• Remembers returning users (with their consent)
• Escalates fraud cases to human agents with a reference ID
• Places outbound Twilio reminder calls for scheme deadlines
• Hands off to Priya, a specialist agent, for deep scheme queries
• Shows a live call analytics dashboard and escalations dashboard
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
How the System Works
Browser mic → Web Speech API (STT)
→ Flask /chat → LLM (OpenRouter) → Tool calls
→ Murf Falcon API (TTS) → Audio URL → Browser plays
The frontend handles speech recognition via the Web Speech API. The text goes to a Flask backend, which runs it through an LLM with tool-calling enabled. The LLM can call tools like
check_eligibility, save_user, create_escalation, or handoff_to_scheme_specialist. The response text is sent to Murf Falcon for TTS, and the audio URL is played back in the browser.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The Most Important Features
1. Indian Voice with Personality — Murf Falcon (hi-IN-shweta)
Ashley uses Murf Falcon GEN2 with the hi-IN-shweta voice — warm, natural, and unmistakably Indian. The system prompt enforces a strict personality: warm, unhurried, under 2 sentences
per response, mirrors the user's language exactly.
python
MURF_VOICE_ID = "hi-IN-shweta"
def murf_tts(text: str) -> str:
r = requests.post(
"https://api.murf.ai/v1/speech/generate",
headers={"api-key": MURF_API_KEY, "Content-Type": "application/json"},
json={"voiceId": MURF_VOICE_ID, "text": text, "format": "MP3", "modelVersion": "GEN2"},
)
r.raise_for_status()
return r.json()["audioFile"]
2. Safety Guardrails
Ashley will never ask for OTPs, PINs, or account numbers. If someone tries, she says exactly:
│ "Main aapka OTP ya PIN kabhi nahi maangunga. Koi bhi yeh maange toh fraud ho sakta hai — turant call kaatein."
3. Scheme Eligibility Tool
A local dataset of 5 schemes with eligibility rules. The LLM calls check_eligibility as soon as it has one relevant data point — age, farmer status, business ownership — without
waiting to collect everything first.
python
def check_eligibility(answers: dict) -> dict:
eligible = []
for name, scheme in SCHEMES.items():
try:
if scheme"rules":
eligible.append({
"scheme": name,
"description": scheme["description"],
"documents": scheme["documents"],
"apply_at": scheme["apply_at"],
})
except (TypeError, ValueError):
continue
return {"eligible_schemes": eligible, "data_as_of": "August 2025 (local dataset)"}
4. Consent-Gated Memory
Ashley asks before saving anything:
│ "Kya main yeh yaad rakh sakta hoon aapke liye?"
Only name, language preference, and scheme facts are stored — never account numbers or Aadhaar.
5. Human Escalation + Outbound Calls
Fraud reports and blocked account cases get escalated with a reference ID. Users can also request a Twilio outbound reminder call to their phone for scheme deadlines.
6. Specialist Handoff
When a user needs detailed scheme guidance, Ashley hands off to Priya — a specialist agent with a different system prompt — without making the user repeat themselves.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Challenges and How I Overcame Them
The Overlapping Audio Problem
The biggest bug: when a user typed a message before clicking "Start Call", two audio responses would play simultaneously — the greeting and the reply — completely overlapping.
Root cause: speakAgent() was creating a new Audio() object every time without stopping the previous one. Also, the chat input depended on an active voice session — if /start hadn't
been called, /chat had no session history and silently failed.
Fix: Made speakAgent() return a Promise and always call stopAudio() first. Created a single handleUserMessage(text, source) function that both voice and chat feed into. If no session
exists when a chat message arrives, it auto-starts one, awaits the greeting, then sends the message — in sequence, never in parallel.
javascript
function stopAudio() {
if (agentAudio) {
agentAudio.onended = null;
agentAudio.onerror = null;
agentAudio.pause();
agentAudio = null;
}
isSpeaking = false;
}
function speakAgent(reply, audioUrl) {
return new Promise((resolve) => {
stopAudio(); // always kill previous before starting new
// ...
});
}
Free LLM Rate Limits
OpenRouter's free tier has a 50 requests/day cap. Hit it mid-demo. Solution: keep a fallback model ready (nvidia/nemotron-3-super-120b-a12b:free) and consider adding $5 credits for
recording days.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
How to Build and Run It
Components You Need
| Component | What it does | Tool used |
|---|---|---|
| STT | Converts speech to text | Web Speech API (browser) |
| LLM | Understands and responds | OpenRouter |
| TTS | Converts text to speech | Murf Falcon API |
| Transport | Connects everything | Flask + fetch |
Setup
bash
git clone https://github.com/your-username/ashley
cd ashley
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
API Keys
Create a .env file — never commit this:
MURF_API_KEY=your_key
OPENROUTER_API_KEY=your_key
TWILIO_ACCOUNT_SID=your_sid
TWILIO_AUTH_TOKEN=your_token
TWILIO_FROM_NUMBER=+1xxxxxxxxxx
PUBLIC_BASE_URL=https://your-ngrok-url.ngrok.io
Run
bash
python server.py
open http://localhost:5000
For outbound calls, run ngrok in a separate terminal:
bash
ngrok http 5000
Then update PUBLIC_BASE_URL in .env with the ngrok URL.
Test a Conversation
- Open http://localhost:5000
- Enter your name or phone number
- Click "Baat Shuru Karein" or just type in the chat box
- Ask: "Main kisan hoon, mujhe kya mil sakta hai?"
- Ashley will ask your age and land size, then show eligible schemes
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
What I'd Improve Next
• Replace Web Speech API with Deepgram or AssemblyAI for better Hindi accuracy
• Add support for regional languages: Tamil, Bengali, Marathi
• Move from SQLite to PostgreSQL for production
• Add a proper job queue so outbound calls don't block the server
• Stream TTS audio instead of waiting for the full clip
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Links
• 🔗 GitHub: https://github.com/25wh1a6678-art/voice_agent
• 📊 Analytics: http://localhost:5000/dashboard
• 🆘 Escalations: http://localhost:5000/escalations
Top comments (0)