🚀 Building Asha AI: A Vernacular Multi-Agent Voice Platform for Kirana Retail (#VoiceForBharat)

Mrittiga M — Sat, 15 Aug 2026 07:57:32 +0000

📌 1. Introduction & The Problem Statement
In India's fast-evolving retail ecosystem, millions of local Kirana store owners and everyday consumers face digital barriers due to complex application interfaces, text-heavy steps, and language differences.
During the 10 Days of Voice Agents — VoiceForBharat Edition challenge hosted by Murf AI, I chose the Local Retail & Kirana Commerce Track to build Asha AI.
Asha AI is a voice-first Kirana assistant designed to bridge this gap. By enabling real-time, hands-free voice interactions in Indian languages (English, Hindi, and Tamil), Asha AI allows customers to inquire about product prices, check stock availability, request returns/refunds, and automatically escalate complex queries.
🛠️ 2. The 10-Day Building Journey
Here is how Asha AI evolved step-by-step over the 10-day sprint:

Days 1–2 (Foundation & Voice Setup): Configured real-time Web Speech Speech-to-Text (STT) and integrated low-latency Indian voice synthesis (en-IN-aarav) powered by the Murf Falcon TTS API.
Days 3–4 (Personality, Objectives & Guardrails): Defined system prompts, safety guardrails, and customer interaction objectives for regional Indian retail.
Days 5–6 (Multilingual Support & Live UI): Added language switching for English, Hindi (हिंदी), and Tamil (தமிழ்), paired with a glassmorphism frontend dashboard displaying active state and live call metrics.
Days 7–8 (Memory, Tools & Outbound / Escalation Systems): Integrated SQLite for database lookups (stock, pricing, customer history), structured call logging, and automated store manager escalation tickets (#HUM-XXXX).
Day 9 (Multi-Agent Handoff): Implemented sub-agent orchestration where the Asha Main Agent hands off refund/damaged goods queries to a specialized Returns & Refunds Agent.
Day 10 (Documentation & Showcase): Consolidated architecture, code, performance metrics, and build learnings into this public guide. ⚙️** 3. Complete Architecture & Workflow Diagram** Here is how audio, user text, backend logic, database queries, and voice output flow through Asha AI: ┌─────────────────────────────────────────────────────────────┐ │ USER INTERFACE │ │ [ Browser Mic Input / Text Chat / Button Triggers ] │ └──────────────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ ASHA AI CORE PROCESSING ENGINE │ │ (python: day9_asha_full_dashboard.py) │ └───────┬──────────────────────┬──────────────────────┬───────┘ │ │ │ ▼ ▼ ▼ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ SQLite Inventory │ │ Sentiment Engine │ │ Session Manager │ │ & Call Logging │ │ & Escalation │ │ & Agent Handoff │ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │ │ │ └──────────────────────┼──────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ MURF FALCON TTS API │ │ (Low-Latency Speech Generation via en-IN-aarav) │ └──────────────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ AUDIO & UI FEEDBACK │ │ [ Real-Time Voice Output & Glassmorphism Dashboard ] │ └─────────────────────────────────────────────────────────────┘

📋 Detailed Step-by-Step Workflow:

User Voice Input: The user clicks the Voice Mic button or types a query in English, Hindi, or Tamil.
Intent Parsing & Sentiment Engine: Asha's backend analyzes input keywords (e.g., "price", "honey", "damaged", "human") and logs tone sentiment (Positive, Neutral, Negative).
Database & Multi-Agent Routing:
- Inventory Enquiries: Queries the local SQLite inventory table.
- Returns & Refunds: Hands context over to the specialized Returns Agent.
- Frustration / Human Help: Triggers an explicit Human Escalation event.
TTS Generation & UI Feedback: Transformed text is rendered into speech via the Murf Falcon TTS API, while live metrics update instantly on the glassmorphism dashboard. 🎯 4. Key Project Features Highlight
Murf Falcon TTS Integration: Lightning-fast text-to-speech audio rendering using Murf AI's conversational voice endpoints.
Sub-Agent Handoff: Seamless transition between main agent conversation and specialized sub-agents while retaining session context.
SQLite Live Inventory Engine: Direct database lookups for product stock and order logs.
Real-time Glassmorphism Analytics: Displays live audio visualizer state, call success rates, customer sentiment metrics, and stock updates. 💥** 5. Challenges Faced & Solutions** ❌ Challenge 1: Latency & Audio Buffering on Fast Responses
Root Cause: Generating audio responses in real-time caused small playback buffers when processing long strings or switching languages quickly.
Solution: Priority queuing with direct streaming from the Murf Falcon API while implementing a fallback browser Web Speech audio pipeline to guarantee uninterrupted voice response fallback. ❌ Challenge 2: Context Retention During Sub-Agent Handoffs
Root Cause: Session context (user details, prior items mentioned) was clearing when transitioning between Asha Main Agent and Returns Specialist Agent.
Solution: Created a global Python session state manager (session_state) that preserves conversational context, order history, and sentiment logs across agent boundaries. ❌ Challenge 3: Vernacular Code-Mixed Speech Processing
Root Cause: Handling Hinglish/Tanglish mixed phrases (e.g., "Honey oda price enna?") resulted in misclassified database queries.
Solution: Normalized incoming transcripts into target item keywords using a dictionary mapper before executing SQLite queries. 💻** 6. How Readers Can Setup & Run Asha AI** Follow these steps to test the project locally: Step 1: Clone the Repository git clone https://github.com/mrittiga/voice-for-bharat-challenge-2026.git cd voice-for-bharat-challenge-2026

Step 2: Configure Environment Variables
Create a .env file in your root folder (never commit API keys publicly!):
MURF_API_KEY=your_actual_murf_api_key_here

Step 3: Run the Application
python day9_asha_full_dashboard.py

Open your browser and navigate to http://127.0.0.1:8000 to interact with the voice agent interface.
🔮 7. Future Enhancements

Adding WebSocket streaming audio transport for near zero-latency full duplex voice communication.
Expanding voice support to additional Indian regional languages (Telugu, Kannada, Marathi).
WhatsApp Business API integration for sending instant invoice receipts. 🔗** 8. Links & References**
GitHub Repository: mrittiga/voice-for-bharat-challenge-2026
Murf AI Falcon Documentation: Falcon API Docs

DEV Community: Mrittiga M

🚀 Building Asha AI: A Vernacular Multi-Agent Voice Platform for Kirana Retail (#VoiceForBharat)