DEV Community

Kavan Bhavsar
Kavan Bhavsar

Posted on

I Built a Voice-First Grocery Agent for Bharat in 10 Days — Here's Everything I Learned

#ai #startup #agents #buildinpublic

What I made

Saathi (साथी) — a voice AI assistant that lets your neighbourhood kirana store take orders, handle disputes, and call customers back. In Hindi. In Gujarati. In whatever you speak.

The Problem and the People

India has over 12 million kirana stores. Most of them are run by a single person — the owner — who is simultaneously billing a customer at the counter, answering a phone order, and mentally tracking which shelf ran out of Aashirvaad Atta. There is no staff for a customer support line. There is no app. There is a phone number, and there is trust built over years.

The problem I chose to solve: what if that phone number could be answered by an intelligent voice agent, 24×7, in the customer's own language?

For the Voice for Bharat Challenge 2026, I built Ratan Kirana & General Store's voice agent — Saathi — over 10 days. Saathi can:

  • Take grocery orders via voice, check live stock, and confirm delivery
  • Remember returning customers and their usual orders
  • Switch between Hindi and English mid-conversation without missing a beat
  • Call customers back proactively for payment reminders and order confirmations
  • Hand off disputes to a specialist agent (Sahayak) when needed
  • Log every call outcome for the store owner's dashboard

This is the story of how I built it, what broke, and what I'd do differently.

What the Agent Does

A customer calls (or opens the web app) and Saathi greets them. If they've called before, Saathi remembers their name, saved address, and usual order. If they're new, Saathi introduces itself warmly and asks how it can help.

A typical successful flow looks like this:
Saathi voice agent order flow sequence diagram showing customer conversation, tool calls to lookup_user, lookup_catalogue, check_stock, and place_order, with responses from SQLite database

When something goes wrong — a payment dispute, a missing delivery — Saathi hands off to Sahayak, the returns specialist, with full conversation context so the customer never repeats themselves.

How the System Works

The architecture is a real-time voice pipeline. Audio from the browser or phone travels through LiveKit's WebRTC infrastructure, hits a Python backend, and returns as synthesised speech — all in under 200ms for the first word.
Ratan Kirana voice agent system architecture showing audio pipeline from customer browser through LiveKit WebRTC to Deepgram STT, Gemini LLM, Python tools layer, SQLite databases, and Murf Falcon TTS

The key components:

Speech-to-text (STT): Deepgram nova-3 with language="multi". This single setting handles Hindi, English, Gujarati, and code-mixed speech like "mujhe 2 bag Atta chahiye" without any extra configuration.

LLM: Google Gemini 3.5 Flash Lite. Fast, cheap, and surprisingly good at following complex Hindi-language instructions.

TTS: Murf Falcon via the murf-livekit plugin. The Anisha voice is the heart of the project — it's what makes Saathi sound like a real neighbourhood shop assistant rather than a corporate bot. The time-to-first-byte on Murf's API was consistently under 90ms, which is what makes the conversation feel natural.

Real-time transport: LiveKit handles WebRTC for browser sessions and SIP trunking for phone calls through Linphone.

Turn detection: LiveKit's MultilingualModel — critical for Hindi, which has different prosody patterns than English.

The Features I Built, Day by Day

Voice Pipeline & Personality (Days 1–2)

The first thing I set up was the voice pipeline and Saathi's system prompt. The prompt is where personality lives. I spent a lot of time on it because a badly written prompt produces an agent that sounds robotic, invents prices, or ignores the customer's language.

Key decisions:

  • Script enforcement: The prompt explicitly says "Hindi → Devanagari (नमस्ते), never romanized (never 'namaste')." Without this, the LLM defaults to romanized Hindi constantly.
  • Short sentences: "Keep each sentence under 20 words whenever possible." TTS sounds better with shorter utterances.
  • Guardrails: Never ask for OTP, PIN, card details. Never invent a discount. Never confirm an order without stock validation.

Live Catalogue & Stock Validation (Days 3–5)

I built a SQLite product database with 53 kirana items — staples, oils, dairy, spices, beverages. Every order goes through two tools in sequence:

  1. lookup_catalogue — fuzzy search by name, returns price and availability
  2. check_stock — validates the exact quantity requested

The agent is explicitly instructed never to confirm an order based only on the catalogue. It must call check_stock with the exact quantity, every time, for every product.

Bug I hit: The LLM would call check_stock("Sprite 750ml", 3) but the DB had name = "Sprite" (no size suffix). The tool was using an exact-match query and returning "not found", which the agent treated as out of stock — even though 20 units sat in the DB. Fix: switched get_product_stock() to use search_products() which does a LIKE match.

Customer Memory (Day 4)

Every browser session gets a unique user_id stored in localStorage. On every call, Saathi runs lookup_user(user_id) first. If the customer is known, it greets them by name and mentions their usual order. If they're new, it asks for their name with explicit consent before saving anything.

The consent flow matters. India has strong norms around privacy. Saathi always asks: "क्या मैं आपका नाम और पता अगली बार के लिए याद रख सकता हूँ?" and only calls save_user_profile if the customer clearly says yes.

Outbound Calling (Day 6)

This was the hardest day. Saathi can initiate phone calls for payment reminders, order confirmations, and delivery updates. The architecture uses LiveKit's SIP outbound trunk through Linphone.

I ran into a 404 SIP error that took hours to debug. Root causes:

  1. The outbound-trunk.json was missing auth_username and auth_password fields. Linphone's free SIP requires authenticated INVITEs.
  2. dial.py was passing sip:kavan@sip.linphone.org as sip_call_to, but LiveKit's API expects just the username (kavan) and constructs the full URI from the trunk config itself.

The call trigger is simple once the trunk is configured:

uv run python src/telephony/outbound/dial.py \
    --to kavan \
    --name "Rahul" \
    --reason payment_reminder \
    --metadata '{"amount": 450, "order_id": "ORD-20260811-001"}'
Enter fullscreen mode Exit fullscreen mode

Human Escalation & Agent Handoff (Days 7–9)

This is the most architecturally interesting part of the project.

Day 7 introduced a create_escalation tool. When a customer reported a payment dispute or undelivered order, Saathi would log the issue and send an email to the store owner. Simple, but it meant Saathi was still trying to be a dispute resolution agent — which it isn't good at.

Days 8–9 replaced this with a proper multi-agent handoff. When Saathi detects a dispute, it says "मैं आपको हमारे रिटर्न्स विशेषज्ञ से जोड़ रहा हूँ" and calls transfer_to_returns_specialist. This returns a new ReturnsAgent instance (named Sahayak, with the Pooja voice from Murf Falcon) along with the full conversation context:

@function_tool()
async def transfer_to_returns_specialist(
    self, context: RunContext
) -> tuple[Agent, str]:
    """Transfer the customer to the returns and refunds specialist."""
    returns_agent = ReturnsAgent(
        chat_ctx=self.chat_ctx.copy(exclude_instructions=True),
        user_id=self.user_id,
    )
    return returns_agent, "Connecting you to our returns specialist now."
Enter fullscreen mode Exit fullscreen mode

The chat_ctx.copy(exclude_instructions=True) is the key line. It passes the full conversation history — everything the customer said to Saathi — into Sahayak's context, but strips Saathi's system prompt so Sahayak runs under its own instructions. The customer never has to repeat themselves.

The voice change from Anisha → Pooja makes the handoff audible without any UI change.

Multi-agent handoff diagram showing Saathi routing normal orders to order flow and disputes to Sahayak returns specialist, with escalation database and store owner email alert

Bug I hit here: I initially defined transfer_to_returns_specialist as a standalone function with self as its first parameter, then added it to a bare Agent(tools=[...]). LiveKit's schema builder tried to introspect it as a regular function, hit self as an untyped parameter, and threw KeyError: 'self'. Fix: the handoff tool must be a method on a class that inherits from Agent, not a standalone function.

The Hardest Problems

1. SIP 404 on Outbound Calls

Already described above. TL;DR: Linphone needs credentials in the trunk config, and LiveKit expects a bare username for sip_call_to, not a full SIP URI. Two separate issues that produced the same 404 error, which made diagnosis slower.

2. Stock Appearing Out of Stock When It Wasn't

The LLM would pass "Sprite 750ml" to check_stock, but the DB had "Sprite". The tool used an exact match, found nothing, and the agent apologised for being out of stock. Twenty units sitting there. Fixed by switching to fuzzy search (LIKE '%sprite%').

3. Agent Handoff KeyError: 'self'

The handoff tool signature had self as a parameter because it was meant to be a class method. When defined outside the class and passed as a standalone tool, the framework tried to build a Pydantic schema from it and couldn't resolve the self type hint. Moved it inside Assistant as a proper method and it worked immediately.

4. Hindi Script Drift

Without explicit script enforcement in the prompt, Gemini 3.5 Flash Lite would frequently respond in romanized Hindi ("namaste", "shukriya") rather than Devanagari (नमस्ते, शुक्रिया). Added a hard rule to the system prompt: "Hindi → Devanagari, never romanized." This is especially important for TTS — Murf Falcon reads Devanagari correctly, but romanized transliteration produces mangled pronunciation.

How to Build and Run This

Prerequisites 

# Clone the repo
git clone https://github.com/KavanBhavsar35/voice-for-bharat-challenge-2026
cd murf-livekit-starter/backend

# Install dependencies (uses uv)
uv sync
Enter fullscreen mode Exit fullscreen mode

API Keys

Create backend/.env.local:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_api_key
LIVEKIT_API_SECRET=your_api_secret
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_gemini_key
MURF_API_KEY=your_murf_key

# For outbound calls (optional)
LIVEKIT_SIP_OUTBOUND_TRUNK_ID=ST_your_trunk_id

# For escalation emails (optional)
GMAIL_USER=your_store_email@gmail.com
GMAIL_APP_PASSWORD=your_app_password
ESCALATION_TO_EMAIL=owner@example.com
Enter fullscreen mode Exit fullscreen mode

Never commit .env.local. It's in .gitignore by default.

Run the Backend 

cd backend
uv run python src/agent.py dev
Enter fullscreen mode Exit fullscreen mode

Run the Frontend 

cd frontend
npm install
npm run dev
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000, click the mic button, and talk to Saathi.

Test an Outbound Call (Optional)

First make sure your outbound trunk is configured in LiveKit. Then:

uv run python src/telephony/outbound/dial.py \
    --to your_linphone_username \
    --name "Your Name" \
    --reason customer_callback
Enter fullscreen mode Exit fullscreen mode

What I Would Improve Next

Real phone numbers. Linphone's free SIP tier works for demos but has reliability issues. The next step is a Twilio Elastic SIP Trunk with a real Indian number, which LiveKit supports with the same configuration.

Gujarati support. The architecture supports it — Deepgram nova-3 handles Gujarati, and Murf has Gujarati voices. The gap is the product catalogue Hindi-to-English translation map, which currently has no Gujarati entries, and the system prompt, which doesn't explicitly handle Gujarati script.

Reduce the prompt size. The current system prompt is enormous. A lot of the order flow logic could move into the tool docstrings, which the LLM reads anyway. A smaller prompt means faster inference and lower cost per call.

Analytics frontend. The call database logs every outcome — success, failure reason, duration, order ID. Right now that data sits in SQLite. A simple dashboard on top of it would let the store owner see at a glance which calls converted and which didn't.

Links

Built for the 10 Days of Voice Agents — VoiceForBharat Edition, August 2026.
Stack: Python · LiveKit · Deepgram nova-3 · Gemini 3.5 Flash Lite · Murf Falcon (Anisha + Pooja voices) · SQLite · Next.js

Top comments (0)