DEV Community

Bhavesh M
Bhavesh M

Posted on

I built a voice agent that answers a kirana shop's phone in Hinglish — here's everything I learned

#voiceagent #ai #automation

10 Days of Voice Agents — VoiceForBharat Edition

The problem

Sharma ji runs a kirana store in Pune. His phone rings all day, and most calls are the same four questions: pyaz ka bhav kya hai? (what's the onion rate), do kilo cheeni bhej dena, kal ka order aadha hi aaya, kya aap Amul butter rakhte ho?

He is usually weighing something with one hand when the phone rings. Half the calls he misses. The ones he takes cost him a customer standing at the counter.

The people calling him are not going to install an app. Many won't type an order in English. They are comfortable doing exactly one thing: talking on a phone, in whatever mix of Hindi and English they normally speak. That is why this had to be a voice agent and not a chatbot — voice is not a nicer interface here, it is the only interface these users already have.

So I built Meera, a voice agent for Sharma Kirana Store, on the Local Commerce track. She answers the shop's phone, quotes real mandi rates, remembers regulars, takes orders, escalates to Sharma ji when she should not decide, calls customers back, and hands complaints to a returns specialist.

Repo: https://github.com/bhaveshmadisetty/voice-for-bharat-bhavesh

What the agent actually does

A real call sounds like this:

Caller: Hello, pyaz ka kya bhav hai?
Meera: Namaste, main Meera bol rahi hoon Sharma Kirana Store se. Onion thirty-two rupaye kilo hai — aaj ka mandi rate.
Caller: Do kilo bhej dena. Main Raj.
Meera: Ji Raj bhai. Do kilo pyaz. Aapka naam aur order yaad rakh loon agli baar ke liye?

She is not reciting a price list from her prompt. get_market_price hits the Government of India's daily mandi feed on every single ask, because an onion rate moves several rupees a day and a shop assistant who quotes yesterday's number is worse than one who says "I'll check."

How the system works

Four moving pieces, and every voice agent is some version of this:

🎙️ user speaks
   ↓ audio
Deepgram STT  →  Gemini LLM (+ tools)  →  Murf Falcon TTS
   ↑                                          ↓ audio
   └──────────── LiveKit (WebRTC / SIP) ──────┘
                       ↓
              🔊 user hears the reply
Enter fullscreen mode Exit fullscreen mode
  • STT — Deepgram nova-3, which handles code-mixed Hindi-English input.
  • LLM — Gemini Flash, with tools for prices, memory, escalation, delivery slots, opt-out, and handoff.
  • TTS — Murf Falcon, voice Anisha (Indian English female). 55ms model latency, ~130ms time to first audio. On a phone call this is the whole product: a 900ms gap before the agent speaks makes a caller say "hello?" and talk over her.
  • Transport — LiveKit, which carries browser audio over WebRTC and real telephone audio over a SIP trunk. Same agent code for both.

The frontend is Next.js and shows the agent's live state — listening, thinking, speaking — so you can see where a delay came from instead of guessing.

The features that tell the story

1. Live mandi prices with a three-source fallback

Prices come from three places, tried in order, and the caller is always told how fresh the number is:

# Source Live? Answers when
1 data.gov.in daily mandi prices Live Normally
2 A CSV the shop uploaded at /prices Local API timed out or rate-limited
3 The shop's standing list of 14 staples Local Nothing else answered

Mandis quote wholesale per quintal; a kirana sells retail per kilo. That conversion is done in Python, never by the LLM:

₹2,400/quintal → ₹24/kg wholesale → ×1.35 margin → ₹32/kg shelf rate
Enter fullscreen mode Exit fullscreen mode

Because a caller is listening to silence while a request runs, the whole lookup has a 6-second budget. Stale data is labelled out loud — "yesterday's closing rate" — and an unknown item is never guessed. She says Sharma ji will call back with that rate.

2. Memory, with consent

A returning caller is greeted by name, in their preferred language, with their usual items ready. It's a SQLite file, it stores a first name, language, delivery preference and past items — and it is written only after the customer says yes, and deleted on request.

3. Escalation to a human, with a spelled-out reference

Two triggers only: a complaint, or a price the tool genuinely could not fetch. An item the shop doesn't stock is not an escalation — nothing was blocked.

Before anything is filed, two sentences are required: name what would be sent, then ask.

"Main sirf itna bhejungi — aapka naam Raj, aur ki do kilo sugar mein se ek hi kilo aaya. Sharma ji ko bhej doon?"

create_escalation refuses to file unless the customer's own last turn contains a real yes, checked against the session transcript. UPI handles, OTPs, PINs, Aadhaar and card numbers are scrubbed before the row is written, not on the way out — a secret that reached storage has already leaked.

The caller gets a reference like SK-4F2B. The alphabet has no I, O, 0 or 1, because this gets said down a phone line and written on the back of a receipt.

4. Outbound calls, and the do-not-call list

Meera also rings customers back to confirm orders. This inverts the consent story — an inbound caller chose to ring the shop; nobody chose this one. So two rules live in code, not in the prompt:

  • The opening line must state who, why, and how to stop in the first two sentences.
  • "Stop calling me" hits the disk the moment it's said, before the confirmation is even spoken. The list is checked three times: when queuing, before dialling, and again once the room is live.

Retries are decided from the actual SIP status, not from vibes:

Outcome SIP Retried?
busy / no_answer / unreachable 486, 480, 404… Yes, up to 3
declined 603, 403 No — they pressed reject
voicemail answered, silence No
opted_out — Never

Voicemail gets a shorter message that names the shop but not the items or the total — a recording can be played back by anyone.

5. A dashboard that can show failure

Three numbers at /dashboard. A call counts as successful only if a tool actually ran: a rate was quoted, an order was saved with consent, or a ticket reached a human with a reference. Never by asking the model how it thought the call went — a verdict the model grades itself on drifts towards flattering.

Failures are recorded with a reason: unresolved, opted_out, no_conversation. A shop that only counts the calls it likes has a dashboard that always looks fine.

The table stores a truncated room id, browser-or-phone, verdict, reason, duration. No name, no number, no transcript — because call_outcomes has no column to put them in, rather than filtering them out. A filter is one refactor away from being removed.

6. Handoff to a specialist

When a customer complains about an order that already arrived, Meera announces one sentence and puts them through to Kavita, the returns and refunds desk. The chat context travels with them, so the customer does not explain themselves twice.

The hard parts (this is the useful section)

The agent never joined the room, and the logs said nothing

Day 1. The frontend connected to LiveKit fine. No agent ever appeared. Session ended with agent didn't join the room.

AgentServer defaults initialize_process_timeout to 10 seconds. That's the budget for prewarming — which on my machine meant Silero VAD plus a 394 MB multilingual turn-detector model. Measured cold start: 16.76s. The worker hit the deadline, logged a bare worker failed, and exited before registering with LiveKit Cloud. A leftover prewarm subprocess kept python.exe visible in Task Manager, so it looked alive.

The fix was one line:

server = AgentServer(initialize_process_timeout=60.0)
Enter fullscreen mode Exit fullscreen mode

The lesson: the giveaway was the absence of the registered worker log line. When debugging a worker, check for the line that should be there, not just the errors that are.

She said the wrong number out loud, and every test passed

Day 5. The price tool returned ₹32 for onion and Meera said "tees" (30). ₹38 became "aath tees", which is not a number in any language.

Hindi numerals between twenty and a hundred are irregular — you cannot compose battees the way you compose thirty-two. The model was assembling them from parts.

Fix attempt 1: list the correct Hindi words in the prompt, and tell her to fall back to English digits if unsure. It got most of the way there, then produced "Oniontees rupees kilo hai" — battees losing its first syllable and merging with the word before it. Being confident was not the same as being right, so the "if you are certain" escape hatch was itself the bug.

The rule is now unconditional: rupee figures are spoken in English, inside the Hindi sentence. "Onion thirty-two rupaye kilo hai." Quantities stay Hindi ("do kilo pyaz"). Every Pune shopkeeper talks this way. Measured over five runs: 5/5 correct, versus wrong roughly one time in three before.

The lesson: remove a failure mode, don't narrow it. And forty passing unit tests told me nothing — I found this by driving four real conversations through the agent and reading the transcript. Do that before you record a demo, not after.

Adding one prompt section silently broke a feature from three days earlier

Day 7. The escalation block is the longest section in the prompt. Adding it broke a Day 4 test that had passed for two days: Meera stopped calling lookup_caller, and greeted a two-call regular as a stranger.

I verified it was a real regression by stashing the change — passed 3/3 before, failed 3/3 after. Strengthening the memory instruction got it to 2/3, and that was the tell: the real conflict was with the greeting rule at the top. On turn one with a name, both rules fire and the greeting wins. Naming the precedence explicitly — look them up first, then introduce yourself in the same breath — got it to 4/4.

The lesson: a prompt is one document, not a pile of independent sections. A new feature can break a finished one at a distance.

The prompt fix that failed twice

Day 9. The handoff was announced in Hindi to an English-speaking customer. The instruction had a language rule plus a worked example in each language — and the model copied the Hindi example. The example outweighed the rule it existed to illustrate. Two rounds of stronger wording did not move it.

The fix was to stop writing prompt. The model has already classified the language to fill in the tool's language argument. So the announcement's language variant is now selected in code, and only the matching one is ever put in front of the model. There is nothing left to copy from.

The lesson: when a prompt fix fails twice and the information already exists in code, decide it in code.

A test can be wrong in the agent's favour

My degradation test asked about imported olive oil and demanded the escalation line. Meera instead said the shop doesn't stock it — honest, useful, and exactly what a real shop assistant says. The test was asserting a script instead of the requirement. I rewrote it to assert the narrower thing that must hold: no invented number, and no dead end.

One more, for anyone doing PII redaction

A sixteen-digit card number written in groups of four opens with a valid twelve-digit Aadhaar match. With the Aadhaar pattern running first, the leading twelve got replaced and the last four — the part a card is identified by — sat in the ticket looking harmless. Longest shape has to win. There is now a test named after that failure.

Build your own — start here

The four components

Every voice agent is the same loop. Pick one provider per box and you have a working agent:

  1. STT — turns speech into text (Deepgram, AssemblyAI, Whisper)
  2. LLM — decides what to say and which tools to call (Gemini, GPT, Claude)
  3. TTS — turns text back into speech (Murf Falcon — 55ms, 150+ voices, 35+ languages)
  4. Real-time transport — moves audio both ways with low latency (LiveKit, over WebRTC for browsers and SIP for phones)

The trick is that these run streaming and overlapping, not one after another. TTS starts speaking the first sentence while the LLM is still writing the second. That's why TTS latency dominates perceived responsiveness, and why Falcon matters more than it looks on a spec sheet.

Setup

Prerequisites: Python 3.10+, uv, Node 18+, pnpm, and a free LiveKit Cloud project.

git clone https://github.com/bhaveshmadisetty/voice-for-bharat-bhavesh.git
cd voice-for-bharat-bhavesh

# Backend
cd backend
uv sync
uv run python src/agent.py download-files

# Frontend
cd ../frontend
pnpm install
Enter fullscreen mode Exit fullscreen mode

Where API keys go — and where they do not

Create backend/.env.local (copy from .env.example). It is gitignored. Keep it that way.

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
MURF_API_KEY=...          # murf.ai/api/dashboard
DEEPGRAM_API_KEY=...      # deepgram.com
GOOGLE_API_KEY=...        # or OPENAI_API_KEY
Enter fullscreen mode Exit fullscreen mode

Three rules I'd give anyone starting:

  1. Never commit .env.local. Check your .gitignore covers .env.* before your first push, not after.
  2. The frontend only needs three of these — LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET — for its token endpoint. I initially had all six in frontend/.env.local. Nothing leaked (Next.js does not ship unprefixed vars to the browser), but there was no reason for those keys to be in that file. Give each service only the keys it uses.
  3. Never put a secret in a NEXT_PUBLIC_* variable. That prefix means "send this to the browser."

Run and test a conversation 

# Terminal 1 — the agent
cd backend && uv run python src/agent.py dev
#   ⚠️ wait for the line: registered worker
#   If you don't see it, the agent is NOT connected — see the prewarm bug above.

# Terminal 2 — shop service (prices, escalations, dashboard)
cd backend && uv run python src/price_server.py

# Terminal 3 — frontend
cd frontend && pnpm dev
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000, click Start talking, allow the mic, and say "pyaz ka bhav kya hai?"

Then try the other pages: /prices to upload the shop's CSV, /escalations for the open-case queue, /dashboard for how the calls are going.

To make a real phone ring, Linphone is the free path — a softphone plus one script:

cd backend
uv run python setup_trunk.py    # writes SIP_OUTBOUND_TRUNK_ID for you
uv run python call_me.py        # your phone rings
Enter fullscreen mode Exit fullscreen mode

Twilio works too if you need an actual PSTN number; setup is in the README.

Change it into your own agent

The whole personality lives in backend/src/prompts.py, deliberately kept out of the pipeline code in agent.py. Rewrite that file and you have a different agent — a clinic receptionist, a language tutor, a delivery tracker — on the same plumbing. Swap the Murf voice with one line: tts=murf.TTS(voice="Samar").

What I'd improve next

  • A shopkeeper-facing view of the day. The dashboard answers "are the calls working"; Sharma ji also wants "what did people ask for today that I don't stock."
  • More languages. Marathi is the obvious next one for Pune. The pipeline supports it; the prompt's numeral rules would need the same treatment Hindi got.
  • Barge-in tuning. Callers in a noisy shop interrupt constantly, and the turn detector is tuned for a quieter room than a kirana at 7pm.
  • Fewer judge-backed tests, more transcript reviews. Every real bug I shipped was found by reading a transcript, not by a green suite.

The one thing I'd tell someone starting Day 1

Your unit tests will pass while your agent says something wrong out loud. Voice agents fail in the gap between "the tool returned 32" and "the customer heard thirty-two." Read the transcripts. Say the numbers out loud yourself. That gap is where the whole product lives.

Code: https://github.com/bhaveshmadisetty/voice-for-bharat-bhavesh
Built with Murf Falcon · LiveKit · Deepgram

#VoiceForBharat
`

Top comments (0)