I Built a Voice Shopping Assistant for Local Kirana Stores in 10 Days — Here's Everything That Broke

Anant Jain — Sat, 15 Aug 2026 06:55:56 +0000

Part of 10 Days of Voice Agents — VoiceForBharat Edition by Murf AI.

The problem, and who it's for

Walk into a small kirana store in most Indian towns and the interaction is entirely voice — you ask what's in stock, you ask the price, you ask if they deliver. There's no app, no menu, no typing. That's not a limitation; it's how local commerce has always worked here, and for a lot of customers — older shoppers, people more comfortable in Hindi than English, people without a smartphone habit of tapping through apps — voice is the natural interface, not a fallback for one.

So for this challenge I built Bazaar Mitra ("market friend"), a voice agent for the Local Commerce track: something a shop could put a phone number behind, that helps a customer find a product, get a real price, start a return, and — critically — knows when to stop pretending it can help and hand the problem to a person.

What it actually does

By the end of 10 days, a call with Bazaar Mitra can do all of this, in Hindi, English, or the code-mixed way people actually talk:

Recognize a returning caller and pick the conversation back up ("last time you asked about atta — need more?") instead of starting cold every time
Look up real prices and stock from an actual catalogue, and price out a multi-item order — never a guessed number
Make outbound calls itself (e.g. confirming an order), and open every one of them by clearly stating who's calling, why, and how to make it stop
Recognize the two situations it genuinely shouldn't handle alone — a payment/refund dispute, or someone asking for a human — and create a real ticket for a person, with the caller's consent, never silently
Hand off returns and refund questions to a separate specialist agent mid-conversation, without making the caller repeat themselves
Record whether each call actually succeeded, and show that on a small live dashboard

How the system works

Caller (phone or browser)
        │  real-time audio
        ▼
   LiveKit Room  ───────────────────────────────►  Agent Session
                                                      STT → LLM → TTS
                                                   (Deepgram → Gemini → Murf Falcon)
                                                            │
                                          ┌─────────────────┴─────────────────┐
                                          ▼                                   ▼
                                  Function tools                        spoken reply
                             (memory, catalogue,                      streams back to
                              escalation, returns)                     the caller
                                          │
                                          ▼
                                       SQLite
                              (callers · catalogue · escalations
                               · call outcomes · returns)
                                          │
                                          ▼
                                 Human dashboard
                              (open escalations + call
                                  analytics, Flask)

Four pieces do all the work:

Speech-to-text (Deepgram, nova-3, multi-language) turns what the caller says into text the model can read.
The LLM (Gemini flash-lite) is the actual brain — it decides what to say and, more importantly, when to call a tool instead of just talking.
Text-to-speech (Murf Falcon) turns the reply back into audio. This is the piece that has to be fast, because every millisecond here is a millisecond of dead air on a live call — in testing it consistently came in around 110ms time-to-first-audio, which is the difference between a conversation and a phone call with a bot bolted on.
LiveKit is the real-time transport underneath all of it — the room both the caller and the agent join, over WebRTC for browser callers or SIP for phone calls.

Everything the agent knows — a caller's name, the product catalogue, open escalations, past call outcomes — lives in one SQLite file, read and written through a small set of Python modules the agent calls as tools.

The features that mattered most

Memory, with consent baked in. The agent can save a caller's name and shopping preferences — but only after explicitly asking and getting a "yes." Say no, and nothing is written. This isn't a nice-to-have; for a track like this, an agent that silently
remembers everything you say is a worse product, not a better one.

Real data over guessing. Early on it would have been easy to let the LLM just generate a price when asked. Instead, prices and stock come from an actual lookup tool, and if that "catalogue service" times out, the agent says so out loud instead of inventing a number. Same principle for order totals — the tool does the math, the model never does.

Escalation as a first-class outcome, not a failure. A call that ends in a properly created human ticket — with the caller's consent, a short factual summary, no transcript, no sensitive data — counts as successful on the analytics dashboard, the
same as a call that resolved on its own. Knowing when to hand off is the feature, not a
fallback.

A specialist that doesn't try to do everything. The Returns & Refunds specialist has exactly one job. It doesn't touch the catalogue or order tools. When the main agent
hands off to it, the caller doesn't repeat themselves — the specialist gets the prior conversation and introduces itself before continuing.

The hardest part: making outbound calls actually work

Everything up to this point ran in a browser tab. Outbound calling is where the project
left "code that runs in one predictable environment" and hit the much messier reality of a real Windows dev machine — and this is where I lost the most time, so it's the part
most worth writing down.

Bug one: two scripts, two different ideas of where .env.local lives.
python -m dotenv resolves a bare ".env.local" relative to the current working
directory — not the script's location. One script lived in backend/src/, my actual
.env.local lived in backend/, and depending on which folder I happened to run the
command from, the same script would either find its config or throw a bare KeyError
with no useful context. The fix wasn't a one-off patch — it was a small shared helper
that checks a short list of likely locations (next to the script, one directory up,
current directory) and loads whichever one actually exists, printing exactly where it
looked if none did. Once I stopped guessing a single path and started checking the
realistic candidates, the problem stopped being a problem for every script, not just
the one I'd just debugged.

Bug two, right after: pip install succeeding into the wrong Python entirely.
With the venv apparently active — prompt and all — pip install flask reported success,
and python dashboard.py still said ModuleNotFoundError: No module named 'flask'.
The giveaway was in the install log itself: the package script had been written to
AppData\Local\Python\pythoncore-3.14-64\Scripts, not backend\.venv\Scripts. pip
and python were quietly resolving to two different interpreters. The real fix was
boring but reliable: stop trusting a bare pip command, and always run
python -m pip install ... instead, so the install is guaranteed to land in whichever
interpreter python itself resolves to. (And when even that failed with "No module
named pip," it turned out the venv had never had pip bootstrapped into it at all —
python -m ensurepip --upgrade fixed that layer.)

Neither of these was a LiveKit problem, or even really a "hard" problem — they were
environment-hygiene issues that happen to be very easy to hit on Windows and very confusing to diagnose from the error message alone. If you're building on Windows: check sys.executable before you trust that an install worked, and never assume a script's config file is where you think it is.

How to build your own

You don't need to solve any of the above to get started — here's the actual minimum path.

1. Get the starter project.
The Murf LiveKit Starter gives you a working STT → LLM → TTS pipeline out of the box; I built on top of it rather than starting from an empty file, and I'd recommend the same.

2. Set up the environment.

cd backend
python -m venv .venv
.venv\Scripts\Activate.ps1      # Windows
# source .venv/bin/activate     # macOS/Linux
python -m pip install -r requirements.txt

3. Add your API keys — never to code, always to .env.local.
Create backend/.env.local (note: .local, and make sure it's in .gitignore — never
commit this file):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
DEEPGRAM_API_KEY=...
GOOGLE_API_KEY=...
MURF_API_KEY=...

If you add outbound calling later, the Twilio trunk credentials go in the same file —
not in the script, not in a committed config.

4. Run it.

python src/agent.py dev

This starts a worker that waits for a room to join. Pair it with the starter's frontend
(or LiveKit's own Agents Playground) to actually get a microphone into a room and talk
to it.

5. Have a real conversation.
Ask it something ordinary first. Then push on the edges — ask about a product it
doesn't have, ask it to remember you, hang up and call back. The gaps show up fast, and
they're the most useful part of testing a voice agent: you can't unit-test "does this
feel like talking to someone helpful."

Code: [https://github.com/Anantjain-infinite/voice-agent-local_commerce]

No API keys, phone numbers, or caller data are in the repo — every secret loads from a git-ignored .env.local, and the SQLite database file itself is also excluded.

What I'd improve next

Swap the hand-built catalogue and return-policy data for a real POS/inventory integration — they're isolated behind single functions specifically so that swap is small when a real one exists.
Add basic auth to the human dashboard before it's anything but a local demo.
Expand the outbound triggers beyond order confirmation — a restock nudge based on a caller's own order rhythm is sitting right there in the memory data already.
Get real numbers on the analytics dashboard beyond a handful of test calls, and see whether the success/failure split actually matches what a human reviewing the calls would call a "good" conversation.

Built as part of 10 Days of Voice Agents — VoiceForBharat Edition, using Murf Falcon for text-to-speech.

DEV Community: Anant Jain