Building Bazaar Mitra: My 10-Day Journey Building a Voice AI Agent for Local Commerce

1.INTRODUCTION

Bazaar Mitra is a voice-first AI assistant designed to help local-commerce customers with product enquiries, memory, outbound calls, human escalation, analytics, and specialist handoffs.

2.THE PROBLEMS

1. Persistent user memory Making the agent remember a caller's name across different calls. Solved using SQLite with user_id, name, language preference, and facts. Also added explicit consent before saving information 2.Product data integration Initially the product lookup used a manually created local catalogue. You then moved toward real external catalogue data using DummyJSON API. The challenge was handling API failures, timeouts, missing products, and preventing the agent from guessing prices. 3.Outbound calling Browser-based calls were straightforward, but outbound SIP calling required configuring LiveKit SIP and Linphone. You had to deal with SIP URI, trunk ID, environment variables, and room/agent connection. 4.Frontend/debugging You encountered frontend/runtime issues such as missing references and component-related errors while customizing the LiveKit starter. This is a good section because it shows actual debugging rather than pretending everything worked immediately. 5.Human escalation The agent needed to know when not to solve something itself. You implemented escalation for situations requiring human assistance, including permission before sharing information. 6.Call analytics You needed actual call results rather than hardcoded dashboard numbers. The dashboard tracks: Total calls Successful calls Failed calls 7.Specialist handoff The main Bazaar Mitra agent shouldn't handle every type of request. You added a specialist-agent concept so specific requests can be transferred while maintaining conversation context.

3.FEATURE MAP

Bazaar Mitra is a Local Commerce AI voice assistant that can:

converse using voice
follow business and safety guardrails
understand Hindi, English, and Hinglish
remember returning callers with consent
look up product information
make outbound calls
escalate payment/order issues to humans
track call outcomes through analytics
hand return/refund conversations to a specialist agent

• Voice + Guardrails - Defined identity, objectives, safety boundaries and concise voice responses.
• Indian / Mixed Language - Hindi, English and Hinglish communication based on the caller.
• Memory - Caller information is stored persistently only after explicit consent.
• Product Lookup - The agent calls a tool/API instead of inventing product information.
• Outbound Calls - The agent can initiate an outbound voice interaction through SIP.
• Human Escalation - Payment/refund or order-dispute issues can become a human-help request.
• Call Analytics - Real call outcomes feed Total, Successful and Failed call counts.
• Specialist Handoff - Returns/refunds conversations move from the general agent to a focused specialist.

4.ARCHITECTURE

User
↓
LiveKit
↓
Deepgram STT
↓
Gemini LLM
↓
Tools / Memory / Handoff
↓
Murf Falcon TTS
↓
User

Supporting Components :
SQLite → persistent customer memory
Product API → product lookup
SIP/Linphone → outbound calls
Escalation DB → human-help requests
Analytics DB → call outcomes
Specialist Agent → returns/refunds

5.DIFFICULT PARTS

Memory persistence problem:
Initially the agent could save information but couldn't remember it across calls. The issue was traced through the caller identity and database flow. Stable user IDs and SQLite persistence were required.

Tool integration problems:
While adding Day 5 tools, tool definitions and API handling caused failures. Debugging logs helped identify whether the tool was being called, whether the API responded, and whether the result reached the voice pipeline.

Outbound SIP setup:
Outbound calling required understanding the difference between the LiveKit SIP endpoint, SIP trunk, and destination SIP address. Environment-variable configuration was also important.

6. PRACTICAL BUILD GUIDE

Prerequisites: Python environment, LiveKit project, API credentials for the services used, and a public repository.

Basic Stack :
Speech-to-Text → Deepgram
LLM → Gemini
Text-to-Speech → Murf Falcon
Real-time transport → LiveKit
Database → SQLite

Setup Sequence :

1. Clone the repository
2. Install dependencies
3. Create .env.local
4. Add API keys there
5. Never commit .env.local
6. Start the LiveKit agent
7. Start the frontend
8. Open the browser
9. Start a voice session
10. Test a simple conversation

7. Evidence

Bazaar Mitra frontend

product tool being called

Call analytics dashboard

Specialist handoff clip

8. LEARNING

Voice agents are not just an LLM plus TTS. Reliability depends on turn detection, STT quality, tool design,
state management, database persistence, permissions, fallback behavior, and clear agent boundaries.

the next version could include:

real merchant inventory instead of demo/catalogue data
proper order-management integration
better multilingual voice support
richer analytics
real CRM/help-desk integration
improved specialist routing
authentication and production-grade data protection.

9. LINKS

GitHub Repository : https://github.com/insyncerick/murf-livekit-starter
My Linkedin Profile : www.linkedin.com/in/tilak-raj-roy-83b85440a
(there is uploaded my day1 to day 10 full journey posts )