DEV Community: Gaurav Shinde

Building Dukaan Mitra: A Voice AI Assistant for Kirana Stores

Gaurav Shinde — Sat, 15 Aug 2026 15:26:00 +0000

Building Dukaan Mitra: A Voice AI Assistant for Kirana Stores

Over the last 10 days, I took part in 10 Days of Voice Agents — VoiceForBharat Edition, where the goal was to build a capable voice AI agent from scratch. Here's a look at what I built, the challenges I ran into, and how you can build one too.

The Problem and the Users

Local grocery shops, known as "Kirana" stores in India, run on personal relationships with regular customers. A large share of their business comes from customers calling in their monthly restock of staples like rice, dal, and oil.

But during peak hours, shop owners miss these calls or forget to follow up — and that means lost sales. I wanted to fix this by building Dukaan Mitra, an AI voice assistant that proactively calls regular customers to remind them about their monthly restock, takes their order, and saves it directly to a database.

What Dukaan Mitra Does

Dukaan Mitra is designed to sound and act like a real shop assistant:

It makes outbound calls to customers and greets them by name.
It understands natural, code-mixed conversation — Hinglish, Hindi, and English.
It checks today's prices and stock for items in real time.
It confirms the order, generates an Order ID, and logs it.
If a customer raises a complaint, dispute, or return request, it hands the call over seamlessly to a specialist agent named Pooja.

The Most Important Features

A few key features are what made this agent feel robust and realistic rather than a scripted demo.

1. Indian voices with Murf Falcon
To make the agent sound local, I used Murf Falcon for text-to-speech — the main assistant runs on the voice "Anisha," and the returns specialist runs on "Pooja." Falcon's sub-130ms latency is what made the conversation feel genuinely responsive instead of laggy and robotic.

2. Deep guardrails and personality
I instructed the LLM to strictly use female grammatical forms when speaking Hindi/Hinglish (e.g., "main kar dungi," not "main kar dunga"). I also gave it a hard rule to never invent prices — if an item isn't in the database, the agent says so honestly instead of guessing.

3. Real-time tools and memory
The agent connects to local SQLite databases (inventory.db and users.db) and uses tools like check_price_and_stock, confirm_order, and save_caller to interact with that data mid-call.

4. Multi-agent handoff
Returns are tricky to handle well in one flow. When a customer asks for a refund, Dukaan Mitra calls a transfer_to_returns_specialist tool that routes the conversation to a completely separate agent instance — different prompt, different voice — dedicated entirely to logging return requests.

The Difficult Parts: Lessons Learned

Building this wasn't without hurdles. The biggest one was a race condition right at the start of every outbound call.

Since the agent needs to know who it's calling, I initially had it call the lookup_caller tool as its very first action. But I discovered that Gemini 3.5 Flash Lite would sometimes crash or hang if it tried to invoke a tool as its absolute first action, before any conversational turns had happened.

How I fixed it: I changed the backend initialization flow. Instead of letting the agent look up the user itself, I wrote a Python script to pre-fetch the caller's information from SQLite before the agent even initializes, then injected that data directly into the system prompt as CURRENT CALLER INFO. This removed the need for an initial tool call entirely — it stopped the crashes and also made the agent's first response noticeably faster.

How to Build and Run Your Own

If you want to build something similar, here are the main components:

Speech-to-Text (STT): Deepgram — turns user speech into text
LLM: Google Gemini — the brain of the agent
Text-to-Speech (TTS): Murf Falcon — turns text back into voice
Real-Time Transport: LiveKit — handles the WebRTC audio streaming

Prerequisites:

Python 3.10+ (with uv for fast package management)
Node.js 18+ (with pnpm)
A free LiveKit Cloud account and project

Step 1: Clone the repository

git clone https://github.com/gauravshinde2162006-hash/Day-1-Get-Your-Voice-Agent-Talking.git
cd Day-1-Get-Your-Voice-Agent-Talking

Step 2: Add your API keys

Create a .env.local file in both the backend/ and frontend/ directories with the following:

LIVEKIT_URL=wss://...
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_secret
MURF_API_KEY=your_murf_key
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_gemini_key

.env.local is already in .gitignore — never commit real keys to the repo.

Step 3: Install dependencies and run

In one terminal, start the backend agent:

cd backend
uv sync
uv run python src/agent.py dev

In a second terminal, start the frontend:

cd frontend
pnpm install
pnpm dev

Step 4: Test a conversation

Open http://localhost:3000, click Start talking, and allow microphone access. Since this simulates an outbound call, Dukaan Mitra greets you first — just like it would greet a real customer.

What I'd Improve Next

Dukaan Mitra works well as a proof of concept, but a few rough edges are honestly still there:

Hardcoded caller identification. The backend currently initializes each session with a fixed caller_id. In production, this should come dynamically from the incoming SIP trunk data or the frontend session instead.
Mocked WhatsApp integration. The send_whatsapp_message tool doesn't actually send anything in the background yet — it opens a pre-filled WhatsApp Web tab via webbrowser.open(). Swapping this for a real Twilio or WhatsApp Business API call is next on the list.
Simulated error handling. To test the agent's failure guardrails during a database outage, I currently check for a local force_db_fail.flag file. A real version needs proper HTTP timeout and try/except handling against a live inventory API.

Links and Demos

The full source code for Dukaan Mitra is on GitHub:
Dukaan Mitra Voice Agent Repository

Built as part of 10 Days of Voice Agents — VoiceForBharat Edition, powered by Murf AI.

Captain Cool: Building an Enterprise Multi-Agent IPL Match Strategist with Google Gemini 🏏🏆

Gaurav Shinde — Sun, 17 May 2026 12:53:13 +0000

🌟 Introduction: The High-Stakes World of T20 Cricket
In the ultra-competitive arena of the Indian Premier League (IPL), matches are won and lost in the margins. A single tactical miscalculation—bowling a spinner in the death overs against a power-hitter, misplacing a fielder at cow corner, or misjudging the heavy dew factor—can alter the fate of a championship.

While modern IPL franchises employ armies of data analysts, real-time tactical decision-making in the dugout remains incredibly stressful. Captains must synthesize vast amounts of historical data, pitch conditions, and gut instinct in seconds.

To solve this, we built Captain Cool—an enterprise-grade, virtual IPL captain powered by a multi-agent Google Gemini AI think-tank. Designed for the Google Build with AI Hackathon (#gdgcloudpune), Captain Cool simulates an elite dugout debate in real-time to deliver the ultimate tactical call.

🧠 The Core Innovation: Multi-Agent Debate Loop
Single-prompt LLMs often suffer from "sycophancy"—they agree with whatever parameters the user inputs without challenging underlying assumptions. In cricket strategy, blind agreement is a recipe for disaster.

To build a robust tactical engine, Captain Cool implements an advanced Multi-Agent Debate Architecture using four autonomous Gemini 3.1 Flash Lite agents orchestrated via a sequential LangChain-style debate loop:

[ Live Cricbuzz Scraper / Form Input ]
│
▼
📊 Agent 1: Stats Analyst (Tool Calling)
│
▼
🎯 Agent 2: Strategist (Initial Call)
│
▼
⚔️ Agent 3: Devil's Advocate (Dissent & Challenge)
│
▼
💪 Agent 2: Strategist (Revised Defense)
│
▼
🎙️ Agent 4: Match Commentator (Final Verdict & Confidence %)

📊 The Stats Analyst (Tool-Calling Head) Acting as the head of data intelligence, this agent is equipped with custom JavaScript tools to fetch real-time match context:

getWinProbability(): Calculates dynamic win percentages based on target, balls remaining, and wickets.
getPlayerProfile(): Pulls elite batting/bowling stats (strike rates, death over economy, weakness against spin/pace).
getPitchAssessment(): Evaluates venue history (Wankhede dew, Chepauk turn) and current phase dynamics.

🎯 The Strategist (Tactical Captain)
Taking the data report from the Stats Analyst, the Strategist formulates the initial over-by-over plan, selecting the ideal bowler, field placements, and bowling lines.
⚔️ The Devil’s Advocate (Opposition Analyst)
This is where the magic happens. The Devil's Advocate is strictly instructed to act as the opposition head coach. It inspects the Strategist's plan, identifies glaring tactical blind spots (e.g., "You're bowling Bumrah now, but who bowls the 20th over?" or "Kohli loves stepping out to spin, your mid-off is too deep"), and aggressively challenges the captain.
🎙️ The Match Commentator (Broadcast Verdict)
Finally, the Commentator observes the debate, synthesizes the opposing viewpoints, and delivers an official broadcast-quality verdict complete with a calculated Tactical Confidence Percentage.

🚀 Key Technical Features
🔴 1. Automated Live Cricbuzz Web Scraper
To ensure the AI operates on absolute real-time data, we built a native Node.js web scraper directly into the Stats Analyst agent.

How it works: When a user pastes a live Cricbuzz match URL, the backend instantly fetches the live HTML, strips out all DOM overhead, extracts the pure live scorecard and commentary text (up to 15KB), and injects it into Gemini's context window.
Absolute Truth: The AI is instructed to treat this scraped live Cricbuzz data as the primary source of truth, overriding any conflicting manual form inputs.
📍 2. Interactive 2D/3D Fielding Oval
Field placements are critical to T20 bowling plans. We implemented a custom, interactive top-down cricket ground UI directly in the command console.

Users can click anywhere on the grass to place up to 9 fielders.
Using advanced polar coordinate math, the frontend instantly calculates the exact cricket position clicked (e.g., Deep Point, Slips / Gully, Cow Corner, Short Fine Leg).
This exact fielding list is fed into the AI prompts, allowing the agents to analyze physical field gaps and suggest bowling lines to match!
🌌 3. Cinematic 3D Enterprise UI (Vercel/Apple Inspired)
We elevated the application from a standard dashboard into an immersive, executive-grade sports broadcast platform:

Photorealistic Championship Trophy: Replaced cartoon emojis with a hyper-realistic 3D golden championship trophy cup rendered with professional studio lighting.
3D Mouse Parallax: As users move their cursor across the screen, background glassmorphism stat cards (⚡ REQ RATE: 12.5 RPO, 🎯 MATCHUP INTEL) and geometric rings shift at varying speeds, creating a stunning illusion of multi-layered depth.
Auto-Scroll Choreography: Clicking Analyze & Decide ignites a glowing neon progress bar and smoothly auto-scrolls down into the Live Debate Arena, spotlighting each agent card as it slides up with 3D tilt effects.
🛠️ Tech Stack & Architecture
AI Engine: Google Gemini 3.1 Flash Lite (via @google/genai SDK)
Backend: Node.js, Express.js, native fetch HTML scraping
Frontend: HTML5, Vanilla JavaScript, CSS3 (Glassmorphism, 3D Perspective Transforms, CSS Grid/Flexbox)
Version Control: Git & GitHub
🏆 Conclusion & Hackathon Journey
Building Captain Cool for the Google Build with AI Hackathon (#gdgcloudpune) was an incredible journey in exploring the power of agentic workflows. By combining tool-calling, multi-agent debate, live web scraping, and premium 3D UI design, we demonstrated how Gemini can transform complex sports analytics into an engaging, real-time decision-making platform.

🔗 Check out the Project:
GitHub Repository: https://github.com/gauravshinde2162006-hash/Captain-cool
We would love to hear your thoughts, feedback, and tactical ideas in the comments below! 👇

Built with ❤️ for #gdgcloudpune