DEV Community

Nikhil Sharma
Nikhil Sharma

Posted on

Building Kisan Mitra: How I Built an Ultra-Fast Voice AI for Indian Farmers in 10 Days

From zero to a full-stack, multilingual agricultural voice agent with caller memory, real-time mandi tools, outbound price alert calls, human escalation, and specialist agent handoffs — powered by Murf Falcon & LiveKit.

🌟 The Problem & The Mission

In rural India, millions of farmers make critical livelihood decisions every day: When should I harvest? Will it rain before I spray pesticides? Which nearby mandi (market) is offering the best price for my cotton crop?

While agricultural data exists across various portals, accessing it through complex web interfaces or text-heavy apps is challenging for farmers out in the field.

Voice is the natural, frictionless interface for Bharat. A farmer standing in an orchard or driving a tractor doesn't want to type queries into a search bar; they want to speak naturally in their native language or conversational Hinglish and get instant, reliable answers.

For the 10 Days of Voice Agents (VoiceForBharat Edition), I chose the Farm & Field track and built Kisan Mitra (⤕ā¤ŋā¤¸ā¤žā¤¨ ā¤Žā¤ŋ⤤āĨā¤°) — an empathetic, real-time AI voice assistant tailored specifically for Indian agriculture.

đŸ—ī¸ Architecture & Core Components

A production-grade voice agent is fundamentally different from a text chatbot. Latency is the single biggest factor in conversational realism: if the agent takes more than 1–1.5 seconds to reply, the human conversation breaks down.

mermaid
flowchart LR
A[đŸŽ™ī¸ Farmer Speaks] -->|Audio Stream| B(Deepgram Nova-3 STT)
B -->|Transcribed Text| C(Gemini 2.5 Flash LLM)
C -->|Streamed Tokens| D(Murf Falcon TTS)
D -->|Real-time Audio| E(LiveKit WebRTC)
E -->|Ultra-low Latency Audio| F[🔊 Farmer Hears Answer]
C <-->|Tools & Memory| G[(SQLite & External APIs)]

The 4 Pillars of the Pipeline:
Real-time Transport (LiveKit): Manages ultra-low-latency, bidirectional audio WebRTC streaming and turn detection.
Speech-to-Text (Deepgram Nova-3): Accurately transcribes spoken Indian English and accented Hindi.
LLM Brain (Google Gemini 2.5 Flash): Handles intent detection, domain reasoning, guardrails, and tool calling.
Fast Text-to-Speech (Murf Falcon): The game changer. With sub-100ms time-to-first-audio, Murf Falconstreams natural, warm Indian voices (Anisha / hi-IN) without robotic pauses.

🚀 Key Features Built Across the 10 Days

  1. Authentic Indian Voice & Conversational Guardrails Using Murf Falcon's hi-IN voice with conversational styling and sentence tokenization, Kisan Mitra sounds warm, respectful, and native. The system prompt strictly enforces:

Conversational, concise spoken responses (no markdown syntax or raw JSON read aloud).
Absolute refusal to fabricate market rates or weather data.

  1. Live Agricultural Tools (Mandi Prices & Open-Meteo Weather) Kisan Mitra is armed with domain tools that fetch real-world data:

lookup_mandi_prices(crop, district): Queries market prices across key Indian APMCs (e.g., Yavatmal, Nagpur, Lasalgaon) and explicitly cites timestamps (e.g., "As of today's Agmarknet live update...").
get_district_weather(district): Leverages Open-Meteo live satellite feeds to deliver temperature, rain probability, and actionable agronomic advice (e.g., "Rain probability is 65% today; postpone chemical spraying").

  1. Graceful Failure & Out-Loud Transparency
    External APIs fail in the real world. Rather than hallucinating rates or hanging silently, Kisan Mitra catches timeouts and announces the service outage out loud to the caller:

  2. Caller Memory & Privacy-First Persistence
    Kisan Mitra remembers returning farmers (e.g., their land size, crops grown, district) across sessions via SQLite. Crucial guardrail: The agent never saves data without first asking: "May I save these details so I can remember you for our next call?"

  3. Outbound Telephony & Proactive Price Alerts
    When mandi rates cross a farmer's predefined threshold (e.g., Cotton crossing ₹7,000/quintal in Yavatmal), Kisan Mitra autonomously places a phone call via Twilio & TwiML:

Compliance in the first 2 sentences: Explains who is calling, why, and how to opt out (Press 9 to unsubscribe, 1 for details).
Outcome tracking: Automatically handles busy lines, no-answers (retry in 2h), and short hang-ups.

  1. Human Escalation (Krishi Vigyan Kendra Officer Support)
    For emergencies (severe pest attacks like Pink Bollworm or crop blight), the agent prompts the farmer for consent and logs a structured escalation ticket with sanitized PII, assigning a reference number like ESC-48291 for agricultural officer callbacks.

  2. Specialist Agent Handoffs & Analytics Dashboard
    Handoffs: Complex agronomic pathology queries are handed off seamlessly to a dedicated CropSpecialist sub-agent.
    Analytics: Complete call outcome tracking (completed, success, failed, reason) stored in SQLite and visualizable via an admin dashboard.

đŸ› ī¸ The Hardest Challenges & Lessons Learned

  1. Eliminating Conversational Latency
    The Problem: Combining STT + LLM reasoning + TTS synthesis often introduces awkward 2–3 second silences.
    The Solution: Streaming tokens incrementally from Gemini into Murf Falcon using SentenceTokenizer(min_sentence_len=2) and preemptive_generation=True. Audio synthesis begins before the LLM finishes generating the full paragraph.

  2. Handling Code-Mixed Hindi & Indian English (Hinglish)
    The Problem: Standard VAD (Voice Activity Detection) models often cut off speakers mid-sentence when Indian language filler words ("haanji", "achha", "matlab") were used.
    The Solution: Integrated LiveKit's MultilingualModel turn detector paired with Silero VAD to maintain natural listening rhythms.

  3. Out-Loud Tool Failures vs. Silent Errors
    The Problem: When an external weather API timed out, the LLM initially tended to guess yesterday's temperature.
    The Solution: Enforced strict system instructions where tools return explicit FAILURE strings that instruct the LLM: "State out loud that the service is unreachable right now. Do not guess."

GITHUB :- https://github.com/codebynikhil08/murf-livekit-starter

Top comments (0)