Building KisanMitra AI: A Full-Stack Voice Agent for Indian Farmers 🌾🎙️

Rishabh Kumar — Sat, 15 Aug 2026 17:00:11 +0000

Day 10 of the **10 Days of Voice Agents – VoiceForBharat Edition* Challenge | Farm & Field Track*

Imagine holding a smartphone in the middle of a mustard field in rural India.

Your hands are covered in soil. The sun makes the screen difficult to read. You need an answer about your crop, weather, or mandi prices — but typing a long query or navigating through complicated portals isn't practical.

Now imagine simply pressing a button and saying:

"भैया, कल बारिश होगी क्या? और धान में भूरा धब्बा रोग लगा है, क्या करें?"

Instead of searching through menus, you can simply talk.

That's the idea behind KisanMitra AI (किसान मित्र AI) — a full-stack, real-time bilingual voice agent designed specifically around agricultural assistance.

The project combines:

🎙️ Murf Falcon TTS
⚡ LiveKit Agents
📝 Deepgram STT
🧠 Google Gemini
🌐 Next.js 15
🗄️ SQLite
📞 SIP-based calling
🚨 Human escalation
🤝 Specialist agent handoff

This article is the complete story of how I built KisanMitra AI during the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

🇮🇳 1. Why Voice for Indian Farmers?

Voice assistants are often treated as convenience tools.

But in agriculture, voice can solve a much more practical problem: accessibility.

Farmers may be working outdoors, using their hands, dealing with sunlight, speaking in Hindi or regional languages, and needing information quickly.

The interaction shouldn't look like this:

Open App
   ↓
Find Menu
   ↓
Select Category
   ↓
Fill Form
   ↓
Type Question
   ↓
Wait for Result

It should feel more like this:

👨‍🌾 Farmer
    ↓
🎙️ "मेरी फसल में ये समस्या हो रही है..."
    ↓
🤖 KisanMitra AI
    ↓
🔊 Natural Voice Response

For the Farm & Field track, my objective was simple:

Make agricultural assistance as conversational and accessible as possible.

🏗️ 2. End-to-End System Architecture

KisanMitra AI connects a real-time voice pipeline with AI reasoning, tools, memory, analytics, telephony and human escalation.

                         👨‍🌾 FARMER
                             │
                 ┌───────────┴───────────┐
                 │                       │
          🌐 Web Browser            📞 Phone / SIP
                 │                       │
                 └───────────┬───────────┘
                             ▼
                       ⚡ LIVEKIT
                    Real-Time Audio
                             │
                             ▼
                    🎙️ Silero VAD
                  Turn Detection
                             │
                             ▼
                    📝 Deepgram STT
                    Hindi / Hinglish
                             │
                             ▼
                   🧠 KISANMITRA AGENT
                             │
                ┌────────────┼────────────┐
                ▼            ▼            ▼
             Gemini       Memory        Tools
                │            │            │
                │            ▼            ├── Weather
                │         SQLite           ├── Farmer Profile
                │                           └── Escalation
                │
                ▼
             Decision
                │
        ┌───────┴────────┐
        ▼                ▼
   Normal Answer    Complex Problem
        │                │
        │                ▼
        │          🌾 FASAL DOCTOR
        │          Specialist Agent
        │
        └────────┬───────────────┘
                 ▼
            🔊 MURF FALCON
             Text-to-Speech
                 │
                 ▼
              LIVEKIT
                 │
                 ▼
           👨‍🌾 FARMER HEARS

The architecture and integration layers are based on the project structure documented during the build.

🔄 3. How a Voice Conversation Works

Let's follow one conversation from beginning to end.

Step 1 — Farmer speaks

The farmer speaks through the browser microphone or a supported phone/SIP connection.

Step 2 — LiveKit transports audio

LiveKit handles the real-time audio communication.

Step 3 — Voice Activity Detection

Silero VAD detects when the farmer starts and stops speaking.

Step 4 — Speech-to-Text

Deepgram Nova-3 STT converts the audio into text.

It is designed to handle Hindi/Hinglish conversations and code-switching.

For example:

"सरसों का Mandi Bhav क्या है?"

Step 5 — Gemini processes the request

Google Gemini receives the transcript and relevant context.

The agent decides whether it needs to:

Answer directly
Access memory
Call a tool
Escalate the issue
Hand the conversation to a specialist

Step 6 — Murf Falcon generates the voice

The generated response is sent to Murf Falcon TTS.

The response is then streamed back as audio.

Step 7 — Farmer hears the response

The audio travels through LiveKit back to the user.

So the complete pipeline is:

🎙️ Speech
   ↓
LiveKit
   ↓
Silero VAD
   ↓
Deepgram STT
   ↓
Google Gemini
   ↓
Tools / Memory
   ↓
Murf Falcon
   ↓
LiveKit
   ↓
🔊 Speech

⚡ 4. Why Low Latency Matters

Voice conversations feel very different from text conversations.

If an AI takes several seconds before responding, users may think:

"Call cut gaya kya?"

That's why latency became an important part of this project.

With streaming TTS, the project targeted approximately 114–135 ms TTFB for the Murf Falcon voice layer in the tested setup.

The goal was to make the response feel immediate rather than waiting for the entire response to be generated first.

🎨 5. Building the Frontend

I didn't want KisanMitra AI to look like a basic chatbot.

I built a custom Next.js 15 + Tailwind CSS + shadcn/ui interface with an agriculture-inspired emerald/green theme.

🌿 Main UI Features

Glassmorphism

Frosted glass cards and layered panels give the interface a modern look.

🎙️ Listening State

The UI visually indicates when the agent is listening.

🌱 Thinking State

A separate state indicates that the AI is processing the request.

🔊 Speaking State

Audio visualizations show when the agent is responding.

💬 Live Transcript

The user can see the conversation while speaking with the agent.

📦 6. Six Agricultural Modules

The main dashboard contains six smart agricultural modules.

Module	Purpose
⛅ Weather Forecast	Weather, rain and temperature information
🌱 Crop Advisory	Crop and farming guidance
📈 Mandi Bhav	Market-related information
🐛 Disease Control	Crop disease and pest assistance
🐄 Livestock Care	Animal and dairy-related assistance
🏛️ Government Schemes	Agricultural scheme information

Example voice prompts:

"क्या कल मेरे खेत में बारिश होगी?"

"गेहूं की पहली सिंचाई कब करनी चाहिए?"

"धान की पत्तियों में धब्बे आ गए हैं, क्या करूं?"

"आज मंडी में सरसों का भाव क्या है?"

These modules are part of the frontend experience documented in the project.

🚀 7. The 9-Day Technical Journey

KisanMitra AI wasn't built in one day.

It evolved throughout the challenge.

Day 1 — Give the Agent a Voice 🎙️

The first step was creating the basic real-time voice agent.

I connected the voice pipeline and integrated Murf Falcon TTS.

This transformed the project from a text-based concept into a conversational assistant.

Day 2 — Build the User Interface 🎨

I moved beyond the backend and created a proper frontend.

The goal was to give users a simple way to:

Start a conversation
See the agent state
View transcripts
Interact with agricultural modules

Day 3 — Add Memory 🧠

A useful assistant shouldn't forget everything after every call.

I introduced persistent storage using SQLite so that useful farmer information and interaction data could be retained.

For example:

User
 ├── Name
 ├── Language
 ├── District
 ├── Crop
 └── Previous Interaction

Day 4 — Connect Real-World Tools 🔧

The agent needed access to information outside the LLM.

I added domain tools that allow KisanMitra AI to perform actions such as:

Farmer profile lookup
Profile updates
Weather lookup
Escalation
Call status logging

The core tools are documented around the FarmerTools layer in the project.

Day 5 — Make the Agent Proactive 📞

Instead of always waiting for a farmer to start a conversation, I explored outbound SIP calling.

This opens the possibility of proactive alerts.

For example:

Weather API
     ↓
Severe Weather Detected
     ↓
KisanMitra AI
     ↓
📞 Outbound Voice Call
     ↓
👨‍🌾 Farmer

Day 6 — Human Escalation 🚨

AI shouldn't pretend to know everything.

For complex or serious cases, KisanMitra AI can escalate the issue.

The system can create a unique reference such as:

KISAN-7842

and send the relevant escalation information through a Discord webhook.

This creates a human-in-the-loop workflow.

Day 7 — Analytics Dashboard 📊

Once the agent could handle conversations, I needed to understand how it was performing.

So I built an analytics dashboard using:

Next.js + SQLite

The dashboard can track:

Total calls
Successful calls
Failed calls
Call logs
Session information
Conversation data

Day 8 — Specialist Agent Handoff 🤝

One of the most interesting additions was the Fasal Doctor.

Instead of forcing one agent to handle every agricultural problem:

                    KisanMitra AI
                         │
             ┌───────────┴───────────┐
             │                       │
       General Query          Complex Crop Issue
             │                       │
             ▼                       ▼
         Response              🌾 Fasal Doctor

The specialist-agent concept allows complex crop-related conversations to be routed to a dedicated agent.

Day 9 — Bring Everything Together 🚀

By this stage, KisanMitra AI combined:

Voice + Memory + Tools + Calling + Human Escalation + Analytics + Specialist Handoff

The project had evolved from a basic voice assistant into a much more complete voice-agent architecture.

📂 8. Project Folder Structure

The project follows a monorepo-style structure separating the Python voice-agent backend from the Next.js frontend.

murf-livekit-starter/
│
├── backend/
│   ├── src/
│   │   ├── agent.py
│   │   └── db.py
│   │
│   ├── tests/
│   │   ├── test_agent.py
│   │   ├── test_day5.py
│   │   └── test_day8.py
│   │
│   ├── .env.example
│   ├── .env.local
│   ├── pyproject.toml
│   └── kisan_mitra.db
│
├── frontend/
│   ├── app/
│   │   ├── api/
│   │   │   ├── token/
│   │   │   │   └── route.ts
│   │   │   └── analytics/
│   │   │       └── route.ts
│   │   │
│   │   ├── dashboard/
│   │   │   └── page.tsx
│   │   │
│   │   ├── layout.tsx
│   │   ├── page.tsx
│   │   └── globals.css
│   │
│   ├── components/
│   │   ├── app/
│   │   ├── agents-ui/
│   │   └── ui/
│   │
│   ├── app-config.ts
│   ├── package.json
│   └── .env.local
│
├── start_app.sh
├── start_app.ps1
├── AGENTS.md
├── README.md
└── BLOG_DEV_COMMUNITY.md

This structure follows the backend/frontend separation documented in the source project.

🧠 9. Important Backend Components

`backend/src/agent.py`

This is the core of the voice agent.

It connects:

STT
 ↓
LLM
 ↓
Tools
 ↓
TTS

It also contains the agent instructions and farmer-specific tools.

`backend/src/db.py`

This manages SQLite persistence.

The project uses tables such as:

users
call_logs

The users table can store useful profile information, while call_logs stores session-related information.

`frontend/app/api/token/route.ts`

This API route generates LiveKit access tokens and handles agent dispatch configuration.

This is important because the frontend needs a secure way to establish a LiveKit session.

`frontend/app/dashboard/page.tsx`

This page provides the analytics dashboard.

It displays information such as:

Total calls
Success rate
Active sessions
Call logs
Transcripts

🚧 10. The Developer Struggles

No real project is complete without debugging.

These were two of my biggest lessons.

💥 Struggle #1 — The Silent Agent

On Day 1, everything appeared to be running.

The backend was active.

The frontend connected.

The microphone worked.

But the agent was completely silent.

After hours of debugging, I found the issue.

My .env.local contained:

# ❌ Wrong
AGENT_NAME=

The agent name was empty.

I changed it to:

# ✅ Fixed
AGENT_NAME=my-agent

After that, the correct agent dispatch was triggered and the voice agent finally started speaking.

Lesson learned:

Always check environment variables before assuming the entire voice pipeline is broken.

The project documentation specifically identifies explicit agent routing through AGENT_NAME as part of the LiveKit token/dispatch setup.

💥 Struggle #2 — The Beautiful but Useless Mic Button 😅

I created a beautiful animated microphone button.

It looked amazing.

But when I clicked it...

Nothing happened.

The problem?

I had designed the UI but hadn't connected the button to the actual LiveKit session lifecycle.

The fix was connecting it with useSessionContext():

import { useSessionContext } from '@livekit/components-react';

export function ActiveMicButton() {
  const session = useSessionContext();

  const handleToggle = async () => {
    if (session.state !== 'connected') {
      await session.start();
    } else {
      await session.end();
    }
  };

  return (
    <button onClick={handleToggle}>
      Start Conversation
    </button>
  );
}

Once connected to:

session.start()

the button actually started the voice session.

Lesson learned:

A beautiful UI is useless if it isn't connected to the actual application logic.

🛠️ 11. How You Can Build Your Own Voice Agent

If you're new to voice AI, start with four major components.

┌──────────────┐
│     STT      │
│  Deepgram    │
└──────┬───────┘
       ↓
┌──────────────┐
│     LLM      │
│ Gemini / GPT │
└──────┬───────┘
       ↓
┌──────────────┐
│     TTS      │
│ Murf Falcon  │
└──────┬───────┘
       ↓
┌──────────────┐
│  LiveKit     │
│  WebRTC      │
└──────────────┘

1. Speech-to-Text

Convert the user's voice into text.

Example: Deepgram

2. LLM

The brain of the agent.

Example: Google Gemini

3. Text-to-Speech

Convert the generated answer back into speech.

Example: Murf Falcon

4. Real-Time Transport

Move audio between the user and agent.

Example: LiveKit

The project uses this four-part STT → LLM → TTS → real-time transport architecture.

🔐 12. Security: Don't Leak Your API Keys

This is extremely important when building AI applications.

Never commit:

.env
.env.local

to GitHub.

Instead:

.env.example

should contain only the required variable names.

For example:

MURF_API_KEY=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
DEEPGRAM_API_KEY=
GOOGLE_API_KEY=

And the real values should remain in your local environment.

🚨 Never publish real API keys in a public GitHub repository.

🔮 13. What's Next for KisanMitra AI?

KisanMitra AI is only the beginning.

📱 WhatsApp Voice Notes

The next step is making the assistant accessible through WhatsApp voice messages.

A farmer could simply send:

🎙️ Voice Note

and receive:

🔊 Voice Answer

🗣️ Regional Indian Languages

I want to expand beyond Hindi/Hinglish into languages and dialects such as:

Bhojpuri
Maithili
Haryanvi
Bundelkhandi
Marathi

The long-term goal is to make voice AI feel natural across different parts of India.

📸 Multimodal Crop Diagnosis

Imagine combining:

📸 Crop Image
     +
🎙️ Farmer's Voice
     +
🧠 AI

A farmer could show an infected leaf while explaining the symptoms verbally.

🛰️ Smarter Agricultural Data

Another future direction is integrating richer agricultural and environmental data sources to make recommendations more context-aware.

📚 14. What I Learned From This Challenge

The biggest lesson from these 10 days is:

Building a voice agent is much more than connecting an LLM to a microphone.

A useful voice agent needs:

🎙️ Natural voice interaction
🧠 Memory
🔧 Real-world tools
📞 Communication capabilities
🚨 Human fallback
📊 Analytics
🤝 Specialist agents
🎨 Good UX
🔐 Secure configuration

But above everything else:

It needs to solve a real problem.

For KisanMitra AI, that problem is making agricultural assistance easier to access through voice.

🔗 15. Project Links

💻 GitHub Repository

👉 [PASTE YOUR ACTUAL GITHUB LINK HERE]

🌐 Live Demo

👉 [PASTE YOUR ACTUAL LIVE DEMO LINK HERE]

🎥 Demo Video

👉 [PASTE YOUR YOUTUBE / LOOM VIDEO LINK HERE]

Don't forget to replace all three placeholders before publishing.

🙏 16. Final Thoughts

Building KisanMitra AI has been one of my most rewarding experiences with voice AI.

In these 10 days, I went from a basic voice interaction to building a system combining:

🎙️ Voice

↓

🧠 Memory

↓

🔧 Tools

↓

📞 Outbound Calls

↓

🚨 Human Escalation

↓

📊 Analytics

↓

🤝 Specialist Agent Handoff

This challenge taught me that the most exciting part of AI isn't just making models smarter.

It's making technology easier for people to use.

For a farmer, technology shouldn't feel complicated.

Sometimes the best interface is simply:

"भैया, मेरी फसल में ये समस्या हो रही है..."

And the AI should be ready to listen.

🌾 That's the idea behind KisanMitra AI.

❤️ Special Thanks

A huge thank you to Murf AI and LiveKit for organizing the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

The challenge gave me an opportunity to learn, experiment, break things, debug them, and build a voice AI system around a real-world Indian use case.

If you're also building with voice AI, I'd love to hear what you're creating.

What kind of voice agent are you building? 👇

Let's connect and build the future of voice AI together.

#10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AI #GenerativeAI #IndianAI #Agriculture #KisanMitraAI #LiveKit

DEV Community: Rishabh Kumar

Building KisanMitra AI: A Full-Stack Voice Agent for Indian Farmers 🌾🎙️

🇮🇳 1. Why Voice for Indian Farmers?

🏗️ 2. End-to-End System Architecture

🔄 3. How a Voice Conversation Works

Step 1 — Farmer speaks

Step 2 — LiveKit transports audio

Step 3 — Voice Activity Detection

Step 4 — Speech-to-Text

Step 5 — Gemini processes the request

Step 6 — Murf Falcon generates the voice

Step 7 — Farmer hears the response

⚡ 4. Why Low Latency Matters

🎨 5. Building the Frontend

🌿 Main UI Features

Glassmorphism

🎙️ Listening State

🌱 Thinking State

🔊 Speaking State

💬 Live Transcript

📦 6. Six Agricultural Modules

🚀 7. The 9-Day Technical Journey

Day 1 — Give the Agent a Voice 🎙️

Day 2 — Build the User Interface 🎨

Day 3 — Add Memory 🧠

Day 4 — Connect Real-World Tools 🔧

Day 5 — Make the Agent Proactive 📞

Day 6 — Human Escalation 🚨

Day 7 — Analytics Dashboard 📊

Day 8 — Specialist Agent Handoff 🤝

Day 9 — Bring Everything Together 🚀

📂 8. Project Folder Structure

🧠 9. Important Backend Components

backend/src/agent.py

backend/src/db.py

frontend/app/api/token/route.ts

frontend/app/dashboard/page.tsx

🚧 10. The Developer Struggles

💥 Struggle #1 — The Silent Agent

Lesson learned:

💥 Struggle #2 — The Beautiful but Useless Mic Button 😅

Lesson learned:

🛠️ 11. How You Can Build Your Own Voice Agent

1. Speech-to-Text

2. LLM

3. Text-to-Speech

4. Real-Time Transport

🔐 12. Security: Don't Leak Your API Keys

🔮 13. What's Next for KisanMitra AI?

📱 WhatsApp Voice Notes

🗣️ Regional Indian Languages

📸 Multimodal Crop Diagnosis

🛰️ Smarter Agricultural Data

📚 14. What I Learned From This Challenge

🔗 15. Project Links

💻 GitHub Repository

🌐 Live Demo

🎥 Demo Video

🙏 16. Final Thoughts

❤️ Special Thanks

`backend/src/agent.py`

`backend/src/db.py`

`frontend/app/api/token/route.ts`

`frontend/app/dashboard/page.tsx`