Day 10 of the **10 Days of Voice Agents โ VoiceForBharat Edition* Challenge | Farm & Field Track*
Imagine holding a smartphone in the middle of a mustard field in rural India.
Your hands are covered in soil. The sun makes the screen difficult to read. You need an answer about your crop, weather, or mandi prices โ but typing a long query or navigating through complicated portals isn't practical.
Now imagine simply pressing a button and saying:
"เคญเฅเคฏเคพ, เคเคฒ เคฌเคพเคฐเคฟเคถ เคนเฅเคเฅ เคเฅเคฏเคพ? เคเคฐ เคงเคพเคจ เคฎเฅเค เคญเฅเคฐเคพ เคงเคฌเฅเคฌเคพ เคฐเฅเค เคฒเคเคพ เคนเฅ, เคเฅเคฏเคพ เคเคฐเฅเค?"
Instead of searching through menus, you can simply talk.
That's the idea behind KisanMitra AI (เคเคฟเคธเคพเคจ เคฎเคฟเคคเฅเคฐ AI) โ a full-stack, real-time bilingual voice agent designed specifically around agricultural assistance.
The project combines:
- ๐๏ธ Murf Falcon TTS
- โก LiveKit Agents
- ๐ Deepgram STT
- ๐ง Google Gemini
- ๐ Next.js 15
- ๐๏ธ SQLite
- ๐ SIP-based calling
- ๐จ Human escalation
- ๐ค Specialist agent handoff
This article is the complete story of how I built KisanMitra AI during the 10 Days of Voice Agents โ VoiceForBharat Edition challenge.
๐ฎ๐ณ 1. Why Voice for Indian Farmers?
Voice assistants are often treated as convenience tools.
But in agriculture, voice can solve a much more practical problem: accessibility.
Farmers may be working outdoors, using their hands, dealing with sunlight, speaking in Hindi or regional languages, and needing information quickly.
The interaction shouldn't look like this:
Open App
โ
Find Menu
โ
Select Category
โ
Fill Form
โ
Type Question
โ
Wait for Result
It should feel more like this:
๐จโ๐พ Farmer
โ
๐๏ธ "เคฎเฅเคฐเฅ เคซเคธเคฒ เคฎเฅเค เคฏเฅ เคธเคฎเคธเฅเคฏเคพ เคนเฅ เคฐเคนเฅ เคนเฅ..."
โ
๐ค KisanMitra AI
โ
๐ Natural Voice Response
For the Farm & Field track, my objective was simple:
Make agricultural assistance as conversational and accessible as possible.
๐๏ธ 2. End-to-End System Architecture
KisanMitra AI connects a real-time voice pipeline with AI reasoning, tools, memory, analytics, telephony and human escalation.
๐จโ๐พ FARMER
โ
โโโโโโโโโโโโโดโโโโโโโโโโโโ
โ โ
๐ Web Browser ๐ Phone / SIP
โ โ
โโโโโโโโโโโโโฌโโโโโโโโโโโโ
โผ
โก LIVEKIT
Real-Time Audio
โ
โผ
๐๏ธ Silero VAD
Turn Detection
โ
โผ
๐ Deepgram STT
Hindi / Hinglish
โ
โผ
๐ง KISANMITRA AGENT
โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโ
โผ โผ โผ
Gemini Memory Tools
โ โ โ
โ โผ โโโ Weather
โ SQLite โโโ Farmer Profile
โ โโโ Escalation
โ
โผ
Decision
โ
โโโโโโโโโดโโโโโโโโโ
โผ โผ
Normal Answer Complex Problem
โ โ
โ โผ
โ ๐พ FASAL DOCTOR
โ Specialist Agent
โ
โโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โผ
๐ MURF FALCON
Text-to-Speech
โ
โผ
LIVEKIT
โ
โผ
๐จโ๐พ FARMER HEARS
The architecture and integration layers are based on the project structure documented during the build.
๐ 3. How a Voice Conversation Works
Let's follow one conversation from beginning to end.
Step 1 โ Farmer speaks
The farmer speaks through the browser microphone or a supported phone/SIP connection.
Step 2 โ LiveKit transports audio
LiveKit handles the real-time audio communication.
Step 3 โ Voice Activity Detection
Silero VAD detects when the farmer starts and stops speaking.
Step 4 โ Speech-to-Text
Deepgram Nova-3 STT converts the audio into text.
It is designed to handle Hindi/Hinglish conversations and code-switching.
For example:
"เคธเคฐเคธเฅเค เคเคพ Mandi Bhav เคเฅเคฏเคพ เคนเฅ?"
Step 5 โ Gemini processes the request
Google Gemini receives the transcript and relevant context.
The agent decides whether it needs to:
- Answer directly
- Access memory
- Call a tool
- Escalate the issue
- Hand the conversation to a specialist
Step 6 โ Murf Falcon generates the voice
The generated response is sent to Murf Falcon TTS.
The response is then streamed back as audio.
Step 7 โ Farmer hears the response
The audio travels through LiveKit back to the user.
So the complete pipeline is:
๐๏ธ Speech
โ
LiveKit
โ
Silero VAD
โ
Deepgram STT
โ
Google Gemini
โ
Tools / Memory
โ
Murf Falcon
โ
LiveKit
โ
๐ Speech
โก 4. Why Low Latency Matters
Voice conversations feel very different from text conversations.
If an AI takes several seconds before responding, users may think:
"Call cut gaya kya?"
That's why latency became an important part of this project.
With streaming TTS, the project targeted approximately 114โ135 ms TTFB for the Murf Falcon voice layer in the tested setup.
The goal was to make the response feel immediate rather than waiting for the entire response to be generated first.
๐จ 5. Building the Frontend
I didn't want KisanMitra AI to look like a basic chatbot.
I built a custom Next.js 15 + Tailwind CSS + shadcn/ui interface with an agriculture-inspired emerald/green theme.
๐ฟ Main UI Features
Glassmorphism
Frosted glass cards and layered panels give the interface a modern look.
๐๏ธ Listening State
The UI visually indicates when the agent is listening.
๐ฑ Thinking State
A separate state indicates that the AI is processing the request.
๐ Speaking State
Audio visualizations show when the agent is responding.
๐ฌ Live Transcript
The user can see the conversation while speaking with the agent.
๐ฆ 6. Six Agricultural Modules
The main dashboard contains six smart agricultural modules.
| Module | Purpose |
|---|---|
| โ Weather Forecast | Weather, rain and temperature information |
| ๐ฑ Crop Advisory | Crop and farming guidance |
| ๐ Mandi Bhav | Market-related information |
| ๐ Disease Control | Crop disease and pest assistance |
| ๐ Livestock Care | Animal and dairy-related assistance |
| ๐๏ธ Government Schemes | Agricultural scheme information |
Example voice prompts:
"เคเฅเคฏเคพ เคเคฒ เคฎเฅเคฐเฅ เคเฅเคค เคฎเฅเค เคฌเคพเคฐเคฟเคถ เคนเฅเคเฅ?"
"เคเฅเคนเฅเค เคเฅ เคชเคนเคฒเฅ เคธเคฟเคเคเคพเค เคเคฌ เคเคฐเคจเฅ เคเคพเคนเคฟเค?"
"เคงเคพเคจ เคเฅ เคชเคคเฅเคคเคฟเคฏเฅเค เคฎเฅเค เคงเคฌเฅเคฌเฅ เค เคเค เคนเฅเค, เคเฅเคฏเคพ เคเคฐเฅเค?"
"เคเค เคฎเคเคกเฅ เคฎเฅเค เคธเคฐเคธเฅเค เคเคพ เคญเคพเคต เคเฅเคฏเคพ เคนเฅ?"
These modules are part of the frontend experience documented in the project.
๐ 7. The 9-Day Technical Journey
KisanMitra AI wasn't built in one day.
It evolved throughout the challenge.
Day 1 โ Give the Agent a Voice ๐๏ธ
The first step was creating the basic real-time voice agent.
I connected the voice pipeline and integrated Murf Falcon TTS.
This transformed the project from a text-based concept into a conversational assistant.
Day 2 โ Build the User Interface ๐จ
I moved beyond the backend and created a proper frontend.
The goal was to give users a simple way to:
- Start a conversation
- See the agent state
- View transcripts
- Interact with agricultural modules
Day 3 โ Add Memory ๐ง
A useful assistant shouldn't forget everything after every call.
I introduced persistent storage using SQLite so that useful farmer information and interaction data could be retained.
For example:
User
โโโ Name
โโโ Language
โโโ District
โโโ Crop
โโโ Previous Interaction
Day 4 โ Connect Real-World Tools ๐ง
The agent needed access to information outside the LLM.
I added domain tools that allow KisanMitra AI to perform actions such as:
- Farmer profile lookup
- Profile updates
- Weather lookup
- Escalation
- Call status logging
The core tools are documented around the FarmerTools layer in the project.
Day 5 โ Make the Agent Proactive ๐
Instead of always waiting for a farmer to start a conversation, I explored outbound SIP calling.
This opens the possibility of proactive alerts.
For example:
Weather API
โ
Severe Weather Detected
โ
KisanMitra AI
โ
๐ Outbound Voice Call
โ
๐จโ๐พ Farmer
Day 6 โ Human Escalation ๐จ
AI shouldn't pretend to know everything.
For complex or serious cases, KisanMitra AI can escalate the issue.
The system can create a unique reference such as:
KISAN-7842
and send the relevant escalation information through a Discord webhook.
This creates a human-in-the-loop workflow.
Day 7 โ Analytics Dashboard ๐
Once the agent could handle conversations, I needed to understand how it was performing.
So I built an analytics dashboard using:
Next.js + SQLite
The dashboard can track:
- Total calls
- Successful calls
- Failed calls
- Call logs
- Session information
- Conversation data
Day 8 โ Specialist Agent Handoff ๐ค
One of the most interesting additions was the Fasal Doctor.
Instead of forcing one agent to handle every agricultural problem:
KisanMitra AI
โ
โโโโโโโโโโโโโดโโโโโโโโโโโโ
โ โ
General Query Complex Crop Issue
โ โ
โผ โผ
Response ๐พ Fasal Doctor
The specialist-agent concept allows complex crop-related conversations to be routed to a dedicated agent.
Day 9 โ Bring Everything Together ๐
By this stage, KisanMitra AI combined:
Voice + Memory + Tools + Calling + Human Escalation + Analytics + Specialist Handoff
The project had evolved from a basic voice assistant into a much more complete voice-agent architecture.
๐ 8. Project Folder Structure
The project follows a monorepo-style structure separating the Python voice-agent backend from the Next.js frontend.
murf-livekit-starter/
โ
โโโ backend/
โ โโโ src/
โ โ โโโ agent.py
โ โ โโโ db.py
โ โ
โ โโโ tests/
โ โ โโโ test_agent.py
โ โ โโโ test_day5.py
โ โ โโโ test_day8.py
โ โ
โ โโโ .env.example
โ โโโ .env.local
โ โโโ pyproject.toml
โ โโโ kisan_mitra.db
โ
โโโ frontend/
โ โโโ app/
โ โ โโโ api/
โ โ โ โโโ token/
โ โ โ โ โโโ route.ts
โ โ โ โโโ analytics/
โ โ โ โโโ route.ts
โ โ โ
โ โ โโโ dashboard/
โ โ โ โโโ page.tsx
โ โ โ
โ โ โโโ layout.tsx
โ โ โโโ page.tsx
โ โ โโโ globals.css
โ โ
โ โโโ components/
โ โ โโโ app/
โ โ โโโ agents-ui/
โ โ โโโ ui/
โ โ
โ โโโ app-config.ts
โ โโโ package.json
โ โโโ .env.local
โ
โโโ start_app.sh
โโโ start_app.ps1
โโโ AGENTS.md
โโโ README.md
โโโ BLOG_DEV_COMMUNITY.md
This structure follows the backend/frontend separation documented in the source project.
๐ง 9. Important Backend Components
backend/src/agent.py
This is the core of the voice agent.
It connects:
STT
โ
LLM
โ
Tools
โ
TTS
It also contains the agent instructions and farmer-specific tools.
backend/src/db.py
This manages SQLite persistence.
The project uses tables such as:
users
call_logs
The users table can store useful profile information, while call_logs stores session-related information.
frontend/app/api/token/route.ts
This API route generates LiveKit access tokens and handles agent dispatch configuration.
This is important because the frontend needs a secure way to establish a LiveKit session.
frontend/app/dashboard/page.tsx
This page provides the analytics dashboard.
It displays information such as:
- Total calls
- Success rate
- Active sessions
- Call logs
- Transcripts
๐ง 10. The Developer Struggles
No real project is complete without debugging.
These were two of my biggest lessons.
๐ฅ Struggle #1 โ The Silent Agent
On Day 1, everything appeared to be running.
The backend was active.
The frontend connected.
The microphone worked.
But the agent was completely silent.
After hours of debugging, I found the issue.
My .env.local contained:
# โ Wrong
AGENT_NAME=
The agent name was empty.
I changed it to:
# โ
Fixed
AGENT_NAME=my-agent
After that, the correct agent dispatch was triggered and the voice agent finally started speaking.
Lesson learned:
Always check environment variables before assuming the entire voice pipeline is broken.
The project documentation specifically identifies explicit agent routing through AGENT_NAME as part of the LiveKit token/dispatch setup.
๐ฅ Struggle #2 โ The Beautiful but Useless Mic Button ๐
I created a beautiful animated microphone button.
It looked amazing.
But when I clicked it...
Nothing happened.
The problem?
I had designed the UI but hadn't connected the button to the actual LiveKit session lifecycle.
The fix was connecting it with useSessionContext():
import { useSessionContext } from '@livekit/components-react';
export function ActiveMicButton() {
const session = useSessionContext();
const handleToggle = async () => {
if (session.state !== 'connected') {
await session.start();
} else {
await session.end();
}
};
return (
<button onClick={handleToggle}>
Start Conversation
</button>
);
}
Once connected to:
session.start()
the button actually started the voice session.
Lesson learned:
A beautiful UI is useless if it isn't connected to the actual application logic.
๐ ๏ธ 11. How You Can Build Your Own Voice Agent
If you're new to voice AI, start with four major components.
โโโโโโโโโโโโโโโโ
โ STT โ
โ Deepgram โ
โโโโโโโโฌโโโโโโโโ
โ
โโโโโโโโโโโโโโโโ
โ LLM โ
โ Gemini / GPT โ
โโโโโโโโฌโโโโโโโโ
โ
โโโโโโโโโโโโโโโโ
โ TTS โ
โ Murf Falcon โ
โโโโโโโโฌโโโโโโโโ
โ
โโโโโโโโโโโโโโโโ
โ LiveKit โ
โ WebRTC โ
โโโโโโโโโโโโโโโโ
1. Speech-to-Text
Convert the user's voice into text.
Example: Deepgram
2. LLM
The brain of the agent.
Example: Google Gemini
3. Text-to-Speech
Convert the generated answer back into speech.
Example: Murf Falcon
4. Real-Time Transport
Move audio between the user and agent.
Example: LiveKit
The project uses this four-part STT โ LLM โ TTS โ real-time transport architecture.
๐ 12. Security: Don't Leak Your API Keys
This is extremely important when building AI applications.
Never commit:
.env
.env.local
to GitHub.
Instead:
.env.example
should contain only the required variable names.
For example:
MURF_API_KEY=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
DEEPGRAM_API_KEY=
GOOGLE_API_KEY=
And the real values should remain in your local environment.
๐จ Never publish real API keys in a public GitHub repository.
๐ฎ 13. What's Next for KisanMitra AI?
KisanMitra AI is only the beginning.
๐ฑ WhatsApp Voice Notes
The next step is making the assistant accessible through WhatsApp voice messages.
A farmer could simply send:
๐๏ธ Voice Note
and receive:
๐ Voice Answer
๐ฃ๏ธ Regional Indian Languages
I want to expand beyond Hindi/Hinglish into languages and dialects such as:
- Bhojpuri
- Maithili
- Haryanvi
- Bundelkhandi
- Marathi
The long-term goal is to make voice AI feel natural across different parts of India.
๐ธ Multimodal Crop Diagnosis
Imagine combining:
๐ธ Crop Image
+
๐๏ธ Farmer's Voice
+
๐ง AI
A farmer could show an infected leaf while explaining the symptoms verbally.
๐ฐ๏ธ Smarter Agricultural Data
Another future direction is integrating richer agricultural and environmental data sources to make recommendations more context-aware.
๐ 14. What I Learned From This Challenge
The biggest lesson from these 10 days is:
Building a voice agent is much more than connecting an LLM to a microphone.
A useful voice agent needs:
- ๐๏ธ Natural voice interaction
- ๐ง Memory
- ๐ง Real-world tools
- ๐ Communication capabilities
- ๐จ Human fallback
- ๐ Analytics
- ๐ค Specialist agents
- ๐จ Good UX
- ๐ Secure configuration
But above everything else:
It needs to solve a real problem.
For KisanMitra AI, that problem is making agricultural assistance easier to access through voice.
๐ 15. Project Links
๐ป GitHub Repository
๐ [PASTE YOUR ACTUAL GITHUB LINK HERE]
๐ Live Demo
๐ [PASTE YOUR ACTUAL LIVE DEMO LINK HERE]
๐ฅ Demo Video
๐ [PASTE YOUR YOUTUBE / LOOM VIDEO LINK HERE]
Don't forget to replace all three placeholders before publishing.
๐ 16. Final Thoughts
Building KisanMitra AI has been one of my most rewarding experiences with voice AI.
In these 10 days, I went from a basic voice interaction to building a system combining:
๐๏ธ Voice
โ
๐ง Memory
โ
๐ง Tools
โ
๐ Outbound Calls
โ
๐จ Human Escalation
โ
๐ Analytics
โ
๐ค Specialist Agent Handoff
This challenge taught me that the most exciting part of AI isn't just making models smarter.
It's making technology easier for people to use.
For a farmer, technology shouldn't feel complicated.
Sometimes the best interface is simply:
"เคญเฅเคฏเคพ, เคฎเฅเคฐเฅ เคซเคธเคฒ เคฎเฅเค เคฏเฅ เคธเคฎเคธเฅเคฏเคพ เคนเฅ เคฐเคนเฅ เคนเฅ..."
And the AI should be ready to listen.
๐พ That's the idea behind KisanMitra AI.
โค๏ธ Special Thanks
A huge thank you to Murf AI and LiveKit for organizing the 10 Days of Voice Agents โ VoiceForBharat Edition challenge.
The challenge gave me an opportunity to learn, experiment, break things, debug them, and build a voice AI system around a real-world Indian use case.
If you're also building with voice AI, I'd love to hear what you're creating.
What kind of voice agent are you building? ๐
Let's connect and build the future of voice AI together.
#10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AI #GenerativeAI #IndianAI #Agriculture #KisanMitraAI #LiveKit
Top comments (0)