Building a voice agent is easy to describe. Building one that can remember, use real-world tools, make calls, involve humans, and hand off to specialists is a very different engineering problem.
Introduction
What if a farmer could simply speak to an AI assistant instead of navigating through multiple applications, menus, and forms?
That was the idea behind Krishi Mitra – AI Voice Assistant for Indian Farmers, which I built as part of the 10 Days of Voice Agents – VoiceForBharat Edition.
I chose the Farm & Field track because agriculture is a domain where timely information can be valuable. Farmers may need information about weather, rainfall, crops, agricultural guidance, government schemes, market information, or a specific farming problem.
A voice interface makes these interactions more natural.
Instead of typing:
"What is the weather forecast for tomorrow?"
a farmer can simply ask:
"कल बारिश की संभावना कितनी है?"
And the answer can come back as voice.
But I quickly realized that making an AI speak was only the beginning.
The interesting engineering work was everything around the conversation:
memory, tools, guardrails, outbound calls, human escalation, analytics, and specialist agents.
That became the real story of Krishi Mitra.
What Is Krishi Mitra?
Krishi Mitra is a voice-based agricultural assistant designed to help Indian farmers through natural voice conversations.
It can assist with topics such as:
- Crop information
- Weather
- Rain alerts
- Agricultural guidance
- Government schemes
- Market information
- Farmer-specific assistance
The system supports Hindi and English interaction and is designed to handle code-mixed conversations as well.
The goal is not to replace agricultural experts.
Instead, Krishi Mitra is designed to provide useful assistance while knowing when it should use a tool, when it should use a specialist, and when it should involve a human expert.
The Architecture
At its core, Krishi Mitra follows a standard real-time voice-agent pipeline:
The main technologies are:
| Layer | Technology | Role |
|---|---|---|
| Real-time voice | LiveKit Agents | Handles the voice session and agent execution |
| Speech-to-text | Deepgram | Converts farmer speech into text |
| LLM | Google Gemini | Understands requests and generates responses |
| Text-to-speech | Murf Falcon | Converts responses into natural voice |
| Memory | SQLite | Stores persistent farmer information |
| Weather tool | Open-Meteo | Provides live weather information |
| Dashboard | Streamlit | Provides analytics and human-help monitoring |
| Telephony | SIP / Linphone | Enables outbound voice calls |
The important point is that the LLM is only one part of the system.
A useful voice agent is a combination of speech, reasoning, memory, tools, communication infrastructure, and application logic.
Understanding the Voice Pipeline
1. Speech-to-Text
The first step is converting the farmer's voice into text.
Krishi Mitra uses Deepgram for speech-to-text.
Conceptually:
Farmer speaks
↓
Microphone / Voice Session
↓
Deepgram
↓
Text
For example:
"आज Mumbai का मौसम कैसा है?"
becomes text that the agent can process.
2. LLM — Understanding the Request
The text is passed to Google Gemini.
Gemini determines what the farmer is asking and what should happen next.
For a simple conversation, it can generate a response directly.
For a weather question, however, the system should not simply generate an answer from the model.
It needs a real tool.
That is where tool calling becomes important.
3. Text-to-Speech
Once the response is generated, Murf Falcon converts the text into voice.
This was particularly important for the Farm & Field use case because the interaction should feel like a natural conversation rather than a text chatbot being read aloud.
The overall flow becomes:
Farmer Voice
↓
Deepgram
↓
Google Gemini
↓
Murf Falcon
↓
Farmer Voice
4. Real-Time Transport
Voice applications are different from normal request-response applications.
The user expects to speak and receive a response naturally.
Krishi Mitra uses LiveKit Agents as the real-time voice infrastructure connecting the conversation components.
This provides the foundation for browser-based voice conversations and later enabled the project to explore outbound voice calls.
Giving the Agent a Job and Guardrails
One of the first lessons I learned was that an AI voice agent needs more than an LLM.
It needs a clearly defined identity and responsibility.
Krishi Mitra was given:
- A clear agricultural identity
- Specific objectives
- Agricultural responsibilities
- Conversational personality
- Language behavior
- Code-mixing behavior
- Safety guardrails
- Escalation behavior
This matters because an agricultural assistant should not confidently answer every possible question.
For example, if a farmer asks for current market information and the system does not have a verified current source, it should not simply invent a price.
The principle is:
If current information cannot be verified, don't guess.
This became an important design principle throughout the project.
Giving Krishi Mitra Memory
A normal chatbot can forget the conversation once the session ends.
For a farmer assistant, that can become frustrating.
Imagine telling an assistant:
"मैं Jaipur में गेहूं उगाता हूँ और मेरे पास 5 acres जमीन है।"
Then having to repeat the same information every time.
I added persistent memory using SQLite.
The farmer profile can contain:
user_id
name
language_preference
crops_grown
land_size
district
irrigation_type
last_interaction
The basic flow is:
First Conversation
↓
Farmer provides information
↓
Permission to remember
↓
SQLite Database
↓
Future Conversation
↓
Retrieve Farmer Context
For example, in a later conversation, Krishi Mitra can recognize that the farmer previously mentioned their district and crop.
This makes the system feel less like a collection of disconnected conversations and more like a continuing assistant.
Connecting the Agent to Real-World Data
This was one of the most important technical improvements in Krishi Mitra.
Consider the question:
"आज Mumbai का मौसम कैसा है?"
An LLM may generate a plausible-sounding response, but plausibility is not the same as current information.
Weather is dynamic.
So I connected Krishi Mitra to a real weather service using Open-Meteo.
The flow is:
Farmer asks weather question
↓
Google Gemini
↓
Weather Function Tool
↓
Open-Meteo
↓
Weather Result
↓
Google Gemini
↓
Murf Falcon
↓
Voice Response
The weather tool can retrieve information such as:
- Location
- Temperature
- Humidity
- Rainfall
- Wind speed
- Latest available weather data time
The raw API response is then converted into a natural response for the farmer.
For example, instead of exposing JSON, the agent can communicate the information naturally through voice.
Tool Failure Is Also Part of the Design
What happens if the weather service is unavailable?
This was an important part of the implementation.
The correct behavior is not:
"I think it will rain today."
The correct behavior is to communicate that the weather service is temporarily unavailable.
This leads to a broader lesson:
An AI system should fail safely rather than confidently inventing information.
Tool failure handling is therefore not an optional extra.
It is part of the agent architecture.
From Reactive AI to Proactive AI
Another major capability I explored was outbound calling.
A normal voice assistant waits for the farmer to start the conversation.
But some agricultural situations are naturally proactive.
For example, if a farmer needs an important weather or rain warning, the system can initiate a call.
Krishi Mitra's outbound setup involved:
- SQLite farmer database
- LiveKit Agents
- LiveKit room and dispatch
- Linphone SIP
- Open-Meteo
- Murf AI TTS
- Hindi and English interaction
The conceptual flow is:
Farmer Profile
↓
Weather Information
↓
Relevant Alert
↓
Outbound Voice Call
↓
Farmer
The interesting part is that this is not just a prerecorded announcement.
The farmer can continue the conversation.
For example:
"कल बारिश की संभावना कितनी है?"
Krishi Mitra can retrieve the forecast and respond.
The farmer can also ask the system to end the call.
That changes the design from a simple notification system into an interactive voice agent.
When AI Should Ask a Human for Help
One of the most important lessons from building Krishi Mitra was understanding when not to let the AI continue autonomously.
Suppose a farmer reports a serious crop disease or crop-damage problem.
Giving an uncertain diagnosis could be worse than admitting that specialist help is needed.
So Krishi Mitra can use a human escalation workflow.
Farmer reports serious crop problem
↓
AI recognizes need for human help
↓
AI asks for permission
↓
Escalation request created
↓
Reference ID generated
↓
Human expert sees the request
Permission is important.
The system asks the farmer before sharing the relevant information for escalation.
The escalation can contain useful context such as:
- Farmer
- District
- Crop
- Language preference
- Reason for escalation
- Urgency
- What Krishi Mitra already checked
This information can then be presented through the Human Help Dashboard.
The principle is simple:
A good AI agent should know when to stop acting autonomously and involve a human.
Moving From One Agent to a Specialist
The next architectural step was introducing a Crop Problem Specialist.
Instead of making the main agent responsible for every agricultural problem, a specialist can focus on crop-health-related conversations.
The flow becomes:
Farmer
↓
Krishi Mitra
↓
Crop Problem Specialist
↓
Specialized Assistance
↓
Human Escalation if Required
An important part of this design is context preservation.
The specialist receives the existing conversation context.
The farmer therefore does not need to repeat the entire problem from the beginning.
This is one of the reasons I find multi-agent architectures interesting.
Rather than building one enormous agent that tries to handle everything, specialized agents can focus on specific responsibilities.
Observability: Building a Call Analytics Dashboard
Once an agent starts handling real conversations, another question becomes important:
What is happening inside the system?
I created a Streamlit-based monitoring dashboard for Krishi Mitra.
It provides visibility into:
- Total calls
- Successful calls
- Failed calls
- Success rate
- Browser calls
- SIP outbound calls
- Human-help requests
- Priority counts
- Detailed escalation information
- Refresh functionality
The system tracks browser and SIP calls separately.
This gives a clearer view of how the voice agent is being used across different communication channels.
The dashboard also connects the voice-agent workflow with human support.
That means monitoring is not limited to:
"How many calls happened?"
It can also show:
"Which conversations require human attention?"
I have intentionally not included fabricated numerical performance results here. The dashboard supports these metrics, but the project documentation does not provide a verified benchmark that should be presented as a result.
The Most Difficult Technical Parts
Building the project was not a straight line from idea to finished system.
Several parts required thinking about the system beyond the LLM.
Challenge 1: Current Data vs. Generated Answers
Problem
The agent needed to answer weather questions.
Cause
An LLM is not a live weather database.
Solution
I implemented a weather function tool connected to Open-Meteo.
Lesson
Use external tools when the information changes in the real world.
Challenge 2: External Service Failure
Problem
The weather service can become unavailable.
Solution
The tool provides a failure response that the agent can communicate naturally instead of generating a made-up weather answer.
Lesson
Graceful failure is part of reliable AI design.
Challenge 3: Conversation Continuity
Problem
A returning farmer should not have to repeat basic information.
Solution
I introduced persistent farmer memory using SQLite.
Lesson
Memory can make a conversational system significantly more useful.
Challenge 4: Knowing When to Escalate
Problem
The AI should not attempt to confidently solve every serious crop problem.
Solution
I implemented a permission-based human-help workflow and later connected it with the specialist-agent flow.
Lesson
Human-in-the-loop is not a failure of AI. It can be an important part of responsible agent design.
How You Can Build a Voice Agent
If you want to build a voice agent yourself, start with four fundamental components:
Speech-to-Text
+
LLM
+
Text-to-Speech
+
Real-Time Transport
For Krishi Mitra, these became:
Farmer Speech
↓
Deepgram
↓
Google Gemini
↓
Tools / Memory / Specialist
↓
Murf Falcon
↓
Farmer Voice
Then build incrementally.
A practical sequence is:
- Build the basic voice conversation.
- Give the agent a clear role.
- Add personality and guardrails.
- Create the user interface.
- Add persistent memory.
- Connect one real-world tool.
- Implement tool failure handling.
- Add outbound calling if required.
- Add human escalation.
- Add monitoring.
- Introduce specialist agents where they provide clear value.
This approach makes debugging much easier than trying to build the entire system at once.
Setting Up the Project
The exact setup should always follow the commands in the project's repository.
The general process is:
1. Clone the repository
2. Create a Python environment
3. Install dependencies
4. Configure environment variables
5. Add API keys
6. Start the agent
7. Start/connect to the voice interface
8. Test a basic conversation
9. Test the weather tool
10. Test memory
GitHub Repository:
https://github.com/roma2020-app/krishimitra-voice-agent/
I recommend following the repository README for the exact installation and startup commands rather than copying generic commands from another project.
Protecting API Keys and User Data
A public technical project must never expose secrets.
Keep API keys in environment variables.
For example:
GEMINI_API_KEY=your_key_here
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
The actual values should never be committed to GitHub.
Use:
.env
locally and add it to:
.gitignore
Also avoid publishing:
- API keys
- Production credentials
- Real phone numbers
- Caller data
- Private farmer information
For demonstrations and articles, use anonymized data.
Testing the Agent
A voice agent should be tested through actual conversations.
Test 1 — Normal Conversation
Ask a basic agricultural question and verify that the agent responds naturally.
Test 2 — Weather
Ask:
"आज Mumbai का मौसम कैसा है?"
Verify that the agent uses the weather tool rather than inventing current weather.
Test 3 — Memory
First conversation:
"मैं Jaipur में गेहूं उगाता हूँ।"
Later conversation:
Verify that the saved farmer context can be retrieved.
Test 4 — Tool Failure
Temporarily make the weather service unavailable and verify that the agent provides a safe fallback.
Test 5 — Human Escalation
Report a serious crop problem and verify that the agent asks for permission before creating the human-help request.
Test 6 — Specialist Handoff
Test whether the Crop Problem Specialist receives the existing conversation context without forcing the farmer to repeat the complete problem.
Evidence From the Build
1. Voice Interface
Krishi Mitra during a farmer voice conversation.
2. Live Weather Tool
Krishi Mitra retrieving current weather information through the weather tool.
3. Outbound Call / Human Escalation
Krishi Mitra moving from proactive voice interaction toward human assistance.
4. Analytics / Specialist Dashboard
Monitoring voice calls and human-help requests.
5. Architecture
Browser Voice Call ──┐
│
├──> Call Analytics Database ──> Streamlit Dashboard
SIP Outbound Call ───┘ │
├── Total Calls
├── SuccessfulCalls
├── Failed Calls
├── Success Rate
├── Browser Metrics
└── SIP Metrics
Voice Agent ──> Human Escalation ──> Escalation Database ──> Human Help Dashboard
│
├── Open Requests
├── High Priority
├── Medium Priority
└── Detailed Farmer Request
What I Learned
Building Krishi Mitra changed how I think about voice agents.
1. A voice agent needs a clear job
An LLM alone does not define an application.
The agent needs objectives, boundaries, tools, and responsibilities.
2. Real-world data needs real-world tools
If information changes frequently, connect the agent to an appropriate source instead of asking the model to guess.
3. Memory changes the experience
A returning user should not always feel like a new user.
4. Guardrails are part of the architecture
Safety should not be added after everything else is finished.
5. AI should know when to ask for help
Human escalation provides a controlled path when the AI should not continue independently.
6. Specialist agents can simplify complex systems
A focused specialist can be more useful than one giant agent responsible for every possible task.
7. Voice agents need observability
Once the agent starts handling conversations, you need visibility into calls, failures, and human-help requirements.
What I Would Build Next
Krishi Mitra is currently a learning/prototype project, so there is plenty of room for improvement.
Some directions I would explore next include:
- More Indian languages
- More agricultural tools
- More verified agricultural information sources
- Additional specialist agents
- Better farmer profile management
- Improved analytics
- More robust telephony
- Production deployment
- Better human expert workflows
These are future directions, not features I am claiming as already completed.
Conclusion
When I started this project, the goal sounded simple:
Make an AI voice agent talk to a farmer.
But building Krishi Mitra taught me that voice is only the beginning.
A useful voice agent needs to:
Listen
↓
Understand
↓
Remember
↓
Use Tools
↓
Act
↓
Escalate
↓
Hand Off
↓
Be Monitored
That was the biggest takeaway from building Krishi Mitra during the 10 Days of Voice Agents – VoiceForBharat Edition.
I learned how speech-to-text, an LLM, text-to-speech, real-time communication, memory, tools, specialist agents, and human workflows can come together to create something much more useful than a simple chatbot.
For me, the most important lesson was this:
AI becomes more useful when it knows not only how to answer, but also what to remember, what to verify, what to do, and when to ask a human for help.
Krishi Mitra is still evolving, but this project gave me a much deeper understanding of how to design and build voice-first AI systems.
Thanks to Murf AI for the 10 Days of Voice Agents – VoiceForBharat Edition challenge and for the opportunity to build with Murf Falcon.
🌾🎙️🤖
Build. Learn. Experiment. Share.
Project Links
GitHub Repository:
https://github.com/roma2020-app/krishimitra-voice-agent
Suggested DEV Community Tags
#ai #voiceai #generativeai #agenticai #python #agritech #livekit #gemini #murfai #voiceforbharat
Top comments (0)