Building Krishi Mitra: An AI Voice Agent for Indian Farmers with LiveKit, Gemini and Murf Falcon 🌾🎙️

Roma Gupta — Sat, 15 Aug 2026 00:50:09 +0000

Building a voice agent is easy to describe. Building one that can remember, use real-world tools, make calls, involve humans, and hand off to specialists is a very different engineering problem.

Introduction

What if a farmer could simply speak to an AI assistant instead of navigating through multiple applications, menus, and forms?

That was the idea behind Krishi Mitra – AI Voice Assistant for Indian Farmers, which I built as part of the 10 Days of Voice Agents – VoiceForBharat Edition.

I chose the Farm & Field track because agriculture is a domain where timely information can be valuable. Farmers may need information about weather, rainfall, crops, agricultural guidance, government schemes, market information, or a specific farming problem.

A voice interface makes these interactions more natural.

Instead of typing:

"What is the weather forecast for tomorrow?"

a farmer can simply ask:

"कल बारिश की संभावना कितनी है?"

And the answer can come back as voice.

But I quickly realized that making an AI speak was only the beginning.

The interesting engineering work was everything around the conversation:

memory, tools, guardrails, outbound calls, human escalation, analytics, and specialist agents.

That became the real story of Krishi Mitra.

What Is Krishi Mitra?

Krishi Mitra is a voice-based agricultural assistant designed to help Indian farmers through natural voice conversations.

It can assist with topics such as:

Crop information
Weather
Rain alerts
Agricultural guidance
Government schemes
Market information
Farmer-specific assistance

The system supports Hindi and English interaction and is designed to handle code-mixed conversations as well.

The goal is not to replace agricultural experts.

Instead, Krishi Mitra is designed to provide useful assistance while knowing when it should use a tool, when it should use a specialist, and when it should involve a human expert.

The Architecture

At its core, Krishi Mitra follows a standard real-time voice-agent pipeline:

The main technologies are:

Layer	Technology	Role
Real-time voice	LiveKit Agents	Handles the voice session and agent execution
Speech-to-text	Deepgram	Converts farmer speech into text
LLM	Google Gemini	Understands requests and generates responses
Text-to-speech	Murf Falcon	Converts responses into natural voice
Memory	SQLite	Stores persistent farmer information
Weather tool	Open-Meteo	Provides live weather information
Dashboard	Streamlit	Provides analytics and human-help monitoring
Telephony	SIP / Linphone	Enables outbound voice calls

The important point is that the LLM is only one part of the system.

A useful voice agent is a combination of speech, reasoning, memory, tools, communication infrastructure, and application logic.

Understanding the Voice Pipeline

1. Speech-to-Text

The first step is converting the farmer's voice into text.

Krishi Mitra uses Deepgram for speech-to-text.

Conceptually:

Farmer speaks
     ↓
Microphone / Voice Session
     ↓
Deepgram
     ↓
   Text

For example:

"आज Mumbai का मौसम कैसा है?"

becomes text that the agent can process.

2. LLM — Understanding the Request

The text is passed to Google Gemini.

Gemini determines what the farmer is asking and what should happen next.

For a simple conversation, it can generate a response directly.

For a weather question, however, the system should not simply generate an answer from the model.

It needs a real tool.

That is where tool calling becomes important.

3. Text-to-Speech

Once the response is generated, Murf Falcon converts the text into voice.

This was particularly important for the Farm & Field use case because the interaction should feel like a natural conversation rather than a text chatbot being read aloud.

The overall flow becomes:

Farmer Voice
     ↓
Deepgram
     ↓
Google Gemini
     ↓
Murf Falcon
     ↓
Farmer Voice

4. Real-Time Transport

Voice applications are different from normal request-response applications.

The user expects to speak and receive a response naturally.

Krishi Mitra uses LiveKit Agents as the real-time voice infrastructure connecting the conversation components.

This provides the foundation for browser-based voice conversations and later enabled the project to explore outbound voice calls.

Giving the Agent a Job and Guardrails

One of the first lessons I learned was that an AI voice agent needs more than an LLM.

It needs a clearly defined identity and responsibility.

Krishi Mitra was given:

A clear agricultural identity
Specific objectives
Agricultural responsibilities
Conversational personality
Language behavior
Code-mixing behavior
Safety guardrails
Escalation behavior

This matters because an agricultural assistant should not confidently answer every possible question.

For example, if a farmer asks for current market information and the system does not have a verified current source, it should not simply invent a price.

The principle is:

If current information cannot be verified, don't guess.

This became an important design principle throughout the project.

Giving Krishi Mitra Memory

A normal chatbot can forget the conversation once the session ends.

For a farmer assistant, that can become frustrating.

Imagine telling an assistant:

"मैं Jaipur में गेहूं उगाता हूँ और मेरे पास 5 acres जमीन है।"

Then having to repeat the same information every time.

I added persistent memory using SQLite.

The farmer profile can contain:

user_id
name
language_preference
crops_grown
land_size
district
irrigation_type
last_interaction

The basic flow is:

First Conversation
       ↓
Farmer provides information
       ↓
Permission to remember
       ↓
SQLite Database
       ↓
Future Conversation
       ↓
Retrieve Farmer Context

For example, in a later conversation, Krishi Mitra can recognize that the farmer previously mentioned their district and crop.

This makes the system feel less like a collection of disconnected conversations and more like a continuing assistant.

Connecting the Agent to Real-World Data

This was one of the most important technical improvements in Krishi Mitra.

Consider the question:

"आज Mumbai का मौसम कैसा है?"

An LLM may generate a plausible-sounding response, but plausibility is not the same as current information.

Weather is dynamic.

So I connected Krishi Mitra to a real weather service using Open-Meteo.

The flow is:

Farmer asks weather question
          ↓
Google Gemini
          ↓
Weather Function Tool
          ↓
Open-Meteo
          ↓
Weather Result
          ↓
Google Gemini
          ↓
Murf Falcon
          ↓
Voice Response

The weather tool can retrieve information such as:

Location
Temperature
Humidity
Rainfall
Wind speed
Latest available weather data time

The raw API response is then converted into a natural response for the farmer.

For example, instead of exposing JSON, the agent can communicate the information naturally through voice.

Tool Failure Is Also Part of the Design

What happens if the weather service is unavailable?

This was an important part of the implementation.

The correct behavior is not:

"I think it will rain today."

The correct behavior is to communicate that the weather service is temporarily unavailable.

This leads to a broader lesson:

An AI system should fail safely rather than confidently inventing information.

Tool failure handling is therefore not an optional extra.

It is part of the agent architecture.

From Reactive AI to Proactive AI

Another major capability I explored was outbound calling.

A normal voice assistant waits for the farmer to start the conversation.

But some agricultural situations are naturally proactive.

For example, if a farmer needs an important weather or rain warning, the system can initiate a call.

Krishi Mitra's outbound setup involved:

SQLite farmer database
LiveKit Agents
LiveKit room and dispatch
Linphone SIP
Open-Meteo
Murf AI TTS
Hindi and English interaction

The conceptual flow is:

Farmer Profile
      ↓
Weather Information
      ↓
Relevant Alert
      ↓
Outbound Voice Call
      ↓
Farmer

The interesting part is that this is not just a prerecorded announcement.

The farmer can continue the conversation.

For example:

"कल बारिश की संभावना कितनी है?"

Krishi Mitra can retrieve the forecast and respond.

The farmer can also ask the system to end the call.

That changes the design from a simple notification system into an interactive voice agent.

When AI Should Ask a Human for Help

One of the most important lessons from building Krishi Mitra was understanding when not to let the AI continue autonomously.

Suppose a farmer reports a serious crop disease or crop-damage problem.

Giving an uncertain diagnosis could be worse than admitting that specialist help is needed.

So Krishi Mitra can use a human escalation workflow.

Farmer reports serious crop problem
          ↓
AI recognizes need for human help
          ↓
AI asks for permission
          ↓
Escalation request created
          ↓
Reference ID generated
          ↓
Human expert sees the request

Permission is important.

The system asks the farmer before sharing the relevant information for escalation.

The escalation can contain useful context such as:

Farmer
District
Crop
Language preference
Reason for escalation
Urgency
What Krishi Mitra already checked

This information can then be presented through the Human Help Dashboard.

The principle is simple:

A good AI agent should know when to stop acting autonomously and involve a human.

Moving From One Agent to a Specialist

The next architectural step was introducing a Crop Problem Specialist.

Instead of making the main agent responsible for every agricultural problem, a specialist can focus on crop-health-related conversations.

The flow becomes:

Farmer
   ↓
Krishi Mitra
   ↓
Crop Problem Specialist
   ↓
Specialized Assistance
   ↓
Human Escalation if Required

An important part of this design is context preservation.

The specialist receives the existing conversation context.

The farmer therefore does not need to repeat the entire problem from the beginning.

This is one of the reasons I find multi-agent architectures interesting.

Rather than building one enormous agent that tries to handle everything, specialized agents can focus on specific responsibilities.

Observability: Building a Call Analytics Dashboard

Once an agent starts handling real conversations, another question becomes important:

What is happening inside the system?

I created a Streamlit-based monitoring dashboard for Krishi Mitra.

It provides visibility into:

Total calls
Successful calls
Failed calls
Success rate
Browser calls
SIP outbound calls
Human-help requests
Priority counts
Detailed escalation information
Refresh functionality

The system tracks browser and SIP calls separately.

This gives a clearer view of how the voice agent is being used across different communication channels.

The dashboard also connects the voice-agent workflow with human support.

That means monitoring is not limited to:

"How many calls happened?"

It can also show:

"Which conversations require human attention?"

I have intentionally not included fabricated numerical performance results here. The dashboard supports these metrics, but the project documentation does not provide a verified benchmark that should be presented as a result.

The Most Difficult Technical Parts

Building the project was not a straight line from idea to finished system.

Several parts required thinking about the system beyond the LLM.

Challenge 1: Current Data vs. Generated Answers

Problem

The agent needed to answer weather questions.

Cause

An LLM is not a live weather database.

Solution

I implemented a weather function tool connected to Open-Meteo.

Lesson

Use external tools when the information changes in the real world.

Challenge 2: External Service Failure

Problem

The weather service can become unavailable.

Solution

The tool provides a failure response that the agent can communicate naturally instead of generating a made-up weather answer.

Lesson

Graceful failure is part of reliable AI design.

Challenge 3: Conversation Continuity

Problem

A returning farmer should not have to repeat basic information.

Solution

I introduced persistent farmer memory using SQLite.

Lesson

Memory can make a conversational system significantly more useful.

Challenge 4: Knowing When to Escalate

Problem

The AI should not attempt to confidently solve every serious crop problem.

Solution

I implemented a permission-based human-help workflow and later connected it with the specialist-agent flow.

Lesson

Human-in-the-loop is not a failure of AI. It can be an important part of responsible agent design.

How You Can Build a Voice Agent

If you want to build a voice agent yourself, start with four fundamental components:

Speech-to-Text
       +
LLM
       +
Text-to-Speech
       +
Real-Time Transport

For Krishi Mitra, these became:

Farmer Speech
      ↓
Deepgram
      ↓
Google Gemini
      ↓
Tools / Memory / Specialist
      ↓
Murf Falcon
      ↓
Farmer Voice

Then build incrementally.

A practical sequence is:

Build the basic voice conversation.
Give the agent a clear role.
Add personality and guardrails.
Create the user interface.
Add persistent memory.
Connect one real-world tool.
Implement tool failure handling.
Add outbound calling if required.
Add human escalation.
Add monitoring.
Introduce specialist agents where they provide clear value.

This approach makes debugging much easier than trying to build the entire system at once.

Setting Up the Project

The exact setup should always follow the commands in the project's repository.

The general process is:

1. Clone the repository
2. Create a Python environment
3. Install dependencies
4. Configure environment variables
5. Add API keys
6. Start the agent
7. Start/connect to the voice interface
8. Test a basic conversation
9. Test the weather tool
10. Test memory

GitHub Repository:

https://github.com/roma2020-app/krishimitra-voice-agent/

I recommend following the repository README for the exact installation and startup commands rather than copying generic commands from another project.

Protecting API Keys and User Data

A public technical project must never expose secrets.

Keep API keys in environment variables.

For example:

GEMINI_API_KEY=your_key_here
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here

The actual values should never be committed to GitHub.

Use:

.env

locally and add it to:

.gitignore

Also avoid publishing:

API keys
Production credentials
Real phone numbers
Caller data
Private farmer information

For demonstrations and articles, use anonymized data.

Testing the Agent

A voice agent should be tested through actual conversations.

Test 1 — Normal Conversation

Ask a basic agricultural question and verify that the agent responds naturally.

Test 2 — Weather

Ask:

"आज Mumbai का मौसम कैसा है?"

Verify that the agent uses the weather tool rather than inventing current weather.

Test 3 — Memory

First conversation:

"मैं Jaipur में गेहूं उगाता हूँ।"

Later conversation:

Verify that the saved farmer context can be retrieved.

Test 4 — Tool Failure

Temporarily make the weather service unavailable and verify that the agent provides a safe fallback.

Test 5 — Human Escalation

Report a serious crop problem and verify that the agent asks for permission before creating the human-help request.

Test 6 — Specialist Handoff

Test whether the Crop Problem Specialist receives the existing conversation context without forcing the farmer to repeat the complete problem.

Evidence From the Build

1. Voice Interface

Krishi Mitra during a farmer voice conversation.

2. Live Weather Tool

Krishi Mitra retrieving current weather information through the weather tool.

3. Outbound Call / Human Escalation

https://www.linkedin.com/posts/roma-gupta-880526416_10daysofvoiceagents-day-murffalcon-ugcPost-7492751782306201600-zmsn/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAGnL62ABCXPHUcwL33BbbmynaukcX_dAI_w

https://www.linkedin.com/posts/roma-gupta-880526416_day-10daysofvoiceagents-voiceforbharat-ugcPost-7493114826727141381-tRy2/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAGnL62ABCXPHUcwL33BbbmynaukcX_dAI_w

Krishi Mitra moving from proactive voice interaction toward human assistance.

4. Analytics / Specialist Dashboard

https://www.linkedin.com/posts/roma-gupta-880526416_10daysofvoiceagents-voiceforbharat-10daysofvoiceagents-ugcPost-7493822168510889986-z0oD/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAGnL62ABCXPHUcwL33BbbmynaukcX_dAI_w

Monitoring voice calls and human-help requests.

5. Architecture

Browser Voice Call ──┐
│
├──> Call Analytics Database ──> Streamlit Dashboard
SIP Outbound Call ───┘ │
├── Total Calls
├── SuccessfulCalls
├── Failed Calls
├── Success Rate
├── Browser Metrics
└── SIP Metrics
Voice Agent ──> Human Escalation ──> Escalation Database ──> Human Help Dashboard
│
├── Open Requests
├── High Priority
├── Medium Priority
└── Detailed Farmer Request

What I Learned

Building Krishi Mitra changed how I think about voice agents.

1. A voice agent needs a clear job

An LLM alone does not define an application.

The agent needs objectives, boundaries, tools, and responsibilities.

2. Real-world data needs real-world tools

If information changes frequently, connect the agent to an appropriate source instead of asking the model to guess.

3. Memory changes the experience

A returning user should not always feel like a new user.

4. Guardrails are part of the architecture

Safety should not be added after everything else is finished.

5. AI should know when to ask for help

Human escalation provides a controlled path when the AI should not continue independently.

6. Specialist agents can simplify complex systems

A focused specialist can be more useful than one giant agent responsible for every possible task.

7. Voice agents need observability

Once the agent starts handling conversations, you need visibility into calls, failures, and human-help requirements.

What I Would Build Next

Krishi Mitra is currently a learning/prototype project, so there is plenty of room for improvement.

Some directions I would explore next include:

More Indian languages
More agricultural tools
More verified agricultural information sources
Additional specialist agents
Better farmer profile management
Improved analytics
More robust telephony
Production deployment
Better human expert workflows

These are future directions, not features I am claiming as already completed.

Conclusion

When I started this project, the goal sounded simple:

Make an AI voice agent talk to a farmer.

But building Krishi Mitra taught me that voice is only the beginning.

A useful voice agent needs to:

Listen
   ↓
Understand
   ↓
Remember
   ↓
Use Tools
   ↓
Act
   ↓
Escalate
   ↓
Hand Off
   ↓
Be Monitored

That was the biggest takeaway from building Krishi Mitra during the 10 Days of Voice Agents – VoiceForBharat Edition.

I learned how speech-to-text, an LLM, text-to-speech, real-time communication, memory, tools, specialist agents, and human workflows can come together to create something much more useful than a simple chatbot.

For me, the most important lesson was this:

AI becomes more useful when it knows not only how to answer, but also what to remember, what to verify, what to do, and when to ask a human for help.

Krishi Mitra is still evolving, but this project gave me a much deeper understanding of how to design and build voice-first AI systems.

Thanks to Murf AI for the 10 Days of Voice Agents – VoiceForBharat Edition challenge and for the opportunity to build with Murf Falcon.

🌾🎙️🤖

Build. Learn. Experiment. Share.

Project Links

GitHub Repository:
https://github.com/roma2020-app/krishimitra-voice-agent

LinkedIn:
https://www.linkedin.com/posts/roma-gupta-880526416_10daysofvoiceagents-voiceforbharat-10daysofvoiceagents-ugcPost-7493822168510889986-z0oD/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAGnL62ABCXPHUcwL33BbbmynaukcX_dAI_w

Suggested DEV Community Tags

#ai #voiceai #generativeai #agenticai #python #agritech #livekit #gemini #murfai #voiceforbharat

DEV Community: Roma Gupta

Building Krishi Mitra: An AI Voice Agent for Indian Farmers with LiveKit, Gemini and Murf Falcon 🌾🎙️

Introduction

What Is Krishi Mitra?

The Architecture

Understanding the Voice Pipeline

1. Speech-to-Text

2. LLM — Understanding the Request

3. Text-to-Speech

4. Real-Time Transport

Giving the Agent a Job and Guardrails

Giving Krishi Mitra Memory

Connecting the Agent to Real-World Data

Tool Failure Is Also Part of the Design

From Reactive AI to Proactive AI

When AI Should Ask a Human for Help

Moving From One Agent to a Specialist

Observability: Building a Call Analytics Dashboard

The Most Difficult Technical Parts

Challenge 1: Current Data vs. Generated Answers

Challenge 2: External Service Failure

Challenge 3: Conversation Continuity

Challenge 4: Knowing When to Escalate

How You Can Build a Voice Agent

Setting Up the Project

Protecting API Keys and User Data

Testing the Agent

Test 1 — Normal Conversation

Test 2 — Weather

Test 3 — Memory

Test 4 — Tool Failure

Test 5 — Human Escalation

Test 6 — Specialist Handoff

Evidence From the Build

1. Voice Interface

2. Live Weather Tool

3. Outbound Call / Human Escalation

4. Analytics / Specialist Dashboard

5. Architecture

What I Learned

1. A voice agent needs a clear job

2. Real-world data needs real-world tools

3. Memory changes the experience

4. Guardrails are part of the architecture

5. AI should know when to ask for help

6. Specialist agents can simplify complex systems

7. Voice agents need observability

What I Would Build Next

Conclusion

Project Links

Suggested DEV Community Tags