Building KrishiMitra: My 10-Day Journey Building a Voice AI Assistant for Indian Farmers

Ashutosh Nayak — Sat, 15 Aug 2026 14:10:40 +0000

Over the last 10 days, I built KrishiMitra, a multilingual voice AI assistant designed for Indian farmers as part of 10 Days of Voice Agents — VoiceForBharat Edition.

The project started as a basic real-time voice agent. By the end of the challenge, it had grown into a system that can remember users with consent, fetch live weather information, make outbound calls, escalate difficult problems to humans, measure call outcomes, and hand conversations to a specialist agent.

This post explains what I built, the architecture behind it, the problems I faced, and how someone else can start building a similar voice agent.

1. The Problem

Farmers often need quick answers about everyday agricultural decisions: irrigation, weather, crop care, pests, and other farming practices.

I chose the Farm & Field track because I wanted to explore how a voice-first interface could make this kind of assistance more natural.

Instead of requiring a farmer to type a detailed question into an application, KrishiMitra is designed for a conversational interaction:

Farmer speaks → KrishiMitra understands → KrishiMitra checks the appropriate information → KrishiMitra responds naturally.

Voice is particularly useful here because the interaction can happen conversationally, in English, Hindi, or a Hindi-English mixed style.

2. What Is KrishiMitra?

KrishiMitra is a voice-first agricultural assistant with several layers of capability.

It can:

Have multilingual voice conversations
Provide general agricultural guidance
Remember useful farmer information after explicit consent
Fetch live weather information
Make outbound calls using LiveKit SIP and Linphone
Create human-help requests when AI assistance is not enough
Track real call outcomes
Hand serious crop-health questions to a dedicated Crop Specialist

The project uses Murf Falcon for text-to-speech, with LiveKit providing the real-time voice transport.

3. How the System Works

At the core, the system follows the standard real-time voice-agent pipeline:

Farmer speaks
      ↓
LiveKit real-time transport
      ↓
Deepgram Speech-to-Text
      ↓
Gemini LLM
      ↓
Tools / Memory / Specialist Handoff
      ↓
Murf Falcon Text-to-Speech
      ↓
Farmer hears the response

The main components are:

Component	Technology
Real-time transport	LiveKit
Speech-to-text	Deepgram
Language model	Gemini
Text-to-speech	Murf Falcon
Backend	Python + LiveKit Agents
Frontend	Next.js
Persistent storage	SQLite
Outbound telephony	LiveKit SIP + Linphone

The complete project is available on GitHub:

https://github.com/Ashu-svg10/murf-livekit-starter

4. Building the Agent Step by Step

Step 1 — Give the agent a clear role

The first important lesson was that a voice agent needs more than a model and a voice.

KrishiMitra has a clear identity, objectives, conversational style, and safety boundaries.

For example, it is instructed to:

Keep responses short and conversational
Ask one question at a time
Mirror the user's language
Use Devanagari for Hindi responses
Avoid pretending to know information it does not have
Escalate when the problem is outside its capabilities

Clear instructions made the behavior much more predictable.

5. Multilingual Voice Interaction

KrishiMitra supports English, Hindi, and code-mixed conversations.

For Hindi, the agent is instructed to respond using Devanagari script rather than Romanized Hindi.

For example:

नमस्ते, आपकी फसल कैसी है?

rather than:

Namaste, aapki fasal kaisi hai?

The goal was not simply to translate every sentence. The agent should naturally follow the language and style used by the farmer.

6. Consent-Based Memory

One of the most useful features I added was persistent memory.

KrishiMitra can remember information such as:

Name
Language preference
Crop
District
Irrigation type

But there is an important rule:

The agent must ask for permission before saving the information.

The flow is:

Learn information
      ↓
Ask for consent
      ↓
YES → Save memory
NO  → Do not save

This was an important design decision because memory should not silently become a way of collecting personal information.

Sensitive information such as Aadhaar numbers, PAN numbers, bank details, passwords, OTPs, and PINs is not intended to be stored.

7. Giving the Agent a Real Tool

A voice agent becomes much more useful when it can access information that the language model does not know reliably.

For KrishiMitra, I added a live weather lookup tool.

For example, a farmer can ask:

"Aaj Khordha mein baarish hone ki kitni possibility hai?"

Instead of generating a weather answer from the model's memory, the agent calls the weather service and speaks the returned information.

The agent also has a failure path. If the weather service is unavailable, it should tell the farmer that live weather information is temporarily unavailable rather than inventing an answer.

This was one of the biggest lessons from the tools day:

The tool description matters.

The model needs a clear description of when a tool should be called and what information it provides.

8. Outbound Calling

After the browser-based voice experience was working, I extended KrishiMitra to outbound calls using LiveKit SIP and Linphone.

The flow became:

KrishiMitra
     ↓
LiveKit SIP
     ↓
Linphone
     ↓
Farmer

This introduced a completely different set of debugging problems compared with browser voice.

One issue I encountered was an invalid SIP destination format. The first attempt used a complete SIP URI where the LiveKit request expected a SIP user/number in the appropriate field.

After correcting the configuration, the SIP call was successfully created and the Linphone client received the call.

This was a good reminder that voice agents are not only about AI. Telephony configuration, transport, authentication, and addressing can be just as important.

9. Knowing When to Ask a Human

I did not want KrishiMitra to pretend it could solve every agricultural problem.

For serious crop-health problems, the agent can create a human-help request.

The flow is:

Farmer reports serious problem
          ↓
KrishiMitra explains the limitation
          ↓
Asks for permission to share a summary
          ↓
Creates human-help request
          ↓
Returns a reference ID

The request is intentionally summarized rather than sending the entire conversation.

The human receives useful information such as:

Who needs help
What happened
What the agent already checked
Urgency
Language and preferred follow-up method

This made the system more realistic because a good assistant should know when to stop pretending to be the expert.

10. Measuring Calls

Once the agent had multiple capabilities, I wanted to know whether calls were actually achieving their intended outcomes.

I added a SQLite-based call analytics system.

The dashboard tracks:

Total Calls
Successful Calls
Failed Calls

For KrishiMitra, I defined a successful call as one where the farmer receives verified weather information or successfully creates a human-help request.

The important part is that the dashboard uses real call records, not hardcoded numbers.

The flow is:

Voice session ends
       ↓
Session outcome is determined
       ↓
SQLite record is created
       ↓
Dashboard reads the records
       ↓
Metrics update

This gave the project a basic observability layer instead of treating the voice experience as a black box.

11. Handing Off to a Specialist Agent

The final major feature was specialist-agent handoff.

The main KrishiMitra agent has a broad agricultural role, but a single agent should not try to be an expert at everything.

So I created a dedicated:

Crop Specialist

The flow is:

Farmer
  ↓
KrishiMitra
  ↓
Serious crop-health problem detected
  ↓
Main agent announces handoff
  ↓
Crop Specialist
  ↓
Continues the same conversation

The specialist has its own instructions and a narrower responsibility.

The existing conversation context is passed to the specialist, so the farmer does not have to repeat the complete problem.

For example:

"Meri tamatar ki fasal ke patton par bahut saare daag aa rahe hain aur paudhe tezi se kharab ho rahe hain."

KrishiMitra can recognize that this needs focused crop-health guidance and hand the conversation to the Crop Specialist.

This separation of responsibilities makes the overall system easier to reason about.

12. The Hard Parts

The project definitely did not work perfectly on the first attempt.

Memory persistence

The first challenge was making sure that information was not merely remembered inside one conversation but persisted for a future conversation.

The important fix was to separate:

Looking up existing memory
Asking for consent
Saving memory

The agent must not assume that learning information means it has permission to store it.

SIP outbound calling

Outbound calling was another major debugging step.

The SIP configuration initially produced errors around the destination and From header. Understanding the distinction between a SIP user, a phone number, and a complete SIP URI was necessary before the call could be created successfully.

Real analytics

It was tempting to simply display numbers on a dashboard, but that would not prove anything.

The dashboard needed to be connected to actual session completion data and a database so that a real call changed the numbers.

Specialist handoff

The handoff also needed more than creating another class.

The main agent needed to:

Know when the specialist was appropriate.
Tell the user that a handoff was happening.
Transfer the conversation.
Preserve the existing context.

That last part is especially important for a natural voice experience.

13. How to Run the Project

The repository is available here:

https://github.com/Ashu-svg10/murf-livekit-starter

Prerequisites

You need:

Python 3.10+
Node.js 18+
uv
pnpm
A LiveKit Cloud project
Murf API key
Deepgram API key
Gemini API key

Clone

git clone https://github.com/Ashu-svg10/murf-livekit-starter.git
cd murf-livekit-starter

Install backend dependencies

cd backend
uv sync

Install frontend dependencies

In another terminal:

cd frontend
pnpm install

Configure environment variables

Create local .env.local files.

Backend:

backend/.env.local

Frontend:

frontend/.env.local

Store API credentials there.

Never commit API keys or secrets to GitHub.

Start the backend

cd backend
uv run python src/agent.py dev

Start the frontend

cd frontend
pnpm dev

Then open:

http://localhost:3000

Click Start Call, allow microphone access, and begin speaking with KrishiMitra.

The repository README contains the project structure, environment-variable setup, memory behavior, weather tool, escalation, analytics, specialist handoff, and outbound-call information.

14. What I Would Improve Next

There is still a lot I would like to build.

The next improvements would include:

More Indian regional languages
Agricultural RAG using trusted agricultural documents
Dedicated specialists for irrigation, soil, pests, and government schemes
More real-time agricultural data sources
Better call-quality and latency analytics
A production-ready human support dashboard
Better support for low-connectivity environments

I would also like to evaluate the agent systematically with a larger set of real-world farmer questions instead of relying mainly on manual testing.

15. What I Learned

The biggest lesson from these 10 days is that a voice agent is much more than:

STT + LLM + TTS

A useful voice agent also needs:

Good prompts
Clear tool descriptions
Real data
Failure handling
Consent
Memory
Safety guardrails
Human escalation
Specialist routing
Observability
Privacy protection
A good real-time user experience

The most interesting part was watching the project evolve from a basic voice conversation into a system that can make decisions about when to use tools, when to ask for permission, when to involve a human, and when to transfer the conversation to another agent.

16. Final Thoughts

Building KrishiMitra over 10 days taught me how quickly a voice-agent prototype can grow when each capability is built on top of the previous one.

I started with a voice interface.

I ended with an agricultural voice assistant that can:

talk → remember → use real data → call → escalate → measure → hand off

The project is still a prototype, but it gave me a much deeper understanding of what is required to build reliable voice AI for real-world users.

A huge part of the experience was learning by debugging the failures rather than only following the happy path.

Repository

KrishiMitra:

https://github.com/Ashu-svg10/murf-livekit-starter

Built as part of:

10 Days of Voice Agents — VoiceForBharat Edition

DEV Community: Ashutosh Nayak

Building KrishiMitra: My 10-Day Journey Building a Voice AI Assistant for Indian Farmers

1. The Problem

2. What Is KrishiMitra?

3. How the System Works

4. Building the Agent Step by Step

Step 1 — Give the agent a clear role

5. Multilingual Voice Interaction

6. Consent-Based Memory

7. Giving the Agent a Real Tool

8. Outbound Calling

9. Knowing When to Ask a Human

10. Measuring Calls

11. Handing Off to a Specialist Agent

12. The Hard Parts

Memory persistence

SIP outbound calling

Real analytics

Specialist handoff

13. How to Run the Project

Prerequisites

Clone

Install backend dependencies

Install frontend dependencies

Configure environment variables

Start the backend

Start the frontend

14. What I Would Improve Next

15. What I Learned

16. Final Thoughts

Repository