Shreya Patel

Posted on Aug 15

KRISHI MITRA AI

#devops #aws #webdev #ai

🌾 KrishiMitra AI: Building a Voice AI Assistant for Smarter Farming

From a simple voice assistant to a multilingual farming companion with tools, memory, outbound calls, human escalation, analytics, and specialist handoffs.

Over the last 10 days, I took part in 10 Days of Voice Agents — VoiceForBharat Edition, where I built and continuously improved my own voice AI project: KrishiMitra AI.

My project belongs to the Farm & Field track and is designed to help farmers access useful agricultural information through a simple voice conversation.

The idea was simple:

What if a farmer could just speak to an AI assistant instead of navigating complicated applications or typing long questions?

That question became the starting point for KrishiMitra AI.

🌱 What is KrishiMitra AI?

KrishiMitra AI is a voice-first agricultural assistant designed to help farmers with everyday farming-related questions.

Instead of requiring a farmer to type a query, the user can simply speak naturally.

For example:

"Meri soybean ki leaves yellow ho rahi hain."

or:

"What will the weather be like today?"

KrishiMitra AI can understand the request, process it, and respond through voice.

The project focuses on making AI assistance more accessible, especially for users who may be more comfortable speaking than typing.

🎯 Why Farm & Field?

Agriculture involves many decisions that depend on timely information.

Farmers may need help with:

Crop-related problems
Weather information
Irrigation questions
Pest-related issues
Farming guidance
Market-related information
Getting human assistance when AI is not enough

Voice can make these interactions more natural.

A farmer doesn't have to open a keyboard and formulate a perfect question.

They can simply speak.

That is the experience I wanted KrishiMitra AI to provide.

🖥️ The KrishiMitra AI Interface

The frontend was designed around a simple idea:

The user should immediately understand what the agent does and how to start talking to it.

The home screen contains the KrishiMitra AI branding, farming categories, a voice interaction button, and options such as Human Help and Analytics.

WhatsApp Image 2026-08-15 at 4.55.43 PM (1).jpeg

Caption:

KrishiMitra AI home screen — The voice-first interface designed for simple and accessible farming assistance.

🎙️ A Voice-First Experience

Once the user starts talking, KrishiMitra AI moves into an active voice conversation.

The interface clearly shows that the agent is speaking, making the interaction feel more like a real conversation than a traditional chatbot.

Under the hood, the voice pipeline connects several components:

Speech-to-Text
Large Language Model
Text-to-Speech
Real-time communication through LiveKit

The starter architecture uses Deepgram for STT, an LLM for reasoning, Murf Falcon for TTS, and LiveKit for real-time audio transport.

Caption:

KrishiMitra AI in an active voice session — The interface shows the agent speaking to the user.

🧠 How KrishiMitra AI Works

The basic voice flow looks like this:

flowchart LR
    A["🎙️ Farmer speaks"] --> B["Deepgram STT"]
    B --> C["LLM / Agent"]
    C --> D["Decision + Tools"]
    D --> E["Murf Falcon TTS"]
    E --> F["🔊 Farmer hears response"]
    F --> A

    C --> G{"Specialist needed?"}
    G -->|No| D
    G -->|Yes| H["🌱 Crop Problem Specialist"]
    H --> E

    C --> I{"Human help needed?"}
    I -->|Yes| J["👩‍🌾 Human Support"]

The important part is that KrishiMitra AI isn't just a text chatbot.

It is a real-time voice pipeline.

The user speaks → speech is transcribed → the agent understands the request → tools or specialist agents can be used → the response is converted back into speech.

🧩 The Main Components

1. Speech-to-Text

The first step is converting the farmer's speech into text.

For my project, the voice pipeline uses Deepgram STT.

This allows the agent to understand what the user is saying before the LLM processes the request.

2. LLM

The LLM acts as the reasoning layer.

It determines:

What the user is asking
Whether a tool is required
Whether the request needs a specialist
How the response should be formulated
When the conversation should be escalated

This is where the main intelligence of the agent lives.

3. Murf Falcon TTS

The response generated by the LLM needs to become speech again.

This is where Murf Falcon comes in.

One of the biggest highlights of this challenge for me was building with Murf Falcon — the fastest TTS API used in the challenge.

The starter project supports Murf voices including Indian English voices such as Anisha, Pooja, and Samar.

For a voice-first agricultural assistant, natural and fast speech matters because long pauses can make a conversation feel unnatural.

4. LiveKit

LiveKit handles the real-time communication layer between the frontend and the voice agent.

The frontend and backend communicate through the LiveKit infrastructure rather than directly calling each other.

This makes it possible to build an interactive browser-based voice experience.

🛠️ Features I Built During the Challenge

The 10-day challenge was not just about making an AI speak.

Each day added another capability to the project.

Here are some of the important features I worked on.

🌾 1. Farming-Focused Personality

Instead of creating a generic assistant, I gave the agent a specific identity:

KrishiMitra AI — a farming companion.

The agent is designed to be:

Helpful
Simple
Conversational
Farmer-friendly
Clear
Safety-aware

The goal is not to sound like a complicated technical system.

It should feel like an assistant that a farmer can actually talk to.

🌐 2. Multilingual and Hinglish Conversations

Agricultural users may not always communicate in formal English.

Therefore, KrishiMitra AI supports natural language switching.

For example:

User:

"Meri crop mein keede lag gaye hain."

The agent can respond in Hinglish/Hindi.

If the user switches to English, the agent can respond in English.

This makes the conversation more natural and accessible.

🧠 3. Memory

Another important capability was memory.

A useful assistant should not behave like it has forgotten everything every time the user returns.

Memory allows the system to retain useful conversational information and provide a more personalized experience.

For a farming assistant, this can become especially useful when discussing:

Crop type
Previously discussed problems
Farming context
Earlier questions

🔧 4. Tools

The agent can use tools when information needs to be retrieved or computed instead of simply generating an answer.

For example, agricultural assistants can use tools for things like:

Weather information
Market information
Farming-related data

This creates an important distinction:

LLM = reasoning

Tool = real-world data

That combination makes the assistant more useful.

📞 5. Outbound Calls

One of the more challenging parts of the journey was working with outbound voice calls.

The idea was to allow the system to proactively reach users for useful situations such as:

Weather warnings
Important agricultural updates
Price-related alerts
Other farming notifications

This introduced another layer of complexity because voice agents now need to interact with telephony infrastructure in addition to the browser.

👩‍🌾 6. Human Escalation

AI should not pretend to know everything.

For serious or uncertain farming problems, KrishiMitra AI can escalate the request to human support.

I created a Human Help flow where the user can provide:

Name
Reason for help
What happened
Urgency
Preferred language
Follow-up method

The request can then be submitted to the support workflow.

📸 IMAGE 4 — INSERT HERE

Human Help request form — Farmers can provide the problem details and choose urgency, language, and follow-up preferences.

✅ Human Help Confirmation

After submitting the request, the user receives a confirmation with a reference ID and follow-up information.

Caption:

Human Help confirmation — The system generates a reference ID and confirms that the request has been submitted.

📊 7. Call Analytics Dashboard

Another major feature was the analytics dashboard.

Instead of only building an agent that talks, I wanted to understand how the agent was performing.

The dashboard provides metrics such as:

Total calls
Successful calls
Failed calls
Success rate
Recent call outcomes
Call duration
Channel
Details about the outcome

For example, during my testing, the dashboard showed:

4 Total Calls

2 Successful Calls

2 Failed Calls

50% Success Rate

Caption:

KrishiMitra AI Call Analytics Dashboard — Tracking call outcomes, success rate, duration, and recent conversations.

This was useful because it changed the way I looked at the project.

I wasn't only asking:

"Does my agent work?"

I was also asking:

"How well does my agent work?"

🔄 8. Specialist Agent Handoff

One of the most interesting features from the final days was agent handoff.

A single AI agent shouldn't have to be an expert at everything.

So I created a separate:

🌱 Crop Problem Specialist

The main KrishiMitra agent handles general farming requests.

When a farmer reports a specific crop-health problem, the main agent can transfer the conversation to the specialist.

For example:

Farmer:

"Meri soybean ki leaves yellow ho rahi hain."

KrishiMitra AI:

"Samajh gaya. Main aapko Crop Problem Specialist se connect karta hoon."

Then the specialist takes over:

"Namaste! Aapne soybean ki leaves yellow hone ki baat batayi hai. Ye problem kab se dikh rahi hai?"

The important part is that the farmer doesn't need to explain the whole problem again.

The conversation context is preserved during the handoff. My repository documents the implementation using LiveKit's session agent update mechanism and passing the existing chat context to the specialist.

🔀 Specialist Handoff Flow

flowchart TD
    A["🎙️ Farmer asks a question"] --> B["🌾 KrishiMitra Main Agent"]

    B --> C{"Does the request need specialist help?"}

    C -->|No| D["Main Agent answers"]
    D --> E["🔊 Murf Falcon TTS"]

    C -->|Yes| F["Main Agent announces handoff"]
    F --> G["🌱 Crop Problem Specialist"]

    G --> H["Conversation context preserved"]
    H --> I["Specialist asks focused follow-up"]
    I --> E

    E --> J["🎧 Farmer continues conversation"]

🐛 A Problem I Faced During Handoff

This was one of the most useful debugging lessons of the entire challenge.

Initially, the main agent successfully said:

"I'll connect you with our crop problem specialist."

But then...

Nothing happened.

The specialist didn't respond.

At first, it looked like the handoff itself was working because the main agent announced it.

But announcing a handoff and actually transferring an active voice session are two completely different things.

🔍 What Was Actually Wrong?

The problem was related to the timing and session state of the specialist agent.

The specialist's on_enter() lifecycle method could attempt to generate a response before the session was ready to handle it.

This created a situation where:

Main Agent → Handoff message → Specialist activation → No response

The repository's specialist handoff fix addresses this with session readiness checks, deferred response generation, retries, better error handling, and preserved conversation context.

💡 What I Learned From This

This taught me an important lesson:

In real-time AI systems, correct logic is not enough. Timing and state management matter too.

A function can technically execute successfully while the user still experiences silence.

For voice agents, the complete chain has to work:

STT → Agent → Handoff → Session State → Specialist → LLM → TTS → User

If one part breaks, the user experiences a failed conversation.

🛡️ Safety and Guardrails

An agricultural assistant also needs boundaries.

KrishiMitra AI should not confidently claim a crop disease diagnosis based only on a short description.

Instead, it should:

Ask follow-up questions
Explain possible causes carefully
Avoid pretending to be a certified agricultural expert
Recommend local agricultural assistance when appropriate
Avoid unsafe instructions
Keep responses understandable

This is especially important because AI advice can influence real-world decisions.

🏗️ Project Architecture

The overall system can be represented like this:

flowchart LR
    U["👨‍🌾 Farmer"]

    UI["🌐 KrishiMitra AI Frontend"]

    LK["⚡ LiveKit"]

    STT["🎤 Deepgram STT"]

    MAIN["🧠 KrishiMitra Main Agent"]

    TOOLS["🔧 Agriculture Tools"]

    SPEC["🌱 Crop Problem Specialist"]

    HUMAN["👩‍🌾 Human Support"]

    LLM["🤖 LLM"]

    TTS["🔊 Murf Falcon TTS"]

    DASH["📊 Analytics Dashboard"]

    U --> UI
    UI --> LK
    LK --> STT
    STT --> MAIN

    MAIN --> LLM
    MAIN --> TOOLS

    MAIN -->|Crop problem| SPEC
    SPEC --> LLM

    MAIN -->|Needs human help| HUMAN

    LLM --> TTS
    TTS --> LK
    LK --> UI
    UI --> U

    LK --> DASH

🚀 How to Run the Project

The project is based on the Murf LiveKit starter architecture.

The repository contains a backend for the Python voice agent and a frontend for the Next.js voice interface.

Prerequisites

You need:

Python 3.10+
Node.js 18+
uv
pnpm
LiveKit project
Murf API key
Deepgram API key
LLM API key

The starter README documents the required environment variables, including:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY or OPENAI_API_KEY

🔐 Keep API Keys Safe

Never put API keys directly inside your source code.

Use environment files such as:

.env.local

and add them to .gitignore.

Never publish:

API keys
API secrets
Private phone numbers
Personal caller data
Private user information

▶️ Running the Backend

From the backend directory:

uv sync
uv run python src/agent.py download-files
uv run python src/agent.py dev

The starter project uses the backend agent as the LiveKit voice-processing service.

▶️ Running the Frontend

From the frontend directory:

pnpm install
pnpm dev

Then open the local application in your browser.

The original starter documentation uses localhost:3000 for the frontend.

🧪 Testing the Voice Agent

I recommend testing the project using both normal and specialist questions.

Test 1 — Normal Question

User:

"What is the weather today?"

Expected:

Main Agent responds.

No specialist handoff.

Test 2 — Crop Problem

User:

"My soybean crop has yellow leaves."

Expected:

Main Agent:

"I understand. I'll connect you with our Crop Problem Specialist."

Then:

Specialist:

"Hello! I understand that your soybean crop has yellow leaves. When did you first notice the problem?"

Test 3 — Hinglish

User:

"Meri crop mein keede lag gaye hain."

Expected:

Main Agent:

"Samajh gaya. Main aapko Crop Problem Specialist se connect karta hoon."

Then the specialist continues in Hinglish.

📁 Project Repository

The complete project is publicly available on GitHub:

shreyaa002/murf-livekit-starter

It contains the backend, frontend, configuration, and documentation for the project.

Repository:
https://github.com/shreyaa002/murf-livekit-starter

📈 What I Would Improve Next

KrishiMitra AI is still a work in progress.

There are several things I would like to improve next:

1. Better Crop Intelligence

Add more specialized agricultural knowledge for different crops and regions.

2. Image-Based Crop Analysis

Allow farmers to upload or capture a crop image and use visual analysis alongside voice.

3. More Regional Languages

Expand support beyond Hindi/Hinglish and English.

4. Better Personalization

Remember useful farming context such as crops and previous problems while respecting user privacy.

5. More Specialist Agents

The current specialist focuses on crop problems.

Future versions could include:

Weather Specialist
Irrigation Specialist
Market Specialist
Government Scheme Specialist

The main agent could route each question to the appropriate expert.

💭 My Biggest Takeaways From 10 Days

This challenge taught me that building a voice agent is much more than connecting an LLM to a microphone.

I learned how different pieces of a real-time AI system fit together:

Speech recognition + reasoning + tools + memory + TTS + real-time transport + frontend + analytics + human escalation + multi-agent workflows.

I also learned that debugging voice agents is different from debugging a normal application.

Sometimes everything looks correct in the code, but the user experience still fails because of timing, session state, audio, or asynchronous behavior.

The specialist handoff was a perfect example of that.

🌾 Why I Want to Continue Building KrishiMitra AI

Agriculture is an area where technology can be useful only if it is accessible.

A powerful AI system is not very helpful if the user needs to understand complicated interfaces before they can use it.

Voice changes that interaction.

Instead of:

Open → Type → Search → Read → Repeat

the experience becomes:

Speak → Listen → Continue

That is the direction I want to explore further with KrishiMitra AI.

🚀 Final Thoughts

Starting this challenge, my goal was simply to build a voice agent.

By the end, KrishiMitra AI had become much more than that.

It can:

🎙️ Have voice conversations
🌾 Answer farming-related questions
🌐 Support multilingual/code-mixed interaction
🧠 Use conversational context
🔧 Use tools
📞 Support outbound calling workflows
👩‍🌾 Escalate users to human support
📊 Track call analytics
🔄 Hand conversations to a crop specialist
🔊 Respond using Murf Falcon TTS

The biggest lesson I am taking away is:

A useful AI agent is not just one that can answer questions. It is one that knows what to do, when to use a tool, when to ask for help, and when to hand the conversation to someone more specialized.

This is my 10 Days of Voice Agents — VoiceForBharat Edition journey with KrishiMitra AI.

And this is only the beginning. 🌱

🔗 Project

KrishiMitra AI — GitHub Repository

shreyaa002/murf-livekit-starter

🙌 Thanks

A big thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition challenge and providing an opportunity to learn by building.

Building every day, debugging every day, and learning something new every day made this challenge a great experience.

VoiceForBharat #MurfAI #VoiceAI #AI #GenerativeAI #LiveKit #MurfFalcon #Agriculture #AgriTech #FarmTech #ArtificialIntelligence #BuildInPublic