Over the last 10 days, I built KrishiMitra, a multilingual voice AI assistant designed for Indian farmers as part of 10 Days of Voice Agents — VoiceForBharat Edition.
The project started as a basic real-time voice agent. By the end of the challenge, it had grown into a system that can remember users with consent, fetch live weather information, make outbound calls, escalate difficult problems to humans, measure call outcomes, and hand conversations to a specialist agent.
This post explains what I built, the architecture behind it, the problems I faced, and how someone else can start building a similar voice agent.
1. The Problem
Farmers often need quick answers about everyday agricultural decisions: irrigation, weather, crop care, pests, and other farming practices.
I chose the Farm & Field track because I wanted to explore how a voice-first interface could make this kind of assistance more natural.
Instead of requiring a farmer to type a detailed question into an application, KrishiMitra is designed for a conversational interaction:
Farmer speaks → KrishiMitra understands → KrishiMitra checks the appropriate information → KrishiMitra responds naturally.
Voice is particularly useful here because the interaction can happen conversationally, in English, Hindi, or a Hindi-English mixed style.
2. What Is KrishiMitra?
KrishiMitra is a voice-first agricultural assistant with several layers of capability.
It can:
- Have multilingual voice conversations
- Provide general agricultural guidance
- Remember useful farmer information after explicit consent
- Fetch live weather information
- Make outbound calls using LiveKit SIP and Linphone
- Create human-help requests when AI assistance is not enough
- Track real call outcomes
- Hand serious crop-health questions to a dedicated Crop Specialist
The project uses Murf Falcon for text-to-speech, with LiveKit providing the real-time voice transport.
3. How the System Works
At the core, the system follows the standard real-time voice-agent pipeline:
Farmer speaks
↓
LiveKit real-time transport
↓
Deepgram Speech-to-Text
↓
Gemini LLM
↓
Tools / Memory / Specialist Handoff
↓
Murf Falcon Text-to-Speech
↓
Farmer hears the response
The main components are:
| Component | Technology |
|---|---|
| Real-time transport | LiveKit |
| Speech-to-text | Deepgram |
| Language model | Gemini |
| Text-to-speech | Murf Falcon |
| Backend | Python + LiveKit Agents |
| Frontend | Next.js |
| Persistent storage | SQLite |
| Outbound telephony | LiveKit SIP + Linphone |
The complete project is available on GitHub:
https://github.com/Ashu-svg10/murf-livekit-starter
4. Building the Agent Step by Step
Step 1 — Give the agent a clear role
The first important lesson was that a voice agent needs more than a model and a voice.
KrishiMitra has a clear identity, objectives, conversational style, and safety boundaries.
For example, it is instructed to:
- Keep responses short and conversational
- Ask one question at a time
- Mirror the user's language
- Use Devanagari for Hindi responses
- Avoid pretending to know information it does not have
- Escalate when the problem is outside its capabilities
Clear instructions made the behavior much more predictable.
5. Multilingual Voice Interaction
KrishiMitra supports English, Hindi, and code-mixed conversations.
For Hindi, the agent is instructed to respond using Devanagari script rather than Romanized Hindi.
For example:
नमस्ते, आपकी फसल कैसी है?
rather than:
Namaste, aapki fasal kaisi hai?
The goal was not simply to translate every sentence. The agent should naturally follow the language and style used by the farmer.
6. Consent-Based Memory
One of the most useful features I added was persistent memory.
KrishiMitra can remember information such as:
- Name
- Language preference
- Crop
- District
- Irrigation type
But there is an important rule:
The agent must ask for permission before saving the information.
The flow is:
Learn information
↓
Ask for consent
↓
YES → Save memory
NO → Do not save
This was an important design decision because memory should not silently become a way of collecting personal information.
Sensitive information such as Aadhaar numbers, PAN numbers, bank details, passwords, OTPs, and PINs is not intended to be stored.
7. Giving the Agent a Real Tool
A voice agent becomes much more useful when it can access information that the language model does not know reliably.
For KrishiMitra, I added a live weather lookup tool.
For example, a farmer can ask:
"Aaj Khordha mein baarish hone ki kitni possibility hai?"
Instead of generating a weather answer from the model's memory, the agent calls the weather service and speaks the returned information.
The agent also has a failure path. If the weather service is unavailable, it should tell the farmer that live weather information is temporarily unavailable rather than inventing an answer.
This was one of the biggest lessons from the tools day:
The tool description matters.
The model needs a clear description of when a tool should be called and what information it provides.
8. Outbound Calling
After the browser-based voice experience was working, I extended KrishiMitra to outbound calls using LiveKit SIP and Linphone.
The flow became:
KrishiMitra
↓
LiveKit SIP
↓
Linphone
↓
Farmer
This introduced a completely different set of debugging problems compared with browser voice.
One issue I encountered was an invalid SIP destination format. The first attempt used a complete SIP URI where the LiveKit request expected a SIP user/number in the appropriate field.
After correcting the configuration, the SIP call was successfully created and the Linphone client received the call.
This was a good reminder that voice agents are not only about AI. Telephony configuration, transport, authentication, and addressing can be just as important.
9. Knowing When to Ask a Human
I did not want KrishiMitra to pretend it could solve every agricultural problem.
For serious crop-health problems, the agent can create a human-help request.
The flow is:
Farmer reports serious problem
↓
KrishiMitra explains the limitation
↓
Asks for permission to share a summary
↓
Creates human-help request
↓
Returns a reference ID
The request is intentionally summarized rather than sending the entire conversation.
The human receives useful information such as:
- Who needs help
- What happened
- What the agent already checked
- Urgency
- Language and preferred follow-up method
This made the system more realistic because a good assistant should know when to stop pretending to be the expert.
10. Measuring Calls
Once the agent had multiple capabilities, I wanted to know whether calls were actually achieving their intended outcomes.
I added a SQLite-based call analytics system.
The dashboard tracks:
Total Calls
Successful Calls
Failed Calls
For KrishiMitra, I defined a successful call as one where the farmer receives verified weather information or successfully creates a human-help request.
The important part is that the dashboard uses real call records, not hardcoded numbers.
The flow is:
Voice session ends
↓
Session outcome is determined
↓
SQLite record is created
↓
Dashboard reads the records
↓
Metrics update
This gave the project a basic observability layer instead of treating the voice experience as a black box.
11. Handing Off to a Specialist Agent
The final major feature was specialist-agent handoff.
The main KrishiMitra agent has a broad agricultural role, but a single agent should not try to be an expert at everything.
So I created a dedicated:
Crop Specialist
The flow is:
Farmer
↓
KrishiMitra
↓
Serious crop-health problem detected
↓
Main agent announces handoff
↓
Crop Specialist
↓
Continues the same conversation
The specialist has its own instructions and a narrower responsibility.
The existing conversation context is passed to the specialist, so the farmer does not have to repeat the complete problem.
For example:
"Meri tamatar ki fasal ke patton par bahut saare daag aa rahe hain aur paudhe tezi se kharab ho rahe hain."
KrishiMitra can recognize that this needs focused crop-health guidance and hand the conversation to the Crop Specialist.
This separation of responsibilities makes the overall system easier to reason about.
12. The Hard Parts
The project definitely did not work perfectly on the first attempt.
Memory persistence
The first challenge was making sure that information was not merely remembered inside one conversation but persisted for a future conversation.
The important fix was to separate:
- Looking up existing memory
- Asking for consent
- Saving memory
The agent must not assume that learning information means it has permission to store it.
SIP outbound calling
Outbound calling was another major debugging step.
The SIP configuration initially produced errors around the destination and From header. Understanding the distinction between a SIP user, a phone number, and a complete SIP URI was necessary before the call could be created successfully.
Real analytics
It was tempting to simply display numbers on a dashboard, but that would not prove anything.
The dashboard needed to be connected to actual session completion data and a database so that a real call changed the numbers.
Specialist handoff
The handoff also needed more than creating another class.
The main agent needed to:
- Know when the specialist was appropriate.
- Tell the user that a handoff was happening.
- Transfer the conversation.
- Preserve the existing context.
That last part is especially important for a natural voice experience.
13. How to Run the Project
The repository is available here:
https://github.com/Ashu-svg10/murf-livekit-starter
Prerequisites
You need:
- Python 3.10+
- Node.js 18+
- uv
- pnpm
- A LiveKit Cloud project
- Murf API key
- Deepgram API key
- Gemini API key
Clone
git clone https://github.com/Ashu-svg10/murf-livekit-starter.git
cd murf-livekit-starter
Install backend dependencies
cd backend
uv sync
Install frontend dependencies
In another terminal:
cd frontend
pnpm install
Configure environment variables
Create local .env.local files.
Backend:
backend/.env.local
Frontend:
frontend/.env.local
Store API credentials there.
Never commit API keys or secrets to GitHub.
Start the backend
cd backend
uv run python src/agent.py dev
Start the frontend
cd frontend
pnpm dev
Then open:
http://localhost:3000
Click Start Call, allow microphone access, and begin speaking with KrishiMitra.
The repository README contains the project structure, environment-variable setup, memory behavior, weather tool, escalation, analytics, specialist handoff, and outbound-call information.
14. What I Would Improve Next
There is still a lot I would like to build.
The next improvements would include:
- More Indian regional languages
- Agricultural RAG using trusted agricultural documents
- Dedicated specialists for irrigation, soil, pests, and government schemes
- More real-time agricultural data sources
- Better call-quality and latency analytics
- A production-ready human support dashboard
- Better support for low-connectivity environments
I would also like to evaluate the agent systematically with a larger set of real-world farmer questions instead of relying mainly on manual testing.
15. What I Learned
The biggest lesson from these 10 days is that a voice agent is much more than:
STT + LLM + TTS
A useful voice agent also needs:
- Good prompts
- Clear tool descriptions
- Real data
- Failure handling
- Consent
- Memory
- Safety guardrails
- Human escalation
- Specialist routing
- Observability
- Privacy protection
- A good real-time user experience
The most interesting part was watching the project evolve from a basic voice conversation into a system that can make decisions about when to use tools, when to ask for permission, when to involve a human, and when to transfer the conversation to another agent.
16. Final Thoughts
Building KrishiMitra over 10 days taught me how quickly a voice-agent prototype can grow when each capability is built on top of the previous one.
I started with a voice interface.
I ended with an agricultural voice assistant that can:
talk → remember → use real data → call → escalate → measure → hand off
The project is still a prototype, but it gave me a much deeper understanding of what is required to build reliable voice AI for real-world users.
A huge part of the experience was learning by debugging the failures rather than only following the happy path.
Repository
KrishiMitra:
https://github.com/Ashu-svg10/murf-livekit-starter
Built as part of:
10 Days of Voice Agents — VoiceForBharat Edition
Powered by Murf Falcon for text-to-speech.
Top comments (0)