DEV Community

Sudhanshu Singh
Sudhanshu Singh

Posted on

From Zero to Voice Agent: My 10-Day Journey Building an AI Voice Agent

#ai #voice #python #livekit

Building AgentX: An AI Voice Assistant for Indian Farmers Powered by Murf Falcon 2

10 Days of AI Voice Agents Challenge — Day 10 Submission | #VoiceForBharat

Introduction

For millions of farmers across India, timely access to agricultural information—such as crop disease identification, weather updates, and mandi (market) prices—can make the difference between a successful harvest and a heavy financial loss. However, existing digital tools present significant barriers: complex mobile interfaces, text-heavy portals, and a lack of support for regional Indian languages or casual code-mixed conversations (Hinglish/Hindi).

To bridge this digital divide, I built AgentX—a voice-first, multilingual AI assistant tailored specifically for Indian agriculture. AgentX allows farmers to simply speak into their phones in Hindi, Hinglish, or Devanagari script and receive real-time, knowledge-backed voice advice.

High-Level System Architecture

AgentX operates as a real-time, bi-directional voice pipeline built on the LiveKit Agent Framework.

  [ Farmer's Voice Input ]
             │
             ▼
┌──────────────────────────┐
│  LiveKit WebRTC Audio    │  (Low-Latency Real-Time Transport)
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│  Speech-to-Text (STT)    │  (Multilingual & Hinglish Recognition)
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│   Agent Engine + RAG     │ ◄───► [ Weather & Mandi APIs ]
│  (LLM + Vector Database) │ ◄───► [ Agronomy RAG Knowledge Base ]
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│   Murf Falcon 2 TTS      │  (Natural Indian Accent Voice Synthesis)
└────────────┬─────────────┘
             │
             ▼
  [ Expressive Voice Response ]
Enter fullscreen mode Exit fullscreen mode

Key Components:

  1. Real-time Transport: LiveKit WebRTC for ultra-low latency audio streaming.
  2. STT: Deepgram / Whisper tuned for Hindi, Devanagari, and Hinglish phonetics.
  3. LLM Engine: Generative AI backed by RAG (Retrieval-Augmented Generation) to prevent hallucinations and provide accurate agricultural advisories.
  4. TTS Engine: Murf Falcon 2, delivering natural, warm, and human-like Indian accent speech synthesis.

Key Features of AgentX

1. Natural Voice AI (Powered by Murf Falcon 2)

AgentX converses naturally without mechanical or robotic delays. By integrating Murf Falcon 2, the agent produces ultra-realistic Indian accent voice responses that sound trustworthy and easy to understand over mobile speakers.

2. Crop Intelligence & Disease Diagnosis

Farmers can describe symptoms (e.g., "Meri gehun ki fasal ke patte peele ho rahe hain" / "Wheat leaves turning yellow"). AgentX cross-references agronomy guides to suggest exact diagnosis and organic or chemical treatment dosages.

3. Weather Intelligence

AgentX checks hyper-local weather forecasts using live APIs before advising on irrigation, pesticide spraying, or harvesting schedules. For example, if rain is predicted in 24 hours, it warns the farmer not to apply pesticides.

4. Market Intelligence (Mandi Prices)

Farmers can query live commodity rates across regional Mandis (e.g., "Aaj Indore mandi me Gehun ka bhav kya hai?"). AgentX retrieves real-time pricing data to help farmers decide when and where to sell for maximum profit.

5. Multilingual & Code-Mixed Support (Hindi / Hinglish / Devanagari)

AgentX naturally understands code-mixed speech (mixing Hindi and English terms like "pest control", "urea", "irrigation", "weather warning") and responds accurately in Hinglish or fluent Hindi.

6. AI + RAG (Knowledge-Backed Answers)

Agricultural guidance requires high precision. AgentX uses RAG over verified government agricultural bulletins and agronomic research papers to ground all responses in facts, ensuring zero dangerous hallucinations.

The Hardest Challenges & Solutions

Building a real-time voice agent for rural Indian contexts came with technical hurdles. Here is how I solved them:

Challenge 1: Code-Mixed Pronunciation in Text-to-Speech

Problem: Traditional TTS models struggle with Hinglish inputs. When reading mixed text like "Gehun crop me yellow rust hai", standard models either butcher the English terms or mispronounce Hindi words written in Devanagari/Roman script.

Solution: By leveraging Murf Falcon 2, which is natively optimized for Indian speech patterns and multilingual nuances, combined with custom phonetic text normalization before sending prompts to the TTS engine, AgentX delivers smooth, natural Hinglish output.

Challenge 2: Reducing Latency with Parallel RAG & Tool Calling

Problem: Querying weather APIs, searching Mandi price databases, and performing vector RAG lookup sequentially caused an audible 3-4 second pause before the agent spoke.

Solution: Implemented asynchronous parallel execution in Python (asyncio.gather) to fetch live API data and vector embeddings concurrently. We also enabled audio streaming so Murf Falcon 2 begins sending audio frames back to the client while the remainder of the response is still generating.

Step-by-Step Guide: How to Build & Run AgentX

Want to run AgentX locally or build your own voice agent? Follow these steps:

Prerequisites

Step 1: Clone the Repository 

git clone https://github.com/singhsudhanshu22168-web/Voice-Agent-Day9.git
cd Voice-Agent-Day9
Enter fullscreen mode Exit fullscreen mode

Step 2: Set Up Virtual Environment & Install Dependencies 

python -m venv venv
# On Windows:
venv\Scripts\activate
# On Linux/Mac:
source venv/bin/activate

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Step 3: Configure Environment Variables

Create a .env file in the root directory (never commit this file to Git!):

LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_falcon_api_key
OPENAI_API_KEY=your_openai_api_key
Enter fullscreen mode Exit fullscreen mode

Step 4: Run the Agent Worker 

python agent.py dev
Enter fullscreen mode Exit fullscreen mode

Step 5: Connect and Test

Open the LiveKit Agents Playground or your custom frontend client, connect to the room, and start speaking to AgentX in Hindi or Hinglish:

Farmer: "Namaste AgentX, aaj meri fasal ke liye mausam kaisa rahega?"

AgentX: "Namaste! Aaj aapke ilake me halki baarish hone ki sambhavna hai, isliye aaj keeTnaashak ka chhidkaav na karein."

Conclusion & Acknowledgments

Over the 10 Days of AI Voice Agents challenge, building AgentX demonstrated the immense power of voice-first AI for Bharat. Voice bridges literacy and digital access gaps, making advanced crop, weather, and market intelligence accessible to every farmer.

Huge thanks to Murf AI for hosting the #VoiceForBharat Challenge 2026 and providing access to the incredible Murf Falcon 2 text-to-speech model!

If you found this guide helpful, check out the code on GitHub and leave a on the repository!

Top comments (0)