DEV Community: Rishabh Kumar

🌿 NatureQuest AI: Turn Screen Time into Green Time with Open-Source AI

Rishabh Kumar — Mon, 12 Oct 2026 04:40:18 +0000

What if AI didn't ask you to spend more time on your screen—but inspired you to step outside?

NatureQuest AI is an AI-powered outdoor adventure application that transforms everyday nature exploration into fun, interactive missions. It encourages users to put their devices down, explore the world around them, observe nature, and build healthier digital habits.

This is my submission for the Hacktoberfest Open-Source AI Challenge — Week 1: Touch Grass.

🌱 1. The Problem: We Spend Too Much Time on Screens

We use technology for almost everything—learning, communication, entertainment, and work. But sometimes, scrolling through a screen replaces the simple experiences that make life interesting.

We might spend hours indoors without noticing the birds outside, the plants around us, or the small details of our environment.

I wanted to explore a different question:

Can we use AI to encourage people to spend more time in the real world instead of keeping them glued to a screen?

That question inspired me to build NatureQuest AI.

🚀 2. What I Built

NatureQuest AI turns outdoor exploration into a mission-based experience.

Instead of simply opening another app to scroll, users can discover a challenge, go outdoors, complete the activity, and track their progress.

✨ Key Features

🤖 AI-Powered Outdoor Missions: Generate nature-related activities and real-world challenges using an open-weight AI model.
🌳 Explore Your Surroundings: Discover activities involving plants, birds, trees, outdoor observation, and environmental awareness.
📸 Photo Evidence Verification: For missions that require visual evidence, users can submit a photo for verification.
🐦 Bird Sound Verification: Bird-observation missions can use recorded audio as supporting evidence.
🏆 XP and Rewards: Earn experience points, build streaks, and unlock badges through mission completion.
📔 Nature Journal: Keep track of discoveries and outdoor experiences.
📶 Offline-Friendly Missions: Use predefined missions when AI generation is unavailable.
🎮 Gamified Progress: Turn small outdoor activities into an engaging journey.

Feature note: Keep the photo and bird-audio features described as available only if their verification flow has been implemented and tested. Otherwise, label them as work in progress.

🔄 3. How NatureQuest AI Works

Here is the high-level user journey.

flowchart TD
    A["🌿 Open NatureQuest AI"] --> B["Choose or Generate a Mission"]
    B --> C["🤖 AI Suggests an Outdoor Challenge"]
    C --> D["🌳 Go Outside and Explore"]
    D --> E{"Does the Mission Require Evidence?"}
    E -- "Yes" --> F["📸 Photo or 🐦 Audio Evidence"]
    F --> G["🔍 Verify Evidence"]
    G --> H{"Evidence Accepted?"}
    H -- "No" --> I["Try Again"]
    I --> F
    H -- "Yes" --> J["Complete Mission"]
    E -- "No" --> J
    J --> K["🏆 Earn XP and Update Progress"]
    K --> L["📔 Record the Experience"]
    L --> M["Explore the Next Mission"]

The important idea is that AI creates the opportunity, but the real experience happens outside the application.

Evidence verification should be used only for relevant missions. Activities that do not require evidence should continue through the normal completion flow.

🧠 4. AI and System Architecture

NatureQuest AI is designed around an open-weight AI workflow for mission generation, with a separate application layer for mission management and progress tracking.

flowchart TB
    U["👤 User"] --> FE["Next.js + TypeScript Frontend"]
    FE --> API["FastAPI Backend"]
    API --> MS["Mission Service"]
    MS --> AI["Open-Weight AI Model"]
    AI --> MS
    MS --> VAL["Validate Mission Structure"]
    VAL --> FE
    FE --> OUT["Outdoor Activity"]
    OUT --> EV["Optional Evidence Submission"]
    EV --> VER["Evidence Verification"]
    VER --> COMPLETE["Mission Completion Logic"]
    COMPLETE --> XP["XP, Streaks and Badges"]
    XP --> JOURNAL["Nature Journal and Progress"]

What each layer does

Component	Responsibility
Frontend	Displays missions, progress, and the user interface
Backend API	Handles requests and application logic
Mission service	Organizes mission generation and mission data
Open-weight model	Generates outdoor mission ideas
Evidence verification	Evaluates relevant photo or audio submissions
Progress system	Tracks completion, XP, streaks, and badges
Nature Journal	Records the user's outdoor experiences

The exact inference provider and verification models depend on the implementation used in the final application.

🛠️ 5. Tech Stack

Technology	Purpose
Next.js	Frontend application
TypeScript	Type-safe application development
Tailwind CSS	Styling and responsive layouts
Python	Backend development
FastAPI	Backend API framework
Pydantic	Data validation and structured schemas
Gemma / another suitable open-weight model	AI mission generation
Ollama, if configured	Local model inference
Git and GitHub	Version control and collaboration

Important: Keep only the technologies and AI models actually used in your project. If you use a different model or inference setup, update this table accordingly.

🔓 6. Why Open Innovation Matters

Open innovation is important because developers should have choices in how they build and deploy AI-powered applications.

For NatureQuest AI, open-weight models create opportunities to experiment with mission generation, test different models, and explore local inference where the model and hardware support it.

Why I care about open-source AI

Flexibility: Choose a model based on quality, speed, resource requirements, and licensing.
Experimentation: Compare different models for generating practical outdoor missions.
Deployment choices: Explore local inference or hosted deployment according to project requirements.
Community contributions: Allow other developers to suggest missions, improve code, fix bugs, and extend the application.
Greater control: Reduce dependence on one proprietary AI provider, depending on the chosen architecture.

Open-weight models do not automatically make an application free, completely private, or fully offline. Those properties depend on the model licence, deployment, hardware, data handling, and external services.

For me, the most interesting part is using open AI for something beyond another chatbot: encouraging people to reconnect with nature.

🌳 7. A Real-World Example

Imagine a user opens NatureQuest AI on a Saturday morning.

The application suggests a mission:

Mission: Become a Bird Observer 🐦

Go outside to a safe, accessible place.
Spend a few minutes quietly observing your surroundings.
Notice the birds and listen to the sounds around you.
If the mission requires it, record a short audio sample.
Submit the evidence for verification.
Complete the mission and earn XP if it passes the required checks.
Record the experience in your Nature Journal.

The activity is simple, but it encourages observation, curiosity, and time away from endless scrolling.

Evidence verification can help discourage unsupported completion claims, although an AI classifier cannot guarantee that every submission is genuine or correctly identified.

🎨 8. Design Philosophy

I wanted the interface to feel calm, natural, friendly, and inviting—not like a corporate dashboard.

The visual direction uses:

Sage green and earthy colors
Warm cream backgrounds
Rounded cards and approachable layouts
Clear mission instructions
Visible progress and rewards
Nature-inspired visual elements

The design supports the purpose of the application: make outdoor exploration feel accessible and enjoyable.

🧩 9. Challenges and What I Learned

Building NatureQuest AI helped me think about AI as part of a complete product, rather than just a model that generates text.

Some important engineering considerations include:

Structured AI output: Generated missions need a predictable format that the application can validate.
Reliable fallback behavior: Users should still have missions available if AI generation fails.
Evidence quality: A photo or audio clip may be unclear, irrelevant, or insufficient for verification.
Responsible gamification: Rewards should encourage outdoor exploration without making users feel pressured to stay in the app.
Privacy: Camera and microphone access should be requested only when needed, with clear controls for reviewing and deleting evidence.
Hardware limitations: Running an open-weight model locally depends on its size, quantization, and available system resources.

One lesson stands out: AI features are only useful when they support a meaningful user experience and work reliably within the limits of the system.

Before publishing, add one or two challenges you personally encountered and solved during development. Real examples make this section stronger.

🔮 10. What's Next?

NatureQuest AI can grow in several directions:

🌼 More mission categories for different ages and interests
🐦 Improved bird-sound recognition and clearer verification feedback
📷 Better image relevance checking
🌍 Optional location-aware missions with explicit permission
♿ Accessibility improvements for different users
🌱 Community-created mission packs
📊 Personal outdoor activity history and progress insights
🔋 Improved performance and offline support

These are potential improvements, not claims that all of them are already implemented.

Keep a record of outdoor experiences.

Replace these placeholders with real, publicly accessible image URLs. You can also embed a short demo video.

🎬 11. Live Demo

Try NatureQuest AI: https://drive.google.com/file/d/1E4YNQP3uqlSZSev_cjVn-vvIMQ2dOOsF/view?usp=sharing

If the application is not deployed yet, replace this section with a short video demonstration or remove it until the demo is available.

💻 12. Source Code

GitHub Repository: https://github.com/Rishabh-KSingh/NatureQuest-AI

The repository contains the project source code and implementation details.

If you are interested in open-source AI, nature-based applications, or creative uses of technology, feel free to explore the project and share feedback.

How to Contribute

Potential contributions include:

Adding new outdoor missions
Improving mission generation and validation
Testing evidence verification
Improving accessibility and mobile usability
Reporting bugs or suggesting new features
Improving documentation and setup instructions

Please follow the contribution instructions in the repository before submitting changes.

⚙️ 13. Run Locally

Use the repository's actual installation instructions if they differ from this example.

Prerequisites

Node.js and the package manager used by the project
Python and the project's required version
Git
Access to the configured AI model or inference service

Setup

# Clone the repository
git clone YOUR_GITHUB_REPOSITORY_URL

# Enter the project directory
cd YOUR_PROJECT_DIRECTORY

Install frontend and backend dependencies using the instructions and dependency files included in the repository.

Configure the required environment variables, then start the frontend and backend using the project's documented development commands.

Note: The commands above are a starting point, not a complete verified setup guide. Add the exact install, environment configuration, and run commands from your working repository before publishing.

💚 14. Final Thoughts

NatureQuest AI began with a simple idea:

What if we built AI that encouraged people to spend less time looking at screens and more time looking at the world?

I wanted to explore a positive use of technology—one that inspires curiosity, supports outdoor activities, and makes reconnecting with nature more engaging.

This project is my contribution to the Touch Grass challenge, and I hope it inspires other developers to experiment with open-source AI in creative, human-centered ways.

Technology doesn't always have to keep us online. Sometimes, it can give us a reason to step outside.

🌿 Build with AI. Explore with curiosity. Reconnect with nature.

I'd love to hear your feedback and ideas!

Hacktoberfest #OpenSourceAI #TouchGrass #NatureQuestAI #ArtificialIntelligence #OpenSource #NextJS #Gemma

I built my friend an AI speaking partner because knowing English isn't the same as speaking it.

Rishabh Kumar — Mon, 05 Oct 2026 05:59:07 +0000

My friend understands English.

They can read it.

They can watch English videos.

They know a decent number of English words.

But ask them to speak.

There is a pause.

They first think in Hindi, then translate the sentence into English, then worry whether the grammar is correct, and sometimes the sentence never comes out.

And when they do make a mistake, they often don't know why it was wrong.

I noticed something else too.

The problem wasn't a lack of English content.

There are thousands of videos, grammar lessons, vocabulary apps and courses available.

The problem was practice.

There wasn't someone available every day who would patiently listen, correct the important mistakes, explain them, and then say:

"Okay. Now say it again."

So I decided to build that person as an AI.

That's how TalkSaathi AI started.

What TalkSaathi does

TalkSaathi AI is an English-speaking companion designed for someone who understands English but struggles to speak confidently.

Its core idea is simple:

Listen → Understand → Recall → Speak → Correct → Repeat → Converse → Track → Adapt

Instead of completing a grammar exercise and moving on, TalkSaathi turns a mistake into the next practice opportunity.

For example, imagine my friend says:

"I take breakfast at 8 and I go college."

TalkSaathi doesn't just say "wrong."

You said

I take breakfast at 8 and I go college.

A better version

I have breakfast at 8 and I go to college.

Why?

We usually say "have breakfast" in natural English.

We also use "to" before a destination.

More natural

I usually have breakfast at 8 and then go to college.

And then comes the important part:

🎤 Now say it again.

The learner repeats the sentence.

The AI checks the new attempt.

That creates a much more useful loop:

┌─────────────┐
│   Mistake   │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Correction  │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Explanation │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   Repeat    │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Re-check   │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Improvement │
└─────────────┘
       │
       └──────────────► 🔄 Next Practice

The problem I actually wanted to solve

While building this, I kept coming back to one question:

Why can someone understand English but still struggle to speak it?

For my friend, several things were connected.

They were:

translating from Hindi before speaking;
afraid of making grammar mistakes;
unsure how to form sentences naturally;
using the same simple vocabulary repeatedly;
not getting enough speaking practice;
lacking a conversation partner.

So TalkSaathi isn't designed to make someone memorize thousands of words.

It's designed to give them a place where they can actually speak.

The first practice session

When a learner opens TalkSaathi for the first time, there is no complicated registration.

They create a small profile:

English level;
learning goal;
interests;
preferred topics;
daily practice time;
speaking goal.

They can choose:

10, 15, 20 or 30 minutes.

TalkSaathi then creates a practice session based on their level and previous performance.

A typical session might look like:

        ┌────────────┐
        │   Listen   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │ Understand │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │   Recall   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │    Speak   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │   Correct  │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │   Repeat   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │Conversation │
        └────────────┘

The important part is that the practice doesn't end after the first answer.

The learner gets another chance.

I didn't want Hindi to become a dependency

One feature I particularly wanted was a Hindi → English speaking bridge.

Because when someone is learning English, sometimes they know exactly what they want to say—but they don't know how to say it in English.

For example:

"मुझे कल कॉलेज जाना है।"

TalkSaathi can turn that into:

I have to go to college tomorrow.

Then it explains the sentence and asks:

🎤 Now say it in English.

After the learner repeats it, TalkSaathi doesn't stop at the translation.

It continues:

What do you usually do at college?

This is important.

Hindi is not supposed to become the final destination.

The goal is to gradually move from:

Hindi thought
     ↓
English translation
     ↓
English speaking

towards:

English thought
     ↓
English speaking

It also remembers mistakes

Another problem with normal practice is that the same mistake can happen again tomorrow.

And again next week.

TalkSaathi keeps a lightweight local history of recurring mistakes.

For example:

Past tense       8 mistakes
Prepositions     6 mistakes
Articles         4 mistakes
Vocabulary       3 mistakes

If the learner repeatedly says:

"I didn't went."

TalkSaathi can recognize that as a recurring pattern.

The next practice can then include:

Past Tense Challenge

So the system isn't only asking:

"What did you say?"

It's also asking:

"What does this learner need to practice next?"

That's where the personalization comes from.

Conversation mode

Sometimes you don't want a lesson.

You just want to talk.

TalkSaathi has a free conversation mode with topics such as:

Daily life

Family, food, college, hobbies, friends.

Professional

Job interviews, meetings, presentations and workplace situations.

Social

Shopping, restaurants, travel and asking directions.

Education & Technology

Programming, AI, college and learning.

Or you can simply enter your own topic.

The AI doesn't try to correct every single word while you're talking.

That would make the conversation feel like an exam.

Instead, it keeps the conversation natural and saves deeper feedback for the end.

A conversation report can show:

Grammar       78%
Fluency       81%
Vocabulary    74%

Important corrections: 4

Focus next:
1. Past tense
2. Articles
3. Prepositions

I wanted the AI to feel different in different moments

TalkSaathi has four personalities, depending on what the learner is doing.

During learning:

Friendly teacher

During conversation:

Natural friend

During correction:

Supportive tutor

During speaking practice:

Encouraging coach

The AI should never make the learner feel embarrassed about making mistakes.

A mistake is not a failure.

It's the next thing to practice.

What happens when you don't know the sentence at all?

I also wanted TalkSaathi to work outside a normal lesson.

So I added a photo-based grammar feature.

A learner can upload a photo containing English text.

For example:

"I am going market yesterday."

TalkSaathi extracts the text and analyzes it.

It returns:

Original

I am going market yesterday.

Correct

I went to the market yesterday.

Why?

"Yesterday" tells us that the action happened in the past, so we use went.

Natural version

I went to the market yesterday.

And then:

🎤 Try saying the corrected sentence.

So even a photo becomes a speaking exercise.

How I built it

The architecture is intentionally split into two important layers:

The open-weight model is the brain.

ElevenLabs is the voice interface.

╭─────────────────────────────────────────────────────────────╮
│                    🗣️  TALKSAATHI AI                       │
│              Your AI Saathi for English Speaking           │
╰─────────────────────────────┬───────────────────────────────╯
                              │
                              ▼
                    ┌───────────────────┐
                    │  Next.js Frontend │
                    └─────────┬─────────┘
                              │
              ┌───────────────┼────────────────┐
              │               │                │
              ▼               ▼                ▼
        🎤 VOICE INPUT    ⌨️ TEXT INPUT    🖼️ IMAGE INPUT
              │               │                │
              ▼               │                ▼
       ┌─────────────┐        │         ┌─────────────┐
       │ ElevenLabs  │        │         │     OCR     │
       │     STT     │        │         └──────┬──────┘
       └──────┬──────┘        │                │
              └───────────────┼────────────────┘
                              ▼
                    ┌───────────────────┐
                    │      🧠 GEMMA     │
                    │  Open-weight AI   │
                    │       Brain       │
                    └─────────┬─────────┘
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
      ✨ Correction     💬 Conversation   🎯 Personalization
             │                │                │
             └────────────────┼────────────────┘
                              ▼
                    ┌───────────────────┐
                    │  ElevenLabs TTS   │
                    │   AI Voice Output │
                    └─────────┬─────────┘
                              │
                              ▼
                    🔊 SPEAK • PRACTICE
                              │
                              ▼
                       👤 THE LEARNER
                              │
                              └───────────────┐
                                              │
                                  🔄 Repeat & Improve
                                              │
                                              └───► 🧠 Gemma

The application is built with:

Next.js
TypeScript
Tailwind CSS
shadcn/ui
Gemma / compatible open-weight inference
Hugging Face / compatible inference provider
ElevenLabs Speech-to-Text
ElevenLabs Text-to-Speech
OCR
LocalStorage
Vercel

The backend uses Next.js API routes, so API credentials stay on the server.

For the MVP, I deliberately avoided forcing users to create an account.

Learning data such as progress, mistakes, phrases and preferences can be stored locally.

Raw voice recordings and uploaded photos don't need to be permanently stored.

Why an open-weight model?

This was an important choice for me.

I didn't want to build:

Frontend → closed AI API → response

and call that the whole product.

The open-weight model is responsible for the actual language-learning intelligence:

grammar correction;
explanations;
Hindi → English conversion;
conversation;
lesson generation;
mistake analysis;
personalized exercises;
phrase suggestions;
image-text language analysis.

That gives TalkSaathi a separate reasoning layer from its voice layer.

The architecture becomes:

                 🧠 GEMMA
                    │
                    ▼
              ┌───────────┐
              │   BRAIN   │
              └─────┬─────┘
                    │
                    │ Understand
                    │ Correct
                    │ Explain
                    │ Personalize
                    │ Converse
                    │ Generate
                    ▼
           ┌─────────────────┐
           │   ElevenLabs    │
           │      VOICE      │
           └────────┬────────┘
                    │
                    ▼
              🎤 Listen & Speak

This also means the AI layer can evolve independently.

A different open model can be tested later without rebuilding the entire product.

Why not just use Hindi → English translation?

Because translation isn't the goal.

If the app simply translated everything for the learner, the learner could become dependent on translation.

That's why TalkSaathi uses translation as a temporary bridge.

The sequence is:

"I’m stuck"
      ↓
   Hindi
      ↓
English sentence
      ↓
Understand why
      ↓
Say it yourself
      ↓
Continue in English

The final goal is not better translation.

It's less need for translation.

What building this taught me

One thing became obvious while designing the product:

The hardest part isn't getting an AI model to generate a correction.

The harder question is:

What should happen after the correction?

If the app simply says:

"Correct sentence: ..."

the learning moment is over.

But if it says:

"Now you try."

the mistake becomes practice.

Then another question appears:

What if the learner makes the same mistake tomorrow?

That's why TalkSaathi has mistake memory.

And then:

What should they practice tomorrow?

That's why the system uses the learner's history to recommend the next activity.

The product became much more interesting once I stopped thinking about individual AI responses and started thinking about the learning loop.

The part I'm most excited about

It's not the dashboard.

It's not the number of features.

It's this tiny interaction:

"Now say it again."

Because that's the difference between an AI that answers questions and an AI that actually helps someone practice a skill.

TalkSaathi is designed around that idea.

Handing it to my friend

The real test isn't whether I can make the demo work.

The real test is whether my friend can open it tomorrow, press Start Practice, speak without feeling embarrassed, make mistakes, and want to come back again.

That's what I wanted to build.

Not another English course.

Not another chatbot.

A speaking companion.

What's next?

The current version focuses on the core speaking experience.

There are still things I want to improve:

more advanced pronunciation analysis;
better adaptive lesson generation;
cloud sync;
user accounts;
PostgreSQL + Prisma;
mobile support;
more languages;
teacher dashboards;
community speaking challenges.

But I want to validate the most important thing first:

Does regular speaking practice with TalkSaathi actually make my friend more confident?

Prize Category

Best Use of ElevenLabs

TalkSaathi uses ElevenLabs for:

Speech-to-Text;
Text-to-Speech;
spoken learning content;
AI conversation voice.

The reasoning layer remains an open-weight model, while ElevenLabs handles the voice interaction.

Try TalkSaathi

Source Code: https://github.com/Rishabh-KSingh/TalkSaathi-AI

Demo Video: https://drive.google.com/file/d/17Pd4YNAB7QNAFNTbSCoJyoxhqDwOKSQ3/view?usp=sharing

I started with a simple observation:

Knowing English and speaking English are two different skills.

So I built something for someone who needed more opportunities to speak.

TalkSaathi AI — Your AI Saathi for English Speaking.

Don't just learn English. Practice speaking it.

Building KisanMitra AI: A Full-Stack Voice Agent for Indian Farmers 🌾🎙️

Rishabh Kumar — Sat, 15 Aug 2026 17:00:11 +0000

Day 10 of the **10 Days of Voice Agents – VoiceForBharat Edition* Challenge | Farm & Field Track*

Imagine holding a smartphone in the middle of a mustard field in rural India.

Your hands are covered in soil. The sun makes the screen difficult to read. You need an answer about your crop, weather, or mandi prices — but typing a long query or navigating through complicated portals isn't practical.

Now imagine simply pressing a button and saying:

"भैया, कल बारिश होगी क्या? और धान में भूरा धब्बा रोग लगा है, क्या करें?"

Instead of searching through menus, you can simply talk.

That's the idea behind KisanMitra AI (किसान मित्र AI) — a full-stack, real-time bilingual voice agent designed specifically around agricultural assistance.

The project combines:

🎙️ Murf Falcon TTS
⚡ LiveKit Agents
📝 Deepgram STT
🧠 Google Gemini
🌐 Next.js 15
🗄️ SQLite
📞 SIP-based calling
🚨 Human escalation
🤝 Specialist agent handoff

This article is the complete story of how I built KisanMitra AI during the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

🇮🇳 1. Why Voice for Indian Farmers?

Voice assistants are often treated as convenience tools.

But in agriculture, voice can solve a much more practical problem: accessibility.

Farmers may be working outdoors, using their hands, dealing with sunlight, speaking in Hindi or regional languages, and needing information quickly.

The interaction shouldn't look like this:

Open App
   ↓
Find Menu
   ↓
Select Category
   ↓
Fill Form
   ↓
Type Question
   ↓
Wait for Result

It should feel more like this:

👨‍🌾 Farmer
    ↓
🎙️ "मेरी फसल में ये समस्या हो रही है..."
    ↓
🤖 KisanMitra AI
    ↓
🔊 Natural Voice Response

For the Farm & Field track, my objective was simple:

Make agricultural assistance as conversational and accessible as possible.

🏗️ 2. End-to-End System Architecture

KisanMitra AI connects a real-time voice pipeline with AI reasoning, tools, memory, analytics, telephony and human escalation.

                         👨‍🌾 FARMER
                             │
                 ┌───────────┴───────────┐
                 │                       │
          🌐 Web Browser            📞 Phone / SIP
                 │                       │
                 └───────────┬───────────┘
                             ▼
                       ⚡ LIVEKIT
                    Real-Time Audio
                             │
                             ▼
                    🎙️ Silero VAD
                  Turn Detection
                             │
                             ▼
                    📝 Deepgram STT
                    Hindi / Hinglish
                             │
                             ▼
                   🧠 KISANMITRA AGENT
                             │
                ┌────────────┼────────────┐
                ▼            ▼            ▼
             Gemini       Memory        Tools
                │            │            │
                │            ▼            ├── Weather
                │         SQLite           ├── Farmer Profile
                │                           └── Escalation
                │
                ▼
             Decision
                │
        ┌───────┴────────┐
        ▼                ▼
   Normal Answer    Complex Problem
        │                │
        │                ▼
        │          🌾 FASAL DOCTOR
        │          Specialist Agent
        │
        └────────┬───────────────┘
                 ▼
            🔊 MURF FALCON
             Text-to-Speech
                 │
                 ▼
              LIVEKIT
                 │
                 ▼
           👨‍🌾 FARMER HEARS

The architecture and integration layers are based on the project structure documented during the build.

🔄 3. How a Voice Conversation Works

Let's follow one conversation from beginning to end.

Step 1 — Farmer speaks

The farmer speaks through the browser microphone or a supported phone/SIP connection.

Step 2 — LiveKit transports audio

LiveKit handles the real-time audio communication.

Step 3 — Voice Activity Detection

Silero VAD detects when the farmer starts and stops speaking.

Step 4 — Speech-to-Text

Deepgram Nova-3 STT converts the audio into text.

It is designed to handle Hindi/Hinglish conversations and code-switching.

For example:

"सरसों का Mandi Bhav क्या है?"

Step 5 — Gemini processes the request

Google Gemini receives the transcript and relevant context.

The agent decides whether it needs to:

Answer directly
Access memory
Call a tool
Escalate the issue
Hand the conversation to a specialist

Step 6 — Murf Falcon generates the voice

The generated response is sent to Murf Falcon TTS.

The response is then streamed back as audio.

Step 7 — Farmer hears the response

The audio travels through LiveKit back to the user.

So the complete pipeline is:

🎙️ Speech
   ↓
LiveKit
   ↓
Silero VAD
   ↓
Deepgram STT
   ↓
Google Gemini
   ↓
Tools / Memory
   ↓
Murf Falcon
   ↓
LiveKit
   ↓
🔊 Speech

⚡ 4. Why Low Latency Matters

Voice conversations feel very different from text conversations.

If an AI takes several seconds before responding, users may think:

"Call cut gaya kya?"

That's why latency became an important part of this project.

With streaming TTS, the project targeted approximately 114–135 ms TTFB for the Murf Falcon voice layer in the tested setup.

The goal was to make the response feel immediate rather than waiting for the entire response to be generated first.

🎨 5. Building the Frontend

I didn't want KisanMitra AI to look like a basic chatbot.

I built a custom Next.js 15 + Tailwind CSS + shadcn/ui interface with an agriculture-inspired emerald/green theme.

🌿 Main UI Features

Glassmorphism

Frosted glass cards and layered panels give the interface a modern look.

🎙️ Listening State

The UI visually indicates when the agent is listening.

🌱 Thinking State

A separate state indicates that the AI is processing the request.

🔊 Speaking State

Audio visualizations show when the agent is responding.

💬 Live Transcript

The user can see the conversation while speaking with the agent.

📦 6. Six Agricultural Modules

The main dashboard contains six smart agricultural modules.

Module	Purpose
⛅ Weather Forecast	Weather, rain and temperature information
🌱 Crop Advisory	Crop and farming guidance
📈 Mandi Bhav	Market-related information
🐛 Disease Control	Crop disease and pest assistance
🐄 Livestock Care	Animal and dairy-related assistance
🏛️ Government Schemes	Agricultural scheme information

Example voice prompts:

"क्या कल मेरे खेत में बारिश होगी?"

"गेहूं की पहली सिंचाई कब करनी चाहिए?"

"धान की पत्तियों में धब्बे आ गए हैं, क्या करूं?"

"आज मंडी में सरसों का भाव क्या है?"

These modules are part of the frontend experience documented in the project.

🚀 7. The 9-Day Technical Journey

KisanMitra AI wasn't built in one day.

It evolved throughout the challenge.

Day 1 — Give the Agent a Voice 🎙️

The first step was creating the basic real-time voice agent.

I connected the voice pipeline and integrated Murf Falcon TTS.

This transformed the project from a text-based concept into a conversational assistant.

Day 2 — Build the User Interface 🎨

I moved beyond the backend and created a proper frontend.

The goal was to give users a simple way to:

Start a conversation
See the agent state
View transcripts
Interact with agricultural modules

Day 3 — Add Memory 🧠

A useful assistant shouldn't forget everything after every call.

I introduced persistent storage using SQLite so that useful farmer information and interaction data could be retained.

For example:

User
 ├── Name
 ├── Language
 ├── District
 ├── Crop
 └── Previous Interaction

Day 4 — Connect Real-World Tools 🔧

The agent needed access to information outside the LLM.

I added domain tools that allow KisanMitra AI to perform actions such as:

Farmer profile lookup
Profile updates
Weather lookup
Escalation
Call status logging

The core tools are documented around the FarmerTools layer in the project.

Day 5 — Make the Agent Proactive 📞

Instead of always waiting for a farmer to start a conversation, I explored outbound SIP calling.

This opens the possibility of proactive alerts.

For example:

Weather API
     ↓
Severe Weather Detected
     ↓
KisanMitra AI
     ↓
📞 Outbound Voice Call
     ↓
👨‍🌾 Farmer

Day 6 — Human Escalation 🚨

AI shouldn't pretend to know everything.

For complex or serious cases, KisanMitra AI can escalate the issue.

The system can create a unique reference such as:

KISAN-7842

and send the relevant escalation information through a Discord webhook.

This creates a human-in-the-loop workflow.

Day 7 — Analytics Dashboard 📊

Once the agent could handle conversations, I needed to understand how it was performing.

So I built an analytics dashboard using:

Next.js + SQLite

The dashboard can track:

Total calls
Successful calls
Failed calls
Call logs
Session information
Conversation data

Day 8 — Specialist Agent Handoff 🤝

One of the most interesting additions was the Fasal Doctor.

Instead of forcing one agent to handle every agricultural problem:

                    KisanMitra AI
                         │
             ┌───────────┴───────────┐
             │                       │
       General Query          Complex Crop Issue
             │                       │
             ▼                       ▼
         Response              🌾 Fasal Doctor

The specialist-agent concept allows complex crop-related conversations to be routed to a dedicated agent.

Day 9 — Bring Everything Together 🚀

By this stage, KisanMitra AI combined:

Voice + Memory + Tools + Calling + Human Escalation + Analytics + Specialist Handoff

The project had evolved from a basic voice assistant into a much more complete voice-agent architecture.

📂 8. Project Folder Structure

The project follows a monorepo-style structure separating the Python voice-agent backend from the Next.js frontend.

murf-livekit-starter/
│
├── backend/
│   ├── src/
│   │   ├── agent.py
│   │   └── db.py
│   │
│   ├── tests/
│   │   ├── test_agent.py
│   │   ├── test_day5.py
│   │   └── test_day8.py
│   │
│   ├── .env.example
│   ├── .env.local
│   ├── pyproject.toml
│   └── kisan_mitra.db
│
├── frontend/
│   ├── app/
│   │   ├── api/
│   │   │   ├── token/
│   │   │   │   └── route.ts
│   │   │   └── analytics/
│   │   │       └── route.ts
│   │   │
│   │   ├── dashboard/
│   │   │   └── page.tsx
│   │   │
│   │   ├── layout.tsx
│   │   ├── page.tsx
│   │   └── globals.css
│   │
│   ├── components/
│   │   ├── app/
│   │   ├── agents-ui/
│   │   └── ui/
│   │
│   ├── app-config.ts
│   ├── package.json
│   └── .env.local
│
├── start_app.sh
├── start_app.ps1
├── AGENTS.md
├── README.md
└── BLOG_DEV_COMMUNITY.md

This structure follows the backend/frontend separation documented in the source project.

🧠 9. Important Backend Components

`backend/src/agent.py`

This is the core of the voice agent.

It connects:

STT
 ↓
LLM
 ↓
Tools
 ↓
TTS

It also contains the agent instructions and farmer-specific tools.

`backend/src/db.py`

This manages SQLite persistence.

The project uses tables such as:

users
call_logs

The users table can store useful profile information, while call_logs stores session-related information.

`frontend/app/api/token/route.ts`

This API route generates LiveKit access tokens and handles agent dispatch configuration.

This is important because the frontend needs a secure way to establish a LiveKit session.

`frontend/app/dashboard/page.tsx`

This page provides the analytics dashboard.

It displays information such as:

Total calls
Success rate
Active sessions
Call logs
Transcripts

🚧 10. The Developer Struggles

No real project is complete without debugging.

These were two of my biggest lessons.

💥 Struggle #1 — The Silent Agent

On Day 1, everything appeared to be running.

The backend was active.

The frontend connected.

The microphone worked.

But the agent was completely silent.

After hours of debugging, I found the issue.

My .env.local contained:

# ❌ Wrong
AGENT_NAME=

The agent name was empty.

I changed it to:

# ✅ Fixed
AGENT_NAME=my-agent

After that, the correct agent dispatch was triggered and the voice agent finally started speaking.

Lesson learned:

Always check environment variables before assuming the entire voice pipeline is broken.

The project documentation specifically identifies explicit agent routing through AGENT_NAME as part of the LiveKit token/dispatch setup.

💥 Struggle #2 — The Beautiful but Useless Mic Button 😅

I created a beautiful animated microphone button.

It looked amazing.

But when I clicked it...

Nothing happened.

The problem?

I had designed the UI but hadn't connected the button to the actual LiveKit session lifecycle.

The fix was connecting it with useSessionContext():

import { useSessionContext } from '@livekit/components-react';

export function ActiveMicButton() {
  const session = useSessionContext();

  const handleToggle = async () => {
    if (session.state !== 'connected') {
      await session.start();
    } else {
      await session.end();
    }
  };

  return (
    <button onClick={handleToggle}>
      Start Conversation
    </button>
  );
}

Once connected to:

session.start()

the button actually started the voice session.

Lesson learned:

A beautiful UI is useless if it isn't connected to the actual application logic.

🛠️ 11. How You Can Build Your Own Voice Agent

If you're new to voice AI, start with four major components.

┌──────────────┐
│     STT      │
│  Deepgram    │
└──────┬───────┘
       ↓
┌──────────────┐
│     LLM      │
│ Gemini / GPT │
└──────┬───────┘
       ↓
┌──────────────┐
│     TTS      │
│ Murf Falcon  │
└──────┬───────┘
       ↓
┌──────────────┐
│  LiveKit     │
│  WebRTC      │
└──────────────┘

1. Speech-to-Text

Convert the user's voice into text.

Example: Deepgram

2. LLM

The brain of the agent.

Example: Google Gemini

3. Text-to-Speech

Convert the generated answer back into speech.

Example: Murf Falcon

4. Real-Time Transport

Move audio between the user and agent.

Example: LiveKit

The project uses this four-part STT → LLM → TTS → real-time transport architecture.

🔐 12. Security: Don't Leak Your API Keys

This is extremely important when building AI applications.

Never commit:

.env
.env.local

to GitHub.

Instead:

.env.example

should contain only the required variable names.

For example:

MURF_API_KEY=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
DEEPGRAM_API_KEY=
GOOGLE_API_KEY=

And the real values should remain in your local environment.

🚨 Never publish real API keys in a public GitHub repository.

🔮 13. What's Next for KisanMitra AI?

KisanMitra AI is only the beginning.

📱 WhatsApp Voice Notes

The next step is making the assistant accessible through WhatsApp voice messages.

A farmer could simply send:

🎙️ Voice Note

and receive:

🔊 Voice Answer

🗣️ Regional Indian Languages

I want to expand beyond Hindi/Hinglish into languages and dialects such as:

Bhojpuri
Maithili
Haryanvi
Bundelkhandi
Marathi

The long-term goal is to make voice AI feel natural across different parts of India.

📸 Multimodal Crop Diagnosis

Imagine combining:

📸 Crop Image
     +
🎙️ Farmer's Voice
     +
🧠 AI

A farmer could show an infected leaf while explaining the symptoms verbally.

🛰️ Smarter Agricultural Data

Another future direction is integrating richer agricultural and environmental data sources to make recommendations more context-aware.

📚 14. What I Learned From This Challenge

The biggest lesson from these 10 days is:

Building a voice agent is much more than connecting an LLM to a microphone.

A useful voice agent needs:

🎙️ Natural voice interaction
🧠 Memory
🔧 Real-world tools
📞 Communication capabilities
🚨 Human fallback
📊 Analytics
🤝 Specialist agents
🎨 Good UX
🔐 Secure configuration

But above everything else:

It needs to solve a real problem.

For KisanMitra AI, that problem is making agricultural assistance easier to access through voice.

🔗 15. Project Links

💻 GitHub Repository

👉 [PASTE YOUR ACTUAL GITHUB LINK HERE]

🌐 Live Demo

👉 [PASTE YOUR ACTUAL LIVE DEMO LINK HERE]

🎥 Demo Video

👉 [PASTE YOUR YOUTUBE / LOOM VIDEO LINK HERE]

Don't forget to replace all three placeholders before publishing.

🙏 16. Final Thoughts

Building KisanMitra AI has been one of my most rewarding experiences with voice AI.

In these 10 days, I went from a basic voice interaction to building a system combining:

🎙️ Voice

↓

🧠 Memory

↓

🔧 Tools

↓

📞 Outbound Calls

↓

🚨 Human Escalation

↓

📊 Analytics

↓

🤝 Specialist Agent Handoff

This challenge taught me that the most exciting part of AI isn't just making models smarter.

It's making technology easier for people to use.

For a farmer, technology shouldn't feel complicated.

Sometimes the best interface is simply:

"भैया, मेरी फसल में ये समस्या हो रही है..."

And the AI should be ready to listen.

🌾 That's the idea behind KisanMitra AI.

❤️ Special Thanks

A huge thank you to Murf AI and LiveKit for organizing the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

The challenge gave me an opportunity to learn, experiment, break things, debug them, and build a voice AI system around a real-world Indian use case.

If you're also building with voice AI, I'd love to hear what you're creating.

What kind of voice agent are you building? 👇

Let's connect and build the future of voice AI together.

#10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AI #GenerativeAI #IndianAI #Agriculture #KisanMitraAI #LiveKit