DEV Community: Rishabh Kumar

YaadVault AI 🕯️ — Bringing Lost Memories Back to Life with Open-Source AI

Rishabh Kumar — Sat, 31 Oct 2026 14:40:22 +0000

What I Built

YaadVault AI is an AI-powered digital memory vault designed to preserve family stories, personal experiences, and memories before they disappear.

We all have stories worth remembering. Our grandparents have childhood memories, our parents have life experiences, and our families have stories that are often passed down through conversations.

But what happens when those stories are forgotten?

That's the problem I wanted to solve.

YaadVault AI transforms spoken memories into searchable digital records using open-source AI.

Users can record a family member sharing a story, convert that recording into text, generate an AI-powered summary, and save it permanently in their personal memory vault.

The idea behind the project is simple:

Technology should not only help us build the future; it should also help us preserve our past.

Key Features

🎙️ Voice Memory Recorder: Record personal stories and family conversations.
📝 AI Transcription: Convert spoken memories into readable text.
🧠 Smart Summarization: Generate meaningful summaries of long recordings.
📅 Memory Timeline: Organize memories chronologically.
💬 Ask Your Memories: Ask questions about saved stories using AI.
🔍 Semantic Memory Search: Find memories based on their meaning, not just exact keywords.
🔒 Privacy-First Architecture: Keep personal memories and AI processing local.

Demo

📸 Screenshots:

The demo will showcase the complete journey: recording a memory, transcribing it, saving it, and asking questions about it

How I Built It

I wanted YaadVault AI to be more than another application that simply sends user data to a third-party AI API.

That's why I chose an open-source, local-first AI architecture.

Technology Stack

Component	Technology
Frontend	Next.js
Styling	Tailwind CSS
Backend	Python + FastAPI
AI Model	Gemma
Local AI Runtime	Ollama
Speech Recognition	Whisper
Database	SQLite

How the AI Works

A user records a personal story.
Whisper converts the audio into text.
The transcribed text is passed to Gemma.
Gemma generates a summary and extracts useful information.
The memory is stored in the local database.
Users can later ask questions about their saved memories.

For question answering, the application retrieves relevant saved memories and provides them as context to the language model.

This helps keep answers grounded in the user's actual recordings rather than generating fictional family history.

The goal is to make the entire experience work locally, without depending on a closed AI API for its core intelligence.

Why Does Open Innovation Matter?

Open innovation is especially important when building applications that deal with personal memories and sensitive information.

Using open-weight models and local inference makes several things possible:

1. Privacy and Control

Personal stories and family recordings are valuable. Local AI processing gives users more control over where their data goes.

2. Freedom to Experiment

Open-weight models allow developers to experiment with different models, prompts, and inference configurations without being locked into a single provider.

3. Accessibility

With tools like Ollama, developers can experiment with local AI without needing to build an entire AI infrastructure from scratch.

4. Community Collaboration

Open-source development allows other developers to improve the application, suggest features, fix bugs, and make memory preservation accessible to more people.

For me, open innovation means that powerful AI technology should not be limited to companies with expensive infrastructure.

It should be available to everyone who has an idea worth building.

The Spirit of It

Imagine sitting with your grandmother and asking her:

"What was your childhood like?"

She starts telling you stories about her village, her school days, festivals, and people she once knew.

You record that conversation.

Years later, someone in your family opens YaadVault AI and asks:

"What was Grandma's childhood like?"

Instead of searching through old notebooks or trying to remember conversations, they can explore those memories again.

That is what inspired YaadVault AI.

I believe memories are more than old photographs or recordings. They are pieces of our identity, our family history, and our connection to the people we love.

People may leave, but their stories don't have to disappear.

That's the little piece of the future I wanted to build. ❤️

Final Thoughts

Building YaadVault AI has been an opportunity to explore how open-source AI can solve a problem that is deeply personal.

The project is built around one simple belief:

Every memory deserves a place to live.

If you like this idea, feel free to explore the repository, contribute, or share your feedback.

Let's build technology that keeps our memories alive! 🕯️

Hacktoberfest #OpenSourceAI #Gemma #Ollama #Whisper #ArtificialIntelligence #OpenSource #DevRelay

# TidyAI 🦥 — Let AI Organize Your Files While You Relax

Rishabh Kumar — Sat, 24 Oct 2026 19:10:12 +0000

Have you ever opened your Downloads folder and found hundreds of files scattered everywhere?

PDF documents, screenshots, college notes, images, ZIP archives, code files, certificates, and random documents — all mixed together in one place.

You tell yourself, "I'll organize these files later." But later never comes!

Manually sorting files is not difficult, but it is repetitive, time-consuming, and easy to postpone. As developers and students, we should spend more time learning, building, and creating rather than organizing files one by one.

That's the idea behind TidyAI — a privacy-focused, AI-powered file organization assistant designed to make everyday file management easier.

Instead of spending time sorting files manually, imagine having an assistant that helps you understand your files, suggests where they belong, and handles repetitive organization tasks with minimal effort.

The philosophy is simple:

Work smarter, automate the boring stuff, and let AI handle the repetitive tasks. 🦥

💡 What I Built

TidyAI is designed to turn a messy collection of files into a structured and organized workspace.

The project combines regular file-system automation with an open-weight AI model. Traditional file organizers usually depend on file extensions or predefined rules. TidyAI aims to go further by using AI to help interpret file context and recommend meaningful categories when simple rules are not enough.

For example, a folder might contain:

python_notes.pdf
IMG_20261020.png
resume_final_v3.pdf
certificate_cloud_computing.pdf
project_backup.zip
lecture_recording.mp4

Instead of manually reviewing every file, users could receive organization suggestions such as:

Documents: PDFs, Word files, and text documents
Images: Screenshots, photographs, and graphics
Study Materials: Notes, assignments, and learning resources
Certificates: Course certificates and achievements
Archives: ZIP files and compressed folders
Videos: Recordings, tutorials, and other video files

These categories are examples of the intended experience. More specific classifications, such as distinguishing a certificate from ordinary study material, may require inspecting document content rather than relying only on filenames or extensions.

The goal is to make file organization more intelligent, transparent, and convenient without taking control away from the user.

😫 The Problem: Digital Clutter Is Real

Our computers gradually become storage spaces for files we no longer organize.

This creates several everyday problems:

1. Finding files takes longer than expected.

You remember downloading a document, but you cannot remember its exact filename or location.

2. Downloads folders become overwhelming.

Every new download adds another file to an already disorganized collection.

3. Manual organization is repetitive.

Creating folders, checking extensions, renaming files, and moving them individually wastes time that could be spent on more valuable work.

4. Existing automation can be too basic.

Rules based only on file extensions cannot always identify what a document is actually about. Two PDF files might represent completely different things — a college assignment and a course certificate, for example.

5. Privacy matters.

Personal documents can contain sensitive information. Uploading every file to an external AI service just to organize a folder may not be desirable.

TidyAI explores how local AI and ordinary automation can work together to address these challenges.

✨ Key Features

1. Smart File Categorization

TidyAI is designed to use file information to recommend meaningful categories.

Basic classification can use filenames, extensions, and metadata. When a file's purpose is unclear, an AI model can help interpret available context, subject to the information provided to it.

The objective is to reduce the need to inspect every file manually.

2. Automatic Folder Organization

Once categories have been suggested, the application can help organize files into appropriate folders.

For example:

Downloads/ → Documents/

Downloads/ → Images/

Downloads/ → Study Materials/

A safe workflow should let users review proposed changes before applying them. This helps prevent accidental moves, duplicate filenames, and unexpected changes to existing files.

3. Local AI Processing

One of the main ideas behind TidyAI is to explore running an open-weight model locally rather than depending entirely on a hosted AI API.

With a compatible local model and runtime, AI inference can take place on the user's computer. This can reduce the need to send file information to external services.

Local processing still depends on the actual implementation: file operations, telemetry, downloads, or optional integrations must also be considered when evaluating privacy.

4. Simple and Beginner-Friendly Interface

File management tools should not require users to understand complicated commands.

A simple interface can make it easier to select a folder, preview suggested categories, review proposed actions, and organize files.

The focus is on making the workflow understandable for students, developers, professionals, and everyday computer users.

5. User-Controlled File Operations

Automation should not mean losing control.

A responsible file organizer should provide a preview of proposed changes, allow users to confirm them, and handle filename conflicts safely.

The intention is to make organization convenient without silently moving or overwriting important files.

⚙️ How It Works

The proposed workflow combines deterministic file handling with AI-assisted classification.

Step 1: Select a folder

The user chooses a directory they want to organize, such as Downloads or a project folder.

Step 2: Scan the files

Python inspects supported files and collects relevant information, such as filenames, extensions, and available metadata.

Step 3: Analyze and categorize

The application applies straightforward rules first. When additional context is useful, a locally running open-weight model can suggest a category based on the available information.

Step 4: Preview the results

The user reviews the proposed categories and destination folders before any changes are made.

Step 5: Organize the files

After confirmation, the application moves supported files into their destination folders while checking for conflicts and handling errors.

Step 6: Review the outcome

The application can display a summary of completed operations and any files that could not be organized.

This hybrid approach is important. AI is useful for interpreting ambiguous information, but predictable operations such as checking file extensions and moving files should remain under explicit programmatic control.

🛠️ How I Built It

The project is planned around Python-based automation, an open-weight AI model, local inference, and a lightweight user interface.

Technology Stack

Python: Core application logic, file scanning, categorization rules, and file operations.
Gemma: An open-weight model option for AI-assisted categorization and interpretation.
Ollama: A local model runtime for running a compatible Gemma model.
Streamlit: A lightweight interface for interacting with the file organizer.

The final model version and configuration should be selected according to the computer's available memory, performance requirements, and the capabilities of the chosen model.

Why Combine AI with Traditional Programming?

Not every task needs a language model.

Identifying a .png file as an image is a straightforward programming task. Moving a file into a folder is also something Python can handle reliably.

However, understanding that semester3_os_notes.pdf is probably related to operating systems or that a document might be a course certificate requires more contextual interpretation.

TidyAI's approach is to use ordinary code for predictable tasks and AI where contextual assistance can add value.

This keeps the workflow easier to understand and avoids using AI unnecessarily for simple operations.

🔓 Why Does Open Innovation Matter?

Open innovation is particularly valuable when building tools that people may want to customize, inspect, and run on their own machines.

For TidyAI, the choice of an open-weight model is not just about adding AI to an application. It is about exploring a different way to build and use AI-powered software.

1. More Control Over the AI Workflow

A locally available model gives developers the opportunity to experiment with prompts, supported model versions, and inference configurations without depending entirely on a proprietary hosted inference endpoint.

2. Privacy-Focused Design

A local model can process the information supplied to it without requiring that information to be sent to an external model provider.

For a file organizer, this is an important design advantage. However, local inference alone does not guarantee complete privacy; the rest of the application must be designed accordingly.

3. Reduced Dependence on Paid APIs

Local inference can reduce usage-based API costs for supported workloads. Users still need compatible hardware, storage, and electricity, and some models may require substantial system resources.

4. Accessibility for Developers

Open-weight models make it easier for developers and students to experiment with AI-powered applications, learn how inference works, and customize their tools.

They also create opportunities for contributors to improve categorization, usability, error handling, and documentation.

5. Learning Through Building

For me, this project is also an opportunity to explore the practical side of AI engineering: choosing a model, integrating it with a Python application, combining AI with deterministic logic, and designing a workflow that people can actually use.

A closed API can be a useful choice for many applications, but open-weight models offer an alternative when local execution, customization, and greater control over deployment are priorities.

🔐 Privacy and Safety Considerations

Because TidyAI is intended to work with users' personal files, safety should be part of the design rather than an afterthought.

The application should aim to:

Process files locally when using the local inference configuration.
Avoid sending full document contents to the model when filenames or metadata are sufficient.
Ask for confirmation before moving files.
Avoid overwriting existing files without explicit permission.
Handle inaccessible files and unsupported formats gracefully.
Exclude sensitive system directories from automatic organization by default.
Provide clear feedback about what was changed.

These safeguards can help make automation more predictable and give users confidence in the process.

🚀 Future Improvements

There are several directions in which TidyAI could evolve.

Duplicate File Detection: Identify possible duplicate files to help users reclaim storage space.

Custom Categories: Allow users to define categories and organization rules that match their own workflow.

Content-Aware Organization: Explore supported document text extraction so the model can classify files using their contents when necessary.

Undo and Recovery: Maintain an operation history and support safe reversal of file moves.

Batch Processing: Organize larger folders efficiently while reporting progress and errors.

Custom Naming Suggestions: Suggest more descriptive filenames while preserving the original files until users approve the changes.

Cross-Platform Support: Improve compatibility across macOS, Windows, and Linux.

These are potential improvements, not claims that every feature is already implemented.

🎯 Who Can Benefit from TidyAI?

TidyAI is intended for people who regularly download, create, or manage large numbers of files.

Students can keep assignments, notes, and learning resources organized.
Developers can reduce clutter in folders containing documentation, screenshots, and downloaded tools.
Content creators can organize images, recordings, and project assets.
Professionals can manage documents and downloads more systematically.
Everyday computer users can spend less time sorting files manually.

The underlying idea is simple: small repetitive tasks add up, and automating them can make everyday computer use more convenient.

🎥 Demo

Screenshots:

The demo should show the folder selection process, categorization suggestions, preview of proposed changes, and the final organization result.

🌱 What I Learned

Building TidyAI is an opportunity to explore a practical question: how can open-weight AI make an ordinary, repetitive computer task easier?

It also highlights an important lesson about AI development. A useful AI application does not always need a complicated agent or a large collection of tools. Sometimes, combining a suitable model with simple, reliable programming is enough to create a valuable experience.

The challenge is to use AI where it genuinely helps, keep important operations predictable, and make the final tool easy to use.

🦥 Final Thoughts

We often spend too much time doing small tasks that could be automated.

Organizing files is one of them.

TidyAI explores how open-weight AI, local inference, and simple automation can help people spend less time managing digital clutter and more time doing what matters to them.

Because productivity is not always about working harder. Sometimes, it is about making repetitive work disappear.

Less sorting. Less effort. More time to create. 🚀

If you are interested in open-source AI, local models, or practical automation, I'd love to hear your thoughts and suggestions.

What repetitive task would you automate with an open-weight AI model?

devchallenge #hf26challenge #Hacktoberfest #OpenSource #AI #Gemma #Python #Ollama #Automation

TutorSaathi AI: An Open-Source AI Teaching Assistant for English Tutors 🇮🇳

Rishabh Kumar — Sun, 18 Oct 2026 19:47:04 +0000

What I Built

TutorSaathi AI is an AI-powered teaching assistant designed to help English tutors and coaching centers create personalized learning materials for Hindi-speaking students.

Many students understand English grammar rules but struggle to use English in everyday conversations. At the same time, tutors spend hours preparing worksheets, quizzes, translations, and practice exercises for their students.

TutorSaathi AI aims to make this process faster and more accessible.

Key Features

Hindi-to-English Learning: Translate Hindi sentences into natural English with simple explanations and examples.
AI Worksheet Generator: Generate grammar exercises, vocabulary activities, quizzes, and answer keys.
Personalized Practice: Create exercises based on a student's learning level and common mistakes.
Speaking Practice: Provide daily English conversation prompts to help learners build confidence.
Progress Tracking: Help tutors review quiz results and identify areas where students need more practice.

Who Would Pay for It?

Independent English tutors, small coaching centers, and online educators could use TutorSaathi AI to save preparation time and offer more personalized learning experiences.

The long-term goal is to provide a useful free learning experience alongside affordable premium features for tutors who need more worksheets, student management, and progress reports.

Demo

The demo will showcase how a tutor can generate an English worksheet, explain Hindi sentences in English, and prepare practice activities for students.

The project source code, setup instructions, and documentation will be available in the repository.

How I Built It

TutorSaathi AI is planned as a full-stack web application with an interface designed for students and teachers.

Technology Stack

Frontend: Next.js and TypeScript
UI: Tailwind CSS and shadcn/ui
AI Model: Gemma, an open-weight model family from Google
Inference: A compatible inference runtime for generating educational content
Database: MongoDB Atlas, if persistent user accounts and progress tracking are implemented
Deployment: Vercel for the web application

The core AI workflow will use Gemma to generate translations, grammar explanations, worksheets, and practice questions from user-provided topics and sentences.

The application will guide the model using structured prompts and validate generated output before displaying it to learners. Educational content should be reviewed carefully because AI-generated explanations can sometimes be incorrect.

The implementation and final technology choices will be documented in the GitHub repository.

Why Does Open Innovation Matter?

Open innovation matters because educational tools should be accessible to learners and teachers regardless of their budget.

Using an open-weight model such as Gemma offers several potential advantages:

Greater flexibility: Developers can experiment with different model sizes and inference setups.
Deployment choices: Depending on hardware and model requirements, inference may run locally or on infrastructure chosen by the developer.
Transparency and experimentation: Developers can inspect available model documentation, evaluate outputs, and improve their application around the model's strengths and limitations.
Cost control: Self-hosting or selecting an appropriate inference provider can offer different cost trade-offs compared with relying entirely on a closed AI API.
Privacy-conscious design: A suitable deployment can reduce the need to send educational content to an external model provider, although the complete application must also handle logs, databases, and user data responsibly.

A closed API can also be useful, but an open-weight approach gives developers more control over how they experiment, deploy, and adapt their AI-powered products.

For TutorSaathi AI, the goal is to explore how open-source AI can help make personalized English learning more affordable and practical.

My First User

[Optional — complete this section after a real person has tested the application.]

I plan to share TutorSaathi AI with an English tutor or a small coaching center and collect feedback on worksheet quality, usefulness, and time saved during lesson preparation.

The feedback will help me identify which features are genuinely useful and what needs improvement.

Prize Categories

[Add the exact partner categories you are eligible for, as listed on the official challenge page. Remove this section if no category applies.]

What's Next?

My next priorities are to test the generated educational content, improve personalization, gather feedback from real tutors, and explore an affordable pricing model.

I want TutorSaathi AI to become more than another AI wrapper. The goal is to build a practical teaching tool that solves a real problem for educators and their students.

Open-source AI + practical education + real user feedback = a product worth building. 🚀

Thanks for checking out TutorSaathi AI!
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Demo

Code

How I Built It

Why Does Open Innovation Matter?

🌿 NatureQuest AI: Turn Screen Time into Green Time with Open-Source AI

Rishabh Kumar — Mon, 12 Oct 2026 04:40:18 +0000

What if AI didn't ask you to spend more time on your screen—but inspired you to step outside?

NatureQuest AI is an AI-powered outdoor adventure application that transforms everyday nature exploration into fun, interactive missions. It encourages users to put their devices down, explore the world around them, observe nature, and build healthier digital habits.

This is my submission for the Hacktoberfest Open-Source AI Challenge — Week 1: Touch Grass.

🌱 1. The Problem: We Spend Too Much Time on Screens

We use technology for almost everything—learning, communication, entertainment, and work. But sometimes, scrolling through a screen replaces the simple experiences that make life interesting.

We might spend hours indoors without noticing the birds outside, the plants around us, or the small details of our environment.

I wanted to explore a different question:

Can we use AI to encourage people to spend more time in the real world instead of keeping them glued to a screen?

That question inspired me to build NatureQuest AI.

🚀 2. What I Built

NatureQuest AI turns outdoor exploration into a mission-based experience.

Instead of simply opening another app to scroll, users can discover a challenge, go outdoors, complete the activity, and track their progress.

✨ Key Features

🤖 AI-Powered Outdoor Missions: Generate nature-related activities and real-world challenges using an open-weight AI model.
🌳 Explore Your Surroundings: Discover activities involving plants, birds, trees, outdoor observation, and environmental awareness.
📸 Photo Evidence Verification: For missions that require visual evidence, users can submit a photo for verification.
🐦 Bird Sound Verification: Bird-observation missions can use recorded audio as supporting evidence.
🏆 XP and Rewards: Earn experience points, build streaks, and unlock badges through mission completion.
📔 Nature Journal: Keep track of discoveries and outdoor experiences.
📶 Offline-Friendly Missions: Use predefined missions when AI generation is unavailable.
🎮 Gamified Progress: Turn small outdoor activities into an engaging journey.

Feature note: Keep the photo and bird-audio features described as available only if their verification flow has been implemented and tested. Otherwise, label them as work in progress.

🔄 3. How NatureQuest AI Works

Here is the high-level user journey.

flowchart TD
    A["🌿 Open NatureQuest AI"] --> B["Choose or Generate a Mission"]
    B --> C["🤖 AI Suggests an Outdoor Challenge"]
    C --> D["🌳 Go Outside and Explore"]
    D --> E{"Does the Mission Require Evidence?"}
    E -- "Yes" --> F["📸 Photo or 🐦 Audio Evidence"]
    F --> G["🔍 Verify Evidence"]
    G --> H{"Evidence Accepted?"}
    H -- "No" --> I["Try Again"]
    I --> F
    H -- "Yes" --> J["Complete Mission"]
    E -- "No" --> J
    J --> K["🏆 Earn XP and Update Progress"]
    K --> L["📔 Record the Experience"]
    L --> M["Explore the Next Mission"]

The important idea is that AI creates the opportunity, but the real experience happens outside the application.

Evidence verification should be used only for relevant missions. Activities that do not require evidence should continue through the normal completion flow.

🧠 4. AI and System Architecture

NatureQuest AI is designed around an open-weight AI workflow for mission generation, with a separate application layer for mission management and progress tracking.

flowchart TB
    U["👤 User"] --> FE["Next.js + TypeScript Frontend"]
    FE --> API["FastAPI Backend"]
    API --> MS["Mission Service"]
    MS --> AI["Open-Weight AI Model"]
    AI --> MS
    MS --> VAL["Validate Mission Structure"]
    VAL --> FE
    FE --> OUT["Outdoor Activity"]
    OUT --> EV["Optional Evidence Submission"]
    EV --> VER["Evidence Verification"]
    VER --> COMPLETE["Mission Completion Logic"]
    COMPLETE --> XP["XP, Streaks and Badges"]
    XP --> JOURNAL["Nature Journal and Progress"]

What each layer does

Component	Responsibility
Frontend	Displays missions, progress, and the user interface
Backend API	Handles requests and application logic
Mission service	Organizes mission generation and mission data
Open-weight model	Generates outdoor mission ideas
Evidence verification	Evaluates relevant photo or audio submissions
Progress system	Tracks completion, XP, streaks, and badges
Nature Journal	Records the user's outdoor experiences

The exact inference provider and verification models depend on the implementation used in the final application.

🛠️ 5. Tech Stack

Technology	Purpose
Next.js	Frontend application
TypeScript	Type-safe application development
Tailwind CSS	Styling and responsive layouts
Python	Backend development
FastAPI	Backend API framework
Pydantic	Data validation and structured schemas
Gemma / another suitable open-weight model	AI mission generation
Ollama, if configured	Local model inference
Git and GitHub	Version control and collaboration

Important: Keep only the technologies and AI models actually used in your project. If you use a different model or inference setup, update this table accordingly.

🔓 6. Why Open Innovation Matters

Open innovation is important because developers should have choices in how they build and deploy AI-powered applications.

For NatureQuest AI, open-weight models create opportunities to experiment with mission generation, test different models, and explore local inference where the model and hardware support it.

Why I care about open-source AI

Flexibility: Choose a model based on quality, speed, resource requirements, and licensing.
Experimentation: Compare different models for generating practical outdoor missions.
Deployment choices: Explore local inference or hosted deployment according to project requirements.
Community contributions: Allow other developers to suggest missions, improve code, fix bugs, and extend the application.
Greater control: Reduce dependence on one proprietary AI provider, depending on the chosen architecture.

Open-weight models do not automatically make an application free, completely private, or fully offline. Those properties depend on the model licence, deployment, hardware, data handling, and external services.

For me, the most interesting part is using open AI for something beyond another chatbot: encouraging people to reconnect with nature.

🌳 7. A Real-World Example

Imagine a user opens NatureQuest AI on a Saturday morning.

The application suggests a mission:

Mission: Become a Bird Observer 🐦

Go outside to a safe, accessible place.
Spend a few minutes quietly observing your surroundings.
Notice the birds and listen to the sounds around you.
If the mission requires it, record a short audio sample.
Submit the evidence for verification.
Complete the mission and earn XP if it passes the required checks.
Record the experience in your Nature Journal.

The activity is simple, but it encourages observation, curiosity, and time away from endless scrolling.

Evidence verification can help discourage unsupported completion claims, although an AI classifier cannot guarantee that every submission is genuine or correctly identified.

🎨 8. Design Philosophy

I wanted the interface to feel calm, natural, friendly, and inviting—not like a corporate dashboard.

The visual direction uses:

Sage green and earthy colors
Warm cream backgrounds
Rounded cards and approachable layouts
Clear mission instructions
Visible progress and rewards
Nature-inspired visual elements

The design supports the purpose of the application: make outdoor exploration feel accessible and enjoyable.

🧩 9. Challenges and What I Learned

Building NatureQuest AI helped me think about AI as part of a complete product, rather than just a model that generates text.

Some important engineering considerations include:

Structured AI output: Generated missions need a predictable format that the application can validate.
Reliable fallback behavior: Users should still have missions available if AI generation fails.
Evidence quality: A photo or audio clip may be unclear, irrelevant, or insufficient for verification.
Responsible gamification: Rewards should encourage outdoor exploration without making users feel pressured to stay in the app.
Privacy: Camera and microphone access should be requested only when needed, with clear controls for reviewing and deleting evidence.
Hardware limitations: Running an open-weight model locally depends on its size, quantization, and available system resources.

One lesson stands out: AI features are only useful when they support a meaningful user experience and work reliably within the limits of the system.

Before publishing, add one or two challenges you personally encountered and solved during development. Real examples make this section stronger.

🔮 10. What's Next?

NatureQuest AI can grow in several directions:

🌼 More mission categories for different ages and interests
🐦 Improved bird-sound recognition and clearer verification feedback
📷 Better image relevance checking
🌍 Optional location-aware missions with explicit permission
♿ Accessibility improvements for different users
🌱 Community-created mission packs
📊 Personal outdoor activity history and progress insights
🔋 Improved performance and offline support

These are potential improvements, not claims that all of them are already implemented.

Keep a record of outdoor experiences.

Replace these placeholders with real, publicly accessible image URLs. You can also embed a short demo video.

🎬 11. Live Demo

Try NatureQuest AI: https://drive.google.com/file/d/1E4YNQP3uqlSZSev_cjVn-vvIMQ2dOOsF/view?usp=sharing

If the application is not deployed yet, replace this section with a short video demonstration or remove it until the demo is available.

💻 12. Source Code

GitHub Repository: https://github.com/Rishabh-KSingh/NatureQuest-AI

The repository contains the project source code and implementation details.

If you are interested in open-source AI, nature-based applications, or creative uses of technology, feel free to explore the project and share feedback.

How to Contribute

Potential contributions include:

Adding new outdoor missions
Improving mission generation and validation
Testing evidence verification
Improving accessibility and mobile usability
Reporting bugs or suggesting new features
Improving documentation and setup instructions

Please follow the contribution instructions in the repository before submitting changes.

⚙️ 13. Run Locally

Use the repository's actual installation instructions if they differ from this example.

Prerequisites

Node.js and the package manager used by the project
Python and the project's required version
Git
Access to the configured AI model or inference service

Setup

# Clone the repository
git clone YOUR_GITHUB_REPOSITORY_URL

# Enter the project directory
cd YOUR_PROJECT_DIRECTORY

Install frontend and backend dependencies using the instructions and dependency files included in the repository.

Configure the required environment variables, then start the frontend and backend using the project's documented development commands.

Note: The commands above are a starting point, not a complete verified setup guide. Add the exact install, environment configuration, and run commands from your working repository before publishing.

💚 14. Final Thoughts

NatureQuest AI began with a simple idea:

What if we built AI that encouraged people to spend less time looking at screens and more time looking at the world?

I wanted to explore a positive use of technology—one that inspires curiosity, supports outdoor activities, and makes reconnecting with nature more engaging.

This project is my contribution to the Touch Grass challenge, and I hope it inspires other developers to experiment with open-source AI in creative, human-centered ways.

Technology doesn't always have to keep us online. Sometimes, it can give us a reason to step outside.

🌿 Build with AI. Explore with curiosity. Reconnect with nature.

I'd love to hear your feedback and ideas!

Hacktoberfest #OpenSourceAI #TouchGrass #NatureQuestAI #ArtificialIntelligence #OpenSource #NextJS #Gemma

I built my friend an AI speaking partner because knowing English isn't the same as speaking it.

Rishabh Kumar — Mon, 05 Oct 2026 05:59:07 +0000

My friend understands English.

They can read it.

They can watch English videos.

They know a decent number of English words.

But ask them to speak.

There is a pause.

They first think in Hindi, then translate the sentence into English, then worry whether the grammar is correct, and sometimes the sentence never comes out.

And when they do make a mistake, they often don't know why it was wrong.

I noticed something else too.

The problem wasn't a lack of English content.

There are thousands of videos, grammar lessons, vocabulary apps and courses available.

The problem was practice.

There wasn't someone available every day who would patiently listen, correct the important mistakes, explain them, and then say:

"Okay. Now say it again."

So I decided to build that person as an AI.

That's how TalkSaathi AI started.

What TalkSaathi does

TalkSaathi AI is an English-speaking companion designed for someone who understands English but struggles to speak confidently.

Its core idea is simple:

Listen → Understand → Recall → Speak → Correct → Repeat → Converse → Track → Adapt

Instead of completing a grammar exercise and moving on, TalkSaathi turns a mistake into the next practice opportunity.

For example, imagine my friend says:

"I take breakfast at 8 and I go college."

TalkSaathi doesn't just say "wrong."

You said

I take breakfast at 8 and I go college.

A better version

I have breakfast at 8 and I go to college.

Why?

We usually say "have breakfast" in natural English.

We also use "to" before a destination.

More natural

I usually have breakfast at 8 and then go to college.

And then comes the important part:

🎤 Now say it again.

The learner repeats the sentence.

The AI checks the new attempt.

That creates a much more useful loop:

┌─────────────┐
│   Mistake   │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Correction  │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Explanation │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   Repeat    │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Re-check   │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Improvement │
└─────────────┘
       │
       └──────────────► 🔄 Next Practice

The problem I actually wanted to solve

While building this, I kept coming back to one question:

Why can someone understand English but still struggle to speak it?

For my friend, several things were connected.

They were:

translating from Hindi before speaking;
afraid of making grammar mistakes;
unsure how to form sentences naturally;
using the same simple vocabulary repeatedly;
not getting enough speaking practice;
lacking a conversation partner.

So TalkSaathi isn't designed to make someone memorize thousands of words.

It's designed to give them a place where they can actually speak.

The first practice session

When a learner opens TalkSaathi for the first time, there is no complicated registration.

They create a small profile:

English level;
learning goal;
interests;
preferred topics;
daily practice time;
speaking goal.

They can choose:

10, 15, 20 or 30 minutes.

TalkSaathi then creates a practice session based on their level and previous performance.

A typical session might look like:

        ┌────────────┐
        │   Listen   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │ Understand │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │   Recall   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │    Speak   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │   Correct  │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │   Repeat   │
        └─────┬──────┘
              ↓
        ┌────────────┐
        │Conversation │
        └────────────┘

The important part is that the practice doesn't end after the first answer.

The learner gets another chance.

I didn't want Hindi to become a dependency

One feature I particularly wanted was a Hindi → English speaking bridge.

Because when someone is learning English, sometimes they know exactly what they want to say—but they don't know how to say it in English.

For example:

"मुझे कल कॉलेज जाना है।"

TalkSaathi can turn that into:

I have to go to college tomorrow.

Then it explains the sentence and asks:

🎤 Now say it in English.

After the learner repeats it, TalkSaathi doesn't stop at the translation.

It continues:

What do you usually do at college?

This is important.

Hindi is not supposed to become the final destination.

The goal is to gradually move from:

Hindi thought
     ↓
English translation
     ↓
English speaking

towards:

English thought
     ↓
English speaking

It also remembers mistakes

Another problem with normal practice is that the same mistake can happen again tomorrow.

And again next week.

TalkSaathi keeps a lightweight local history of recurring mistakes.

For example:

Past tense       8 mistakes
Prepositions     6 mistakes
Articles         4 mistakes
Vocabulary       3 mistakes

If the learner repeatedly says:

"I didn't went."

TalkSaathi can recognize that as a recurring pattern.

The next practice can then include:

Past Tense Challenge

So the system isn't only asking:

"What did you say?"

It's also asking:

"What does this learner need to practice next?"

That's where the personalization comes from.

Conversation mode

Sometimes you don't want a lesson.

You just want to talk.

TalkSaathi has a free conversation mode with topics such as:

Daily life

Family, food, college, hobbies, friends.

Professional

Job interviews, meetings, presentations and workplace situations.

Social

Shopping, restaurants, travel and asking directions.

Education & Technology

Programming, AI, college and learning.

Or you can simply enter your own topic.

The AI doesn't try to correct every single word while you're talking.

That would make the conversation feel like an exam.

Instead, it keeps the conversation natural and saves deeper feedback for the end.

A conversation report can show:

Grammar       78%
Fluency       81%
Vocabulary    74%

Important corrections: 4

Focus next:
1. Past tense
2. Articles
3. Prepositions

I wanted the AI to feel different in different moments

TalkSaathi has four personalities, depending on what the learner is doing.

During learning:

Friendly teacher

During conversation:

Natural friend

During correction:

Supportive tutor

During speaking practice:

Encouraging coach

The AI should never make the learner feel embarrassed about making mistakes.

A mistake is not a failure.

It's the next thing to practice.

What happens when you don't know the sentence at all?

I also wanted TalkSaathi to work outside a normal lesson.

So I added a photo-based grammar feature.

A learner can upload a photo containing English text.

For example:

"I am going market yesterday."

TalkSaathi extracts the text and analyzes it.

It returns:

Original

I am going market yesterday.

Correct

I went to the market yesterday.

Why?

"Yesterday" tells us that the action happened in the past, so we use went.

Natural version

I went to the market yesterday.

And then:

🎤 Try saying the corrected sentence.

So even a photo becomes a speaking exercise.

How I built it

The architecture is intentionally split into two important layers:

The open-weight model is the brain.

ElevenLabs is the voice interface.

╭─────────────────────────────────────────────────────────────╮
│                    🗣️  TALKSAATHI AI                       │
│              Your AI Saathi for English Speaking           │
╰─────────────────────────────┬───────────────────────────────╯
                              │
                              ▼
                    ┌───────────────────┐
                    │  Next.js Frontend │
                    └─────────┬─────────┘
                              │
              ┌───────────────┼────────────────┐
              │               │                │
              ▼               ▼                ▼
        🎤 VOICE INPUT    ⌨️ TEXT INPUT    🖼️ IMAGE INPUT
              │               │                │
              ▼               │                ▼
       ┌─────────────┐        │         ┌─────────────┐
       │ ElevenLabs  │        │         │     OCR     │
       │     STT     │        │         └──────┬──────┘
       └──────┬──────┘        │                │
              └───────────────┼────────────────┘
                              ▼
                    ┌───────────────────┐
                    │      🧠 GEMMA     │
                    │  Open-weight AI   │
                    │       Brain       │
                    └─────────┬─────────┘
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
      ✨ Correction     💬 Conversation   🎯 Personalization
             │                │                │
             └────────────────┼────────────────┘
                              ▼
                    ┌───────────────────┐
                    │  ElevenLabs TTS   │
                    │   AI Voice Output │
                    └─────────┬─────────┘
                              │
                              ▼
                    🔊 SPEAK • PRACTICE
                              │
                              ▼
                       👤 THE LEARNER
                              │
                              └───────────────┐
                                              │
                                  🔄 Repeat & Improve
                                              │
                                              └───► 🧠 Gemma

The application is built with:

Next.js
TypeScript
Tailwind CSS
shadcn/ui
Gemma / compatible open-weight inference
Hugging Face / compatible inference provider
ElevenLabs Speech-to-Text
ElevenLabs Text-to-Speech
OCR
LocalStorage
Vercel

The backend uses Next.js API routes, so API credentials stay on the server.

For the MVP, I deliberately avoided forcing users to create an account.

Learning data such as progress, mistakes, phrases and preferences can be stored locally.

Raw voice recordings and uploaded photos don't need to be permanently stored.

Why an open-weight model?

This was an important choice for me.

I didn't want to build:

Frontend → closed AI API → response

and call that the whole product.

The open-weight model is responsible for the actual language-learning intelligence:

grammar correction;
explanations;
Hindi → English conversion;
conversation;
lesson generation;
mistake analysis;
personalized exercises;
phrase suggestions;
image-text language analysis.

That gives TalkSaathi a separate reasoning layer from its voice layer.

The architecture becomes:

                 🧠 GEMMA
                    │
                    ▼
              ┌───────────┐
              │   BRAIN   │
              └─────┬─────┘
                    │
                    │ Understand
                    │ Correct
                    │ Explain
                    │ Personalize
                    │ Converse
                    │ Generate
                    ▼
           ┌─────────────────┐
           │   ElevenLabs    │
           │      VOICE      │
           └────────┬────────┘
                    │
                    ▼
              🎤 Listen & Speak

This also means the AI layer can evolve independently.

A different open model can be tested later without rebuilding the entire product.

Why not just use Hindi → English translation?

Because translation isn't the goal.

If the app simply translated everything for the learner, the learner could become dependent on translation.

That's why TalkSaathi uses translation as a temporary bridge.

The sequence is:

"I’m stuck"
      ↓
   Hindi
      ↓
English sentence
      ↓
Understand why
      ↓
Say it yourself
      ↓
Continue in English

The final goal is not better translation.

It's less need for translation.

What building this taught me

One thing became obvious while designing the product:

The hardest part isn't getting an AI model to generate a correction.

The harder question is:

What should happen after the correction?

If the app simply says:

"Correct sentence: ..."

the learning moment is over.

But if it says:

"Now you try."

the mistake becomes practice.

Then another question appears:

What if the learner makes the same mistake tomorrow?

That's why TalkSaathi has mistake memory.

And then:

What should they practice tomorrow?

That's why the system uses the learner's history to recommend the next activity.

The product became much more interesting once I stopped thinking about individual AI responses and started thinking about the learning loop.

The part I'm most excited about

It's not the dashboard.

It's not the number of features.

It's this tiny interaction:

"Now say it again."

Because that's the difference between an AI that answers questions and an AI that actually helps someone practice a skill.

TalkSaathi is designed around that idea.

Handing it to my friend

The real test isn't whether I can make the demo work.

The real test is whether my friend can open it tomorrow, press Start Practice, speak without feeling embarrassed, make mistakes, and want to come back again.

That's what I wanted to build.

Not another English course.

Not another chatbot.

A speaking companion.

What's next?

The current version focuses on the core speaking experience.

There are still things I want to improve:

more advanced pronunciation analysis;
better adaptive lesson generation;
cloud sync;
user accounts;
PostgreSQL + Prisma;
mobile support;
more languages;
teacher dashboards;
community speaking challenges.

But I want to validate the most important thing first:

Does regular speaking practice with TalkSaathi actually make my friend more confident?

Prize Category

Best Use of ElevenLabs

TalkSaathi uses ElevenLabs for:

Speech-to-Text;
Text-to-Speech;
spoken learning content;
AI conversation voice.

The reasoning layer remains an open-weight model, while ElevenLabs handles the voice interaction.

Try TalkSaathi

Source Code: https://github.com/Rishabh-KSingh/TalkSaathi-AI

Demo Video: https://drive.google.com/file/d/17Pd4YNAB7QNAFNTbSCoJyoxhqDwOKSQ3/view?usp=sharing

I started with a simple observation:

Knowing English and speaking English are two different skills.

So I built something for someone who needed more opportunities to speak.

TalkSaathi AI — Your AI Saathi for English Speaking.

Don't just learn English. Practice speaking it.

Building KisanMitra AI: A Full-Stack Voice Agent for Indian Farmers 🌾🎙️

Rishabh Kumar — Sat, 15 Aug 2026 17:00:11 +0000

Day 10 of the **10 Days of Voice Agents – VoiceForBharat Edition* Challenge | Farm & Field Track*

Imagine holding a smartphone in the middle of a mustard field in rural India.

Your hands are covered in soil. The sun makes the screen difficult to read. You need an answer about your crop, weather, or mandi prices — but typing a long query or navigating through complicated portals isn't practical.

Now imagine simply pressing a button and saying:

"भैया, कल बारिश होगी क्या? और धान में भूरा धब्बा रोग लगा है, क्या करें?"

Instead of searching through menus, you can simply talk.

That's the idea behind KisanMitra AI (किसान मित्र AI) — a full-stack, real-time bilingual voice agent designed specifically around agricultural assistance.

The project combines:

🎙️ Murf Falcon TTS
⚡ LiveKit Agents
📝 Deepgram STT
🧠 Google Gemini
🌐 Next.js 15
🗄️ SQLite
📞 SIP-based calling
🚨 Human escalation
🤝 Specialist agent handoff

This article is the complete story of how I built KisanMitra AI during the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

🇮🇳 1. Why Voice for Indian Farmers?

Voice assistants are often treated as convenience tools.

But in agriculture, voice can solve a much more practical problem: accessibility.

Farmers may be working outdoors, using their hands, dealing with sunlight, speaking in Hindi or regional languages, and needing information quickly.

The interaction shouldn't look like this:

Open App
   ↓
Find Menu
   ↓
Select Category
   ↓
Fill Form
   ↓
Type Question
   ↓
Wait for Result

It should feel more like this:

👨‍🌾 Farmer
    ↓
🎙️ "मेरी फसल में ये समस्या हो रही है..."
    ↓
🤖 KisanMitra AI
    ↓
🔊 Natural Voice Response

For the Farm & Field track, my objective was simple:

Make agricultural assistance as conversational and accessible as possible.

🏗️ 2. End-to-End System Architecture

KisanMitra AI connects a real-time voice pipeline with AI reasoning, tools, memory, analytics, telephony and human escalation.

                         👨‍🌾 FARMER
                             │
                 ┌───────────┴───────────┐
                 │                       │
          🌐 Web Browser            📞 Phone / SIP
                 │                       │
                 └───────────┬───────────┘
                             ▼
                       ⚡ LIVEKIT
                    Real-Time Audio
                             │
                             ▼
                    🎙️ Silero VAD
                  Turn Detection
                             │
                             ▼
                    📝 Deepgram STT
                    Hindi / Hinglish
                             │
                             ▼
                   🧠 KISANMITRA AGENT
                             │
                ┌────────────┼────────────┐
                ▼            ▼            ▼
             Gemini       Memory        Tools
                │            │            │
                │            ▼            ├── Weather
                │         SQLite           ├── Farmer Profile
                │                           └── Escalation
                │
                ▼
             Decision
                │
        ┌───────┴────────┐
        ▼                ▼
   Normal Answer    Complex Problem
        │                │
        │                ▼
        │          🌾 FASAL DOCTOR
        │          Specialist Agent
        │
        └────────┬───────────────┘
                 ▼
            🔊 MURF FALCON
             Text-to-Speech
                 │
                 ▼
              LIVEKIT
                 │
                 ▼
           👨‍🌾 FARMER HEARS

The architecture and integration layers are based on the project structure documented during the build.

🔄 3. How a Voice Conversation Works

Let's follow one conversation from beginning to end.

Step 1 — Farmer speaks

The farmer speaks through the browser microphone or a supported phone/SIP connection.

Step 2 — LiveKit transports audio

LiveKit handles the real-time audio communication.

Step 3 — Voice Activity Detection

Silero VAD detects when the farmer starts and stops speaking.

Step 4 — Speech-to-Text

Deepgram Nova-3 STT converts the audio into text.

It is designed to handle Hindi/Hinglish conversations and code-switching.

For example:

"सरसों का Mandi Bhav क्या है?"

Step 5 — Gemini processes the request

Google Gemini receives the transcript and relevant context.

The agent decides whether it needs to:

Answer directly
Access memory
Call a tool
Escalate the issue
Hand the conversation to a specialist

Step 6 — Murf Falcon generates the voice

The generated response is sent to Murf Falcon TTS.

The response is then streamed back as audio.

Step 7 — Farmer hears the response

The audio travels through LiveKit back to the user.

So the complete pipeline is:

🎙️ Speech
   ↓
LiveKit
   ↓
Silero VAD
   ↓
Deepgram STT
   ↓
Google Gemini
   ↓
Tools / Memory
   ↓
Murf Falcon
   ↓
LiveKit
   ↓
🔊 Speech

⚡ 4. Why Low Latency Matters

Voice conversations feel very different from text conversations.

If an AI takes several seconds before responding, users may think:

"Call cut gaya kya?"

That's why latency became an important part of this project.

With streaming TTS, the project targeted approximately 114–135 ms TTFB for the Murf Falcon voice layer in the tested setup.

The goal was to make the response feel immediate rather than waiting for the entire response to be generated first.

🎨 5. Building the Frontend

I didn't want KisanMitra AI to look like a basic chatbot.

I built a custom Next.js 15 + Tailwind CSS + shadcn/ui interface with an agriculture-inspired emerald/green theme.

🌿 Main UI Features

Glassmorphism

Frosted glass cards and layered panels give the interface a modern look.

🎙️ Listening State

The UI visually indicates when the agent is listening.

🌱 Thinking State

A separate state indicates that the AI is processing the request.

🔊 Speaking State

Audio visualizations show when the agent is responding.

💬 Live Transcript

The user can see the conversation while speaking with the agent.

📦 6. Six Agricultural Modules

The main dashboard contains six smart agricultural modules.

Module	Purpose
⛅ Weather Forecast	Weather, rain and temperature information
🌱 Crop Advisory	Crop and farming guidance
📈 Mandi Bhav	Market-related information
🐛 Disease Control	Crop disease and pest assistance
🐄 Livestock Care	Animal and dairy-related assistance
🏛️ Government Schemes	Agricultural scheme information

Example voice prompts:

"क्या कल मेरे खेत में बारिश होगी?"

"गेहूं की पहली सिंचाई कब करनी चाहिए?"

"धान की पत्तियों में धब्बे आ गए हैं, क्या करूं?"

"आज मंडी में सरसों का भाव क्या है?"

These modules are part of the frontend experience documented in the project.

🚀 7. The 9-Day Technical Journey

KisanMitra AI wasn't built in one day.

It evolved throughout the challenge.

Day 1 — Give the Agent a Voice 🎙️

The first step was creating the basic real-time voice agent.

I connected the voice pipeline and integrated Murf Falcon TTS.

This transformed the project from a text-based concept into a conversational assistant.

Day 2 — Build the User Interface 🎨

I moved beyond the backend and created a proper frontend.

The goal was to give users a simple way to:

Start a conversation
See the agent state
View transcripts
Interact with agricultural modules

Day 3 — Add Memory 🧠

A useful assistant shouldn't forget everything after every call.

I introduced persistent storage using SQLite so that useful farmer information and interaction data could be retained.

For example:

User
 ├── Name
 ├── Language
 ├── District
 ├── Crop
 └── Previous Interaction

Day 4 — Connect Real-World Tools 🔧

The agent needed access to information outside the LLM.

I added domain tools that allow KisanMitra AI to perform actions such as:

Farmer profile lookup
Profile updates
Weather lookup
Escalation
Call status logging

The core tools are documented around the FarmerTools layer in the project.

Day 5 — Make the Agent Proactive 📞

Instead of always waiting for a farmer to start a conversation, I explored outbound SIP calling.

This opens the possibility of proactive alerts.

For example:

Weather API
     ↓
Severe Weather Detected
     ↓
KisanMitra AI
     ↓
📞 Outbound Voice Call
     ↓
👨‍🌾 Farmer

Day 6 — Human Escalation 🚨

AI shouldn't pretend to know everything.

For complex or serious cases, KisanMitra AI can escalate the issue.

The system can create a unique reference such as:

KISAN-7842

and send the relevant escalation information through a Discord webhook.

This creates a human-in-the-loop workflow.

Day 7 — Analytics Dashboard 📊

Once the agent could handle conversations, I needed to understand how it was performing.

So I built an analytics dashboard using:

Next.js + SQLite

The dashboard can track:

Total calls
Successful calls
Failed calls
Call logs
Session information
Conversation data

Day 8 — Specialist Agent Handoff 🤝

One of the most interesting additions was the Fasal Doctor.

Instead of forcing one agent to handle every agricultural problem:

                    KisanMitra AI
                         │
             ┌───────────┴───────────┐
             │                       │
       General Query          Complex Crop Issue
             │                       │
             ▼                       ▼
         Response              🌾 Fasal Doctor

The specialist-agent concept allows complex crop-related conversations to be routed to a dedicated agent.

Day 9 — Bring Everything Together 🚀

By this stage, KisanMitra AI combined:

Voice + Memory + Tools + Calling + Human Escalation + Analytics + Specialist Handoff

The project had evolved from a basic voice assistant into a much more complete voice-agent architecture.

📂 8. Project Folder Structure

The project follows a monorepo-style structure separating the Python voice-agent backend from the Next.js frontend.

murf-livekit-starter/
│
├── backend/
│   ├── src/
│   │   ├── agent.py
│   │   └── db.py
│   │
│   ├── tests/
│   │   ├── test_agent.py
│   │   ├── test_day5.py
│   │   └── test_day8.py
│   │
│   ├── .env.example
│   ├── .env.local
│   ├── pyproject.toml
│   └── kisan_mitra.db
│
├── frontend/
│   ├── app/
│   │   ├── api/
│   │   │   ├── token/
│   │   │   │   └── route.ts
│   │   │   └── analytics/
│   │   │       └── route.ts
│   │   │
│   │   ├── dashboard/
│   │   │   └── page.tsx
│   │   │
│   │   ├── layout.tsx
│   │   ├── page.tsx
│   │   └── globals.css
│   │
│   ├── components/
│   │   ├── app/
│   │   ├── agents-ui/
│   │   └── ui/
│   │
│   ├── app-config.ts
│   ├── package.json
│   └── .env.local
│
├── start_app.sh
├── start_app.ps1
├── AGENTS.md
├── README.md
└── BLOG_DEV_COMMUNITY.md

This structure follows the backend/frontend separation documented in the source project.

🧠 9. Important Backend Components

`backend/src/agent.py`

This is the core of the voice agent.

It connects:

STT
 ↓
LLM
 ↓
Tools
 ↓
TTS

It also contains the agent instructions and farmer-specific tools.

`backend/src/db.py`

This manages SQLite persistence.

The project uses tables such as:

users
call_logs

The users table can store useful profile information, while call_logs stores session-related information.

`frontend/app/api/token/route.ts`

This API route generates LiveKit access tokens and handles agent dispatch configuration.

This is important because the frontend needs a secure way to establish a LiveKit session.

`frontend/app/dashboard/page.tsx`

This page provides the analytics dashboard.

It displays information such as:

Total calls
Success rate
Active sessions
Call logs
Transcripts

🚧 10. The Developer Struggles

No real project is complete without debugging.

These were two of my biggest lessons.

💥 Struggle #1 — The Silent Agent

On Day 1, everything appeared to be running.

The backend was active.

The frontend connected.

The microphone worked.

But the agent was completely silent.

After hours of debugging, I found the issue.

My .env.local contained:

# ❌ Wrong
AGENT_NAME=

The agent name was empty.

I changed it to:

# ✅ Fixed
AGENT_NAME=my-agent

After that, the correct agent dispatch was triggered and the voice agent finally started speaking.

Lesson learned:

Always check environment variables before assuming the entire voice pipeline is broken.

The project documentation specifically identifies explicit agent routing through AGENT_NAME as part of the LiveKit token/dispatch setup.

💥 Struggle #2 — The Beautiful but Useless Mic Button 😅

I created a beautiful animated microphone button.

It looked amazing.

But when I clicked it...

Nothing happened.

The problem?

I had designed the UI but hadn't connected the button to the actual LiveKit session lifecycle.

The fix was connecting it with useSessionContext():

import { useSessionContext } from '@livekit/components-react';

export function ActiveMicButton() {
  const session = useSessionContext();

  const handleToggle = async () => {
    if (session.state !== 'connected') {
      await session.start();
    } else {
      await session.end();
    }
  };

  return (
    <button onClick={handleToggle}>
      Start Conversation
    </button>
  );
}

Once connected to:

session.start()

the button actually started the voice session.

Lesson learned:

A beautiful UI is useless if it isn't connected to the actual application logic.

🛠️ 11. How You Can Build Your Own Voice Agent

If you're new to voice AI, start with four major components.

┌──────────────┐
│     STT      │
│  Deepgram    │
└──────┬───────┘
       ↓
┌──────────────┐
│     LLM      │
│ Gemini / GPT │
└──────┬───────┘
       ↓
┌──────────────┐
│     TTS      │
│ Murf Falcon  │
└──────┬───────┘
       ↓
┌──────────────┐
│  LiveKit     │
│  WebRTC      │
└──────────────┘

1. Speech-to-Text

Convert the user's voice into text.

Example: Deepgram

2. LLM

The brain of the agent.

Example: Google Gemini

3. Text-to-Speech

Convert the generated answer back into speech.

Example: Murf Falcon

4. Real-Time Transport

Move audio between the user and agent.

Example: LiveKit

The project uses this four-part STT → LLM → TTS → real-time transport architecture.

🔐 12. Security: Don't Leak Your API Keys

This is extremely important when building AI applications.

Never commit:

.env
.env.local

to GitHub.

Instead:

.env.example

should contain only the required variable names.

For example:

MURF_API_KEY=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
DEEPGRAM_API_KEY=
GOOGLE_API_KEY=

And the real values should remain in your local environment.

🚨 Never publish real API keys in a public GitHub repository.

🔮 13. What's Next for KisanMitra AI?

KisanMitra AI is only the beginning.

📱 WhatsApp Voice Notes

The next step is making the assistant accessible through WhatsApp voice messages.

A farmer could simply send:

🎙️ Voice Note

and receive:

🔊 Voice Answer

🗣️ Regional Indian Languages

I want to expand beyond Hindi/Hinglish into languages and dialects such as:

Bhojpuri
Maithili
Haryanvi
Bundelkhandi
Marathi

The long-term goal is to make voice AI feel natural across different parts of India.

📸 Multimodal Crop Diagnosis

Imagine combining:

📸 Crop Image
     +
🎙️ Farmer's Voice
     +
🧠 AI

A farmer could show an infected leaf while explaining the symptoms verbally.

🛰️ Smarter Agricultural Data

Another future direction is integrating richer agricultural and environmental data sources to make recommendations more context-aware.

📚 14. What I Learned From This Challenge

The biggest lesson from these 10 days is:

Building a voice agent is much more than connecting an LLM to a microphone.

A useful voice agent needs:

🎙️ Natural voice interaction
🧠 Memory
🔧 Real-world tools
📞 Communication capabilities
🚨 Human fallback
📊 Analytics
🤝 Specialist agents
🎨 Good UX
🔐 Secure configuration

But above everything else:

It needs to solve a real problem.

For KisanMitra AI, that problem is making agricultural assistance easier to access through voice.

🔗 15. Project Links

💻 GitHub Repository

👉 [PASTE YOUR ACTUAL GITHUB LINK HERE]

🌐 Live Demo

👉 [PASTE YOUR ACTUAL LIVE DEMO LINK HERE]

🎥 Demo Video

👉 [PASTE YOUR YOUTUBE / LOOM VIDEO LINK HERE]

Don't forget to replace all three placeholders before publishing.

🙏 16. Final Thoughts

Building KisanMitra AI has been one of my most rewarding experiences with voice AI.

In these 10 days, I went from a basic voice interaction to building a system combining:

🎙️ Voice

↓

🧠 Memory

↓

🔧 Tools

↓

📞 Outbound Calls

↓

🚨 Human Escalation

↓

📊 Analytics

↓

🤝 Specialist Agent Handoff

This challenge taught me that the most exciting part of AI isn't just making models smarter.

It's making technology easier for people to use.

For a farmer, technology shouldn't feel complicated.

Sometimes the best interface is simply:

"भैया, मेरी फसल में ये समस्या हो रही है..."

And the AI should be ready to listen.

🌾 That's the idea behind KisanMitra AI.

❤️ Special Thanks

A huge thank you to Murf AI and LiveKit for organizing the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

The challenge gave me an opportunity to learn, experiment, break things, debug them, and build a voice AI system around a real-world Indian use case.

If you're also building with voice AI, I'd love to hear what you're creating.

What kind of voice agent are you building? 👇

Let's connect and build the future of voice AI together.

#10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AI #GenerativeAI #IndianAI #Agriculture #KisanMitraAI #LiveKit