DEV Community: Gautam

NEO AI — Your Offline AI PC Assistant

Gautam — Sat, 24 Oct 2026 16:04:59 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I built NEO AI, an offline-first Windows PC assistant designed to make everyday computer tasks faster and easier through voice commands.

Instead of relying on a cloud AI model for every action, NEO AI routes supported PC commands directly to local handlers. Say “Open Notepad,” “Increase volume,” or “Take a screenshot,” and the assistant can execute the corresponding action without waiting for an LLM response.

NEO AI combines:

🎙️ Offline speech recognition using Faster-Whisper

🧠 Local conversational AI powered by Ollama

⚡ Direct voice-command routing for supported PC actions

🔊 Offline text-to-speech using Windows SAPI5

🛡️ Safety checks and confirmation for destructive operations

🖥️ A desktop interface with voice controls, activity history, and settings

The goal is simple: less clicking, less waiting, and more control over my own computer.

Demo

Code

GitHub repository: https://github.com/gunmasterg9/Neo-AI

Built with Python, PySide6, Faster-Whisper, Ollama, and Windows-native speech synthesis.

How I Built It

I built NEO AI around a local-first architecture using open-source tools and locally running AI models.

The core voice workflow is:

Microphone → Faster-Whisper → Local Command Router → PC Action

For conversational requests:

Text or Speech → Local Ollama Model → Response → Offline Text-to-Speech

The important design decision was separating voice commands from AI conversations.

A straightforward command such as “Open Notepad” should not require an LLM to reason about it. A supported command goes directly to its local handler, while questions and open-ended requests can use the local Ollama model.

The project uses:

Python for application logic and automation

PySide6 for the Windows desktop interface

Faster-Whisper for local speech recognition

Ollama for local conversational AI

Qwen 3 (1.7B) as the configured local model

Windows SAPI5 / pyttsx3 for offline speech output

I also worked on audio resampling, transcription validation, command routing, cancellation, and safeguards against accidental actions from incorrectly recognized speech.

Why Does Open Innovation Matter?

Open innovation made it possible to experiment with a personal AI assistant without making a cloud API the foundation of every interaction.

Using open-source speech tools and a locally running model gives me more control over privacy, latency, customization, and how the assistant behaves.

It also makes experimentation more accessible: I can inspect the pipeline, change the routing logic, test failure cases, and improve the application without depending entirely on a closed AI service.

For me, open innovation is about more than accessing a model. It is about being able to understand, modify, and build on the technology

The Lazy Payoff

The idea is to replace repetitive mouse-and-keyboard interactions with short voice commands.

Instead of manually navigating menus, I can ask NEO AI to open a supported application, change volume, or capture a screenshot.

Direct local routing also avoids unnecessary LLM processing for simple commands, making the experience potentially faster and allowing these actions to work even when Ollama is unavailable.

The next milestone is validating the full voice workflow consistently with real microphone input across English, Hindi, and Gujarati.

Prize Categories

Built by Gautam Vandar.

NEO AI is my exploration of what a useful, privacy-conscious, offline-first personal assistant can become.

DukaanVoice AI

Gautam — Fri, 16 Oct 2026 19:39:10 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

I built DukaanVoice AI, a local-first shop management and point-of-sale application designed for Indian kirana stores and small retailers.

Many small shopkeepers still manage sales, inventory, customer credit (Udhaar), and daily expenses using notebooks or disconnected tools. DukaanVoice AI brings these everyday workflows into one application.

What it can do

Quick POS: Record sales, confirm transactions, update stock, and generate a print-ready receipt preview.
Product Management: Add products without requiring a barcode or SKU.
Udhaar Khata: Track customer credit, record repayments, and generate shareable customer statements.
Expense Register: Record operating expenses and track their effect on the cash drawer.
Daily Cash Closing: Reconcile expected cash with the amount counted at the end of the day.
Inventory Reconciliation: Compare calculated stock with recorded stock.
Multilingual Interface: Support for 23 language dictionaries, including Gujarati, Hindi, Tamil, and right-to-left layouts for supported languages.

The goal is simple: help small retailers manage their daily business with a practical, locally operated application.

Demo

The project includes a five-minute evaluator walkthrough covering the complete retail workflow.

Project repository: https://github.com/gunmasterg9/DukaanVoice-AI

Demo guide: https://github.com/gunmasterg9/DukaanVoice-AI/blob/main/docs/SUBMISSION_DEMO_GUIDE.md
LIVE https://dukaanvoice-ai.vercel.app/
The walkthrough demonstrates product entry, a point-of-sale transaction, customer credit and repayment, expense recording, daily cash closing, and inventory reconciliation.

The application currently runs locally on Windows. A publicly hosted interactive demo is not available in this release.

Code

The complete source code is available on GitHub:

Repository: https://github.com/gunmasterg9/DukaanVoice-AI

Release: v1.0.1

Release notes: https://github.com/gunmasterg9/DukaanVoice-AI/releases/tag/v1.0.1

The project is built with React, Vite, Python, FastAPI, and SQLite.

The repository includes the source code, testing instructions, regression tests, isolated end-to-end test runners, and the evaluator demo guide.

How I Built It

I built DukaanVoice AI using a local-first architecture:

Frontend: React and Vite
Backend: Python and FastAPI
Database: SQLite
Testing: Pytest, frontend tests, Oxlint, and isolated end-to-end test runners

One of the most important engineering decisions was to make retail transactions auditable and predictable.

Money is stored in integer paise, inventory quantities use integer milli-units, stock movements are recorded for auditing, and sales require explicit confirmation. Customer payments and cash-drawer calculations are kept distinct so that digital payments are not incorrectly counted as physical cash.

I also added regression coverage for duplicate product lines, customer-credit reversals, cash refunds, daily closing records, and database persistence.

Current scope: Version 1.0.1 focuses on reliable local-first retail workflows and multilingual usability. It does not yet integrate an open-weight language model for inference. Adding a locally runnable assistant for natural-language sales and inventory commands is a potential future extension, not a feature claimed by this release.

Why Does Open Innovation Matter?

Small businesses need affordable software that they can understand, adapt, and operate without depending entirely on expensive cloud services.

Open-source development makes it possible for developers and local communities to inspect the code, report bugs, contribute improvements, and adapt workflows for their own needs.

For a project like DukaanVoice AI, this matters because retail practices differ across regions and languages. A transparent, extensible application can evolve around the needs of real shopkeepers.

The current release demonstrates the local-first application and its auditable workflows. Integrating open-weight AI in a future version could extend this approach to natural-language product entry, sales drafting, and multilingual assistance while keeping transaction confirmation and financial calculations under deterministic application control.

My Agent Session

DukaanVoice AI was developed through an iterative workflow of implementation, code review, regression testing, and release verification.

The project repository includes the implementation and testing artifacts:

https://github.com/gunmasterg9/DukaanVoice-AI

I have not included a DevRelay session recording in this post.

My First User

The current release has been rehearsed using synthetic retail data to demonstrate product entry, sales, customer credit, repayments, expenses, cash closing, and inventory reconciliation.

Real-world shopkeeper feedback and paid usage have not yet been documented. My next goal is to put DukaanVoice AI in front of a local retailer, observe their daily workflow, and use their feedback to guide future improvements.

Prize Categories

I'm submitting DukaanVoice AI for the overall Hacktoberfest Open-Source AI Challenge: Week 2 prize.

DukaanVoice AI focuses on practical, local-first retail workflows for Indian kirana stores, including point-of-sale transactions, customer Udhaar, expense tracking, cash reconciliation, and multilingual usability.

I am not claiming a partner prize category because I have not verified that the released version meaningfully integrates any of the listed partner technologies.

DukaanVoice AI is an individual project submission. No team members to credit.

Thanks for participating!

JalRakshak Garden AI: Bringing Open-Source AI Into the Garden 🌱

Gautam — Fri, 09 Oct 2026 23:26:46 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

🌱 JalRakshak Garden AI is a local-first, AI-powered gardening assistant designed to connect digital intelligence with real-world plant care.

The idea is simple: instead of spending all our time looking at screens, let's use technology to help us spend more time outdoors, care for plants, understand our gardens, and make better watering decisions.

JalRakshak combines a garden management web application with an IoT hardware prototype. My goal is to create a system that can monitor environmental conditions, collect soil moisture readings, maintain a plant-care journal, and eventually automate watering based on real sensor data.

The project is being built for home gardeners, beginners, and anyone interested in combining open-source AI, IoT, and sustainable gardening.

Hardware I'm using

My current prototype includes:

LOLIN NodeMCU V3 ESP8266: Wi-Fi-enabled microcontroller for collecting sensor data.
DHT11: Temperature and humidity monitoring.
Soil moisture sensor module with LM393 comparator: Provides a moisture threshold output and, depending on the module, an analog reading.
A second fork-shaped soil moisture probe: Intended to provide an additional soil measurement once its interface and output are verified.
Mini DC water pump: Intended for controlled plant watering.
Single-channel relay module: Intended to switch the pump using an appropriately rated external power supply.

The DHT11 has already produced real readings of approximately 30–31 °C and 64–66% relative humidity during testing.

The next hardware milestone is to verify both soil sensor outputs, calibrate their readings, and integrate reliable telemetry into the application. The pump will remain disabled until its driver circuit, power supply, and watering safeguards have been tested.

Demo

Project repository: JalRakshak Garden AI on GitHub

The repository contains the project implementation I'm developing.

The hardware prototype is currently under active development. I have verified DHT11 readings over serial, but the ESP8266-to-backend connection still needs to be validated on a shared network. I don't yet have a verified public deployment or a complete end-to-end hardware demo to share.

My next demo will show live sensor readings reaching the application and a controlled watering test using real soil moisture data.

Code

Explore the source code and follow the project's progress:

GitHub: https://github.com/gunmasterg9/JalRakshak-Garden-AI

The intended data flow is:

DHT11 ───────────────┐
                     │
Soil Sensor 1 ───────┤
                     ├──> NodeMCU ESP8266
Soil Sensor 2 ───────┤          │
                     │          │ Wi-Fi
                     └──────────┘
                                │
                                ▼
                         FastAPI Backend
                                │
                                ▼
                           SQLite Database
                                │
                                ▼
                         React Dashboard

                         ESP8266 GPIO
                                │
                                ▼
                         Relay / Driver
                                │
                                ▼
                           Mini Water Pump

This diagram represents the intended architecture. The complete sensor-to-dashboard and automatic-watering paths are still being integrated and tested.

How I Built It

I wanted JalRakshak to combine practical hardware with an AI system that can run locally instead of depending entirely on paid cloud APIs.

Software stack

React: Responsive web interface for garden management.
FastAPI: Python backend and API endpoints.
SQLite: Local data storage.
Ollama: Local inference runtime for experimenting with open-weight language models.
ESP8266 Arduino framework: Firmware for reading sensors and communicating over Wi-Fi.
DHT11 and soil moisture sensors: Environmental and soil data collection.

I've been experimenting with locally available models, including Llama 3.2, Qwen3 8B, and Gemma 4 12B. The intention is to use a suitable local model for gardening assistance, plant-care explanations, and recommendations without sending private garden notes to an external AI service.

Development milestones

1. Building the application

I started with the garden application's frontend and backend, with plant management, journal functionality, recommendations, and IoT API integration as core parts of the project.

2. Connecting the ESP8266

I configured the NodeMCU to connect to Wi-Fi and read temperature and humidity from the DHT11. The sensor readings are working over serial.

3. Integrating real telemetry

The backend provides an IoT telemetry endpoint. I am working on getting real ESP8266 readings accepted by the backend and displayed by the application. A network connection problem currently prevents me from claiming successful end-to-end telemetry.

4. Adding two soil moisture sensors

I want to compare readings from two soil sensors instead of relying on a single measurement. Before using those values for decisions, I need to identify their output interfaces, check voltage compatibility, and calibrate their readings in dry and wet soil.

5. Preparing safe automatic watering

The mini pump and relay are part of the hardware prototype. Before enabling automatic watering, I plan to add safeguards such as configurable moisture thresholds, a maximum pump runtime, cooldown periods, and a manual emergency stop.

The pump will use a suitable external power supply and a correctly rated switching circuit. It will never be powered directly from an ESP8266 GPIO pin.

Why Does Open Innovation Matter?

Gardening technology should be accessible, understandable, and adaptable.

Open-source software and open-weight AI make it possible to experiment with local inference, inspect implementation details, and adapt the system to different plants, gardens, and hardware configurations.

For a project like JalRakshak, this matters in several ways:

Privacy: Garden journals and personal notes can remain on the local machine when local inference is used.
Affordability: Hobbyists can experiment with inexpensive microcontrollers and sensors.
Transparency: Developers can inspect the code and understand how the system processes sensor readings.
Community collaboration: Others can contribute support for new sensors, plant-care workflows, models, and hardware.
Learning by building: The project connects AI development with electronics, programming, and hands-on gardening.

A closed AI API might provide useful recommendations, but an open and locally runnable architecture gives developers greater control over where data is processed and how the system evolves.

My goal is not simply to build another chatbot. I want to build something that encourages people to step away from their screens, look at their plants, and use technology to support real-world activity.

My Agent Session

I haven't included a DevRelay agent-session recording because I don't yet have a verified session link to share.

If I capture a relevant session during development, I'll add it here using the challenge's required embedding format.

What's Next?

My immediate priorities are:

Verify and calibrate both soil moisture sensors.
Establish reliable Wi-Fi communication between the ESP8266 and FastAPI backend.
Store real sensor readings and distinguish them from simulated data.
Display live temperature, humidity, and soil moisture on the dashboard.
Validate the relay and pump with a suitable external supply and safe switching circuit.
Implement bounded watering rules and an emergency stop.
Record a short demonstration showing the complete system working with a real plant.

JalRakshak Garden AI is a work in progress, but each step brings the project closer to connecting open-source AI with practical, outdoor plant care.

Let's build technology that helps us touch grass — literally. 🌱

Mitra Local: A Private, Local-First AI Voice Memory & Task Assistant Built for My Friend Ketan

Gautam — Sat, 03 Oct 2026 17:03:15 +0000

This article is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend.

What I Built

Meet My Friend Ketan

My childhood friend Ketan runs a textile and garment trading firm in Ahmedabad, Gujarat. If you spend even one morning with him at his shop or warehouse, you will notice three things immediately:

His phone rings unceasingly. Between 9 AM and 7 PM, he takes over 50 phone calls from yarn spinners, dye mills, courier agencies, brokers, and retail store owners.
He switches languages three times in a single sentence. He will greet in Gujarati, negotiate quantities in Hindi, and discuss dispatch timelines or bank transactions in English.
He is constantly making verbal commitments on the go: > "હા Maheshbhai, કાલે સવારે તમને sample swatches મોકલાવી દઉં છું." (Yes Maheshbhai, sending you the sample swatches tomorrow morning.) > "Sureshji, aapka 45,000 ka payment RTGS kar diya hai, shaam tak credit ho jayega." (Sureshji, your 45,000 payment was sent via RTGS, will credit by evening.) > "Priya, please make sure the GST invoice for bill #1042 is cleared before Friday 4 PM."

By evening, half of those verbal promises were forgotten or scribbled hastily on scraps of paper, receipts, or scattered WhatsApp self-chats.

Why Off-the-Shelf Tools Failed Ketan

Ketan tried Siri, Google Keep, Notion, and standard Todoist apps. None of them worked for him:

Language Inflexibility: Siri and Google Assistant choked on code-mixed Gujarati-English sentences ("Maheshbhai ne call kari de" became unintelligible gibberish).
High Friction: Notion and Todoist demanded manual typing with date pickers—impossible while inspecting fabric rolls or driving between warehouses.
Privacy & Business Secrets: Ketan strictly refused to upload his customer contact books, invoice figures, and verbal business memos to public corporate AI clouds.

The Solution: Mitra Local

So I promised him: "I will build an assistant designed specifically for how you speak and how you work—running completely on your laptop, 100% private, with zero subscription fees."

That project became Mitra Local (મિત્ર / मित्र — "Friend").

Mitra Local is an open-source, local-first voice memory and task companion. Ketan simply hits a hotkey or clicks the microphone, talks naturally in whatever mix of Gujarati, Hindi, or English comes out of his mouth, and lets Mitra handle the rest:

🗣️ Native Code-Mixed Understanding: Seamlessly parses Gujarati (ગુજરાતી), Hindi (हिंदी), English, and hybrid mixtures without requiring manual language toggling.
🎯 Automatic Intent & Action Extraction: Distinguishes between actionable tasks (calls, deliveries, payments, follow-ups) and knowledge memories (client preferences, phone numbers, notes).
👤 Entity & Person Graph: Mentions of "Maheshbhai" or "Sureshji" automatically link to contact records with conversation histories and pending to-dos.
⚡ Local Voice Feedback: Mitra speaks back a concise confirmation in natural audio so Ketan knows his commitment was captured without staring at the screen.
🔒 100% Offline & Private: Powered by open-source AI (Ollama + Gemma / Llama) and local speech-to-text. Zero cloud telemetry.

Demo

Here is how Mitra Local works in real life when Ketan speaks to it:

1. Real-World Voice Interaction Scenarios

Language Input	Spoken Command	Extracted Action & Intelligence
Gujarati + Eng	"કાલે Maheshbhai ને call કરી દે, shipment વિશે પૂછવાનું છે."	Call Task linked to contact `Maheshbhai`, scheduled for tomorrow, categorized under shipments.
Hindi + Eng	"Sureshji ko bol dena ki 45,000 ka payment RTGS kar diya hai."	Payment Memory & Verification Task linked to contact `Sureshji` with amount `₹45,000`.
English	"Remind me to file GST tax documents by Friday 3 PM."	High-Priority Task with absolute deadline set to Friday 15:00.
Pure Gujarati	"યાદ રાખજે કે રમેશભાઈની દુકાને નવા સેમ્પલ પહોંચાડવાના છે."	Delivery Task linked to `Rameshbhai`'s shop record.
Pure Hindi	"याद रखना कि राजेश का जन्मदिन 15 तारीख को है."	Personal Memory tagged under `Rajesh` in the local memory bank.

2. The Real-Time Workflow

Speak naturally: Ketan clicks the central microphone button (or triggers the local hotkey) and speaks in Gujarati, Hindi, English, or a mix.
Instant Structured Extraction: Within 400 milliseconds, the dashboard dynamically updates with:
- Extracted task title with original code-mixed transcript preserved
- Categorized action type (Call, Payment, Meeting, Delivery, Note)
- Identified person / contact entity
- Normalized due date & priority level
Verbal Confirmation: The local Web Speech API generates an instant natural voice response: > "Call task for Maheshbhai regarding shipment scheduled for tomorrow."
Relational Organization: The task is instantly filed in the Task Matrix and linked to the Person's profile in the SQLite database.

3. Ketan's Reaction

When I handed Ketan the first build on his laptop, he laughed and immediately tested it with his fastest Kathiyawadi Gujarati:

"કાલે સવારે 11 વાગ્યે શર્માજી સાથે પેમેન્ટનું સેટલમેન્ટ કરવાનું છે, યાદ રાખજે!"

Within less than half a second, the card appeared on his screen:

Title: Payment settlement with Sharmaji
Action: Payment / Financial
Person: Sharmaji
Due: Tomorrow at 11:00 AM
Audio Response: "Payment task created for Sharmaji tomorrow at 11:00 AM."

He looked up and said:

"Gautam, this is the first app that actually talks like an Indian businessman. No complex forms, no English-only barriers, and no fear of my data leaving my laptop."

Code

Mitra Local is 100% open source under the permissive MIT License.

GitHub Repository: https://github.com/gunmasterg9/mitra-local

Quickstart: Running Locally in 3 Minutes

1. Clone the Repository

git clone https://github.com/gunmasterg9/mitra-local.git
cd mitra-local

2. Start the Backend

cd backend
python -m venv .venv
# On Windows:
.venv\Scripts\activate
# On macOS/Linux:
# source .venv/bin/activate

pip install -r requirements.txt
python -m uvicorn app.main:app --reload --port 8000

Backend runs on http://localhost:8000 (Interactive API docs at http://localhost:8000/docs).

3. Start the Frontend

cd ../frontend
npm install
npm run dev

Frontend runs on http://localhost:5173.

4. Automated Verification Suite

We built a comprehensive test suite covering multilingual entity extraction, relative date parsing, contact deduplication, and database transactions:

cd backend
python -m pytest tests/

======================= 36 passed, 16 warnings in 8.02s =======================

All 36 unit and integration tests pass, ensuring zero-regression reliability.

How I Built It

Mitra Local is built from the ground up as a resilient, local-first system designed for zero latency and complete data sovereignty.

Architecture Overview

┌────────────────────────────────────────────────────────┐
│             Modern React 19 + Vite Frontend            │
│  - One-click Voice Mic (Web Audio API / MediaRecorder) │
│  - Real-time Task Matrix & Priority Filters            │
│  - Memory Bank & Person Profiles                       │
│  - Local Browser Speech Synthesis (TTS)                │
└───────────────────────────┬────────────────────────────┘
                            │ REST API
                            ▼
┌────────────────────────────────────────────────────────┐
│                FastAPI Asynchronous Engine             │
│  - Audio Transcription Pipeline (Local Whisper / API)  │
│  - Structured Multilingual Extraction Pipeline         │
│  - Heuristic Fallback Engine (Zero-Downtime Guarantee) │
└───────────────┬────────────────────────┬───────────────┘
                │                        │
                ▼                        ▼
┌──────────────────────────────┐ ┌───────────────────────┐
│     Local AI with Ollama     │ │  Local SQLite Engine  │
│  - Gemma 4 12B / Llama 3.2   │ │  - aiosqlite Async DB │
│  - Zero Cloud Telemetry      │ │  - Full-Text Search   │
│  - Structured JSON Schema    │ │  - People & Memories  │
└──────────────────────────────┘ └───────────────────────┘

Technical Stack

Frontend: React 19, Vite, Lucide Icons, Vanilla CSS design tokens with sleek dark mode, Web Speech Synthesis API.
Backend: FastAPI (Python 3.12), Pydantic v2 schemas, SQLAlchemy with aiosqlite for asynchronous local database transactions.
Local AI Inference: Ollama running open-weight models (gemma4:12b, llama3.2, or mistral).
Storage: Local SQLite database stored securely in backend/data/mitra.db on Ketan's machine.

Solving the Multilingual Code-Switching Challenge

Indian business conversations freely mix Gujarati, Hindi, and English with honorifics ("-bhai", "-ji", "-ben"). Conventional prompts often fail by either translating names into English words (e.g. converting "Sureshji" into "Sir Suresh") or dropping dates.

To achieve bulletproof parsing, Mitra combines two layers:

Structured Few-Shot Prompting with Open LLMs: We prompt local models with strict JSON output schemas that explicitly understand Indic code-mixing:

   {
     "intent": "create_task",
     "task": {
       "title": "Call Maheshbhai regarding shipment",
       "title_original": "કાલે Maheshbhai ને call કરી દે, shipment વિશે પૂછવાનું છે.",
       "action_type": "call",
       "person_name": "Maheshbhai",
       "due_date": "2026-10-04",
       "priority": "medium",
       "language": "gu-en"
     },
     "speech_response": "Call task for Maheshbhai regarding shipment scheduled for tomorrow."
   }

Heuristic Linguistic Fallback Engine: Laptops in the field don't always have GPU acceleration, and background tasks might temporarily tie up resources. Mitra includes an integrated regex- and rule-based Indic NLP engine that recognizes common action verbs in Gujarati ("call kari de", "moklavano chhe", "poochhvanu chhe"), Hindi ("bol dena", "bhej do", "yaad rakhna"), and English ("remind me", "pay"), along with relative temporal expressions ("kaale", "parso", "kal"). Even with Ollama paused, Mitra processes inputs flawlessly in under 5 milliseconds.

Why Does Open Innovation Matter?

When starting this project for Ketan, the easiest route would have been wiring up a quick commercial cloud API (OpenAI or Claude) with a credit card. But for Ketan, and millions of independent business owners like him, closed cloud AI is fundamentally the wrong answer.

Here is why open innovation and open-source AI were strictly essential to solving this problem:

1. Absolute Privacy and Data Sovereignty

In the textile trading world, commercial relationships and pricing are hard-won trade secrets. A typical memo contains:

Supplier identities and credit balances
GST numbers and pending payment amounts
Private personal numbers and commitments

Sending audio snippets or text logs of these conversations to third-party public clouds exposes small business owners to data harvesting, telemetry profiling, and compliance vulnerabilities. With open-weight models running on Ollama and local SQLite storage, Ketan has 100% mathematical certainty that not a single byte of his business data ever leaves his hardware.

2. Zero Recurring Overhead for Everyday People

Proprietary AI SaaS platforms charge $20 to $50 per user per month, plus metered API token fees. For an independent merchant, recurring software subscriptions add up quickly and become a financial burden. Open-source models (like Gemma and Llama) democratize state-of-the-art intelligence: once downloaded, they run indefinitely with zero subscription costs, zero token meters, and zero paywalls.

3. Freedom to Adapt for Underrepresented Languages

Commercial AI models are optimized primarily for formal Western corporate English. They frequently misinterpret code-mixed regional Indian dialects like Gujarati-English or Hinglish, treating common honorifics as foreign errors or hallucinations.
Because open models and open-source code can be inspected, steered, and fine-tuned locally, we were able to build customized prompt pipelines and hybrid heuristic engines tailored specifically to Gujarati merchants. Open innovation puts the power to build AI in the hands of the communities that need it most, rather than waiting for Silicon Valley giants to prioritize regional dialects.

4. True Offline Autonomy

Wholesalers and traders frequently operate in basement warehouses, noisy textile markets, and transport hubs with spotty cellular reception. A tool that depends on roundtrips to an external cloud server will fail right when it is needed most. Open-source local inference delivers instant, reliable performance anywhere—on a flight, in a godown, or during an internet outage.

Conclusion & What's Next

Building Mitra Local for Ketan showed me that the most impactful AI tools aren't generic chatbots—they are personalized, empathetic utilities built for the specific rhythms of real people's lives.

Our upcoming roadmap includes:

Fine-tuning a quantized Whisper model on regional Gujarati and Kathiyawadi accents
Local WhatsApp Desktop webhook bridge for automated reminder dispatch
Offline vector search using DuckDB / ChromaDB for multi-year business archives

Thank you to the DEV Community and Hacktoberfest for championing open innovation that makes technology truly personal and accessible.

Links:

GitHub Repository: https://github.com/gunmasterg9/mitra-local
License: MIT License

devchallenge #hacktoberfest #hf26challenge

Building Bharat Voice AI: My 10-Day Journey from a Voice Agent to a Multilingual Multi-Agent System

Gautam — Sat, 15 Aug 2026 08:29:00 +0000

# Building Bharat Voice AI: My 10-Day Journey from a Voice Agent to a Multilingual Multi-Agent System

Over the last 10 days, I took a simple idea, a voice assistant, and gradually turned it into a more complete multilingual voice-agent system.

This project is called Bharat Voice AI.

The goal was to build a voice agent that can communicate naturally with users in India, understand different languages and code-mixed speech, remember returning users, use real-world tools, make outbound calls, escalate to humans, measure its performance, and hand specialized tasks to another agent.

This journey was part of:

10 Days of Voice Agents — VoiceForBharat Edition

The project uses Murf Falcon for voice generation together with LiveKit, Gemini, Deepgram, Python, SQLite, and real external tools.

What is Bharat Voice AI?

Bharat Voice AI is a multilingual voice assistant designed around the idea that voice interfaces should feel natural for Indian users.

Instead of forcing users to interact only through a text interface, the system allows them to speak naturally.

It supports:

English
Hindi
Gujarati
Hinglish and code-mixed conversations
Persistent user memory
Real-time tools
Human escalation
Outbound voice calls
Call analytics
Specialist-agent handoff

The system is designed so that the main agent does not have to do everything itself.

When a task requires specialized knowledge, it can hand the conversation to a specialist agent.

Why Voice?

India has a huge diversity of languages, communication styles, and levels of digital literacy.

A voice interface can make technology easier to access because users do not have to type everything.

For Bharat Voice AI, I wanted the interaction to feel closer to a normal conversation.

For example:

"Veraval mein aaj weather kaisa hai?"

The system should understand the intent even though the sentence mixes Hindi and English.

It should then respond naturally in the appropriate language.

Architecture

The basic architecture looks like this:

                    Bharat Voice AI
                           |
                     LiveKit Agents
                           |
              +------------+------------+
              |                         |
           Deepgram                   Gemini
             STT                       LLM
              |                         |
              +------------+------------+
                           |
                         Tools
              +------------+------------+
              |            |            |
           Weather      Memory      Escalation
              |            |            |
          Open-Meteo     SQLite       SQLite
                           |
                    Specialist Agents
                           |
                     Murf Falcon TTS
                           |
                         User

Day 1 - Building the Voice Agent
The first step was getting the basic voice pipeline working.
The core pipeline became:

User Speech
     |
     v
Deepgram STT
     |
     v
Gemini
     |
     v
Murf Falcon
     |
     v
User Voice

The agent could listen to the user and respond using voice.
The main technologies were:
LiveKit Agents
Deepgram
Gemini
Murf Falcon
Day 2 - Persona and Guardrails

The next step was defining who the agent actually is.

The agent became:

Bharat Voice AI

I added clear instructions covering:

Identity
Objectives
Knowledge boundaries
Language behavior
Guardrails
Escalation
Conversation style

The agent should not pretend to know something it does not know.

It should also know when a request is outside its role.

This became especially important later when I added human escalation and specialist agents.
Multilingual Conversations

One of the important goals was supporting Indian languages.

The agent supports:

English

"What's the weather today?"

Hindi

"आज वेरावल में मौसम कैसा है?"

Gujarati

"આજે વેરાવળમાં હવામાન કેવું છે?"

I also tested code-mixed speech such as:

"Veraval mein aaj weather kaisa hai?"

The important part is not simply detecting the language.

The agent should also respond in the appropriate script.

For example, Hindi should be written and spoken naturally rather than being forced into Romanized Hindi.
Day 3 - Frontend

After the voice pipeline worked, I customized the frontend.

The frontend clearly represents the state of the agent.

The main states are:

Ready
  |
  v
Connecting
  |
  v
Listening
  |
  v
Speaking
  |
  v
Call Ended

t
Day 4 - Persistent Memory

A voice agent that forgets everything after every call isn't very useful for returning users.

So I added persistent memory using SQLite.

The profile stores information such as:

user_id
name
language_preference
facts
last_interaction

The important design decision was that memory is handled by backend functions rather than being written into the LLM prompt.

The agent can:

Look up a caller.
Ask permission before saving information.
Save approved information.
Retrieve the profile during a later conversation.

For example:

First conversation:

"My name is Gautam."

The agent asks whether it should remember the name.

After permission is granted, the information is stored.

During a later conversation, the agent can recognize the returning user.

Day 5 - Real Tools

The agent needed to do more than generate answers from the LLM.

I added a real weather tool.

The weather flow is:

User
 |
 | "What is the weather in Veraval today?"
 v
Gemini
 |
 | tool call
 v
get_weather()
 |
 v
Weather API
 |
 v
Real weather data
 |
 v
Gemini
 |
 v
Murf Falcon
 |
 v
User

The important rule is:

The agent must never invent current weather information.

If the weather service is unavailable, the agent should say that it could not retrieve the latest information.

It should not guess.

One Real Debugging Lesson

One of the problems I encountered during development was a Deepgram connection failure.

The voice pipeline produced an error similar to:

APIConnectionError:
failed to connect to deepgram

Instead of assuming the code was broken, I tested the network connection from Windows PowerShell.

I used:

Test-NetConnection api.deepgram.com -Port 443

Initially the connection failed.

After investigating the network/DNS path, I eventually got:

TcpTestSucceeded : True

This was a good reminder that voice-agent problems are not always LLM problems.

The failure can be caused by:

Network connectivity
DNS
Firewall
API availability
WebSocket connections
Authentication
Tool execution
Frontend state

Debugging the complete pipeline is essential.

Day 6 - Outbound Voice Calls

The next step was making the agent call a user instead of waiting for the user to open the browser.

I integrated outbound calling using LiveKit telephony/SIP.

I also tested the Linphone route.

The basic flow became:

Bharat Voice AI
      |
      v
LiveKit SIP
      |
      v
Linphone
      |
      v
    Phone

The outbound conversation needs a different opening from a browser conversation.

The agent must immediately explain:

Who is calling
Why it is calling
How the user can end the call

I also tested the SIP connection and call lifecycle.

Day 7 - Human Escalation

A voice agent should not try to solve everything.

I added a human escalation workflow.

When a user says:

"I want to talk to a human."

the agent can start the escalation process.

But it does not automatically share information.

It first asks for permission.

For example:

"I can create a request for human assistance. Before I do, I would share your name, the issue you described, what I checked, your preferred language, and the urgency. Would you like me to create the request?"

If the user says yes, the backend creates a real escalation record.

A reference ID is generated by the backend.

The important rule is:

Never tell the user that an escalation was created unless the database operation actually succeeded.

This avoids fake success messages.

Day 8 - Call Analytics

After adding many features, I needed a way to measure what was happening.

So I built a Call Analytics Dashboard.

The dashboard tracks:

The data comes from real calls stored in SQLite.

It is not hardcoded.

A call is considered successful when the user's intended task is successfully completed.

For example:

User asks for weather
        |
Weather tool succeeds
        |
Weather information delivered
        |
     SUCCESS

If the required tool fails or the task is incomplete, the call can be recorded as failed or incomplete.

The dashboard can therefore show actual agent performance.

Day 9 - Specialist Agent Handoff

This was one of the most interesting parts of the project.

Instead of forcing the main agent to handle every type of request, I created a specialist:

Bharat Weather Specialist

The architecture became:

                   User
                     |
                     v
              Bharat Voice AI
                     |
              Weather request?
                 /       \
               No         Yes
               |           |
               v           v
          Main Agent   Weather Specialist
                           |
                           v
                      get_weather()
                           |
                           v
                       Real Data

python
For example:

User:

"What is the weather today in Veraval?"

The main agent says:

"For detailed weather information, I'll connect you with our weather specialist."

The specialist then takes over.

The important part is that the user should not have to explain the entire question again.

The conversation context is passed to the specialist.

Native Agent Handoff

The LiveKit agent-handoff pattern allows the main agent to return a specialist agent together with an announcement.

Conceptually:

@function_tool
async def handoff_to_weather_specialist(
    self,
    context: RunContext,
) -> tuple[Agent, str]:
    specialist = BharatWeatherSpecialist(
        chat_ctx=self.chat_ctx.copy(
            exclude_instructions=True
        )
    )

    return (
        specialist,
        "I'll connect you with our weather specialist."
    )

The actual implementation in the repository follows the project's LiveKit version and architecture.

The important concept is that the specialist receives the previous conversation context.

A Day 9 Bug That Taught Me Something

During testing, the handoff itself worked.

However, I encountered a problem where the internal function-call information could appear as visible text instead of the actual weather response.

For example, the user could see something similar to:

The function called is get_weather
location = Veraval
forecast_days = 1

That is not acceptable for a real voice assistant.

The user should hear:

"The latest weather in Veraval is..."

not internal tool-call information.

This led me to investigate the difference between:

Function registration
Tool execution
LLM output
LiveKit agent handoff
Frontend message rendering

It was another reminder that a production voice agent is a complete system, not simply an LLM with speech.

The Complete SystemAfter the nine days, Bharat Voice AI looks roughly like this:

                         USER
                          |
                    Voice / Browser
                          |
                          v
                    LiveKit Agents
                          |
                    Bharat Voice AI
                          |
        +-----------------+------------------+
        |                 |                  |
      Memory           Tools              Routing
        |                 |                  |
      SQLite       +------+-------+          |
                   |              |          |
                Weather      Other Tools     |
                   |                         |
              Open-Meteo                     |
                                             |
                                      Specialist Agent
                                             |
                                      Weather Specialist
                                             |
                                      get_weather()
                                             |
        +------------------------------------+
        |
   Human Escalation
        |
      SQLite
        |
    Murf Falcon
        |
        v
       USER

Outbound calls extend the system through SIP/telephony.

Analytics records the outcome of calls and makes it visible through the dashboard.

Technology Stack

The main technologies used in the project include:

Python
LiveKit Agents
Gemini
Deepgram
Murf Falcon
SQLite
Open-Meteo
SIP / Linphone
Browser frontend
GitHub

Each component has a different responsibility.

Deepgram
Speech → Text

Gemini
Reasoning + Conversation

Tools
Real-world data and actions

SQLite
Persistent state

LiveKit
Real-time voice transport and agent orchestration

Murf Falcon
Text → Natural Voice

Security

One of the most important lessons is to never put API credentials directly into the source code.

Use environment variables.

For example:

DEEPGRAM_API_KEY=your_key_here
MURF_API_KEY=your_key_here
LIVEKIT_API_KEY=your_key_here
LIVEKIT_API_SECRET=your_secret_here

The real .env file should never be committed to GitHub.

Use:

.env

locally and:

.env.example

for documentation.

Never publish:

API keys
SIP credentials
passwords
OTPs
PINs
caller information
private database records

Running the Project

The complete project is available on GitHub:

https://github.com/gunmasterg9/bharat-voice-ai

A typical setup is:

git clone https://github.com/gunmasterg9/bharat-voice-ai.git

cd bharat-voice-ai

cd backend

uv sync

Create your environment configuration:

.env

Add the required API credentials.

Then start the backend using the project's configured startup command.

Start the frontend and open the browser interface.

Allow microphone access.

Then start a conversation.

For example:

"Hello Bharat Voice AI."

Then test:

"What is the weather today in Veraval?"

Then test multilingual interaction:

"આજે વેરાવળમાં હવામાન કેવું છે?"

Then test memory:

"My name is Gautam."

Then restart the application and verify that the saved profile can be retrieved.

Testing

I created tests covering important parts of the system.

The project includes testing for:

Agent behavior
Memory
Persistent memory
Language switching
Weather tools
Escalation
Outbound calls
Linphone
Analytics
Specialist handoff

The Day 9 development test suite included specialist handoff tests alongside the previous functionality.

The important lesson was that every new feature should be tested without breaking the previous days' work.

What I Learned

The biggest lesson from this challenge is that building a voice agent is much more than connecting an LLM to a TTS API.

A reliable voice agent needs:

Voice
+
Reasoning
+
Memory
+
Tools
+
State
+
Error Handling
+
Security
+
Human Handoff
+
Observability

A model can generate a great response, but the surrounding system determines whether the product is actually reliable.

I also learned to debug the entire pipeline instead of assuming every problem is caused by the LLM.

What I Would Build Next

There is still a lot I would like to add.

Future improvements could include:

More specialist agents
Better interruption handling
More Indian languages
Better low-bandwidth support
Advanced call analytics
Conversation quality scoring
Better tool observability
More real-world Indian datasets
Improved outbound-call workflows
Specialist-to-specialist routing
More sophisticated RAG
Production deployment and monitoring

The long-term goal would be to turn Bharat Voice AI into a platform where different specialized voice agents can work together.

Final Thoughts

The most interesting part of the challenge wasn't building the first voice conversation.

It was everything that came afterward.

Making the agent remember.

Making it use real data.

Making it call a phone.

Making it know when to ask a human.

Measuring whether conversations actually succeeded.

And finally, teaching one agent when another agent is better suited to help.

That progression changed how I think about voice AI.

A voice agent isn't just a chatbot that speaks.

It can become a complete software system with memory, tools, workflows, specialized agents, and real-world actions.

That is what I wanted to explore with Bharat Voice AI.

Project Links
GitHub

https://github.com/gunmasterg9/bharat-voice-ai
All Videos in Linkedin
https://www.linkedin.com/in/gautam-vandar-71a75632b/

Challenge

10 Days of Voice Agents — VoiceForBharat Edition

Voice Technology

Murf Falcon

Thank You

Thank you to Murf AI for organizing the 10 Days of Voice Agents challenge and providing the opportunity to build, experiment, debug, and learn through a real voice-agent project.

Building Bharat Voice AI over these 10 days was a great experience, and I hope this project helps someone else start building their own voice agent.