DEV Community: KUSAL SARKAR

NOVA THE LOCAL AI ASSISTANT

KUSAL SARKAR — Mon, 26 Oct 2026 04:28:31 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I built NOVA (Neural Operational Virtual Assistant), a local-first, JARVIS-inspired AI desktop assistant designed to take care of repetitive computer tasks and make everyday digital work easier.

Think about how often we switch between applications, search for files, repeat the same instructions, check system information, or explain the same project context to an AI assistant.

NOVA aims to bring these interactions together in one place.

It supports:

Local AI conversations: Ask questions and get responses from locally running language models.
Offline voice interaction: Talk to NOVA using local speech recognition and speech synthesis.
Persistent memory: Remember user preferences, projects, conversations, tasks, and useful knowledge.
Desktop automation: Perform supported application, file, and system operations through registered tools.
Visual intelligence: Understand relevant desktop context using local screen capture and OCR.
Multi-agent orchestration: Coordinate research, development, system, vision, and memory tasks.
Proactive automation: Schedule reminders, monitor configured conditions, and prepare useful briefings.
Personalization: Adapt recommendations and context based on user preferences.

The idea is simple: instead of manually handling every small digital task, I want to interact with one assistant that can understand context and help execute the work.

NOVA is built to make the user a little lazier, without taking control away from them.

Demo

Project: NOVA — Neural Operational Virtual Assistant

Platform: Windows Desktop

Version: v1.0 Release Candidate

The release will showcase:

NOVA's desktop command center.
Local AI conversation using Ollama.
Push-to-talk voice interaction and speech recognition.
Persistent project and conversation memory.
Permission-controlled desktop operations.
Visual intelligence and system monitoring.

Code

GitHub: https://github.com/kusalar

NOVA Repository: https://github.com/kusalar/nova-local

The project is developed using a modular architecture, with separate desktop, backend, AI, memory, voice, automation, and orchestration components.

How I Built It

NOVA is built around open-source technologies and open-weight AI models. My goal was to avoid making the assistant dependent on a single closed AI provider.

Local AI and Open-Weight Models

I use Ollama as the local model runtime, with Qwen 2.5 models for language generation.

This allows NOVA to run inference directly on the user's machine. Models can be changed, tested, and optimized depending on available hardware.

Offline Voice Pipeline

NOVA's voice architecture combines:

sounddevice for microphone audio capture.
openWakeWord for wake-word detection.
Vosk for offline speech recognition.
Ollama for local language generation.
Piper TTS for offline speech synthesis.

This creates a local voice pipeline without depending on cloud speech APIs.

Persistent Semantic Memory

I implemented a SQLite-based memory system with five categories:

User profile memory
Project memory
Conversation memory
Task memory
Knowledge memory

For semantic retrieval, NOVA uses FastEmbed with BAAI/bge-small-en-v1.5, with a lightweight fallback mechanism.

This helps NOVA retain useful context between conversations instead of treating every interaction as a completely new session.

Desktop Automation and Permissions

NOVA includes a registered desktop tool system with 23 tools covering application, file, and system operations.

The execution architecture includes:

Permission-based tool execution.
Operation previews and user approval.
Execution logging.
Cancellation and rollback support.
Restricted access to sensitive locations and operations.

I wanted NOVA to be useful without allowing an AI model to execute arbitrary actions without authorization.

Multi-Agent Orchestration

NOVA uses five logical agents:

Research Agent
Developer Agent
System Agent
Vision Agent
Memory Agent

These agents share a local model runtime. A workflow orchestrator coordinates tasks, manages dependencies, and applies execution permissions.

Visual Intelligence

I added local screen awareness using Windows screen capture and OCR.

It can extract relevant desktop context while applying privacy filters to sensitive information. The subsystem is designed to be read-only.

Technology Stack

Layer	Technologies
Desktop	Tauri, React, TypeScript
Backend	Python, FastAPI
Local AI	Ollama, Qwen 2.5
Speech recognition	Vosk
Wake-word detection	openWakeWord
Speech synthesis	Piper TTS
Memory	SQLite, FastEmbed
System monitoring	psutil
Visual intelligence	Windows OCR

Why Does Open Innovation Matter?

For NOVA, open innovation is not just about choosing free technologies. It is about having control over the entire AI experience.

1. Privacy Comes First

A desktop assistant may interact with personal files, conversations, application information, and project data.

I wanted NOVA to process its core AI interactions locally rather than requiring every request to be sent to an external server.

Local inference, offline speech recognition, offline speech synthesis, and local persistent memory make this possible.

2. No Mandatory Cloud AI Subscription

Running open-weight models through Ollama means NOVA does not require a paid hosted AI API for its core language-model functionality.

Once the required models and dependencies are installed, its core AI pipeline can operate without continuous internet access.

3. Freedom to Change Models

One of the biggest advantages of open-weight models is flexibility.

I can experiment with different models, compare their performance, adjust context sizes, change generation settings, and select models that work better with my hardware.

I am not locked into a single provider's API, pricing, or model limitations.

4. Building AI for Ordinary Hardware

My development laptop is not a dedicated AI workstation.

It runs on an Intel Core i3-1115G4 processor, 12 GB RAM, and integrated graphics.

This limitation became an important engineering challenge. I had to consider CPU utilization, memory consumption, model size, inference settings, and background workloads.

It made me realize that building useful AI isn't always about using the biggest model. Architecture and optimization matter just as much.

5. Open Components Make Experimentation Possible

Because NOVA uses modular open-source components, I can improve individual subsystems without rebuilding the entire assistant.

I can replace a speech recognition engine, switch language models, modify the memory architecture, or improve the automation system independently.

That freedom is one of the biggest advantages of building on open innovation rather than depending entirely on a closed API.

My Agent Session

NOVA was developed incrementally with AI-assisted coding and engineering workflows.

The development progressed from a basic local assistant into a system with offline voice, persistent semantic memory, desktop automation, visual intelligence, multi-agent orchestration, proactive automation, personalization, and reliability monitoring.

The Lazy Payoff

The real payoff is reducing repeated effort.

Instead of opening multiple applications, repeating instructions, and manually maintaining context, I can use NOVA as a central interface for supported tasks.

For example:

Persistent memory reduces the need to explain projects repeatedly.
Voice interaction provides an alternative to typing every instruction.
Desktop tools reduce repetitive application and file operations.
Multi-agent orchestration helps coordinate more complex workflows.
Proactive automation can prepare reminders and briefings without waiting for me to request them.

I am also testing and optimizing NOVA on my actual laptop rather than assuming that a local AI assistant will automatically be fast.

One important lesson from this project is that local AI requires careful resource management. Model inference, speech recognition, speech synthesis, and background tasks all compete for limited CPU and memory.

I would rather have a smaller assistant that works reliably than a feature-heavy assistant that struggles on ordinary hardware.

The goal isn't to make the computer work harder. It's to make the user do less repetitive work.

Stay slow. Stay steady. Let NOVA handle the busywork. 🦥

HireLens AI: Building an Open-Weight AI Recruitment Platform for Small Businesses

KUSAL SARKAR — Sun, 18 Oct 2026 11:10:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

I built HireLens AI, an AI-powered recruitment and applicant tracking platform designed for small businesses, startups, and recruitment agencies.

Hiring is expensive and time-consuming, especially for small teams that cannot afford dedicated recruitment departments or expensive enterprise hiring software.

Recruiters often spend hours:

Reading and extracting information from resumes.
Comparing candidates against job requirements.
Shortlisting applicants.
Tracking candidates through recruitment stages.
Making sense of different candidate profiles.

HireLens AI brings these workflows into one recruitment workspace.

Its key capabilities include:

Resume Intelligence: Extract structured information from uploaded resumes.
Job Management: Create and manage job openings.
AI-Assisted Matching: Compare candidate profiles against job requirements.
Explainable Matching: Understand why a candidate matches a particular role.
Candidate Comparison: Compare multiple applicants side by side.
Recruitment Pipeline: Track applicants throughout the hiring process.
Analytics: Understand recruitment activity and pipeline performance.

The goal is simple: give smaller recruitment teams access to useful hiring intelligence without the complexity and cost of enterprise recruitment platforms.

Demo

Live Application: https://hire-lens-black.vercel.app

The application includes a one-click demo experience.

Visitors can explore the platform as a fictional demo recruiter, Alex Morgan, without creating an account or entering credentials.

The demo includes synthetic candidates, sample job openings, resume workflows, and explainable matching results.

The platform also protects its demo workspace against destructive operations and isolates demo data from other tenants.

Note: The free backend hosting may take a little time to wake up after inactivity.

Code

GitHub Repository: https://github.com/kusalar/HireLens

The project uses a full-stack architecture:

Frontend: Next.js 14, React and Tailwind CSS.
Backend: Python, FastAPI and Pydantic.
Database: PostgreSQL with SQLAlchemy and Alembic.
Background processing: Celery and Redis.
AI: Open-weight model(Ollama) integration for resume intelligence and grounded explanations.
Matching: Deterministic Python-based scoring engine.

How I Built It

I wanted HireLens AI to be more than another chatbot that generates recruitment recommendations.

I designed it around a combination of open-weight AI, conventional backend engineering, and deterministic decision logic.

1. Resume Intelligence

The platform extracts information from PDF and DOCX resumes and converts it into structured candidate information.

The architecture supports open-weight model inference, including local development with Ollama and hosted open-weight inference.

2. Deterministic Candidate Matching

One of the most important architectural decisions was separating candidate scoring from language model generation.

The matching score is calculated using deterministic Python logic.

The LLM does not decide who should be hired, does not autonomously reject candidates, and does not control the final matching score.

Instead, AI assists with grounded explanations that help recruiters understand the matching results.

3. Recruitment Workspace

I built the recruitment workflow around jobs, candidates, applications, resume processing, matching, comparison, and pipeline management.

The backend uses FastAPI, PostgreSQL, SQLAlchemy, and tenant-aware data access to support multiple recruitment organizations.

4. Production Deployment

I deployed the frontend on Vercel, the backend on Render, and the PostgreSQL database on Neon.

I also implemented a publicly accessible one-click demo mode so evaluators can explore the actual application.

The current deployment uses free-tier infrastructure, making it possible to demonstrate the product without maintaining expensive hosting services.

Why Does Open Innovation Matter?

For HireLens AI, open innovation is not just about using a different AI model. It is about having control over the technology that powers the product.

Recruitment platforms process sensitive information. Resumes can contain contact details, employment histories, education records, and other personal information.

Building around open-weight models gives us more control over where inference happens and how models are deployed.

There are several important advantages:

1. Model Independence

I do not want the entire product to depend on a single proprietary AI provider.

Open-weight models give us the flexibility to experiment with different models, change inference providers, and adapt the system as the product grows.

2. Cost Control

A recruitment SaaS for small businesses needs sustainable economics.

Depending entirely on expensive proprietary inference APIs could make the product difficult to operate at scale.

Open-weight inference gives us the option to explore local and self-hosted deployments and better control inference costs.

3. Data Control

Local inference can provide an additional privacy option for organizations that do not want resume content sent to external AI providers.

It also creates opportunities for future private deployments.

4. Transparent and Controlled Decisions

I deliberately kept candidate scoring deterministic rather than allowing an LLM to make hiring decisions.

This makes matching more reproducible and easier to test, while AI-generated explanations help recruiters understand the results.

For me, open innovation means being able to inspect, adapt, replace, and control important parts of the technology instead of building the entire business around someone else's API.

Built with the idea that powerful recruitment intelligence should be accessible to smaller teams, not just enterprises.

Thanks for checking out HireLens AI!

🌿 WildSense — Look Closer. Then Look Away.

KUSAL SARKAR — Sat, 10 Oct 2026 17:27:00 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

🌿 WildSense — Look Closer. Then Look Away.

What I Built

WildSense is a privacy-first, offline AI plant exploration companion designed to turn screen time into real-world nature time.

We carry incredibly powerful technology in our pockets, but we often use it to scroll through endless digital content instead of noticing the world around us. WildSense takes a different approach: use AI as a bridge to the outdoors, not another reason to stay online.

📱 Take a photo. Discover a plant. Explore the world around you.

WildSense combines local AI-powered plant recognition with a botanical encyclopedia and nature-exploration features to encourage people to step outside, observe their surroundings, and learn from the natural world.

🌱 Key Features

🔬 Offline AI Plant Recognition: Uses a locally running FastViT ONNX model with a 4,271-taxon taxonomy to suggest plant identities without sending photos to a cloud AI service.
🌍 Botanical Encyclopedia: Explore an offline collection of 114 botanical species, including regional-language names and toxicity notes.
📸 Multi-Photo Exploration: Analyze multiple plant photographs and select the plant part you're observing to support identification.
🧭 Nature Missions: Turn outdoor exploration into small, engaging challenges that encourage observation and discovery.
📓 Personal Field Journal: Keep track of exploration sessions and discoveries locally.
🌿 Grass Score: Make outdoor exploration more engaging through progress and nature-focused activities.
🔊 Audio Narration: Listen to botanical information using browser-based speech synthesis, including offline-capable system voices where available.
📵 Phone-Free Field Mode: Use the app to learn what you need, then put your phone away and experience nature directly.
🔐 Local-First Privacy: Plant recognition runs on-device, and journal data is stored in a local SQLite database.

🎯 Who is it for?

WildSense is for students, nature enthusiasts, beginner botanists, hikers, families, and anyone who wants to reconnect with the outdoors without depending on a constant internet connection.

The philosophy is simple:

Technology should help us notice more of the real world—not distract us from it.

Demo

🎥 Video Demo: [Watch the WildSense walkthrough on YouTube] https://youtu.be/_LjfTU0UkzQ

The project is designed to run locally. For the full local-inference experience, follow the setup instructions in the repository.

Code

💻 GitHub Repository: https://github.com/kusalar/WildSense

The repository includes the frontend, FastAPI backend, local ONNX inference pipeline, botanical database, specialist-classifier experimentation, evaluation tools, and documentation.

The project is built to be explored, tested, and improved by the open-source community.

How I Built It

WildSense is built around local AI inference rather than a cloud-dependent identification API.

🧠 Open-Source AI at the Core

FastViT + ONNX Runtime: Runs the plant-image inference pipeline locally on the CPU.
4,271-Taxon Classification Space: Provides a broad taxonomy for candidate plant identification.
Experimental Indian Flora Specialist: An optional specialist classifier uses FastViT image embeddings and a conservative arbitration layer to improve selected regional plant identifications.
FastAPI + Python: Powers the backend, inference pipeline, API endpoints, and application logic.
SQLite: Stores the offline botanical encyclopedia, field journal, and exploration data.
Next.js + React + TypeScript + Tailwind CSS: Powers the interactive frontend.
Browser Speech Synthesis: Provides audio narration without requiring a cloud speech service.

🏗️ Architecture

The basic workflow is:

The user photographs a plant.
The image is processed locally by the AI inference pipeline.
The model produces candidate matches using image embeddings and taxonomy labels.
When enabled, the specialist classifier passes through additional confidence, margin, and contradiction checks.
WildSense displays a prediction or falls back when the specialist result does not meet its acceptance criteria.
The user can explore botanical information, record discoveries, and continue their exploration offline.

The specialist is experimental and disabled by default. The global model and specialist arbitration have been evaluated on a small held-out dataset, so real-world identification accuracy still needs broader validation. Predictions should not be used to determine whether a plant is edible or safe for medicinal use.

I also designed WildSense to work on modest hardware. The local inference approach avoids requiring a dedicated GPU or a paid AI API for the core plant-recognition workflow.

Why Does Open Innovation Matter?

Open innovation made WildSense possible as a project that people can inspect, run locally, evaluate, and improve instead of treating AI as an inaccessible black box.

A closed cloud API might offer convenient image recognition, but it would not automatically give users the same combination of local processing, data control, offline access, and freedom to experiment with the model pipeline.

With open tools and locally runnable models, I can:

Keep plant photos private: Images can be processed locally instead of being sent to a remote AI provider.
Build for limited connectivity: The core recognition workflow does not depend on a constant internet connection.
Experiment beyond a black-box response: I can evaluate embeddings, compare candidate matches, test specialist classifiers, and introduce conservative rejection rules.
Make the project accessible: People can explore a local-first AI application without needing a paid inference API.
Invite community improvements: Developers can contribute better regional datasets, improve recognition and abstention, expand the botanical encyclopedia, and add languages and nature missions.

Open innovation is especially important for regional biodiversity. Communities should be able to contribute local plant knowledge and improve tools for their environments instead of relying entirely on centralized systems.

The goal is not to make AI the center of the outdoor experience. It's to use open AI to help people look up from their screens.

Prize Categories

Used ' Eleven Labs ' for Speech to Text

🌿 Final Thought

WildSense is an experiment in building technology that encourages people to spend less time staring at technology.

Identify a plant. Learn something new. Notice the details. Put the phone away.

WildSense — Look closer. Then look away.

Hacktoberfest #OpenSourceAI #TouchGrass #LocalAI #ONNX #FastViT #NatureTech #AI

FRIEND.OS: An AI Memory That Remembers What Was Waiting for What

KUSAL SARKAR — Sat, 03 Oct 2026 18:15:37 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built FRIEND.OS, a local-first AI memory system for a close friend who tends to remember individual things but sometimes forgets the connection between them.

The problem is simple:

“I’ll do X after Y happens.”

We remember X.

We remember Y.

But when Y finally happens, we may forget that X was waiting for it.

FRIEND.OS is designed to catch that missing connection.

For example:

“I’ll submit my internship application after I receive my certificate.”

Later:

“I received my certificate today.”

FRIEND.OS recognizes that the certificate was the dependency for the internship application and proactively surfaces:

Something just clicked

You can now submit your internship application.

But FRIEND.OS also handles the opposite situation: when you are about to do something before completing what it depends on.

For example:

“I’ll eat dessert after I finish rice.”

Then:

“I’ll eat dessert now.”

FRIEND.OS can detect that the dependent action is being attempted before its prerequisite has been satisfied and surface:

You're forgetting something…

Eat rice first.

You said you'd eat dessert after finishing rice.

So FRIEND.OS isn't just storing memories. It tracks the relationships and state between memories.

The core idea is:

MEMORY → CONNECTION → REALIZATION

FRIEND.OS doesn't wait for you to ask what you forgot. It notices when something you've been waiting for finally happens — or when you're about to skip something you said you needed to do first.

I built it around a real problem experienced by someone close to me rather than starting with a generic AI chatbot and searching for a use case afterward.

Demo

Demo video: https://youtu.be/M7XgCTY7Xig

Windows download: https://github.com/kusalar/FriendOS/releases/tag/v1.0.0

The demo shows two sides of FRIEND.OS.

1. Something just clicked

I create:

“I'll submit my internship application after I receive my certificate.”

Then:

“I received my certificate today.”

FRIEND.OS identifies the new evidence, connects it to the waiting commitment, and proactively surfaces the next action.

2. You're forgetting something…

I create:

“I'll eat dessert after I finish rice.”

Then:

“I'll eat dessert now.”

FRIEND.OS detects that the dependent action is being attempted before its prerequisite has been satisfied.

It surfaces:

You're forgetting something…

Eat rice first.

The user can also inspect why FRIEND.OS produced the insight and see the relevant memory relationship.

The project also includes a 60-second Demo Mode with three realistic scenarios:

Internship application
VLSI project report
Research internship

Code

GitHub: https://github.com/kusalar/FriendOS

FRIEND.OS is structured as a local web application with:

Next.js
React
TypeScript
Tailwind CSS
FastAPI
SQLite
Ollama
Qwen3-4B
Deterministic dependency matching

A packaged Windows installer and portable build are also provided through the GitHub Release.

How I Built It

The AI at the core of FRIEND.OS is Qwen3-4B, running locally through Ollama.

The architecture is:


text
Browser
   ↓
Next.js / React / TypeScript
   ↓
FastAPI
   ↓
SQLite
   ↓
Qwen3-4B via Ollama
   ↓
Structured Memory
   ↓
Deterministic Dependency Engine
   ↓
Proactive Insight

Building a Multilingual Voice Agent for Local Commerce with Murf Falcon

KUSAL SARKAR — Sat, 15 Aug 2026 10:59:30 +0000

What if you could call a local business, speak naturally in Hindi, Bengali, English, or a mix of them, ask about a product or order, and get help without navigating complicated menus?

That was the idea behind my project for 10 Days of Voice Agents — VoiceForBharat Edition.

Over the course of the challenge, I evolved a basic real-time voice agent into a multilingual Local Commerce Voice Assistant with live product lookup, consent-based memory, outbound calling, human escalation, specialist-agent handoffs, and a call analytics dashboard.

The project is built using Murf Falcon, LiveKit, Deepgram, an LLM, Python and Next.js.

🔗 GitHub: https://github.com/kusalar/agentvoice

🎯 The Problem

Local businesses often need to answer repetitive customer questions:

Is this product available?
What is the price?
Where is my order?
When will my order arrive?
Can I return this product?
I have a problem with my order. Can someone help me?

Traditional chat interfaces aren't always convenient, especially for users who are more comfortable speaking than typing.

This is where voice can make the interaction more natural.

My goal was to build an assistant that could provide a more conversational experience while still being useful, reliable and safe.

I chose the local commerce/customer support direction because it allowed me to combine voice AI with real business tools and workflows.

🗣️ Why Voice?

Voice is particularly interesting for the Indian context because conversations don't always happen in a single language.

A customer might say:

"Bhai, mera order kab tak deliver hoga?"

or:

"Product available hai kya? Price bhi bata do."

Or they may switch between Hindi and English during the same conversation.

Instead of forcing users to type formal queries, a voice interface allows them to interact naturally.

My agent is designed to handle English, Hindi, Bengali and code-mixed conversations, while trying to mirror the user's language and conversational style.

"The final Local Commerce Assistant interface: a voice-first shopping experience for discovering local products and interacting with the agent."

🏗️ How the Voice Agent Works

The core architecture looks like this:

                 ┌─────────────────────┐
                 │      Customer       │
                 │    Browser / Phone  │
                 └──────────┬──────────┘
                            │
                    Voice / SIP / Chat
                            │
                            ▼
                 ┌─────────────────────┐
                 │      LiveKit        │
                 │   Real-time RTC     │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │    Deepgram STT     │
                 │    Speech → Text    │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │   LiveKit Agent     │
                 │   LLM + Reasoning   │
                 │      + Tools        │
                 └──────────┬──────────┘
                            │
              ┌─────────────┼─────────────┐
              │             │             │
              ▼             ▼             ▼
        Product Tool    Memory DB    Escalation Tool
              │             │             │
              ▼             ▼             ▼
       Local Catalogue  Caller Memory  SQLite Tickets
                                          │
                                          ▼
                                   Human Escalation
                                          │
                                          ▼
                                Returns Specialist
                                          │
                                          ▼
                                 ┌─────────────────────┐
                                 │    Murf Falcon      │
                                 │    Text → Speech    │
                                 └──────────┬──────────┘
                                            │
                                            ▼
                                       🔊 Customer

And specialist path: 

Main Agent
    │
    │ refund / returns issue
    ▼
Returns & Refunds Specialist
    │
    ▼
Resolution / Escalation

The four fundamental components are:

1. Speech-to-Text

I use Deepgram to convert the user's speech into text.

2. LLM

The LLM understands the request, decides how to respond and determines when it needs to use one of the available tools.

3. Text-to-Speech

I use Murf Falcon to convert the agent's responses back into natural speech.

4. Real-time Transport

LiveKit handles the real-time communication between the user and the voice agent.

🇮🇳 Indian Voice with Murf Falcon

One of the key parts of the project is the voice itself.

I used Murf Falcon for the TTS layer and experimented with Indian voices including voices such as Pooja, Anisha and Samar.

The goal wasn't simply to make the agent speak.

I wanted the interaction to feel natural enough for an Indian customer-support scenario.

Fast TTS matters in voice applications because long delays between the user finishing their sentence and the agent responding can make the conversation feel unnatural.

🧠 Agent Personality and Safety

A voice agent needs more than an LLM.

I defined a specific personality and behavioral rules for the assistant.

The agent is designed to be:

Friendly
Helpful
Concise
Professional
Conversational

But it also has important guardrails.

For example:

It should not invent product prices or availability.

If information needs to be verified, the agent should use the appropriate tool.

Similarly, sensitive actions such as storing customer information or creating an escalation should require user consent.

This helped me move from simply building a chatbot to thinking about how a voice agent should behave in a real customer-facing environment.

🛒 Live Product Lookup

One of the most useful features I built was a product lookup tool.

Instead of allowing the LLM to guess product information, the agent can query actual catalogue data.

The project uses Open Food Facts as a live source and also maintains a local catalogue as a fallback.

For example, a customer can ask:

"Do you have organic honey?"

The agent can use the tool to check the product rather than generating a random answer.

If the external service isn't available, the system can fall back to the local catalogue.

This taught me an important lesson:

An AI agent should use tools for facts instead of relying on the LLM to remember business data.

"Live product lookup: the agent retrieves catalogue information such as product name, seller, pack size and price instead of guessing product details."

🧠 Memory with User Consent

I also implemented memory for returning users.

The agent can remember useful information such as:

Customer name
Language preference
Relevant customer information
Previous interaction information

But I didn't want the system to automatically store everything a user says.

The agent asks for permission before saving information.

The basic workflow is:

Customer shares information
          ↓
Agent asks for permission
          ↓
       ┌──┴──┐
       │     │
      YES    NO
       │     │
       ▼     ▼
     Save   Don't save

This made memory more privacy-conscious and also made me think about consent as part of the agent's design rather than an afterthought.

"Persistent memory in action: after consent, the assistant remembers the customer's name and previous interaction to provide a more personalized conversation."

📊 Call Analytics Dashboard

A voice application also needs visibility into what is happening during conversations.

During the challenge, I built a call analytics dashboard to monitor and analyze voice interactions.

Instead of only seeing whether the agent works, the dashboard provides a way to understand call activity and conversation performance.

This is important for a real deployment because businesses need to answer questions such as:

How many calls are happening?
How are conversations performing?
What is happening during calls?
Where are users encountering problems?
How is the voice agent being used?

Building this dashboard helped me understand that production voice AI isn't only about the conversation itself.

It also needs observability and analytics.

"Call analytics dashboard: tracking call outcomes, success rates, failure categories, channels and recent conversation activity."

📞 Outbound Calling

The project also goes beyond incoming browser-based conversations.

I implemented outbound calling using LiveKit SIP.

This can be useful for scenarios such as:

Customer follow-ups
Order notifications
Restock notifications
Callback requests
Proactive customer communication

This was an interesting part of the project because it introduced telephony into the voice-agent architecture.

"Outbound Calling: Call from team to customer for support."

👨‍💼 Human Escalation

AI shouldn't try to solve every problem.

For difficult cases such as:

Refund disputes
Return problems
Damaged products
Missing orders
Serious delivery issues

the assistant can escalate the issue to human support.

Before creating an escalation, the agent asks the customer for consent.

The escalation can contain relevant information such as the issue description and what has already been checked.

The customer can then receive a reference ID for the support request.

Sensitive information such as passwords, OTPs and payment credentials should never be collected or exposed.

"Human-in-the-loop escalation dashboard: support tickets created by the voice agent are tracked through open, in-progress and resolved states."

🔄 Specialist Agent Handoff

I also implemented a specialist Returns Agent.

Instead of forcing one agent to handle every possible scenario, certain conversations can be handed to a specialist.

For example:

                 Main Agent
                     │
             Return / Refund?
                     │
                     ▼
             Returns Specialist
                     │
                     ▼
              Specialist Agent

This introduced me to a more scalable approach to voice agents:

multiple specialized agents working together instead of one enormous agent trying to do everything.

"Specialist-agent handoff: the main commerce assistant recognizes a returns/refund issue and transfers the conversation to a dedicated Returns & Refunds Specialist."

😅 The Difficult Parts

The hardest part wasn't simply getting the agent to speak.

The difficult part was making the system behave reliably.

One of the biggest problems was preventing the LLM from confidently producing information that wasn't actually verified.

For example, if someone asked:

"Is this product available?"

the LLM could theoretically generate a plausible answer even if it had no actual catalogue information.

I solved this by introducing dedicated tools for business data.

The agent uses the product lookup tool when it needs product information instead of relying on its own knowledge.

I also added fallback behavior when external product data isn't available.

Another challenge was coordinating multiple components:

Frontend
   ↓
LiveKit
   ↓
STT
   ↓
LLM
   ↓
Tools
   ↓
TTS
   ↓
Frontend

A problem in any one of these layers can affect the entire conversation.

Working through these issues taught me that a voice agent is really a system of connected components, not just an LLM with a voice.

🛠️ How to Run the Project

If you want to experiment with the project yourself, the complete source code is available on GitHub:

🔗 https://github.com/kusalar/agentvoice

Requirements

You'll need:

Python 3.10+
Node.js 18+
uv
pnpm
LiveKit
Murf API credentials
Deepgram API credentials
LLM API credentials

Clone the repository:

git clone https://github.com/kusalar/agentvoice.git
cd agentvoice

Install the backend dependencies:

cd backend
uv sync

Install the required model/files:

uv run python src/agent.py download-files

Then install the frontend dependencies:

cd ../frontend
pnpm install

🔐 Environment Variables

Never put API keys directly into the source code.

Create the appropriate environment file and add your credentials there.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_llm_api_key

Use your actual credentials locally, but never commit them to GitHub or publish them in screenshots.

▶️ Running the Agent

Start the LiveKit development server:

livekit-server --dev

Run the backend agent:

cd backend
uv run python src/agent.py dev

Start the frontend:

cd frontend
pnpm dev

Then open:

http://localhost:3000

Allow microphone access and start a conversation with the agent.

🧪 Testing the Agent

A simple test conversation could be:

User:
"Do you have organic honey?"

Agent:
Checks the product tool.

User:
"How much is it?"

Agent:
Returns the verified catalogue information.

You can also test:

Switching between Hindi and English
Bengali conversations
Code-mixed sentences
Product availability
Memory consent
Return/refund escalation
Specialist-agent handoff
Outbound calling

For testing screenshots and demos, use dummy customer information rather than real personal data.

📸 Evidence From My Build

The project evolved throughout the 10-day challenge, and I documented different capabilities as I built them.

The most useful evidence includes:

The final voice-agent interface
A real conversation with the agent
Product lookup in action
The call analytics dashboard
Memory/consent flow
Human escalation
Specialist-agent handoff
Outbound calling

I don't need to include every screenshot from every day.

Instead, I want the screenshots to show the progression from a simple voice agent into a more complete voice system.

🚀 What I Learned

The biggest lesson from this challenge is that building a voice agent isn't just about making AI talk.

A useful voice agent needs:

Voice + Reasoning + Tools + Memory + Safety + Real-time communication + Observability + Human handoff

The voice is what the user experiences.

But the tools, data validation, consent, fallback behavior, analytics and escalation logic are what make the system useful.

Over these 10 days, I learned how these different pieces can come together to create something much closer to a real-world voice application.

🔮 What's Next?

There is still a lot I would like to improve:

More Indian languages
Better production telephony
More business integrations
More advanced call analytics
Better observability
More specialist agents
Production deployment
Automated voice-agent evaluation

But this challenge gave me a strong foundation for exploring all of these areas.

Final Thoughts

What started as a basic voice agent became a multilingual local-commerce assistant capable of using tools, remembering users with consent, making calls, escalating problems, handing conversations to specialists and analyzing calls.

The most important thing I learned is:

A good voice agent shouldn't just sound intelligent. It should be useful, accurate, safe and measurable.

I'm grateful to Murf AI for organizing 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build and experiment with voice AI over 10 days.

🔗 Project: https://github.com/kusalar/agentvoice

Built with:

Murf Falcon + LiveKit + Deepgram + LLM + Python + Next.js

VoiceForBharat #MurfAI #VoiceAI #GenerativeAI #AI #BuildInPublic