DEV Community: Vishal Kumar

SOLO: A Local-First AI Developer Agent That Works With Gemma 4

Vishal Kumar — Mon, 26 Oct 2026 12:59:23 +0000

This is my submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

I built SOLO, a local-first AI developer agent designed to help developers work with AI without being locked into a single model or cloud provider.

I wanted to explore something beyond a basic AI chatbot. Something that could actually interact with a development environment, understand project context, use tools, and help with everyday engineering tasks.

Most AI coding assistants depend heavily on hosted APIs. That's useful, but I also wanted the option to work locally, keep development context on my own machine, and switch between different AI models whenever needed.

That's where SOLO comes in.

With SOLO, developers can use a locally running Gemma 4 model through Ollama or connect to hosted AI providers. The same agent runtime handles both.

Some things SOLO can do:

Read and search project files.
Write and modify files through controlled tools.
Execute calculations.
Run restricted shell commands with permission checks.
Use skills to provide additional instructions and capabilities.
Connect external tools through MCP.
Maintain sessions, memory, and execution traces.
Switch between local and hosted AI providers.

The idea is simple: Your development agent, your models, your environment, your control.

Demo

[Add your SOLO demo video or deployed dashboard link here]

A short walkthrough showing local Gemma 4 running through Ollama, interacting with the agent, using tools, and switching providers would be a good demonstration.

Code

GitHub Repository: [Add your SOLO GitHub repository URL]

I'd love for developers to explore the implementation, experiment with different models, and contribute ideas.

How I Built It

I built SOLO around a shared agent runtime rather than creating separate implementations for different AI providers.

This was important because I didn't want the agent's behavior to change completely whenever someone switched models.

1. Gemma 4 + Ollama (Local AI)

For the local AI setup, I integrated Gemma 4 through Ollama.

I'm currently using gemma4:e2b locally.

Ollama makes it possible to run the model on my own machine and communicate with it through a local API.

This gives SOLO an offline-capable model option without requiring every interaction to go through a hosted AI service.

2. Multiple AI Providers

I didn't want SOLO to depend on just one model.

The provider architecture supports:

Ollama local provider.
OpenAI-compatible API provider.
Optional LiteLLM adapter.

This makes it possible to experiment with different models without rebuilding the entire agent.

3. The Agent Runtime

The core of SOLO is a shared runtime implemented in Python.

Both the CLI and API use the same SoloContext runtime.

The agent follows a controlled execution flow:

Receive the user's request.
Validate the request.
Load relevant skills.
Send context to the selected model.
Validate requested tool calls.
Check permissions.
Request approval when required.
Execute approved tools.
Persist the result and execution trace.

I wanted the agent to have actual boundaries instead of blindly executing whatever the model suggests.

4. Tools and Permissions

SOLO includes tools for:

File listing and reading.
File writing.
Project search.
Calculations.
Restricted shell execution.

Shell execution is disabled by default and uses an allowlist.

The agent also supports approval workflows, so sensitive operations are not automatically executed without authorization.

5. MCP Integration

Another interesting part of this project is Model Context Protocol (MCP).

SOLO supports MCP connections through:

stdio
SSE
Streamable HTTP

This allows the agent to connect to external tools and services instead of being limited to its built-in capabilities.

6. Memory and Sessions

I also implemented persistent sessions, memory, and execution traces using SQLite.

This helps keep track of previous interactions and makes it easier to inspect what the agent actually did.

7. Dashboard and CLI

SOLO isn't limited to a terminal.

The project includes:

Typer-based CLI.
FastAPI backend.
React + Vite dashboard.

The dashboard provides an interface for interacting with the agent and managing its functionality.

Tech Stack

Component	Technology
AI Model	Gemma 4 (gemma4:e2b)
Local Inference	Ollama
Backend	Python, FastAPI
CLI	Typer
Frontend	React, Vite
Database	SQLite
AI Providers	Ollama, OpenAI-compatible APIs, LiteLLM
Tool Integration	MCP
Testing	Pytest

Why Does Open Innovation Matter?

One of the biggest reasons I wanted to work with open-weight models was control.

Developers should have the freedom to experiment with AI without always depending on external APIs.

Using Gemma 4 with Ollama gives SOLO a local execution option. It makes experimentation easier and allows developers to keep model interactions on their own machine when using the local provider.

I also like that open models encourage developers to understand what's happening behind the interface rather than treating AI as a black box.

For me, this challenge was a good opportunity to explore how open-weight models can become part of real developer tools.

The Lazy Payoff

The biggest benefit of SOLO is that I don't have to build a completely different agent whenever I want to try another model.

I can switch between a local Gemma 4 model and hosted providers while keeping the same agent runtime, tools, and workflow.

It also gives me a structured way to experiment with tool execution, project context, permissions, and MCP integrations.

I'm still improving the project, but building it has taught me a lot about what goes into making an AI agent useful beyond simply generating text.

What I Learned

Building SOLO helped me understand that an AI agent is much more than a model connected to a chat interface.

The interesting engineering challenges are around:

Tool execution and validation.
Provider abstraction.
Permission management.
Session persistence.
Error handling and retries.
Keeping local and hosted model workflows consistent.

These are the parts I enjoyed exploring the most.

What's Next?

I want to continue improving SOLO by making its tool ecosystem more extensible, strengthening its security, improving the developer experience, and exploring more open-weight models.

My long-term goal is to make SOLO a practical agent harness that developers can run locally, customize, and extend according to their own workflows.

Built with curiosity, Python, Ollama, and Gemma 4.

Thanks for checking out SOLO!

Hacktoberfest #Gemma4 #Ollama #OpenSource #AI #DeveloperTools

Community Copilot: An AI-Powered Discord Assistant for Developer Communities

Vishal Kumar — Mon, 19 Oct 2026 06:29:26 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

What I Built

Developer communities on Discord are full of valuable knowledge: bug reports, troubleshooting discussions, questions, solutions, and helpful conversations. But as a community grows, finding that information becomes increasingly difficult.

Important solutions get buried in long threads. Moderators spend time reviewing repeated issues, and newcomers often ask questions that have already been answered.

I built Community Copilot, an AI-powered Discord assistant designed to help communities organize discussions, understand technical problems, and turn past conversations into searchable knowledge.

Instead of treating every conversation as disposable, Community Copilot helps preserve useful information for the next person who encounters the same problem.

What can it do?

Thread summarization: Turn long Discord discussions into structured summaries containing the main problem, findings, attempted solutions, blockers, unanswered questions, and next steps.
Image analysis: Analyze screenshots shared in technical discussions, including error messages and debugging information.
Issue tracking: Help moderators propose solutions and manage their approval or rejection.
Searchable knowledge base: Search past discussions and approved solutions using /kb-search.
Human-reviewed solutions: Keep proposed solutions separate from approved knowledge so moderators can review them before they become searchable.
Cancellation controls: Allow users to cancel their own pending or running summarization requests.
Source links: Keep references to the original discussions so users can revisit the context behind a summary.
Privacy-conscious interactions: Use private command responses by default and enforce access controls for supported operations.

Community Copilot currently supports 11 Discord slash commands, covering status checks, summarization, knowledge search, moderation workflows, insights, and privacy-related operations.

The goal is simple: help communities spend less time searching through old conversations and more time solving problems together.

Potential users include developer communities, open-source project maintainers, student tech clubs, programming servers, and volunteer moderators.

Demo

Community Copilot runs as a Discord bot, so its interface lives directly inside Discord.

The current demo covers:

Checking the bot's health using /status.
Summarizing a technical discussion with /summarize.
Analyzing an error screenshot as part of a discussion.
Searching previous discussions using /kb-search.
Proposing, approving, and rejecting solutions through moderation commands.
Cancelling a running summarization task.

I tested these workflows in a Discord test server, including image analysis, knowledge-base search, solution approval and rejection, and cancellation of a running task.

Demo: Add a link to your demo video or screenshots here if available.

Code

GitHub repository: https://github.com/codewithvishuuu/community-copilot

The repository contains the bot implementation, configuration examples, documentation, and automated tests.

Please make sure the repository is public and the latest intended changes are pushed before publishing this article.

How I Built It

Community Copilot is built with Python and discord.py, with a local language model handling AI inference.

Technology stack

Python — core application and bot logic.
discord.py — Discord slash commands and interactions.
Ollama — local model inference.
Gemma 4 E2B — the local model used for analysis and summarization.
SQLite and FTS5 — persistent storage and searchable knowledge.
pytest — automated tests.
Ruff — code quality and lint checks.

How the system works

The bot collects messages from a Discord thread, processes the available text and images, and passes the relevant context to the local model. The resulting analysis is validated and organized into a structured summary.

The application stores relevant information in SQLite. Full-text search helps retrieve past discussions and approved solutions when someone searches for a related issue.

Moderation commands provide a human-review step before proposed solutions become approved knowledge. This separation matters because an AI-generated suggestion should not automatically become an authoritative answer.

The project also includes an inference queue, ownership checks for cancellation, privacy controls, source references, and automated tests for important workflows.

I used an agent-assisted development workflow to build and test the application, then verified the major Discord interactions manually.

At the time of my latest local verification, the automated suite reported 228 tests passed and 2 skipped, and Ruff checks passed. These are local verification results, not a claim of production deployment.

Why Does Open Innovation Matter?

Open innovation matters because developer communities work best when people can inspect, improve, adapt, and share the tools they use.

Building Community Copilot around an open-weight model and local inference gives the project several useful properties:

More control over data flow: Inference can run locally rather than requiring every conversation to be sent to a hosted model API.
Model flexibility: The model and inference setup can evolve as better open-weight options become available.
Lower dependency on hosted AI services: The core analysis workflow can use a locally running model, although the Discord bot itself still requires network access.
Community-driven improvement: Other developers can inspect the implementation, report bugs, contribute features, and adapt the bot for their own communities.
Transparency: Tests and source code make it easier to understand how the application behaves.

Local inference does not automatically guarantee privacy or eliminate every cost. The deployment environment, Discord integration, permissions, and data-retention policies still matter.

For me, open innovation means building something that others can understand, run, improve, and make their own.

My Agent Session

This section is optional.

If I publish a session using DevRelay, I will link or embed it here so readers can see how the project was developed.

Why This Project Matters to Me

I wanted to build something practical with AI—not just another chatbot that answers isolated questions.

Community Copilot focuses on the knowledge already created by a community: the discussions people have, the problems they solve, and the lessons worth preserving.

The interesting part is connecting AI summarization with search, moderation, and a persistent knowledge base. A useful answer should not disappear when a Discord conversation gets buried.

This is an early-stage project, and there is still room to improve reliability, search quality, error handling, and the overall moderation experience.

I hope other developers will explore the code, share feedback, and contribute ideas that make it more useful.

Prize Categories

Check the challenge rules and list only the partner categories that your project genuinely qualifies for. Remove this section if none apply or if you have not verified the eligible categories.

Thanks for checking out Community Copilot! If you have ideas for improving AI-powered knowledge management in developer communities, I'd love to hear your feedback.

Verdantra: An AI-Powered Field Journal That Gets You Outdoors 🌿

Vishal Kumar — Sun, 11 Oct 2026 05:48:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

🌿 Verdantra — Look closer outside.

Verdantra is an open-source, AI-powered botanical field journal designed to encourage people to step away from their screens and explore the natural world.

The idea is simple: take a photo of a leaf, plant, rock, puddle, or another outdoor subject, and use AI to guide your observation rather than replace it.

Key Features

Photo-based observations: Upload an outdoor photo for AI-assisted analysis.
Visible evidence: Receive descriptions of features visible in the image, such as leaf colors, shapes, and patterns.
Honest uncertainty: The AI highlights what cannot reliably be determined from a photo instead of presenting guesses as facts.
Safe mini-experiments: Get short observation activities that encourage real-world exploration without picking, tasting, or damaging plants.
Personal field journal: Save your photos, AI predictions, and your own observations for later review.
Online and offline AI modes: Use Google's Gemini API in online mode or a local Gemma model through Ollama in offline mode. Each mode can be selected explicitly.

Verdantra is built around an important principle: AI should encourage curiosity about the real world, not replace our own observations.

Demo

GitHub Repository: https://github.com/codewithvishuuu/verdantra

The project currently runs locally. A public, deployed demo is not available yet.

The repository includes setup instructions for running the backend and frontend locally, configuring the AI provider, and testing the application.

Code

Explore the complete source code:

View Verdantra on GitHub

The repository contains the React frontend, FastAPI backend, AI integration, journal functionality, tests, and individual project documentation files.

How I Built It

I built Verdantra using a modern web stack with an emphasis on practical AI integration, privacy awareness, and reliable local development.

Technology stack

Frontend: React, TypeScript, Vite, and Tailwind CSS
Backend: Python and FastAPI
AI — Online: Google's Gemini API using gemma-4-26b-a4b-it
AI — Offline: Gemma 4 E2B through Ollama using the gemma4:e2b model
Data storage: SQLite for journal entries
Testing: Pytest for backend tests, TypeScript checks, and Vite production builds

The application uses a shared AI service interface so that the observation workflow can work with either provider without requiring separate frontends.

For online use, the backend sends the image to the configured Gemini API. For offline use, the backend communicates with the local Ollama server, allowing image analysis without an internet connection once the model is installed and running.

The AI instructions emphasize photo-grounded evidence, plain language, explicit uncertainty, and safe observation activities. Predictions are labelled unverified so users understand that AI-generated descriptions can be wrong.

I also added automated tests for backend behavior and output constraints. Passing tests verify implementation behavior; they do not guarantee that every AI-generated observation is botanically correct.

Why Does Open Innovation Matter?

Nature exploration should not depend on expensive hardware or a single online AI service.

Open-source tools made it possible to build and inspect the complete application, combine different AI inference options, and give users a choice between hosted and local processing.

Local inference through Ollama also provides an alternative when internet access is unavailable. The hosted option offers a different model and inference path, while the local option keeps image processing on the user's machine.

Open innovation makes projects like Verdantra easier for other developers to inspect, learn from, improve, and adapt for their own communities.

I hope other contributors will help improve the quality of observations, accessibility, offline usability, and the overall field-journaling experience.

My Agent Session

This section is optional. I do not have a verified, shareable DevRelay agent-session link to include, so I am leaving it out rather than linking to a nonexistent session.

Prize Categories

My submission is for the Hacktoberfest Open-Source AI Challenge — Week 1: Touch Grass.

I have not added any additional partner prize categories because I have not verified which partner categories apply to this project.

Built with curiosity, open-source tools, and a little more time outdoors.

Verdantra — Look closer outside. 🌱

I Built StudyMate for a Friend Who Was Tired of Studying From Scattered PDFs

Vishal Kumar — Sun, 04 Oct 2026 16:40:31 +0000

What I Built

I built StudyMate, a grounded AI study companion for a friend who was struggling with scattered PDFs, notes, and revision material.

The problem was simple: studying required constantly switching between PDFs, searching for specific topics, asking questions in separate AI tools, creating revision notes, and figuring out what was actually remembered.

StudyMate brings this workflow into one focused study space.

Its core loop is:

Learn → Ask → Practice → Identify Weakness → Revise → Improve

Students can upload their study PDFs and use them to:

Ask questions grounded in their own study material
Generate revision-ready summaries
Create quizzes with different difficulty levels
Practice with AI-generated flashcards
Build a study plan based on their exam date and available study time
Track their study progress

The most important design rule is grounding.

StudyMate should not pretend to know something that is not present in the student's uploaded material. If the required information cannot be found, it responds:

"I couldn't find this information in your uploaded study material."

It also preserves source and page information so that answers can be traced back to the student's actual notes instead of using fabricated citations.

I built StudyMate specifically for a real student workflow rather than creating another general-purpose AI chatbot.

Demo

Try StudyMate here:

Live Demo: https://studymate-frontend-hbi2.onrender.com

You can upload a study PDF, select your study material, ask grounded questions, generate summaries, create quizzes, practice flashcards, and build a personalized study plan.

The live demo is deployed on Render so anyone can open it and try the application directly in their browser.

Note: The current deployment is running on Render's free infrastructure, so uploaded study data should not be considered permanent storage. The live deployment is primarily intended for demonstrating and testing the application.

Code

The complete source code for StudyMate is available on GitHub:

GitHub Repository: https://github.com/codewithvishuuu/studymate

The project is open source and released under the MIT License.

The repository contains the complete React frontend, FastAPI backend, AI/RAG pipeline, document processing, database logic, tests, and deployment configuration.

The project was built as a full-stack application rather than a frontend-only AI demo, with the backend handling AI orchestration, retrieval, document processing, and application logic.

How I Built It

StudyMate is built as a full-stack AI application using React, TypeScript, Vite, Tailwind CSS, Python, FastAPI, ChromaDB, and Gemma.

The core AI workflow is based on Retrieval-Augmented Generation (RAG).

When a student uploads a PDF, StudyMate processes it through this pipeline:

PDF
↓
Text Extraction
↓
Cleaning
↓
Page-Aware Chunking
↓
Embeddings
↓
ChromaDB

Why Does Open Innovation Matter?

For StudyMate, open innovation means having more flexibility and control over the AI layer instead of building the entire product around a closed, black-box system.

I chose Gemma as the generation model because it gives me an open-weight model that I can experiment with as the project evolves.

With this approach, I can explore different possibilities in the future, such as:

Different Gemma variants
Local inference
Different retrieval strategies
Different embedding models
Fine-tuning
Different deployment environments

This flexibility is important for StudyMate because the AI model is only one part of the system. The document processing, retrieval pipeline, source attribution, and study features can continue evolving independently.

The project also helped me understand that open innovation is not only about using an open model. It is about having the freedom to experiment, understand the technology, and build a system that can evolve with the needs of its users.

For a study application, that flexibility can eventually mean better control over cost, privacy, deployment, and model choice.

Building for a Friend

I built StudyMate around a real problem faced by a friend: studying from scattered PDFs and notes was taking more effort than it should.

Instead of starting with the question, "What AI feature can I build?", I started with:

"What would actually make studying easier for my friend?"

That changed the direction of the project.

I focused on making the student's own study material the center of the experience.

That led to a few important decisions:

Answers should be grounded in the uploaded study material.
Source and page information should be preserved.
The system should never create fake citations.
Students should be able to turn their notes into summaries.
Quizzes should be generated from the same study material.
Flashcards should support active recall.
A study plan should help organize revision around an exam date.
If information is not available in the uploaded material, the AI should clearly say so.

The goal was not to build another general-purpose chatbot.

The goal was to build something my friend could actually use as part of their study routine.

That is why the core experience of StudyMate is:

Upload → Understand → Ask → Practice → Revise → Improve

What I Learned

One of the biggest lessons from building StudyMate was that building an AI feature is very different from building a reliable AI product.

Getting an AI model to answer a question is relatively easy.

The harder part is making sure that the answer is based on the correct source, that relevant context is retrieved, that source information is preserved, and that the system knows when it does not have enough information to answer.

Working on the RAG pipeline also taught me how important retrieval quality is.

A powerful model with poor context can still produce a poor answer. Giving the model relevant and well-structured context makes the generation much more useful.

I also learned that the parts of an AI application that users don't immediately see are extremely important.

During the project, I spent significant time working on:

PDF processing
Page-aware chunking
Retrieval
Source attribution
API error handling
Loading states
Responsive UI
Mobile layouts
CORS
Production routing
Deployment
Testing

The project also reinforced an important principle for me:

AI should not just generate answers. It should behave reliably within the context of the problem it is solving.

For StudyMate, that means being grounded in the student's material and being honest when the material does not contain the answer.

Testing

I wanted StudyMate to be more than a happy-path AI demo, so I tested both the backend and the frontend throughout the build.

The backend currently has:

46 passing tests

I also tested the frontend across different screen sizes:

1440px desktop
1024px tablet
768px tablet
390px mobile

The application was checked for:

Horizontal overflow
Console errors
Failed API requests
Responsive layouts
Real API flows
Grounded answers
Not-found behavior
Source and page citations
Quiz generation
Flashcards
Study plans
Settings persistence

I also tested the production deployment.

During deployment, I found a client-side routing issue where refreshing an internal React route could return a Not Found page. I fixed this by adding a Render SPA rewrite so routes such as /notes, /ask, /quiz, and /flashcards correctly load the React application when refreshed.

The goal was to make sure StudyMate works as an actual application that someone can use, rather than only working during development on my local machine.

Tech Stack

Frontend

React
TypeScript
Vite
Tailwind CSS

Backend

Python
FastAPI
Pydantic

AI & RAG

Gemma 4
Google Gemini API
Embeddings
Retrieval-Augmented Generation (RAG)

Data & Storage

ChromaDB
SQLite
PDF/document storage

Deployment

Render

Best Use of Gemma

I am entering StudyMate in the Best Use of Gemma category.

Gemma is not being used as a superficial add-on. It is part of the core generation pipeline that powers the grounded study experience.

When a student asks a question, StudyMate first retrieves relevant information from the student's uploaded study material using embeddings and ChromaDB. The retrieved context is then passed to Gemma to generate the final grounded response.

This architecture allows Gemma to work with the student's actual study material instead of treating StudyMate as a general-purpose chatbot.

For the current deployed version, Gemma is served through the Google Gemini API, while the backend handles the AI orchestration and retrieval pipeline.

Using Gemma also gives me room to explore future directions such as local inference, different Gemma variants, improved retrieval strategies, and other deployment approaches.

The model is therefore directly connected to the core problem StudyMate is solving: helping students understand and practice from their own study material.

Final Thoughts

I started with a simple problem: studying from scattered PDFs and notes was harder than it needed to be.

That problem became StudyMate.

Instead of building another general-purpose AI chatbot, I wanted to create a study companion that understands the student's own material, helps them practice, and supports their revision workflow.

The project taught me that building with AI is not only about choosing a powerful model. It is also about grounding, reliability, source attribution, good product design, and solving a real problem for a real person.

The core idea behind StudyMate is simple:

Learn → Ask → Practice → Revise → Improve

I hope StudyMate can make studying a little more focused, practical, and useful for the person I built it for.

Thanks for checking it out!

Live Demo: https://studymate-frontend-hbi2.onrender.com

GitHub: https://github.com/codewithvishuuu/studymate

My Agent Session

I used AI-assisted development throughout the process of designing, implementing, testing, and polishing StudyMate.

The development process included planning the architecture, building the RAG pipeline, implementing the study features, testing the application across screen sizes, debugging deployment issues, and preparing the project for release.

I am not adding a DevRelay session link here because I do not want to publish an unverified or placeholder URL.

Additional Notes

StudyMate is an open-source project built specifically around the idea of using AI to solve a practical study problem.

The project is intentionally focused on grounded AI rather than generic AI generation. The student's own study material remains at the center of the experience, while Gemma, RAG, retrieval, and the study tools work together to support learning and revision.

The project is still evolving, and future improvements will focus on better retrieval, stronger study analytics, more local inference options, and feedback from actual users.

Thank You

Thank you for taking the time to check out StudyMate.

I hope this project shows how open AI and thoughtful product design can come together to solve a simple problem for a real person.

If you try StudyMate, I'd love to hear what you think and what I can improve next.

Building an AI Bug Triage Agent with Swytchcode and Gemini

Vishal Kumar — Fri, 21 Aug 2026 13:52:42 +0000

𝐓𝐮𝐫𝐧𝐢𝐧𝐠 𝐮𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐛𝐮𝐠 𝐫𝐞𝐩𝐨𝐫𝐭𝐬 𝐢𝐧𝐭𝐨 𝐚𝐜𝐭𝐢𝐨𝐧𝐚𝐛𝐥𝐞 𝐭𝐫𝐢𝐚𝐠𝐞 𝐫𝐞𝐩𝐨𝐫𝐭𝐬 𝐰𝐢𝐭𝐡 𝐀𝐈, 𝐫𝐞𝐭𝐫𝐢𝐞𝐬, 𝐭𝐢𝐦𝐞𝐨𝐮𝐭𝐬, 𝐚𝐧𝐝 𝐢𝐝𝐞𝐦𝐩𝐨𝐭𝐞𝐧𝐜𝐲.

𝟏.𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚𝐧 𝐀𝐈 𝐁𝐮𝐠 𝐓𝐫𝐢𝐚𝐠𝐞 𝐀𝐠𝐞𝐧𝐭 𝐰𝐢𝐭𝐡 𝐒𝐰𝐲𝐭𝐜𝐡𝐜𝐨𝐝𝐞 𝐚𝐧𝐝 𝐆𝐞𝐦𝐢𝐧𝐢

Software bugs are a normal part of development, but analyzing bug reports manually can take a lot of time.

A developer may receive a bug report containing an error message, incomplete context, or a description of unexpected behavior. Before fixing the issue, they usually need to determine:

What is the actual problem?
How severe is it?
Which part of the application is affected?
What could be causing it?
What should be checked first?
How can the issue be fixed?

To solve this problem, I built an AI Bug Triage Agent using Swytchcode and Gemini.

The agent takes a bug report as input and automatically generates a structured triage report containing the summary, severity, category, likely root cause, recommended fix, and debugging steps.

I also implemented two bonus features:

Production-oriented execution policies
Application-level idempotency

Project structure of the AI Bug Triage Agent

𝟐.𝐖𝐡𝐚𝐭 𝐃𝐨𝐞𝐬 𝐭𝐡𝐞 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 𝐃𝐨?

The workflow is simple:

Bug Report → Python Agent → Swytchcode → Gemini → Structured Bug Triage Report

The user provides a bug report through a text file or standard input.

The Python agent sends the report to Gemini through the Swytchcode runtime. Gemini analyzes the issue and returns structured information.

The final output contains:

Summary
Severity
Category
Likely root cause
Recommended fix
Debugging steps

This makes the output much easier for a developer to understand and act on.

Example bug report used as input

𝟑.𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞

The important part of this architecture is that the Python application does not directly call the Gemini API using an HTTP library or SDK.

The AI execution goes through:
swytchcode_runtime.exec("gemini.models.generateContent", ...)

Gemini is accessed through the Swytchcode runtime

𝟒. 𝐖𝐡𝐲 𝐒𝐰𝐲𝐭𝐜𝐡𝐜𝐨𝐝𝐞?

One of the main goals of this project was to use Swytchcode as the execution layer for the AI integration.

Instead of adding a direct HTTP client or an SDK to the Python application, the project uses the Swytchcode runtime.

This provides a clean separation:

Python handles the application logic.
Swytchcode handles the integration execution.
Gemini handles the AI analysis.

The project therefore keeps the AI provider integration separate from the core bug-triage logic.

𝟓. 𝐆𝐞𝐦𝐢𝐧𝐢 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧

For the AI model, I used Gemini.

The integration is defined as a custom Swytchcode integration using the Gemini generateContent endpoint.

The API key is stored locally using an environment variable:
env GOOGLE_GEMINI_KEY=your_api_key_here

𝟔.𝐁𝐨𝐧𝐮𝐬: 𝐈𝐝𝐞𝐦𝐩𝐨𝐭𝐞𝐧𝐜𝐲

Another bonus feature I implemented is application-level idempotency.

The goal is simple:

If the exact same bug report is submitted again, the application should recognize that it has already been processed and avoid making another unnecessary AI call.

To achieve this, the application generates a deterministic SHA-256 hash from the normalized bug report.

Conceptually:

text
Bug Report
↓
Normalize content
↓
SHA-256
↓
Idempotency Key

Idempotency prevents duplicate processing of the same bug report

𝟕..gitignore 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲

API credentials should never be hardcoded into an application or committed to GitHub.

For this project:

The Gemini API key is stored in an environment variable.
.env is ignored by Git.
The API key is never printed in the normal output.
Dry-run output redacts the API key.
The idempotency key is generated only from the bug report content and does not contain the API key.
The local cache file is also ignored by Git.

Before pushing the project to GitHub, I verified that the real .env file was not included in the Git commit.

𝟖.𝐓𝐞𝐬𝐭𝐢𝐧𝐠

I performed several tests before publishing the project.

Basic validation

bash
python -m py_compile bug_triage_agent.py

The Python compilation test passed successfully.

python bug_triage_agent.py sample_bug.txt --dry-run

𝟗.𝐖𝐡𝐚𝐭 𝐈 𝐋𝐞𝐚𝐫𝐧𝐞𝐝

What I Learned

Building this project taught me several practical lessons.

1. AI integration is more than calling a model

A useful AI application needs input validation, response parsing, error handling, and predictable output.

2. Execution policies matter

Timeouts and controlled retries make external AI calls more reliable without creating unnecessary repeated requests.

3. Idempotency is useful for AI workflows

Repeated requests can waste API resources. Detecting identical inputs can prevent unnecessary processing.

4. Secrets must be handled carefully

API keys should stay outside source code and should never be committed to a public repository.

5. Integration layers can simplify application architecture

Using Swytchcode as the execution layer allowed me to keep the application logic separate from the AI integration.

𝟏𝟎.𝐆𝐢𝐭𝐇𝐮𝐛

Project Repository

The complete source code is available on GitHub:

👉 https://github.com/codewithvishuuu/swytchcode-ai-bug-triage

The repository contains the Python agent, Swytchcode Gemini integration, configuration, documentation, and sample bug report.

𝟏𝟏.𝐃𝐞𝐦𝐨

I also created a short demo showing the complete workflow:

Bug Report → Swytchcode → Gemini → Bug Triage Report

The demo also shows the idempotency feature and the generated result.

𝟏𝟐.𝐂𝐨𝐧𝐜𝐥𝐮𝐬𝐢𝐨𝐧

The AI Bug Triage Agent demonstrates how AI can be used to automate an everyday software-development task.

Instead of manually reading and categorizing every bug report, the agent can quickly generate a structured analysis containing severity, category, root cause, recommended fixes, and debugging steps.

By combining Gemini with Swytchcode and adding execution policies and idempotency, the project goes beyond a basic AI API demo and focuses on reliability, security, and practical developer workflows.

This project was built as part of Task 3 and was a great opportunity to explore AI agents, integration management, and production-oriented patterns.

𝙏𝙝𝙖𝙣𝙠 𝙮𝙤𝙪 𝙛𝙤𝙧 𝙧𝙚𝙖𝙙𝙞𝙣𝙜! 𝙁𝙚𝙚𝙡 𝙛𝙧𝙚𝙚 𝙩𝙤 𝙧𝙚𝙖𝙘𝙝 𝙤𝙪𝙩 𝙤𝙧 𝙘𝙤𝙣𝙣𝙚𝙘𝙩 𝙤𝙣 𝙇𝙞𝙣𝙠𝙚𝙙𝙄𝙣 / 𝙂𝙞𝙩𝙃𝙪𝙗🤝
𝑽𝒊𝒔𝒉𝒂𝒍 𝑲𝒖𝒎𝒂𝒓💖

Building Aarogya Sahayak: My 10-Day Journey Building a Voice AI Health Assistant

Vishal Kumar — Sat, 15 Aug 2026 14:44:25 +0000

𝐅𝐫𝐨𝐦 𝐚 𝐛𝐚𝐬𝐢𝐜 𝐯𝐨𝐢𝐜𝐞 𝐚𝐬𝐬𝐢𝐬𝐭𝐚𝐧𝐭 𝐭𝐨 𝐡𝐮𝐦𝐚𝐧 𝐞𝐬𝐜𝐚𝐥𝐚𝐭𝐢𝐨𝐧, 𝐜𝐚𝐥𝐥 𝐚𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬, 𝐦𝐮𝐥𝐭𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐜𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧𝐬, 𝐚𝐧𝐝 𝐬𝐩𝐞𝐜𝐢𝐚𝐥𝐢𝐬𝐭 𝐡𝐚𝐧𝐝𝐨𝐟𝐟𝐬.

𝟏. 𝐓𝐡𝐞 𝐏𝐫𝐨𝐛𝐥𝐞𝐦 𝐚𝐧𝐝 𝐭𝐡𝐞 𝐔𝐬𝐞𝐫𝐬

Healthcare can be difficult to navigate, especially when users are not sure whether they need general health guidance, a clinic appointment, or support from a real person.

For this project, I chose the Health Access track and built "𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤", a voice-based health and wellness assistant designed to make basic health conversations easier and more accessible.

The idea was not to build an AI doctor or replace medical professionals. Instead, "𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤" is designed to help users with general health and wellness questions, guide them toward appropriate next steps, and connect them with human support when a situation requires it.

I chose voice because users can speak naturally instead of typing everything into a chat box. This makes the interaction feel more like a real conversation and can be especially useful for people who prefer speaking over typing.

The project also focuses on safety. When a conversation involves diagnosis requests, emergency or red-flag situations, or needs human intervention, the agent follows dedicated safety and escalation flows instead of trying to handle everything itself.

The assistant also supports scheduled reminder calls, allowing users to request reminders naturally and receive a follow-up call at the requested time. This extends the experience beyond a single conversation while keeping the existing safety and consent rules in place.

𝟐. 𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐕𝐨𝐢𝐜𝐞 𝐀𝐠𝐞𝐧𝐭 𝐃𝐨𝐞𝐬

𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 is built as a conversational voice assistant rather than a simple question-and-answer bot.

A user can start a natural conversation about general health and wellness. The main agent can provide basic guidance while following clear safety boundaries. It does not try to diagnose medical conditions or act as a replacement for a healthcare professional.

𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗦𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻

One of the important improvements I added was a language-selection flow.

Before starting a conversation, the user can choose between "𝗘𝗻𝗴𝗹𝗶𝘀𝗵" 𝗮𝗻𝗱 "𝗛𝗶𝗻𝗱𝗶". Once selected, the preferred language is carried through the agent session so the conversation stays consistent instead of randomly switching languages.

For Hindi conversations, the agent is instructed to respond using "𝗗𝗲𝘃𝗮𝗻𝗮𝗴𝗮𝗿𝗶 𝘀𝗰𝗿𝗶𝗽𝘁" rather than Romanized Hindi. This makes the conversation more natural for users who prefer reading and speaking Hindi.

𝗛𝘂𝗺𝗮𝗻 𝗦𝘂𝗽𝗽𝗼𝗿𝘁

When a situation needs human assistance, 𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 can create a human support request and provide a unique reference ID.

The user can later use that reference ID on the Human Support page to check the status of the request.

The public support page is intentionally privacy-focused. It displays only safe request metadata such as the reference ID, status, timestamps, urgency, and follow-up information. Conversation content and sensitive health details are not exposed on the public page.

𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗔𝗴𝗲𝗻𝘁 𝗛𝗮𝗻𝗱𝗼𝗳𝗳

Another important feature is the "𝗖𝗹𝗶𝗻𝗶𝗰 & 𝗔𝗽𝗽𝗼𝗶𝗻𝘁𝗺𝗲𝗻𝘁 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁".

When a user clearly wants to book or prepare for a clinic appointment, the main agent can hand the conversation to this dedicated specialist instead of trying to handle every task itself.

The specialist can help with appointment-related questions, preparation, preferred appointment timing, and general visit guidance.

The user does not have to repeat the entire problem because relevant context is transferred during the handoff.

𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗛𝗮𝗻𝗱𝗯𝗮𝗰𝗸

The specialist can also hand the conversation back to the main health assistant when the appointment-related task is complete or when the user wants to discuss a general health topic.

This creates a simple multi-agent flow where each agent has a focused responsibility.

𝗖𝗮𝗹𝗹 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀

I also built a Call Analytics Dashboard to make the voice agent measurable instead of relying only on how the conversation feels.

The dashboard tracks real call outcomes such as:

Total calls
Successful calls
Failed calls
Call outcome trends

The success and failure logic is connected to actual call outcomes rather than hardcoded numbers.

This helped me understand whether the agent was actually completing useful conversations.

𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗖𝗮𝗹𝗹𝘀

A new feature I added is scheduled reminder calling.

Users can ask the agent to schedule a reminder naturally, such as "Remind me in 5 minutes to drink water." The agent understands relative or specific times, validates the requested timezone when needed, and creates a reminder with a unique reference ID.

The reminder is processed by the existing outbound calling system instead of using a separate dialing implementation. An exactly-once mechanism prevents the same reminder from being dialed multiple times.

Users can also open the Scheduled Reminders page to check the reminder status and cancel a reminder while it is still pending.

𝗪𝗲𝗹𝗹𝗻𝗲𝘀𝘀 𝗧𝗶𝗽𝘀

I also added a dedicated Wellness Tips section for users who want simple general wellness guidance outside the live conversation.

It includes five areas:

Hydration
Sleep & Rest
Healthy Eating
Daily Activity
Stress & Relaxation

The section is designed for general wellness information and includes a clear disclaimer that it is not medical advice.

𝟑.𝐇𝐨𝐰 𝐭𝐡𝐞 𝐒𝐲𝐬𝐭𝐞𝐦 𝐖𝐨𝐫𝐤𝐬

𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 is built around a real-time voice conversation pipeline. The goal was to keep the interaction simple for the user while allowing different parts of the system to handle different responsibilities.

At a high level, the flow looks like this:

𝟭.𝗨𝘀𝗲𝗿 𝗮𝗻𝗱 𝗙𝗿𝗼𝗻𝘁𝗲𝗻𝗱 The user starts the conversation from the web interface and can choose their preferred language before the voice session begins.

The frontend handles the conversation interface, voice controls, language selection, agent status, and other user-facing states.

𝟮.𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗩𝗼𝗶𝗰𝗲 𝗧𝗿𝗮𝗻𝘀𝗽𝗼𝗿𝘁 The voice conversation runs through "𝗟𝗶𝘃𝗲𝗞𝗶𝘁", which provides the real-time communication layer between the user and the agent.

This allows the agent to receive the user's audio, process it, and return spoken responses during the same conversation.

𝟯. 𝗦𝗽𝗲𝗲𝗰𝗵-𝘁𝗼-𝗧𝗲𝘅𝘁 When the user speaks, their audio is converted into text so the language model can understand the request.

This text becomes the input for the agent's reasoning and routing logic.

𝟰.𝗠𝗮𝗶𝗻 𝗔𝗴𝗲𝗻𝘁 The main 𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 agent is responsible for general health and wellness conversations.

It follows a system prompt containing its role, objectives, safety rules, language instructions, and routing decisions.

The main agent does not try to handle every request itself. Depending on the conversation, it can use tools or hand the conversation to another part of the system.

𝟱. 𝗧𝗼𝗼𝗹𝘀 𝗮𝗻𝗱 𝗥𝗼𝘂𝘁𝗶𝗻𝗴 The agent can use dedicated tools when a task requires an action rather than a simple response.

For example, a clear clinic or appointment request can trigger a handoff to the "Clinic & Appointment Specialist".

Emergency or red-flag situations follow the existing escalation flow instead of being sent to the appointment specialist.

This separation keeps the responsibilities of each agent clear.

𝟲.𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗙𝗹𝗼𝘄

The agent can also create scheduled reminders from natural-language requests.

When a user asks for a reminder, the request is parsed and stored with a unique reference ID. The scheduler checks for due reminders and triggers the existing outbound calling system at the scheduled time.

The reminder system uses an exactly-once claiming mechanism to prevent duplicate calls. Users can also check the reminder status or cancel a reminder while it is still pending.

This flow reuses the existing outbound dialer instead of introducing a separate calling system.

𝟳.𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗔𝗴𝗲𝗻𝘁 The Clinic & Appointment Specialist is a separate agent with its own instructions and responsibilities.

During a handoff, only relevant context is transferred instead of exposing the entire conversation unnecessarily.

The specialist can continue the conversation without asking the user to repeat the original request.

When its task is complete or the user changes back to a general health topic, the specialist can hand the conversation back to the main agent.

𝟴.𝗧𝗲𝘅𝘁-𝘁𝗼-𝗦𝗽𝗲𝗲𝗰𝗵 After the agent generates a response, it is converted back into speech so the user can hear the answer naturally.

For this project, I used "𝗠𝘂𝗿𝗳 𝗙𝗮𝗹𝗰𝗼𝗻" for the voice experience.

One of the things I wanted to achieve was a conversational experience that feels closer to talking to an assistant rather than interacting with a traditional text chatbot.

Architecture Overview

The overall architecture can be represented as:

𝟒. 𝐓𝐡𝐞 𝐌𝐨𝐬𝐭 𝐈𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐭 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬

Over the course of the challenge, I gradually added features that made "Aarogya Sahayak" more useful, safer, and easier to evaluate.

𝗦𝗮𝗳𝗲𝘁𝘆 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 The agent has clear boundaries around what it can and cannot do.

It is designed for general health and wellness guidance, not medical diagnosis. Emergency or red-flag situations follow a separate escalation flow instead of being routed to the appointment specialist.

The agent also avoids asking users for sensitive information such as passwords, OTPs, PINs, or card details.

𝟭.𝗛𝘂𝗺𝗮𝗻 𝗘𝘀𝗰𝗮𝗹𝗮𝘁𝗶𝗼𝗻
When a conversation needs human intervention, the agent can create a support request and generate a unique reference ID.

The user can use this ID to check the request status later through the Human Support page.

The public status page is intentionally privacy-focused and does not expose the user's conversation or sensitive health information.

𝟮.𝗖𝗹𝗶𝗻𝗶𝗰 𝗮𝗻𝗱 𝗔𝗽𝗽𝗼𝗶𝗻𝘁𝗺𝗲𝗻𝘁 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁
Instead of making the main agent responsible for everything, I created a dedicated "Clinic & Appointment Specialist".

The specialist focuses on appointment-related tasks such as:

Understanding appointment requests
Helping with clinic visit preparation
Discussing preferred appointment timing
Providing general appointment guidance

The main agent announces the handoff before switching the conversation, and the specialist introduces itself after taking over.

𝟯.𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗛𝗮𝗻𝗱𝗯𝗮𝗰𝗸
The specialist can also return the conversation to the main health assistant.

For example, if the user finishes discussing an appointment and asks for general wellness advice, the specialist can hand the conversation back instead of continuing outside its scope.

Relevant context is preserved during both handoff and handback so the user does not have to repeat everything.

𝟰.𝗠𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝘀
The agent supports both English and Hindi through a language-selection flow.

The user chooses the preferred language before starting the conversation. The selected language is then carried through the agent session.

For Hindi conversations, the agent is instructed to use "𝗗𝗲𝘃𝗮𝗻𝗮𝗴𝗮𝗿𝗶 𝘀𝗰𝗿𝗶𝗽𝘁" rather than randomly switching to Romanized Hindi.

𝟱.𝗖𝗮𝗹𝗹 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀
I built a Call Analytics Dashboard that records real call outcomes.

It tracks metrics such as:

Total calls
Successful calls
Failed calls
Call outcome trends

The success condition is based on actual conversation outcomes, such as successful health guidance or successful human escalation, rather than simply counting every completed call as successful.

This made the system measurable and helped me evaluate whether the agent was actually completing useful conversations.

𝟲.𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗖𝗮𝗹𝗹𝘀

One of the major features I added was scheduled reminder calling.

Users can ask "Aarogya Sahayak" to schedule a reminder using natural language, for example:

"Remind me in 5 minutes to drink water."

The agent understands the requested time, handles timezone and AM/PM clarification when required, and creates a reminder with a unique reference ID.

The reminder is then stored and monitored by a background scheduler. When the scheduled time arrives, the system reuses the existing outbound calling infrastructure to place the reminder call.

A key reliability feature is the exactly-once mechanism, which prevents the same reminder from being triggered more than once.

Users can also check the status of a reminder using its reference ID and cancel a reminder while it is still pending.

This turned reminders from a simple notification concept into an actual scheduled voice-call workflow connected to the existing agent infrastructure.

𝟳.𝗪𝗲𝗹𝗹𝗻𝗲𝘀𝘀 𝗧𝗶𝗽𝘀

I also added a dedicated Wellness Tips section for users who want simple health and wellness information without starting a full voice conversation.

The section provides practical, general wellness tips across five areas:

Hydration
Sleep & Rest
Healthy Eating
Daily Activity
Stress & Relaxation

Each category contains short and easy-to-follow suggestions so users can quickly find relevant information.

The feature is intentionally focused on general wellness rather than diagnosis or treatment. A clear disclaimer reminds users that the information is not medical advice.

This gives users another way to interact with Aarogya Sahayak when they simply want quick wellness guidance.

𝟓. 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 𝐚𝐧𝐝 𝐇𝐨𝐰 𝐈 𝐎𝐯𝐞𝐫𝐜𝐚𝐦𝐞 𝐓𝐡𝐞𝐦

Building a voice agent over multiple days was not always straightforward. A few problems taught me more than the features themselves.

𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗔𝗴𝗲𝗻𝘁 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀 𝗦𝗲𝗽𝗮𝗿𝗮𝘁𝗲 Initially, it would have been easy to let one agent handle everything.

Instead, I separated responsibilities between the main health assistant and the Clinic & Appointment Specialist.

The main agent handles general health conversations and safety routing, while the specialist focuses only on clinic and appointment-related tasks.

This made the routing logic clearer and reduced the chance of the specialist answering questions outside its intended role.

𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗗𝘂𝗿𝗶𝗻𝗴 𝗛𝗮𝗻𝗱𝗼𝗳𝗳𝘀 Another challenge was making sure the specialist understood what the user had already asked.

I did not want the user to repeat the entire conversation after every handoff.

The solution was to transfer only the relevant context, including a short request summary and the user's latest request, instead of transferring the entire conversation.

The same principle is used when handing the conversation back to the main agent.

𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 Another issue appeared when the agent sometimes mixed English and Hindi or used Romanized Hindi.

I added an explicit language-selection flow and made the selected language part of the agent session context.

For Hindi, the instructions explicitly require "Devanagari script" and prevent the agent from switching back to Romanized Hindi.

𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗖𝗮𝗹𝗹𝘀 Adding scheduled reminder calls introduced another engineering challenge.

The system needed to understand natural-language time requests such as "in 5 minutes" while handling timezone and AM/PM ambiguity safely.

Another important requirement was preventing the same reminder from triggering more than once. Since the scheduler checks for due reminders repeatedly, simply checking the reminder status was not enough.

I implemented an atomic claiming mechanism so that each reminder is claimed exactly once before the outbound call is triggered. Reminders then move through states such as pending, triggered, completed, failed, or cancelled.

I also reused the existing outbound calling system instead of creating a separate dialing implementation. This kept the reminder feature integrated with the project's existing calling infrastructure.

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗟𝗶𝗺𝗶𝘁𝗲𝗱 𝗟𝗟𝗠 𝗤𝘂𝗼𝘁𝗮 Some behavioral tests depend on LiveKit hosted inference and could not always be executed because the available free inference quota was exhausted.

Instead of treating those failures as application failures, I separated deterministic tests from LLM-dependent tests.

The deterministic test suite, linting, formatting, type checking, and frontend build could still be verified locally.

This was an important lesson: an external service failure and an application bug are not necessarily the same thing.

𝟔. 𝐇𝐨𝐰 𝐑𝐞𝐚𝐝𝐞𝐫𝐬 𝐂𝐚𝐧 𝐁𝐮𝐢𝐥𝐝 𝐚𝐧𝐝 𝐑𝐮𝐧 𝐈𝐭:

If you want to build a similar voice agent, the architecture can be broken into four main parts:

𝟭. 𝗦𝗽𝗲𝗲𝗰𝗵-𝘁𝗼-𝗧𝗲𝘅𝘁 (𝗦𝗧𝗧) 𝘁𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝘂𝘀𝗲𝗿 𝘀𝗮𝘆𝘀

𝟮. 𝗟𝗟𝗠 𝘁𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗿𝗲𝗾𝘂𝗲𝘀𝘁 𝗮𝗻𝗱 𝗱𝗲𝗰𝗶𝗱𝗲 𝘄𝗵𝗮𝘁 𝘁𝗼 𝗱𝗼

𝟯. 𝗧𝗲𝘅𝘁-𝘁𝗼-𝗦𝗽𝗲𝗲𝗰𝗵 (𝗧𝗧𝗦) 𝘁𝗼 𝘀𝗽𝗲𝗮𝗸 𝘁𝗵𝗲 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗲

𝟰. 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝘁𝗿𝗮𝗻𝘀𝗽𝗼𝗿𝘁 𝘁𝗼 𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝘁𝗵𝗲 𝘂𝘀𝗲𝗿 𝗮𝗻𝗱 𝗮𝗴𝗲𝗻𝘁 𝗱𝘂𝗿𝗶𝗻𝗴 𝗮 𝗹𝗶𝘃𝗲 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻

For Aarogya Sahayak, I used LiveKit for the real-time agent session and Murf Falcon for the voice experience.

𝐂𝐥𝐨𝐧𝐞 𝐭𝐡𝐞 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 Start by cloning the repository:

Bash:--
git clone https://github.com/codewithvishuuu/murf-livekit-starter.git
cd murf-livekit-starter

𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗲 𝗔𝗣𝗜 𝗞𝗲𝘆𝘀 API keys should never be hardcoded into the source code or committed to GitHub.

Store them in the project's local environment file, such as .env.local, according to the variables required by the project.

𝗙𝗼𝗿 𝗲𝘅𝗮𝗺𝗽𝗹𝗲: GOOGLE_API_KEY=your_key_here

Use your actual keys only in your local environment.

Make sure .env.local is included in .gitignore and never publish the file containing real credentials.

𝗦𝘁𝗮𝗿𝘁 𝘁𝗵𝗲 𝗟𝗼𝗰𝗮𝗹 𝗟𝗶𝘃𝗲𝗞𝗶𝘁 𝗦𝗲𝗿𝘃𝗲𝗿
If you are running LiveKit locally, start the local LiveKit server first. The backend agent connects to this server for the real-time voice session.
𝗙𝗼𝗿 𝗲𝘅𝗮𝗺𝗽𝗹𝗲:
cd backend
uv run python src/agent.py dev

Then start the frontend...

𝗦𝘁𝗮𝗿𝘁 𝘁𝗵𝗲 𝗕𝗮𝗰𝗸𝗲𝗻𝗱 𝗔𝗴𝗲𝗻𝘁
cd backend
uv run python src/agent.py dev
𝗦𝘁𝗮𝗿𝘁 𝘁𝗵𝗲 𝗙𝗿𝗼𝗻𝘁𝗲𝗻𝗱
Then start the frontend using the project's frontend development command.

Open the local application in your browser and start a conversation.

𝐓𝐞𝐬𝐭𝐢𝐧𝐠 𝐭𝐡𝐞 𝐂𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧:

A simple test flow is:

Start the application.
Select English or Hindi.
Ask a normal health or wellness question.
Ask for a clinic appointment.
Verify that the main agent announces the specialist handoff.
Continue the appointment conversation with the specialist.
Ask for a general wellness topic and verify the specialist can hand the conversation back.
Test the Human Support flow and verify that a reference ID is generated.
Check the Call Analytics Dashboard after completing calls.
Schedule a reminder using a natural-language request, such as "Remind me in 5 minutes to drink water."
Verify that a unique reminder reference ID is generated.
Open the Scheduled Reminders page and check the reminder status.
Create a pending reminder and verify that it can be cancelled.
Leave a reminder active and verify that the scheduled outbound call is triggered at the requested time.
Verify that the same reminder is not triggered more than once.

This type of testing helped me verify the actual user journey across voice conversations, specialist handoffs, human support, scheduled reminders, and call analytics instead of relying only on automated tests.

𝗺𝗮𝗿𝗸𝗱𝗼𝘄𝗻
𝟕. 𝐖𝐡𝐚𝐭 𝐈 𝐖𝐨𝐮𝐥𝐝 𝐈𝐦𝐩𝐫𝐨𝐯𝐞 𝐍𝐞𝐱𝐭:

There are still several things I would like to improve.

𝗕𝗲𝘁𝘁𝗲𝗿 𝗔𝗽𝗽𝗼𝗶𝗻𝘁𝗺𝗲𝗻𝘁 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻
The current specialist focuses on appointment guidance and preparation. A future version could integrate real clinic availability and appointment booking instead of only guiding the user through the process.
𝗠𝗼𝗿𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁
The current language-selection flow focuses on English and Hindi. More Indian languages could be added while keeping the same strict native-script approach.
𝗕𝗲𝘁𝘁𝗲𝗿 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆
The analytics dashboard could be expanded with more operational metrics such as latency, handoff rates, tool usage, and failure reasons.
𝗠𝗼𝗿𝗲 𝗥𝗼𝗯𝘂𝘀𝘁 𝗧𝗲𝘀𝘁𝗶𝗻𝗴
I would also like to expand automated behavioral testing with more real-world conversation scenarios and improve testing around external inference-service failures.
𝗠𝗼𝗿𝗲 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀
The current reminder system supports natural-language scheduling, status tracking, cancellation, and outbound reminder calls. A future version could support recurring reminders, richer scheduling options, user-configurable reminder preferences, and more advanced delivery controls.
𝗠𝗼𝗿𝗲 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗔𝗴𝗲𝗻𝘁𝘀
The same architecture could support additional focused agents, such as a pharmacy information specialist or a health-services navigation specialist, while keeping the main agent responsible for routing.

𝟖. 𝐋𝐢𝐧𝐤𝐬 𝐭𝐨 𝐭𝐡𝐞 𝐂𝐨𝐝𝐞 𝐚𝐧𝐝 𝐃𝐞𝐦𝐨:

The project is open for others to explore and learn from.

𝗚𝗶𝘁𝗛𝘂𝗯 𝗥𝗲𝗽𝗼𝘀𝗶𝘁𝗼𝗿𝘆:
https://github.com/codewithvishuuu/murf-livekit-starter

The repository contains the agent implementation, prompts, specialist handoff logic, scheduled reminder calls, tests, frontend, analytics, wellness tips, and support flows.

I also recommend looking at the project structure and tests if you are building your own voice agent. The tests were especially useful for checking routing, safety rules, handoffs, handbacks, language behavior, and privacy boundaries.

𝐅𝐢𝐧𝐚𝐥 𝐓𝐡𝐨𝐮𝐠𝐡𝐭𝐬:

Building Aarogya Sahayak over these 10 days taught me that a useful voice agent is much more than connecting an LLM to speech.

The difficult part is designing the system around the model: defining clear responsibilities, creating safety boundaries, routing conversations to the right agent, preserving context, protecting user information, and measuring whether the system actually works.

As the project evolved, I also learned how features such as scheduled reminder calls and accessible wellness guidance can extend a voice agent beyond a single conversation while still keeping safety and privacy at the center.

The biggest lesson I learned is that a good voice agent should know not only "how to answer", but also "when not to answer itself".

That is where tools, specialist agents, analytics, scheduled workflows, and human support become important.

I'm grateful to "Murf AI" for organizing the "10 Days of Voice Agents — #VoiceForBharat Edition" and giving me the opportunity to build, test, break, fix, and improve a real voice-agent project.

I'm building voice agents using the fastest TTS API — "Murf Falcon".

Have you experimented with real-time voice pipelines or LiveKitintegrations yet?🛠️ What’s the biggest latency hurdle you’ve run into? Drop your thoughts in the comments below!👇💬

𝙏𝙝𝙖𝙣𝙠 𝙮𝙤𝙪 𝙛𝙤𝙧 𝙧𝙚𝙖𝙙𝙞𝙣𝙜! 𝙁𝙚𝙚𝙡 𝙛𝙧𝙚𝙚 𝙩𝙤 𝙧𝙚𝙖𝙘𝙝 𝙤𝙪𝙩 𝙤𝙧 𝙘𝙤𝙣𝙣𝙚𝙘𝙩 𝙤𝙣 𝙇𝙞𝙣𝙠𝙚𝙙𝙄𝙣 / 𝙂𝙞𝙩𝙃𝙪𝙗🤝
𝑽𝒊𝒔𝒉𝒂𝒍 𝑲𝒖𝒎𝒂𝒓💖

How I Turned an Old Movie Recommendation Project Into a Cinematic AI Platform

Vishal Kumar — Sun, 24 May 2026 16:35:48 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

I built CineMatch, an AI-powered movie recommendation platform designed with a cinematic Netflix-inspired interface.

The project started as a small recommendation engine, but during this challenge I focused on transforming it into a much more polished and immersive experience. I redesigned the UI, improved the recommendation logic, added responsive layouts, integrated TMDB live metadata, and reorganized the project architecture into modular components.

The platform uses TF-IDF vectorization and cosine similarity to recommend movies based on storyline similarity instead of only genres.

Some major features include:

AI-powered semantic movie recommendations
Typo-correcting search system
Live TMDB integration
Dynamic spotlight movie banners
Responsive mobile-first layouts
Personalized watchlist system
Glassmorphism cinematic UI
Mood-based recommendation steering

The app currently works with a local database of more than 45,000 movies.

Demo

Live Demo

https://cinematch-movie-recommender-bqzsppgvepmahksda8qxg9.streamlit.app/

GitHub Repository

https://github.com/codewithvishuuu/cinematch-movie-recommender

Screenshots

Home Landing	AI Matcher	Responsive Layout

The Comeback Story

Originally this project was a very simple recommendation prototype with basic UI and minimal functionality.

During the challenge I completely reworked large parts of the project. I reorganized the structure into modular folders, improved the frontend design, added reusable UI components, optimized mobile responsiveness, and connected live TMDB APIs for trailers and metadata.

One of the biggest improvements was making the interface work properly on smaller screens. Earlier many sections broke on mobile devices, especially buttons, grids, and text layouts. I redesigned the layouts using responsive spacing, adaptive sizing, and flexible containers.

I also improved the recommendation engine by adding better search handling, typo correction, and mood-based filtering.

The final result feels much more like a complete product instead of just a small ML experiment.

My Experience with GitHub Copilot

GitHub Copilot helped me speed up repetitive development tasks and UI restructuring.

I mainly used it while:

reorganizing components
improving responsive CSS
generating repetitive layout sections
refactoring utility logic

It helped reduce development time, especially during UI polishing and architecture cleanup, while I still manually customized the recommendation logic and overall design decisions.

Overall this challenge helped me improve both my frontend design skills and project organization workflow.