DEV Community: Vishal Kumar

Building an AI Bug Triage Agent with Swytchcode and Gemini

Vishal Kumar — Fri, 21 Aug 2026 13:52:42 +0000

𝐓𝐮𝐫𝐧𝐢𝐧𝐠 𝐮𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐛𝐮𝐠 𝐫𝐞𝐩𝐨𝐫𝐭𝐬 𝐢𝐧𝐭𝐨 𝐚𝐜𝐭𝐢𝐨𝐧𝐚𝐛𝐥𝐞 𝐭𝐫𝐢𝐚𝐠𝐞 𝐫𝐞𝐩𝐨𝐫𝐭𝐬 𝐰𝐢𝐭𝐡 𝐀𝐈, 𝐫𝐞𝐭𝐫𝐢𝐞𝐬, 𝐭𝐢𝐦𝐞𝐨𝐮𝐭𝐬, 𝐚𝐧𝐝 𝐢𝐝𝐞𝐦𝐩𝐨𝐭𝐞𝐧𝐜𝐲.

𝟏.𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚𝐧 𝐀𝐈 𝐁𝐮𝐠 𝐓𝐫𝐢𝐚𝐠𝐞 𝐀𝐠𝐞𝐧𝐭 𝐰𝐢𝐭𝐡 𝐒𝐰𝐲𝐭𝐜𝐡𝐜𝐨𝐝𝐞 𝐚𝐧𝐝 𝐆𝐞𝐦𝐢𝐧𝐢

Software bugs are a normal part of development, but analyzing bug reports manually can take a lot of time.

A developer may receive a bug report containing an error message, incomplete context, or a description of unexpected behavior. Before fixing the issue, they usually need to determine:

What is the actual problem?
How severe is it?
Which part of the application is affected?
What could be causing it?
What should be checked first?
How can the issue be fixed?

To solve this problem, I built an AI Bug Triage Agent using Swytchcode and Gemini.

The agent takes a bug report as input and automatically generates a structured triage report containing the summary, severity, category, likely root cause, recommended fix, and debugging steps.

I also implemented two bonus features:

Production-oriented execution policies
Application-level idempotency

Project structure of the AI Bug Triage Agent

𝟐.𝐖𝐡𝐚𝐭 𝐃𝐨𝐞𝐬 𝐭𝐡𝐞 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 𝐃𝐨?

The workflow is simple:

Bug Report → Python Agent → Swytchcode → Gemini → Structured Bug Triage Report

The user provides a bug report through a text file or standard input.

The Python agent sends the report to Gemini through the Swytchcode runtime. Gemini analyzes the issue and returns structured information.

The final output contains:

Summary
Severity
Category
Likely root cause
Recommended fix
Debugging steps

This makes the output much easier for a developer to understand and act on.

Example bug report used as input

𝟑.𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞

The important part of this architecture is that the Python application does not directly call the Gemini API using an HTTP library or SDK.

The AI execution goes through:
swytchcode_runtime.exec("gemini.models.generateContent", ...)

Gemini is accessed through the Swytchcode runtime

𝟒. 𝐖𝐡𝐲 𝐒𝐰𝐲𝐭𝐜𝐡𝐜𝐨𝐝𝐞?

One of the main goals of this project was to use Swytchcode as the execution layer for the AI integration.

Instead of adding a direct HTTP client or an SDK to the Python application, the project uses the Swytchcode runtime.

This provides a clean separation:

Python handles the application logic.
Swytchcode handles the integration execution.
Gemini handles the AI analysis.

The project therefore keeps the AI provider integration separate from the core bug-triage logic.

𝟓. 𝐆𝐞𝐦𝐢𝐧𝐢 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧

For the AI model, I used Gemini.

The integration is defined as a custom Swytchcode integration using the Gemini generateContent endpoint.

The API key is stored locally using an environment variable:
env GOOGLE_GEMINI_KEY=your_api_key_here

𝟔.𝐁𝐨𝐧𝐮𝐬: 𝐈𝐝𝐞𝐦𝐩𝐨𝐭𝐞𝐧𝐜𝐲

Another bonus feature I implemented is application-level idempotency.

The goal is simple:

If the exact same bug report is submitted again, the application should recognize that it has already been processed and avoid making another unnecessary AI call.

To achieve this, the application generates a deterministic SHA-256 hash from the normalized bug report.

Conceptually:

text
Bug Report
↓
Normalize content
↓
SHA-256
↓
Idempotency Key

Idempotency prevents duplicate processing of the same bug report

𝟕..gitignore 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲

API credentials should never be hardcoded into an application or committed to GitHub.

For this project:

The Gemini API key is stored in an environment variable.
.env is ignored by Git.
The API key is never printed in the normal output.
Dry-run output redacts the API key.
The idempotency key is generated only from the bug report content and does not contain the API key.
The local cache file is also ignored by Git.

Before pushing the project to GitHub, I verified that the real .env file was not included in the Git commit.

𝟖.𝐓𝐞𝐬𝐭𝐢𝐧𝐠

I performed several tests before publishing the project.

Basic validation

bash
python -m py_compile bug_triage_agent.py

The Python compilation test passed successfully.

python bug_triage_agent.py sample_bug.txt --dry-run

𝟗.𝐖𝐡𝐚𝐭 𝐈 𝐋𝐞𝐚𝐫𝐧𝐞𝐝

What I Learned

Building this project taught me several practical lessons.

1. AI integration is more than calling a model

A useful AI application needs input validation, response parsing, error handling, and predictable output.

2. Execution policies matter

Timeouts and controlled retries make external AI calls more reliable without creating unnecessary repeated requests.

3. Idempotency is useful for AI workflows

Repeated requests can waste API resources. Detecting identical inputs can prevent unnecessary processing.

4. Secrets must be handled carefully

API keys should stay outside source code and should never be committed to a public repository.

5. Integration layers can simplify application architecture

Using Swytchcode as the execution layer allowed me to keep the application logic separate from the AI integration.

𝟏𝟎.𝐆𝐢𝐭𝐇𝐮𝐛

Project Repository

The complete source code is available on GitHub:

👉 https://github.com/codewithvishuuu/swytchcode-ai-bug-triage

The repository contains the Python agent, Swytchcode Gemini integration, configuration, documentation, and sample bug report.

𝟏𝟏.𝐃𝐞𝐦𝐨

I also created a short demo showing the complete workflow:

Bug Report → Swytchcode → Gemini → Bug Triage Report

The demo also shows the idempotency feature and the generated result.

𝟏𝟐.𝐂𝐨𝐧𝐜𝐥𝐮𝐬𝐢𝐨𝐧

The AI Bug Triage Agent demonstrates how AI can be used to automate an everyday software-development task.

Instead of manually reading and categorizing every bug report, the agent can quickly generate a structured analysis containing severity, category, root cause, recommended fixes, and debugging steps.

By combining Gemini with Swytchcode and adding execution policies and idempotency, the project goes beyond a basic AI API demo and focuses on reliability, security, and practical developer workflows.

This project was built as part of Task 3 and was a great opportunity to explore AI agents, integration management, and production-oriented patterns.

𝙏𝙝𝙖𝙣𝙠 𝙮𝙤𝙪 𝙛𝙤𝙧 𝙧𝙚𝙖𝙙𝙞𝙣𝙜! 𝙁𝙚𝙚𝙡 𝙛𝙧𝙚𝙚 𝙩𝙤 𝙧𝙚𝙖𝙘𝙝 𝙤𝙪𝙩 𝙤𝙧 𝙘𝙤𝙣𝙣𝙚𝙘𝙩 𝙤𝙣 𝙇𝙞𝙣𝙠𝙚𝙙𝙄𝙣 / 𝙂𝙞𝙩𝙃𝙪𝙗🤝
𝑽𝒊𝒔𝒉𝒂𝒍 𝑲𝒖𝒎𝒂𝒓💖

Building Aarogya Sahayak: My 10-Day Journey Building a Voice AI Health Assistant

Vishal Kumar — Sat, 15 Aug 2026 14:44:25 +0000

𝐅𝐫𝐨𝐦 𝐚 𝐛𝐚𝐬𝐢𝐜 𝐯𝐨𝐢𝐜𝐞 𝐚𝐬𝐬𝐢𝐬𝐭𝐚𝐧𝐭 𝐭𝐨 𝐡𝐮𝐦𝐚𝐧 𝐞𝐬𝐜𝐚𝐥𝐚𝐭𝐢𝐨𝐧, 𝐜𝐚𝐥𝐥 𝐚𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬, 𝐦𝐮𝐥𝐭𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐜𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧𝐬, 𝐚𝐧𝐝 𝐬𝐩𝐞𝐜𝐢𝐚𝐥𝐢𝐬𝐭 𝐡𝐚𝐧𝐝𝐨𝐟𝐟𝐬.

𝟏. 𝐓𝐡𝐞 𝐏𝐫𝐨𝐛𝐥𝐞𝐦 𝐚𝐧𝐝 𝐭𝐡𝐞 𝐔𝐬𝐞𝐫𝐬

Healthcare can be difficult to navigate, especially when users are not sure whether they need general health guidance, a clinic appointment, or support from a real person.

For this project, I chose the Health Access track and built "𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤", a voice-based health and wellness assistant designed to make basic health conversations easier and more accessible.

The idea was not to build an AI doctor or replace medical professionals. Instead, "𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤" is designed to help users with general health and wellness questions, guide them toward appropriate next steps, and connect them with human support when a situation requires it.

I chose voice because users can speak naturally instead of typing everything into a chat box. This makes the interaction feel more like a real conversation and can be especially useful for people who prefer speaking over typing.

The project also focuses on safety. When a conversation involves diagnosis requests, emergency or red-flag situations, or needs human intervention, the agent follows dedicated safety and escalation flows instead of trying to handle everything itself.

The assistant also supports scheduled reminder calls, allowing users to request reminders naturally and receive a follow-up call at the requested time. This extends the experience beyond a single conversation while keeping the existing safety and consent rules in place.

𝟐. 𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐕𝐨𝐢𝐜𝐞 𝐀𝐠𝐞𝐧𝐭 𝐃𝐨𝐞𝐬

𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 is built as a conversational voice assistant rather than a simple question-and-answer bot.

A user can start a natural conversation about general health and wellness. The main agent can provide basic guidance while following clear safety boundaries. It does not try to diagnose medical conditions or act as a replacement for a healthcare professional.

𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗦𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻

One of the important improvements I added was a language-selection flow.

Before starting a conversation, the user can choose between "𝗘𝗻𝗴𝗹𝗶𝘀𝗵" 𝗮𝗻𝗱 "𝗛𝗶𝗻𝗱𝗶". Once selected, the preferred language is carried through the agent session so the conversation stays consistent instead of randomly switching languages.

For Hindi conversations, the agent is instructed to respond using "𝗗𝗲𝘃𝗮𝗻𝗮𝗴𝗮𝗿𝗶 𝘀𝗰𝗿𝗶𝗽𝘁" rather than Romanized Hindi. This makes the conversation more natural for users who prefer reading and speaking Hindi.

𝗛𝘂𝗺𝗮𝗻 𝗦𝘂𝗽𝗽𝗼𝗿𝘁

When a situation needs human assistance, 𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 can create a human support request and provide a unique reference ID.

The user can later use that reference ID on the Human Support page to check the status of the request.

The public support page is intentionally privacy-focused. It displays only safe request metadata such as the reference ID, status, timestamps, urgency, and follow-up information. Conversation content and sensitive health details are not exposed on the public page.

𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗔𝗴𝗲𝗻𝘁 𝗛𝗮𝗻𝗱𝗼𝗳𝗳

Another important feature is the "𝗖𝗹𝗶𝗻𝗶𝗰 & 𝗔𝗽𝗽𝗼𝗶𝗻𝘁𝗺𝗲𝗻𝘁 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁".

When a user clearly wants to book or prepare for a clinic appointment, the main agent can hand the conversation to this dedicated specialist instead of trying to handle every task itself.

The specialist can help with appointment-related questions, preparation, preferred appointment timing, and general visit guidance.

The user does not have to repeat the entire problem because relevant context is transferred during the handoff.

𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗛𝗮𝗻𝗱𝗯𝗮𝗰𝗸

The specialist can also hand the conversation back to the main health assistant when the appointment-related task is complete or when the user wants to discuss a general health topic.

This creates a simple multi-agent flow where each agent has a focused responsibility.

𝗖𝗮𝗹𝗹 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀

I also built a Call Analytics Dashboard to make the voice agent measurable instead of relying only on how the conversation feels.

The dashboard tracks real call outcomes such as:

Total calls
Successful calls
Failed calls
Call outcome trends

The success and failure logic is connected to actual call outcomes rather than hardcoded numbers.

This helped me understand whether the agent was actually completing useful conversations.

𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗖𝗮𝗹𝗹𝘀

A new feature I added is scheduled reminder calling.

Users can ask the agent to schedule a reminder naturally, such as "Remind me in 5 minutes to drink water." The agent understands relative or specific times, validates the requested timezone when needed, and creates a reminder with a unique reference ID.

The reminder is processed by the existing outbound calling system instead of using a separate dialing implementation. An exactly-once mechanism prevents the same reminder from being dialed multiple times.

Users can also open the Scheduled Reminders page to check the reminder status and cancel a reminder while it is still pending.

𝗪𝗲𝗹𝗹𝗻𝗲𝘀𝘀 𝗧𝗶𝗽𝘀

I also added a dedicated Wellness Tips section for users who want simple general wellness guidance outside the live conversation.

It includes five areas:

Hydration
Sleep & Rest
Healthy Eating
Daily Activity
Stress & Relaxation

The section is designed for general wellness information and includes a clear disclaimer that it is not medical advice.

𝟑.𝐇𝐨𝐰 𝐭𝐡𝐞 𝐒𝐲𝐬𝐭𝐞𝐦 𝐖𝐨𝐫𝐤𝐬

𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 is built around a real-time voice conversation pipeline. The goal was to keep the interaction simple for the user while allowing different parts of the system to handle different responsibilities.

At a high level, the flow looks like this:

𝟭.𝗨𝘀𝗲𝗿 𝗮𝗻𝗱 𝗙𝗿𝗼𝗻𝘁𝗲𝗻𝗱 The user starts the conversation from the web interface and can choose their preferred language before the voice session begins.

The frontend handles the conversation interface, voice controls, language selection, agent status, and other user-facing states.

𝟮.𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗩𝗼𝗶𝗰𝗲 𝗧𝗿𝗮𝗻𝘀𝗽𝗼𝗿𝘁 The voice conversation runs through "𝗟𝗶𝘃𝗲𝗞𝗶𝘁", which provides the real-time communication layer between the user and the agent.

This allows the agent to receive the user's audio, process it, and return spoken responses during the same conversation.

𝟯. 𝗦𝗽𝗲𝗲𝗰𝗵-𝘁𝗼-𝗧𝗲𝘅𝘁 When the user speaks, their audio is converted into text so the language model can understand the request.

This text becomes the input for the agent's reasoning and routing logic.

𝟰.𝗠𝗮𝗶𝗻 𝗔𝗴𝗲𝗻𝘁 The main 𝐀𝐚𝐫𝐨𝐠𝐲𝐚 𝐒𝐚𝐡𝐚𝐲𝐚𝐤 agent is responsible for general health and wellness conversations.

It follows a system prompt containing its role, objectives, safety rules, language instructions, and routing decisions.

The main agent does not try to handle every request itself. Depending on the conversation, it can use tools or hand the conversation to another part of the system.

𝟱. 𝗧𝗼𝗼𝗹𝘀 𝗮𝗻𝗱 𝗥𝗼𝘂𝘁𝗶𝗻𝗴 The agent can use dedicated tools when a task requires an action rather than a simple response.

For example, a clear clinic or appointment request can trigger a handoff to the "Clinic & Appointment Specialist".

Emergency or red-flag situations follow the existing escalation flow instead of being sent to the appointment specialist.

This separation keeps the responsibilities of each agent clear.

𝟲.𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗙𝗹𝗼𝘄

The agent can also create scheduled reminders from natural-language requests.

When a user asks for a reminder, the request is parsed and stored with a unique reference ID. The scheduler checks for due reminders and triggers the existing outbound calling system at the scheduled time.

The reminder system uses an exactly-once claiming mechanism to prevent duplicate calls. Users can also check the reminder status or cancel a reminder while it is still pending.

This flow reuses the existing outbound dialer instead of introducing a separate calling system.

𝟳.𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗔𝗴𝗲𝗻𝘁 The Clinic & Appointment Specialist is a separate agent with its own instructions and responsibilities.

During a handoff, only relevant context is transferred instead of exposing the entire conversation unnecessarily.

The specialist can continue the conversation without asking the user to repeat the original request.

When its task is complete or the user changes back to a general health topic, the specialist can hand the conversation back to the main agent.

𝟴.𝗧𝗲𝘅𝘁-𝘁𝗼-𝗦𝗽𝗲𝗲𝗰𝗵 After the agent generates a response, it is converted back into speech so the user can hear the answer naturally.

For this project, I used "𝗠𝘂𝗿𝗳 𝗙𝗮𝗹𝗰𝗼𝗻" for the voice experience.

One of the things I wanted to achieve was a conversational experience that feels closer to talking to an assistant rather than interacting with a traditional text chatbot.

Architecture Overview

The overall architecture can be represented as:

𝟒. 𝐓𝐡𝐞 𝐌𝐨𝐬𝐭 𝐈𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐭 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬

Over the course of the challenge, I gradually added features that made "Aarogya Sahayak" more useful, safer, and easier to evaluate.

𝗦𝗮𝗳𝗲𝘁𝘆 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 The agent has clear boundaries around what it can and cannot do.

It is designed for general health and wellness guidance, not medical diagnosis. Emergency or red-flag situations follow a separate escalation flow instead of being routed to the appointment specialist.

The agent also avoids asking users for sensitive information such as passwords, OTPs, PINs, or card details.

𝟭.𝗛𝘂𝗺𝗮𝗻 𝗘𝘀𝗰𝗮𝗹𝗮𝘁𝗶𝗼𝗻
When a conversation needs human intervention, the agent can create a support request and generate a unique reference ID.

The user can use this ID to check the request status later through the Human Support page.

The public status page is intentionally privacy-focused and does not expose the user's conversation or sensitive health information.

𝟮.𝗖𝗹𝗶𝗻𝗶𝗰 𝗮𝗻𝗱 𝗔𝗽𝗽𝗼𝗶𝗻𝘁𝗺𝗲𝗻𝘁 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁
Instead of making the main agent responsible for everything, I created a dedicated "Clinic & Appointment Specialist".

The specialist focuses on appointment-related tasks such as:

Understanding appointment requests
Helping with clinic visit preparation
Discussing preferred appointment timing
Providing general appointment guidance

The main agent announces the handoff before switching the conversation, and the specialist introduces itself after taking over.

𝟯.𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗛𝗮𝗻𝗱𝗯𝗮𝗰𝗸
The specialist can also return the conversation to the main health assistant.

For example, if the user finishes discussing an appointment and asks for general wellness advice, the specialist can hand the conversation back instead of continuing outside its scope.

Relevant context is preserved during both handoff and handback so the user does not have to repeat everything.

𝟰.𝗠𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝘀
The agent supports both English and Hindi through a language-selection flow.

The user chooses the preferred language before starting the conversation. The selected language is then carried through the agent session.

For Hindi conversations, the agent is instructed to use "𝗗𝗲𝘃𝗮𝗻𝗮𝗴𝗮𝗿𝗶 𝘀𝗰𝗿𝗶𝗽𝘁" rather than randomly switching to Romanized Hindi.

𝟱.𝗖𝗮𝗹𝗹 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀
I built a Call Analytics Dashboard that records real call outcomes.

It tracks metrics such as:

Total calls
Successful calls
Failed calls
Call outcome trends

The success condition is based on actual conversation outcomes, such as successful health guidance or successful human escalation, rather than simply counting every completed call as successful.

This made the system measurable and helped me evaluate whether the agent was actually completing useful conversations.

𝟲.𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗖𝗮𝗹𝗹𝘀

One of the major features I added was scheduled reminder calling.

Users can ask "Aarogya Sahayak" to schedule a reminder using natural language, for example:

"Remind me in 5 minutes to drink water."

The agent understands the requested time, handles timezone and AM/PM clarification when required, and creates a reminder with a unique reference ID.

The reminder is then stored and monitored by a background scheduler. When the scheduled time arrives, the system reuses the existing outbound calling infrastructure to place the reminder call.

A key reliability feature is the exactly-once mechanism, which prevents the same reminder from being triggered more than once.

Users can also check the status of a reminder using its reference ID and cancel a reminder while it is still pending.

This turned reminders from a simple notification concept into an actual scheduled voice-call workflow connected to the existing agent infrastructure.

𝟳.𝗪𝗲𝗹𝗹𝗻𝗲𝘀𝘀 𝗧𝗶𝗽𝘀

I also added a dedicated Wellness Tips section for users who want simple health and wellness information without starting a full voice conversation.

The section provides practical, general wellness tips across five areas:

Hydration
Sleep & Rest
Healthy Eating
Daily Activity
Stress & Relaxation

Each category contains short and easy-to-follow suggestions so users can quickly find relevant information.

The feature is intentionally focused on general wellness rather than diagnosis or treatment. A clear disclaimer reminds users that the information is not medical advice.

This gives users another way to interact with Aarogya Sahayak when they simply want quick wellness guidance.

𝟓. 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 𝐚𝐧𝐝 𝐇𝐨𝐰 𝐈 𝐎𝐯𝐞𝐫𝐜𝐚𝐦𝐞 𝐓𝐡𝐞𝐦

Building a voice agent over multiple days was not always straightforward. A few problems taught me more than the features themselves.

𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗔𝗴𝗲𝗻𝘁 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀 𝗦𝗲𝗽𝗮𝗿𝗮𝘁𝗲 Initially, it would have been easy to let one agent handle everything.

Instead, I separated responsibilities between the main health assistant and the Clinic & Appointment Specialist.

The main agent handles general health conversations and safety routing, while the specialist focuses only on clinic and appointment-related tasks.

This made the routing logic clearer and reduced the chance of the specialist answering questions outside its intended role.

𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗗𝘂𝗿𝗶𝗻𝗴 𝗛𝗮𝗻𝗱𝗼𝗳𝗳𝘀 Another challenge was making sure the specialist understood what the user had already asked.

I did not want the user to repeat the entire conversation after every handoff.

The solution was to transfer only the relevant context, including a short request summary and the user's latest request, instead of transferring the entire conversation.

The same principle is used when handing the conversation back to the main agent.

𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 Another issue appeared when the agent sometimes mixed English and Hindi or used Romanized Hindi.

I added an explicit language-selection flow and made the selected language part of the agent session context.

For Hindi, the instructions explicitly require "Devanagari script" and prevent the agent from switching back to Romanized Hindi.

𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗖𝗮𝗹𝗹𝘀 Adding scheduled reminder calls introduced another engineering challenge.

The system needed to understand natural-language time requests such as "in 5 minutes" while handling timezone and AM/PM ambiguity safely.

Another important requirement was preventing the same reminder from triggering more than once. Since the scheduler checks for due reminders repeatedly, simply checking the reminder status was not enough.

I implemented an atomic claiming mechanism so that each reminder is claimed exactly once before the outbound call is triggered. Reminders then move through states such as pending, triggered, completed, failed, or cancelled.

I also reused the existing outbound calling system instead of creating a separate dialing implementation. This kept the reminder feature integrated with the project's existing calling infrastructure.

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗟𝗶𝗺𝗶𝘁𝗲𝗱 𝗟𝗟𝗠 𝗤𝘂𝗼𝘁𝗮 Some behavioral tests depend on LiveKit hosted inference and could not always be executed because the available free inference quota was exhausted.

Instead of treating those failures as application failures, I separated deterministic tests from LLM-dependent tests.

The deterministic test suite, linting, formatting, type checking, and frontend build could still be verified locally.

This was an important lesson: an external service failure and an application bug are not necessarily the same thing.

𝟔. 𝐇𝐨𝐰 𝐑𝐞𝐚𝐝𝐞𝐫𝐬 𝐂𝐚𝐧 𝐁𝐮𝐢𝐥𝐝 𝐚𝐧𝐝 𝐑𝐮𝐧 𝐈𝐭:

If you want to build a similar voice agent, the architecture can be broken into four main parts:

𝟭. 𝗦𝗽𝗲𝗲𝗰𝗵-𝘁𝗼-𝗧𝗲𝘅𝘁 (𝗦𝗧𝗧) 𝘁𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝘂𝘀𝗲𝗿 𝘀𝗮𝘆𝘀

𝟮. 𝗟𝗟𝗠 𝘁𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗿𝗲𝗾𝘂𝗲𝘀𝘁 𝗮𝗻𝗱 𝗱𝗲𝗰𝗶𝗱𝗲 𝘄𝗵𝗮𝘁 𝘁𝗼 𝗱𝗼

𝟯. 𝗧𝗲𝘅𝘁-𝘁𝗼-𝗦𝗽𝗲𝗲𝗰𝗵 (𝗧𝗧𝗦) 𝘁𝗼 𝘀𝗽𝗲𝗮𝗸 𝘁𝗵𝗲 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗲

𝟰. 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝘁𝗿𝗮𝗻𝘀𝗽𝗼𝗿𝘁 𝘁𝗼 𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝘁𝗵𝗲 𝘂𝘀𝗲𝗿 𝗮𝗻𝗱 𝗮𝗴𝗲𝗻𝘁 𝗱𝘂𝗿𝗶𝗻𝗴 𝗮 𝗹𝗶𝘃𝗲 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻

For Aarogya Sahayak, I used LiveKit for the real-time agent session and Murf Falcon for the voice experience.

𝐂𝐥𝐨𝐧𝐞 𝐭𝐡𝐞 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 Start by cloning the repository:

Bash:--
git clone https://github.com/codewithvishuuu/murf-livekit-starter.git
cd murf-livekit-starter

𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗲 𝗔𝗣𝗜 𝗞𝗲𝘆𝘀 API keys should never be hardcoded into the source code or committed to GitHub.

Store them in the project's local environment file, such as .env.local, according to the variables required by the project.

𝗙𝗼𝗿 𝗲𝘅𝗮𝗺𝗽𝗹𝗲: GOOGLE_API_KEY=your_key_here

Use your actual keys only in your local environment.

Make sure .env.local is included in .gitignore and never publish the file containing real credentials.

𝗦𝘁𝗮𝗿𝘁 𝘁𝗵𝗲 𝗟𝗼𝗰𝗮𝗹 𝗟𝗶𝘃𝗲𝗞𝗶𝘁 𝗦𝗲𝗿𝘃𝗲𝗿
If you are running LiveKit locally, start the local LiveKit server first. The backend agent connects to this server for the real-time voice session.
𝗙𝗼𝗿 𝗲𝘅𝗮𝗺𝗽𝗹𝗲:
cd backend
uv run python src/agent.py dev

Then start the frontend...

𝗦𝘁𝗮𝗿𝘁 𝘁𝗵𝗲 𝗕𝗮𝗰𝗸𝗲𝗻𝗱 𝗔𝗴𝗲𝗻𝘁
cd backend
uv run python src/agent.py dev
𝗦𝘁𝗮𝗿𝘁 𝘁𝗵𝗲 𝗙𝗿𝗼𝗻𝘁𝗲𝗻𝗱
Then start the frontend using the project's frontend development command.

Open the local application in your browser and start a conversation.

𝐓𝐞𝐬𝐭𝐢𝐧𝐠 𝐭𝐡𝐞 𝐂𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧:

A simple test flow is:

Start the application.
Select English or Hindi.
Ask a normal health or wellness question.
Ask for a clinic appointment.
Verify that the main agent announces the specialist handoff.
Continue the appointment conversation with the specialist.
Ask for a general wellness topic and verify the specialist can hand the conversation back.
Test the Human Support flow and verify that a reference ID is generated.
Check the Call Analytics Dashboard after completing calls.
Schedule a reminder using a natural-language request, such as "Remind me in 5 minutes to drink water."
Verify that a unique reminder reference ID is generated.
Open the Scheduled Reminders page and check the reminder status.
Create a pending reminder and verify that it can be cancelled.
Leave a reminder active and verify that the scheduled outbound call is triggered at the requested time.
Verify that the same reminder is not triggered more than once.

This type of testing helped me verify the actual user journey across voice conversations, specialist handoffs, human support, scheduled reminders, and call analytics instead of relying only on automated tests.

𝗺𝗮𝗿𝗸𝗱𝗼𝘄𝗻
𝟕. 𝐖𝐡𝐚𝐭 𝐈 𝐖𝐨𝐮𝐥𝐝 𝐈𝐦𝐩𝐫𝐨𝐯𝐞 𝐍𝐞𝐱𝐭:

There are still several things I would like to improve.

𝗕𝗲𝘁𝘁𝗲𝗿 𝗔𝗽𝗽𝗼𝗶𝗻𝘁𝗺𝗲𝗻𝘁 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻
The current specialist focuses on appointment guidance and preparation. A future version could integrate real clinic availability and appointment booking instead of only guiding the user through the process.
𝗠𝗼𝗿𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁
The current language-selection flow focuses on English and Hindi. More Indian languages could be added while keeping the same strict native-script approach.
𝗕𝗲𝘁𝘁𝗲𝗿 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆
The analytics dashboard could be expanded with more operational metrics such as latency, handoff rates, tool usage, and failure reasons.
𝗠𝗼𝗿𝗲 𝗥𝗼𝗯𝘂𝘀𝘁 𝗧𝗲𝘀𝘁𝗶𝗻𝗴
I would also like to expand automated behavioral testing with more real-world conversation scenarios and improve testing around external inference-service failures.
𝗠𝗼𝗿𝗲 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀
The current reminder system supports natural-language scheduling, status tracking, cancellation, and outbound reminder calls. A future version could support recurring reminders, richer scheduling options, user-configurable reminder preferences, and more advanced delivery controls.
𝗠𝗼𝗿𝗲 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘀𝘁 𝗔𝗴𝗲𝗻𝘁𝘀
The same architecture could support additional focused agents, such as a pharmacy information specialist or a health-services navigation specialist, while keeping the main agent responsible for routing.

𝟖. 𝐋𝐢𝐧𝐤𝐬 𝐭𝐨 𝐭𝐡𝐞 𝐂𝐨𝐝𝐞 𝐚𝐧𝐝 𝐃𝐞𝐦𝐨:

The project is open for others to explore and learn from.

𝗚𝗶𝘁𝗛𝘂𝗯 𝗥𝗲𝗽𝗼𝘀𝗶𝘁𝗼𝗿𝘆:
https://github.com/codewithvishuuu/murf-livekit-starter

The repository contains the agent implementation, prompts, specialist handoff logic, scheduled reminder calls, tests, frontend, analytics, wellness tips, and support flows.

I also recommend looking at the project structure and tests if you are building your own voice agent. The tests were especially useful for checking routing, safety rules, handoffs, handbacks, language behavior, and privacy boundaries.

𝐅𝐢𝐧𝐚𝐥 𝐓𝐡𝐨𝐮𝐠𝐡𝐭𝐬:

Building Aarogya Sahayak over these 10 days taught me that a useful voice agent is much more than connecting an LLM to speech.

The difficult part is designing the system around the model: defining clear responsibilities, creating safety boundaries, routing conversations to the right agent, preserving context, protecting user information, and measuring whether the system actually works.

As the project evolved, I also learned how features such as scheduled reminder calls and accessible wellness guidance can extend a voice agent beyond a single conversation while still keeping safety and privacy at the center.

The biggest lesson I learned is that a good voice agent should know not only "how to answer", but also "when not to answer itself".

That is where tools, specialist agents, analytics, scheduled workflows, and human support become important.

I'm grateful to "Murf AI" for organizing the "10 Days of Voice Agents — #VoiceForBharat Edition" and giving me the opportunity to build, test, break, fix, and improve a real voice-agent project.

I'm building voice agents using the fastest TTS API — "Murf Falcon".

Have you experimented with real-time voice pipelines or LiveKitintegrations yet?🛠️ What’s the biggest latency hurdle you’ve run into? Drop your thoughts in the comments below!👇💬

𝙏𝙝𝙖𝙣𝙠 𝙮𝙤𝙪 𝙛𝙤𝙧 𝙧𝙚𝙖𝙙𝙞𝙣𝙜! 𝙁𝙚𝙚𝙡 𝙛𝙧𝙚𝙚 𝙩𝙤 𝙧𝙚𝙖𝙘𝙝 𝙤𝙪𝙩 𝙤𝙧 𝙘𝙤𝙣𝙣𝙚𝙘𝙩 𝙤𝙣 𝙇𝙞𝙣𝙠𝙚𝙙𝙄𝙣 / 𝙂𝙞𝙩𝙃𝙪𝙗🤝
𝑽𝒊𝒔𝒉𝒂𝒍 𝑲𝒖𝒎𝒂𝒓💖

How I Turned an Old Movie Recommendation Project Into a Cinematic AI Platform

Vishal Kumar — Sun, 24 May 2026 16:35:48 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

I built CineMatch, an AI-powered movie recommendation platform designed with a cinematic Netflix-inspired interface.

The project started as a small recommendation engine, but during this challenge I focused on transforming it into a much more polished and immersive experience. I redesigned the UI, improved the recommendation logic, added responsive layouts, integrated TMDB live metadata, and reorganized the project architecture into modular components.

The platform uses TF-IDF vectorization and cosine similarity to recommend movies based on storyline similarity instead of only genres.

Some major features include:

AI-powered semantic movie recommendations
Typo-correcting search system
Live TMDB integration
Dynamic spotlight movie banners
Responsive mobile-first layouts
Personalized watchlist system
Glassmorphism cinematic UI
Mood-based recommendation steering

The app currently works with a local database of more than 45,000 movies.

Demo

Live Demo

https://cinematch-movie-recommender-bqzsppgvepmahksda8qxg9.streamlit.app/

GitHub Repository

https://github.com/codewithvishuuu/cinematch-movie-recommender

Screenshots

Home Landing	AI Matcher	Responsive Layout

The Comeback Story

Originally this project was a very simple recommendation prototype with basic UI and minimal functionality.

During the challenge I completely reworked large parts of the project. I reorganized the structure into modular folders, improved the frontend design, added reusable UI components, optimized mobile responsiveness, and connected live TMDB APIs for trailers and metadata.

One of the biggest improvements was making the interface work properly on smaller screens. Earlier many sections broke on mobile devices, especially buttons, grids, and text layouts. I redesigned the layouts using responsive spacing, adaptive sizing, and flexible containers.

I also improved the recommendation engine by adding better search handling, typo correction, and mood-based filtering.

The final result feels much more like a complete product instead of just a small ML experiment.

My Experience with GitHub Copilot

GitHub Copilot helped me speed up repetitive development tasks and UI restructuring.

I mainly used it while:

reorganizing components
improving responsive CSS
generating repetitive layout sections
refactoring utility logic

It helped reduce development time, especially during UI polishing and architecture cleanup, while I still manually customized the recommendation logic and overall design decisions.

Overall this challenge helped me improve both my frontend design skills and project organization workflow.