DEV Community: Harshit Agarwal

The Comeback of Agent Baymax: From Hackathon Prototype to AI Healthcare Platform

Harshit Agarwal — Sat, 06 Jun 2026 09:17:20 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

🩺 Agent Baymax V2 — From a Local AI Assistant to a Full Healthcare Platform

What I Built

Some projects don't need to be replaced—they need to be completed.

Agent Baymax started as a simple terminal-based healthcare companion built during a hackathon. The original version could answer basic health-related questions, log calories, track hydration, and maintain short-term conversational context using a locally running language model.

While the idea worked, it never fully matched the vision I had in mind.

Users had to manually enter meals, there was no personalization across sessions, no visual interface, no image understanding, and no way to retrieve information from trusted healthcare resources.

For the Finish-Up-A-Thon Challenge, I decided to revisit the project and transform it into something much more capable.

The result is Agent Baymax V2 — an AI-powered healthcare companion that combines:

🧠 Conversational AI
📸 Multimodal meal analysis
🍽️ Personalized nutrition planning
🧮 Dynamic BMR and TDEE calculations
📚 Retrieval-Augmented Generation (RAG)
💾 Long-term memory extraction
🔐 Secure authentication
☁️ Cloud-based storage and data management

What started as a simple terminal application evolved into a modern healthcare platform designed around personalization, accessibility, and intelligent health tracking.

Demo

This project is currently under active development and is being demonstrated through screenshots.

Dashboard

The redesigned dashboard provides users with a modern healthcare experience featuring personalized health metrics and quick access to Baymax's capabilities.

AI Healthcare Assistant

Users can chat naturally with Baymax and receive contextual health guidance powered by Gemini.

Vision-Based Meal Analysis

Users can upload meal images and receive calorie estimates, macro breakdowns, health scores, and nutritional insights.

Personalized Health Metrics

Baymax dynamically calculates BMR, TDEE, protein targets, and daily nutrition recommendations based on the user's profile.

Knowledge Base & RAG

Healthcare documents can be uploaded, embedded, and searched to provide evidence-based responses grounded in trusted resources.

The Comeback Story

The original Agent Baymax was built as a lightweight proof of concept.

Version 1 used Google's Gemma 2B instruction-tuned model running locally through Hugging Face Transformers. Users interacted through a terminal interface where they could ask basic health questions, track hydration, and log food consumption.

Although functional, several limitations quickly became apparent:

No graphical user interface
No authentication system
No persistent memory
No image understanding
No document retrieval
No personalization engine
No scalable architecture

Rather than abandoning the project, I chose to rebuild it from the ground up.

The first major step was moving from a terminal application to a modern web platform powered by Next.js and Supabase.

Next came multimodal AI capabilities. Instead of manually logging food, users can now upload meal images and receive structured nutritional analysis.

To improve personalization, I implemented a memory extraction system that identifies important long-term user preferences, dietary restrictions, and lifestyle choices and stores them for future interactions.

I then introduced a Retrieval-Augmented Generation pipeline using embeddings and vector search, allowing Baymax to retrieve information from uploaded healthcare documents instead of relying entirely on model knowledge.

Finally, I redesigned the user experience with glassmorphic interfaces, smooth animations, streaming AI responses, cloud storage, authentication, and scalable backend infrastructure.

Before vs After

Agent Baymax V1	Agent Baymax V2
Terminal-Based Application	Modern Web Platform
Local Gemma 2B Model	Gemini-Powered AI
Text-Only Inputs	Text + Image Understanding
Local File Storage	Cloud Database
Session Context Only	Persistent Long-Term Memory
Manual Food Logging	AI Meal Recognition
No Knowledge Retrieval	RAG + Vector Search
Single-User Local Tool	Scalable Full-Stack Platform

This challenge became less about finishing unfinished code and more about finishing the original vision behind Baymax.

My Experience with GitHub Copilot

As the project grew from a small Python prototype into a full-stack AI application, development complexity increased significantly.

GitHub Copilot helped speed up implementation by assisting with:

API integrations
Authentication flows
Database operations
TypeScript interfaces
React component scaffolding
Utility functions
Backend route generation

For larger features such as memory extraction workflows and vector search integrations, Copilot provided useful starting points that accelerated experimentation and iteration.

Rather than replacing problem-solving, it reduced repetitive work and allowed more focus on architecture decisions, feature design, and user experience.

🛠️ Tech Stack

Frontend

Next.js 16
React 19
Tailwind CSS v4
Framer Motion

Backend & Database

Supabase PostgreSQL
pgvector
Supabase Auth
Supabase Storage

AI & Machine Learning

Google Gemini
Google Embedding Models
Vercel AI SDK

Infrastructure

Upstash Redis
Vector Search
Retrieval-Augmented Generation (RAG)
Long-Term Memory Engine

Final Thoughts

Revisiting Agent Baymax taught me an important lesson.

Not every unfinished project deserves to be abandoned. Sometimes the most rewarding challenge is returning to an old idea and finally building it the way you originally imagined.

Agent Baymax began as a small hackathon experiment.

Agent Baymax V2 represents the vision that experiment was always meant to become.

And this challenge gave me the perfect excuse to finally finish it.

Gemma Wagon - Your Private Ambient AI Desktop Companion Powered by Gemma 4

Harshit Agarwal — Tue, 19 May 2026 17:31:11 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

Gemma Wagon

Executive Summary

Gemma Wagon is a privacy-first, fully local AI desktop assistant designed to transform how users interact with their computers. Instead of functioning as a simple chatbot, Gemma Wagon acts as an intelligent operating system layer capable of seeing the user’s screen, understanding voice commands, reasoning through complex workflows, and executing real desktop actions — all without sending data to the cloud.

Built around the multimodal and agentic capabilities of Gemma 4, Gemma Wagon combines local inference, real-time desktop context awareness, Retrieval-Augmented Generation (RAG), and secure OS-level automation into a single unified experience.

Our goal is to bridge the growing gap between AI utility and user privacy. Current AI assistants often require constant internet connectivity and expose sensitive information to external servers. Gemma Wagon solves this problem by ensuring every interaction happens entirely on-device.

This project demonstrates how Gemma 4 can power the next generation of ambient AI systems that are fast, secure, context-aware, and developer-friendly.

The Problem

Modern AI assistants are powerful, but they still suffer from several major limitations:

Sensitive files and screen data must often be uploaded to external servers
Existing assistants lack persistent contextual awareness of the desktop environment
Local AI solutions are fragmented and difficult for non-technical users
Most assistants cannot safely execute real operating system tasks
Cloud dependency introduces latency, privacy concerns, and internet requirements

Users need an AI system that is:

Local-first
Privacy-preserving
Multimodal
Action-oriented
Lightweight enough to run continuously

Gemma Wagon was designed specifically to solve these challenges.

Our Solution: Gemma Wagon

Gemma Wagon introduces an Ambient AI Desktop Layer that continuously assists users through contextual understanding and local reasoning.

The system includes:

Persistent floating AI overlay (“Orb”)
Local multimodal reasoning using Gemma 4
Voice + screen understanding
OS-level task automation
Local REST API for developers
Offline document intelligence using RAG
Fully private local execution

Instead of forcing users to switch between applications, Gemma Wagon becomes a natural extension of the desktop itself.

Why Gemma 4?

Gemma 4 is the core technology that makes this architecture possible.

1. Native Multimodality

Gemma 4 can process:

Text
Images
Audio

This enables Gemma Wagon to:

Understand screenshots
Analyze UI elements
Process voice commands
Maintain contextual awareness

The multimodal capabilities allow the assistant to “see” and “hear” the user’s environment in real time.

2. Efficient Local Inference

We use optimized GGUF variants of Gemma 4 running through llama.cpp.

This allows:

Fully offline inference
GPU acceleration
Low memory usage
Background execution without disrupting workflows

The Mixture-of-Experts efficiency of Gemma 4 enables high-quality reasoning while remaining lightweight enough for local consumer hardware.

3. Massive Context Window

Gemma 4’s extended context capabilities allow Gemma Wagon to:

Read large PDFs
Analyze repositories
Understand long conversations
Process entire document libraries

This becomes especially powerful when combined with our local RAG pipeline.

4. Agentic Reasoning & Function Calling

Gemma 4’s reasoning capabilities enable safe desktop automation through structured function calls.

Examples include:

Opening applications
Finding files
Organizing folders
Summarizing spreadsheets
Running scripts

The assistant reasons before acting, making automation safer and more reliable.

Technical Architecture

Gemma Wagon is built using a layered architecture optimized for performance, security, and modularity.

1. Core Engine (Python/Rust)

The Core Engine acts as the brain and system controller.

Responsibilities

Screen capture
Audio capture
OS integrations
Function execution
REST API handling

Key Technologies

Python
Rust
mss for screenshots
pyaudio for microphone streaming

The engine runs locally as a background service and exposes an OpenAI-compatible local API endpoint.

2. AI Inference Layer

The inference layer embeds Gemma 4 directly into the application.

Key Implementation Details

GGUF model format
llama.cpp backend
CUDA acceleration
Vulkan/ROCm support
Metal support for macOS

Optimizations

KV-cache memory management
Context retention
Inference throughput
Local GPU utilization

This enables real-time AI interaction directly on-device.

3. Frontend (Tauri + Rust)

The frontend provides the desktop experience.

Features

Floating Orb overlay
Modern chat interface
Markdown rendering
Model configuration
Document management

Why Tauri?

Smaller binaries
Better performance
Higher security
Native desktop integration

Rust handles secure communication between the UI and backend systems.

4. Local Knowledge Base (RAG)

Gemma Wagon includes fully local document intelligence.

Pipeline

Document upload
Chunking
Embedding generation
Vector indexing
Retrieval during inference

Supported Documents

PDFs
PPTs
Notes
Codebases

The vector database uses lightweight local storage for fully offline retrieval.

Communication Flow

The interaction pipeline follows this sequence:

User triggers Gemma Wagon via voice or hotkey
System captures screen/audio context
Gemma 4 processes multimodal input
AI generates either:
- A response
- A function call
Core Engine executes the action
UI provides visual feedback

This architecture enables real-time ambient assistance while remaining fully local.

Privacy & Security

Privacy is the foundation of Gemma Wagon.

Security Model

Fully local inference
No cloud processing
No telemetry
Encrypted local storage
Sandboxed OS function execution

The only internet access required is the initial model download.

Ideal For

Developers
Enterprises
Researchers
Privacy-conscious users

Key Use Cases

Productivity Assistant

“Summarize this spreadsheet and generate action items.”

Developer Copilot

“Analyze this repository and explain the architecture.”

Smart Document Search

“Find the PDF where I discussed vector databases.”

Workflow Automation

“Open VS Code, launch Docker, and summarize yesterday’s notes.”

Accessibility Support

Context-aware voice-based desktop interaction.

Engineering Challenges

Real-Time Multimodal Processing

Running continuous screen + audio analysis locally required careful optimization of:

GPU memory
Context management
Inference latency

Safe Function Calling

We implemented controlled execution pipelines to prevent unsafe automation behavior.

Lightweight Desktop Integration

Creating a persistent desktop assistant without large resource consumption required deep optimization using Rust and Tauri.

Local RAG Performance

Efficient indexing and retrieval were essential for maintaining fast response times on consumer hardware.

What Makes Gemma Wagon Different?

Unlike traditional AI chat applications, Gemma Wagon is:

Ambient instead of reactive
Local instead of cloud-based
Agentic instead of passive
Multimodal instead of text-only
Integrated into the OS instead of isolated in a browser tab

Gemma Wagon demonstrates how Gemma 4 can power truly personal AI systems that remain private, fast, and deeply contextual.

Conclusion

Gemma Wagon represents a new category of AI-native computing.

By combining:

Gemma 4 multimodal reasoning
Local inference
Agentic automation
Desktop integration
Privacy-first architecture

we created a system that transforms AI from a chatbot into a true operating system companion.

This project showcases the real-world potential of Gemma 4 as the foundation for next-generation ambient AI experiences that users can fully trust.

Repository

🔗 GitHub:

https://github.com/Harshitagarwal113/gemma_wagon