DEV Community: Harshit Agarwal

OutsideQuest AI: Screen-Free Nature Quests with On-Device Gemma 3

Harshit Agarwal — Sun, 11 Oct 2026 13:32:47 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Most modern technology traps us in continuous dopamine loops, screen fatigue, and doomscrolling. Even outdoor apps often exacerbate this problem by demanding continuous screen interaction—staring at turn-by-turn navigation, checking GPS maps, or curating photos for social validation.

OutsideQuest AI was built to do the exact opposite: use lightweight, open-weight AI to inspire you to put your phone in your pocket and touch grass.

OutsideQuest AI is an intelligent, privacy-first companion that creates personalized outdoor micro-adventures designed for sensory immersion in living nature. Whether you have 15 minutes between remote work meetings or an hour on the weekend, the app crafts a structured, grounding outdoor mission tailored to:

Available Time: 15, 30, 45, or 60 minutes.
Immediate Landscape: Neighborhood, city park, botanical garden, university campus, or wild forest trail.
Sensory Mood & Movement Pace: Curious, calm, energetic, or social; gentle, moderate, or active.
Companion Mode: Solo mindful walks, friends, family with children, or community groups.
Sensory Interests: Birding & bird song, plant identification, walking rhythms, mindfulness, tactile exploration, or nature photography.
Accessibility Needs: Step-free pathways, flat terrain, frequent rest benches, and sensory-friendly options.

The Screen-Free Philosophy: Phone Away, Adventure On

Once your quest begins, OutsideQuest AI encourages you to keep your screen off:

Paced Audio Chimes: Built-in Web Audio API synthesizers play gentle harmonic chimes (528 Hz and 659 Hz) to signal when each sensory milestone concludes, so you never have to watch a ticking clock.
Sensory Focus Prompts: Every step activates a distinct physical sense—listening for rustling foliage, noticing bark textures, feeling temperature gradients, or tracking shadow angles.
Safety-First Guardrails: The engine programmatically enforces strict nature safety rules: no eating wild plants/mushrooms, no approaching or handling wildlife, and staying on marked public paths.
Nature Journal & Reflections: When you return, you record what you noticed (sensory observations, bird sightings, mental headspace) in a private, persistent nature journal stored locally in your browser.

Demo

🌐 Live Web Application: https://outside-quest-ai.vercel.app/
⚡ Zero Setup Required: Deployed on Vercel with zero required API keys, accounts, or environment variables. Works on desktop, tablet, and mobile browsers.

Code

📦 GitHub Repository: https://github.com/Harshitagarwal113/OutsideQuest-AI
📄 License: MIT License (Open Source)
🏷️ Current Release: v0.1.0

Harshitagarwal113 / OutsideQuest-AI

OutsideQuest AI 🌲

Turn routine neighborhood walks into sensory outdoor micro-adventures with browser-native, open-weight AI.
Built for the Touch Grass Open-Source AI Challenge.

🧭 Product Overview

In an era of hyper-connected screens and infinite AI chat loops, OutsideQuest AI was built to turn artificial intelligence inside out: instead of capturing your gaze, it encourages you to pocket your phone and explore the living world with your physical senses.

OutsideQuest AI runs open-weight Small Language Models (SLMs) directly inside the user's browser via WebGPU and WebAssembly. It personalizes sensory exploration missions tailored to your immediate environment (park, garden, neighborhood, campus), available duration (15 to 60 minutes), social setting (solo, friends, family), mood, and accessibility preferences.

Every quest is private, free, and self-contained:

Zero Cloud Inference APIs: No OpenAI, Anthropic, or external serverless endpoints.
Zero Account Creation: No login walls, passwords, or tracking cookies.
Zero Ongoing Cost: Hosted on Vercel Hobby (Free…

View on GitHub

How I Built It

OutsideQuest AI runs an open-weight Small Language Model directly in the user's browser—no cloud GPU clusters, no external inference APIs, and no server-side execution.

1. On-Device AI Engine: Google Gemma 3

Model: onnx-community/gemma-3-270m-it-ONNX (Google Gemma 3 270M Instruction-Tuned).
Quantization: 4-bit INT4 (q4) weights (~200 MB download footprint), optimized for consumer mobile devices and laptops.
Inference Library: Transformers.js v3 (@huggingface/transformers) coupled with ONNX Runtime Web.
Hardware Acceleration: Automatically targets WebGPU for parallel tensor arithmetic on device silicon, with graceful degradation to multi-threaded WebAssembly (WASM).

2. Multi-Threaded Web Worker Architecture

Running heavy neural network matrix calculations on the browser UI thread can freeze scrolling and animations. We isolated the model lifecycle inside a dedicated Web Worker (src/features/ai/workers/model.worker.ts):

Non-blocking token generation keeps the React user interface at a butter-smooth 60 FPS.
Dynamic Gemma chat template handling formats prompt turns and seamlessly handles system-prompt merging.
Comprehensive progress tracking reports byte-level download and shader compilation progress back to the UI.

3. Dual-Engine Generation & Zero-Drop Resilience

To guarantee that you can always head outside even with zero connectivity:

Primary Generator: Browser-native Google Gemma 3 neural generation.
Deterministic Curated Engine: If the model has not yet been downloaded, or if you are already on a deep trail with cellular reception turned off, a deterministic engine instantly generates rich, sensory-grounded quests in 0 milliseconds with 0 MB downloaded.

4. Local-First Nature Journaling with IndexedDB

Your thoughts and reflections are stored in a sandboxed, client-side database using idb (IndexedDB):

Quests, completion statuses, sensory notes, and nature metrics persist across browser sessions.
Built-in data export (formatted JSON) and print-ready expedition note summaries.

Tech Stack Summary

Framework: Next.js 14 App Router (Full static prerendering)
Language & Runtime: TypeScript (Strict Mode) + React 18
Styling: Tailwind CSS with a curated, nature-inspired palette (Forest, Sage, Sand, Earth Amber)
Validation: Zod schemas for runtime parameter checking and LLM JSON output repair
Audio: Custom Web Audio API synthesizer for serene nature chimes
Testing: 15 Vitest test suites (71 tests passing) + Playwright E2E suites for desktop and mobile viewports

Why Does Open Innovation Matter?

Outdoor exploration and nature connection should be universal, safe, and free. Building OutsideQuest AI around an open-weight, browser-native model unlocks benefits that closed, proprietary cloud APIs simply cannot match:

True Offline Functionality in Living Nature:
Closed APIs require constant high-speed internet. But the best places to "touch grass"—state parks, botanical gardens, coastal headlands, and dense woodlands—often have spotty or zero cellular coverage. Because Gemma 3 is cached on-device, OutsideQuest AI continues generating adventures even when completely disconnected in Airplane Mode.
Absolute Privacy & Data Sovereignty:
Closed APIs log your prompts, location hints, daily walking routines, physical limitations, and sensory reflection journals onto commercial cloud servers. With open weights on-device, your thoughts and personal context never leave your browser sandbox.
Zero Cost & Democratic Access:
There are no token costs, subscription tiers, API credit expirations, or paywalls. Anyone with a smartphone or laptop can experience personalized nature guidance without entering a credit card or registering an account.
Reliability & Immortality:
A closed API can be sunset, rate-limited, or altered overnight. An open-weight model with an open-source web application remains fully operational forever.

My Agent Session

This application was engineered collaboratively with Antigravity AI, Google DeepMind's agentic coding pair programmer.

The full agent workflow covered:

Architecting the client-side Web Worker communication layer and Zod output repair pipeline.
Integrating onnx-community/gemma-3-270m-it-ONNX via Transformers.js v3 with WebGPU/WASM fallback.
Conducting rigorous security and code audits, resolving PostCSS configurations, and achieving 100% test pass rates across 71 unit/integration tests.
Deploying the static bundle to Vercel Hobby with zero environment variables and zero paid compute.

(Transcripts and architecture documents are preserved directly in the repository under docs/ARCHITECTURE.md and docs/AI_DECISIONS.md).

Prize Categories

We are entering OutsideQuest AI into the following track:

Best Use of Gemma (Primary Partner Category)
Grand Prize / Overall Track (Hacktoberfest Open-Source AI Challenge: Week 1 — Touch Grass)

🌲 Step Outside

OutsideQuest AI proves that open-source AI doesn't have to keep us glued to glowing rectangles. By taking advantage of Google Gemma 3 running on your device silicon, we can use artificial intelligence to celebrate and preserve our connection with the real, tactile, beautiful natural world.

Take 15 minutes today, put your phone in your pocket, and go explore:
👉 Launch OutsideQuest AI

The Comeback of Agent Baymax: From Hackathon Prototype to AI Healthcare Platform

Harshit Agarwal — Sat, 06 Jun 2026 09:17:20 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

🩺 Agent Baymax V2 — From a Local AI Assistant to a Full Healthcare Platform

What I Built

Some projects don't need to be replaced—they need to be completed.

Agent Baymax started as a simple terminal-based healthcare companion built during a hackathon. The original version could answer basic health-related questions, log calories, track hydration, and maintain short-term conversational context using a locally running language model.

While the idea worked, it never fully matched the vision I had in mind.

Users had to manually enter meals, there was no personalization across sessions, no visual interface, no image understanding, and no way to retrieve information from trusted healthcare resources.

For the Finish-Up-A-Thon Challenge, I decided to revisit the project and transform it into something much more capable.

The result is Agent Baymax V2 — an AI-powered healthcare companion that combines:

🧠 Conversational AI
📸 Multimodal meal analysis
🍽️ Personalized nutrition planning
🧮 Dynamic BMR and TDEE calculations
📚 Retrieval-Augmented Generation (RAG)
💾 Long-term memory extraction
🔐 Secure authentication
☁️ Cloud-based storage and data management

What started as a simple terminal application evolved into a modern healthcare platform designed around personalization, accessibility, and intelligent health tracking.

Demo

This project is currently under active development and is being demonstrated through screenshots.

Dashboard

The redesigned dashboard provides users with a modern healthcare experience featuring personalized health metrics and quick access to Baymax's capabilities.

AI Healthcare Assistant

Users can chat naturally with Baymax and receive contextual health guidance powered by Gemini.

Vision-Based Meal Analysis

Users can upload meal images and receive calorie estimates, macro breakdowns, health scores, and nutritional insights.

Personalized Health Metrics

Baymax dynamically calculates BMR, TDEE, protein targets, and daily nutrition recommendations based on the user's profile.

Knowledge Base & RAG

Healthcare documents can be uploaded, embedded, and searched to provide evidence-based responses grounded in trusted resources.

The Comeback Story

The original Agent Baymax was built as a lightweight proof of concept.

Version 1 used Google's Gemma 2B instruction-tuned model running locally through Hugging Face Transformers. Users interacted through a terminal interface where they could ask basic health questions, track hydration, and log food consumption.

Although functional, several limitations quickly became apparent:

No graphical user interface
No authentication system
No persistent memory
No image understanding
No document retrieval
No personalization engine
No scalable architecture

Rather than abandoning the project, I chose to rebuild it from the ground up.

The first major step was moving from a terminal application to a modern web platform powered by Next.js and Supabase.

Next came multimodal AI capabilities. Instead of manually logging food, users can now upload meal images and receive structured nutritional analysis.

To improve personalization, I implemented a memory extraction system that identifies important long-term user preferences, dietary restrictions, and lifestyle choices and stores them for future interactions.

I then introduced a Retrieval-Augmented Generation pipeline using embeddings and vector search, allowing Baymax to retrieve information from uploaded healthcare documents instead of relying entirely on model knowledge.

Finally, I redesigned the user experience with glassmorphic interfaces, smooth animations, streaming AI responses, cloud storage, authentication, and scalable backend infrastructure.

Before vs After

Agent Baymax V1	Agent Baymax V2
Terminal-Based Application	Modern Web Platform
Local Gemma 2B Model	Gemini-Powered AI
Text-Only Inputs	Text + Image Understanding
Local File Storage	Cloud Database
Session Context Only	Persistent Long-Term Memory
Manual Food Logging	AI Meal Recognition
No Knowledge Retrieval	RAG + Vector Search
Single-User Local Tool	Scalable Full-Stack Platform

This challenge became less about finishing unfinished code and more about finishing the original vision behind Baymax.

My Experience with GitHub Copilot

As the project grew from a small Python prototype into a full-stack AI application, development complexity increased significantly.

GitHub Copilot helped speed up implementation by assisting with:

API integrations
Authentication flows
Database operations
TypeScript interfaces
React component scaffolding
Utility functions
Backend route generation

For larger features such as memory extraction workflows and vector search integrations, Copilot provided useful starting points that accelerated experimentation and iteration.

Rather than replacing problem-solving, it reduced repetitive work and allowed more focus on architecture decisions, feature design, and user experience.

🛠️ Tech Stack

Frontend

Next.js 16
React 19
Tailwind CSS v4
Framer Motion

Backend & Database

Supabase PostgreSQL
pgvector
Supabase Auth
Supabase Storage

AI & Machine Learning

Google Gemini
Google Embedding Models
Vercel AI SDK

Infrastructure

Upstash Redis
Vector Search
Retrieval-Augmented Generation (RAG)
Long-Term Memory Engine

Final Thoughts

Revisiting Agent Baymax taught me an important lesson.

Not every unfinished project deserves to be abandoned. Sometimes the most rewarding challenge is returning to an old idea and finally building it the way you originally imagined.

Agent Baymax began as a small hackathon experiment.

Agent Baymax V2 represents the vision that experiment was always meant to become.

And this challenge gave me the perfect excuse to finally finish it.

Gemma Wagon - Your Private Ambient AI Desktop Companion Powered by Gemma 4

Harshit Agarwal — Tue, 19 May 2026 17:31:11 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

Gemma Wagon

Executive Summary

Gemma Wagon is a privacy-first, fully local AI desktop assistant designed to transform how users interact with their computers. Instead of functioning as a simple chatbot, Gemma Wagon acts as an intelligent operating system layer capable of seeing the user’s screen, understanding voice commands, reasoning through complex workflows, and executing real desktop actions — all without sending data to the cloud.

Built around the multimodal and agentic capabilities of Gemma 4, Gemma Wagon combines local inference, real-time desktop context awareness, Retrieval-Augmented Generation (RAG), and secure OS-level automation into a single unified experience.

Our goal is to bridge the growing gap between AI utility and user privacy. Current AI assistants often require constant internet connectivity and expose sensitive information to external servers. Gemma Wagon solves this problem by ensuring every interaction happens entirely on-device.

This project demonstrates how Gemma 4 can power the next generation of ambient AI systems that are fast, secure, context-aware, and developer-friendly.

The Problem

Modern AI assistants are powerful, but they still suffer from several major limitations:

Sensitive files and screen data must often be uploaded to external servers
Existing assistants lack persistent contextual awareness of the desktop environment
Local AI solutions are fragmented and difficult for non-technical users
Most assistants cannot safely execute real operating system tasks
Cloud dependency introduces latency, privacy concerns, and internet requirements

Users need an AI system that is:

Local-first
Privacy-preserving
Multimodal
Action-oriented
Lightweight enough to run continuously

Gemma Wagon was designed specifically to solve these challenges.

Our Solution: Gemma Wagon

Gemma Wagon introduces an Ambient AI Desktop Layer that continuously assists users through contextual understanding and local reasoning.

The system includes:

Persistent floating AI overlay (“Orb”)
Local multimodal reasoning using Gemma 4
Voice + screen understanding
OS-level task automation
Local REST API for developers
Offline document intelligence using RAG
Fully private local execution

Instead of forcing users to switch between applications, Gemma Wagon becomes a natural extension of the desktop itself.

Why Gemma 4?

Gemma 4 is the core technology that makes this architecture possible.

1. Native Multimodality

Gemma 4 can process:

Text
Images
Audio

This enables Gemma Wagon to:

Understand screenshots
Analyze UI elements
Process voice commands
Maintain contextual awareness

The multimodal capabilities allow the assistant to “see” and “hear” the user’s environment in real time.

2. Efficient Local Inference

We use optimized GGUF variants of Gemma 4 running through llama.cpp.

This allows:

Fully offline inference
GPU acceleration
Low memory usage
Background execution without disrupting workflows

The Mixture-of-Experts efficiency of Gemma 4 enables high-quality reasoning while remaining lightweight enough for local consumer hardware.

3. Massive Context Window

Gemma 4’s extended context capabilities allow Gemma Wagon to:

Read large PDFs
Analyze repositories
Understand long conversations
Process entire document libraries

This becomes especially powerful when combined with our local RAG pipeline.

4. Agentic Reasoning & Function Calling

Gemma 4’s reasoning capabilities enable safe desktop automation through structured function calls.

Examples include:

Opening applications
Finding files
Organizing folders
Summarizing spreadsheets
Running scripts

The assistant reasons before acting, making automation safer and more reliable.

Technical Architecture

Gemma Wagon is built using a layered architecture optimized for performance, security, and modularity.

1. Core Engine (Python/Rust)

The Core Engine acts as the brain and system controller.

Responsibilities

Screen capture
Audio capture
OS integrations
Function execution
REST API handling

Key Technologies

Python
Rust
mss for screenshots
pyaudio for microphone streaming

The engine runs locally as a background service and exposes an OpenAI-compatible local API endpoint.

2. AI Inference Layer

The inference layer embeds Gemma 4 directly into the application.

Key Implementation Details

GGUF model format
llama.cpp backend
CUDA acceleration
Vulkan/ROCm support
Metal support for macOS

Optimizations

KV-cache memory management
Context retention
Inference throughput
Local GPU utilization

This enables real-time AI interaction directly on-device.

3. Frontend (Tauri + Rust)

The frontend provides the desktop experience.

Features

Floating Orb overlay
Modern chat interface
Markdown rendering
Model configuration
Document management

Why Tauri?

Smaller binaries
Better performance
Higher security
Native desktop integration

Rust handles secure communication between the UI and backend systems.

4. Local Knowledge Base (RAG)

Gemma Wagon includes fully local document intelligence.

Pipeline

Document upload
Chunking
Embedding generation
Vector indexing
Retrieval during inference

Supported Documents

PDFs
PPTs
Notes
Codebases

The vector database uses lightweight local storage for fully offline retrieval.

Communication Flow

The interaction pipeline follows this sequence:

User triggers Gemma Wagon via voice or hotkey
System captures screen/audio context
Gemma 4 processes multimodal input
AI generates either:
- A response
- A function call
Core Engine executes the action
UI provides visual feedback

This architecture enables real-time ambient assistance while remaining fully local.

Privacy & Security

Privacy is the foundation of Gemma Wagon.

Security Model

Fully local inference
No cloud processing
No telemetry
Encrypted local storage
Sandboxed OS function execution

The only internet access required is the initial model download.

Ideal For

Developers
Enterprises
Researchers
Privacy-conscious users

Key Use Cases

Productivity Assistant

“Summarize this spreadsheet and generate action items.”

Developer Copilot

“Analyze this repository and explain the architecture.”

Smart Document Search

“Find the PDF where I discussed vector databases.”

Workflow Automation

“Open VS Code, launch Docker, and summarize yesterday’s notes.”

Accessibility Support

Context-aware voice-based desktop interaction.

Engineering Challenges

Real-Time Multimodal Processing

Running continuous screen + audio analysis locally required careful optimization of:

GPU memory
Context management
Inference latency

Safe Function Calling

We implemented controlled execution pipelines to prevent unsafe automation behavior.

Lightweight Desktop Integration

Creating a persistent desktop assistant without large resource consumption required deep optimization using Rust and Tauri.

Local RAG Performance

Efficient indexing and retrieval were essential for maintaining fast response times on consumer hardware.

What Makes Gemma Wagon Different?

Unlike traditional AI chat applications, Gemma Wagon is:

Ambient instead of reactive
Local instead of cloud-based
Agentic instead of passive
Multimodal instead of text-only
Integrated into the OS instead of isolated in a browser tab

Gemma Wagon demonstrates how Gemma 4 can power truly personal AI systems that remain private, fast, and deeply contextual.

Conclusion

Gemma Wagon represents a new category of AI-native computing.

By combining:

Gemma 4 multimodal reasoning
Local inference
Agentic automation
Desktop integration
Privacy-first architecture

we created a system that transforms AI from a chatbot into a true operating system companion.

This project showcases the real-world potential of Gemma 4 as the foundation for next-generation ambient AI experiences that users can fully trust.

Repository

🔗 GitHub:

https://github.com/Harshitagarwal113/gemma_wagon