DEV Community: shruti rai

Touch Grass with Canopy: The Sovereign Offline-First AI Backcountry Guardian & 3D Topo Explorer

shruti rai — Fri, 09 Oct 2026 20:17:44 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Modern outdoor technology suffers from a deep irony: apps designed to help us explore nature often end up trapping us on our screens. Hikers find themselves craning their necks down at battery-hungry maps, refreshing disconnected cloud services, and obsessing over GPS dots instead of immersing themselves in the wilderness. Worse, when hikers venture into remote backcountry areas where cellular signal drops to zero, proprietary cloud apps freeze completely.

Canopy (Touch Grass / Backcountry Guardian) is an open-source, 100% sovereign, offline-first backcountry AI companion engineered to help hikers, mountaineers, and nature enthusiasts safely disconnect from digital distractions and "Touch Grass".

Engineered to Get You Off the Screen

Canopy is intentionally built to minimize screen time rather than maximize digital engagement:

Pocket Mode & Audio Whispers: Slide your phone into your jacket or backpack. Canopy runs background environmental monitoring and speaks subtle spatial audio chimes only when safety-critical events occur (e.g., approaching a turnaround deadline or sudden barometric drops).
Minimalist OLED Field Mode: In bright alpine sunlight or extreme battery-saving states, Canopy switches to a pure-black, high-contrast OLED interface displaying only essential telemetry: altitude, pressure delta, temperature, and hard turnaround countdowns.
Screen vs. Outdoor Ratio: Actively measures your trail time versus screen-on time, encouraging a 150× ratio of being present in nature.

Key Wilderness Capabilities

Interactive 3D Digital Twin Himalayan Topography (WebGL/Three.js): Procedural 3D alpine terrains for legendary routes (Hampta Pass, Kedarnath Summit Ridge, Triund, Chandrashila Peak) with real-time raycasted elevation inspectors, slope hazard gradients, snowpack coverage, and waypoint flythroughs.
On-Device Spectral Bioacoustics: Real-time Web Audio FFT spectrogram analyzing high-altitude avian calls (Alpine Chough, Himalayan Monal, Snow Partridge, Hermit Thrush) with zero cloud roundtrips.
Microclimate Physics Engine: Local barometric lapse rate and radiative frost prediction modeling temperature drop per 1,000 m of ascent without requiring internet weather forecasts.
Physical Sensor Bridge (Web Serial API): Connects directly over USB/Bluetooth to hardware microcontrollers (such as Arduino UNO Q with BME280) for real-time barometric and temperature ingestion.
Deterministic Safety Engine: A strict guardrail layer that prevents dangerous AI hallucinations—enforcing non-negotiable turnaround curfews, hypothermia warnings, and strictly refusing prescription drug recommendations.

Demo

Live GitHub Repository: https://github.com/shrutirai29/hacktober
Interactive Web App: https://shrutirai29.github.io/hacktober/
Developer Diagnostics: Visit /diagnostics within the app to run the live WebLLM validation contract self-test, verify sensor streams, and inspect the red-team safety suites.

Code

shrutirai29 / hacktober

🌿 Canopy — Backcountry Guardian

Offline-First Open-Weight AI Backcountry Safety & Terrain Intelligence System

Built for the Touch Grass Open-Source AI Challenge

"Build something with open-source/open-weight AI at its core that gets people off the screen and into the world."

🏔️ 1. Project Overview & Philosophy

Outdoor safety cannot depend on cellular signal or cloud APIs. The moment an alpine trekker crosses 3,500 meters or enters an exposed couloir, cloud connectivity drops to zero. If safety intelligence lives exclusively on remote servers, it fails precisely when human lives depend on it most.

Canopy (Backcountry Guardian) is an offline-first backcountry safety, microclimate, and terrain intelligence system built with open-weight AI at its core.

The Touch Grass Architecture

Canopy is architected to get users off screens and into the natural world:

REAL-WORLD SENSORS & ENVIRONMENT
             ↓
CANOPY TELEMETRY CONTEXT ENGINE
             ↓
OPEN-WEIGHT LOCAL AI (SmolLM2 / Gemma 2) + NEURAL FALLBACK

…

View on GitHub

The complete project is open source under the MIT License. It contains zero cloud AI API keys, zero remote tracking scripts, and zero paid subscription gates.

# Clone and launch in 3 steps
git clone https://github.com/shrutirai29/hacktober.git
cd hacktober/t2
npm install
npm run dev

How I Built It

The 3-Tier Sovereign AI Fallback Pipeline

In deep mountain valleys, cellular data does not exist. Canopy solves this with a strictly local, 3-tier sovereign AI architecture:

User Query + Trail Telemetry
           ↓
[Tier 1: WebGPU WebLLM (SmolLM2-135M / Qwen2.5-0.5B)]
     ├─► Success: Instant client-side inference (100% on-device)
     └─► Failure/No WebGPU:
           ↓
     [Tier 2: Localhost Daemon (Ollama / Gemma 2 / Qwen)]
          ├─► Success: Local rugged field tablet / basecamp laptop
          └─► Failure/Unavailable:
                ↓
          [Tier 3: Micro-MLP & Deterministic Safety Engine]
               └─► Guarantees offline, non-crashing safety advice

Tier 1 — In-Browser WebGPU WebLLM (@mlc-ai/web-llm): Runs lightweight open-weight models (SmolLM2-135M-Instruct or Qwen2.5-0.5B-Instruct) compiled to WebAssembly and executed on the client's GPU via WebGPU. Model weights are cached locally in browser CacheStorage.
Tier 2 — Localhost Daemon (Ollama): Connects via local HTTP loopback (http://localhost:11434) to execute open-weight models like gemma2:2b or qwen2.5:0.5b for field workstations.
Tier 3 — Deterministic Safety Engine & Micro-MLP: A browser-native fallback that parses telemetry matrices and delivers instantaneous rule-based safety directives even on low-power devices without WebGPU.

Safety-First: The Deterministic Guardrail Layer

LLMs are creative, but backcountry safety requires deterministic boundaries. Canopy implements an uncompromised safety layer:

Curfew Enforcer: If the clock passes the turnaround time (e.g., 1:30 PM on Hampta Pass), any model output encouraging further ascent is intercepted and overridden with a mandatory TURN BACK IMMEDIATELY directive.
Medical Sanitization Protocol: Strictly strips and blocks prescription drug dosages (e.g., Diamox/Acetazolamide milligrams) and forces conservative hydration, rest, and immediate descent protocols.
Telemetry Fabrication Rejection: If requested to evaluate an unmonitored metric or non-existent sensor, the system refuses to hallucinate data.

Rigorous Red-Team & Unit Verification

Every capability claimed is backed by automated tests:

16 Unit Regression Tests (npm test): Validates the WebLLM self-test contract (CANOPY_WEBLLM_OK), adversarial prompt injection neutralization, medical sanitization, and audio FFT edge cases.
15 Red-Team Audit Tests (npm run test:audit): Validates the 3-tier fallback chain, 0 cloud API key leak audit across all source files, Vite dev server startup, and extreme environmental inputs (-50°C to +50°C, 0 m to 10,000 m).

Why Does Open Innovation Matter?

Open innovation is not just an ideological preference for Canopy—it is the only architecture that makes wilderness survival AI possible.

Closed Cloud APIs Fail in the Wild: If an AI assistant requires an internet connection to reach a cloud server, it is completely useless the second you step off the trailhead. Open-weight models (SmolLM, Gemma, Qwen) allow weights to live directly on personal devices, providing true computational sovereignty.
Safety Must Be Inspectable: When software provides advice on alpine hypothermia, avalanche hazards, or mountain sickness, proprietary black-box APIs are unacceptable. Open-source development enables open-weight inspection, deterministic guardrail wrapping, and transparent peer auditing.
Privacy and Zero Telemetry Tracking: The outdoors is where people go for tranquility and privacy. Canopy uses open-source software to ensure that no biometric data, GPS coordinates, or voice audio ever leave your local machine.

My Agent Session

During development, the entire iterative engineering loop—implementing the WebGPU WebLLM provider, Three.js Himalayan topographic math, deterministic red-team suites, and automated validation tests—was completed with pair-programming assistance. Full audit logs and automated test reports are open and verifiable in the repository under /scripts.

Prize Categories

Main Challenge: Open-Source AI Challenge Week 1: Touch Grass
Best Use of Open-Weight Models: SmolLM2 / Gemma 2 via WebGPU and local Ollama inference
Edge & On-Device AI: 100% in-browser WebGPU WebLLM execution with zero network dependency
Responsible & Safe AI: Open-weight LLMs wrapped in deterministic safety and medical guardrails

CheckMate: A 3D spatial twin of my best friend's hostel room so he stops leaving his charger behind

shruti rai — Sun, 04 Oct 2026 17:26:34 +0000

This is my submission for the Hacktoberfest Weekend Challenge: Build for a Friend.

What I Built

My best friend Kanwal has a habit of leaving things behind.

Not small, insignificant things. Important things. Things he will desperately need approximately 45 minutes after leaving the hostel.

Like his charger.

Picture this: you're preparing for a college presentation. Your laptop is charged, your slides are ready, your bag is packed, and you're already running late. You board the train, settle into your seat, and then it hits you.

"Bhai, charger toh switchboard mein hi reh gaya."

And just like that, your perfectly planned trip has a problem.

The annoying part is that Kanwal doesn't forget because he doesn't know what to pack. He forgets because packing is a physical task, while most reminders live on a screen.

A checklist can tell you to pack your charger. It cannot, by itself, connect that reminder to the charger sitting beside your desk.

I could have made him another checklist app. There are already plenty of those.

Instead, I wanted to try something different.

CheckMate is a trip-aware packing assistant built around a 3D representation of my best friend's actual hostel room. It combines an interactive spatial interface, tracked belongings, AI-assisted packing recommendations, memory of recurring mistakes, and a Hinglish voice coach that reminds him to check before leaving.

Basically, I built software to stop one very specific person from making the same very specific mistake.

And yes, the charger has a starring role.

Demo

🌐 Live app: https://checkmate-ebvi.onrender.com/

💻 GitHub repository: https://github.com/shrutirai29/hacktober

The deployed project brings together the room interface and packing-assistant experience. AI and voice capabilities depend on the relevant backend services and deployment configuration.

Why a 3D Room Instead of Another Checklist?

Here's the idea that got me interested in this project.

Kanwal's room already contains most of the information he needs to pack. His laptop is on the desk, his backpack is beside the chair, and his charger has a favourite spot near the wall socket.

The problem is that his brain doesn't always connect the packing decision to the physical object.

A conventional checklist represents belongings as text:

Laptop
Charger
Water bottle
ID card

CheckMate represents the room as a space that you can explore and interact with.

That changes the experience from simply asking "What do I need?" to asking "What do I still need to collect from my room?"

That's the distinction I wanted to explore.

The 3D Room: Getting the Details Right

The main interface is a custom 3D room built using Three.js.

I modelled the scene around the reference photograph of Kanwal's hostel room instead of starting with a generic bedroom template.

The scene includes a birchwood study desk, black desk legs, a purple swivel chair, a bed with a lavender duvet, a floating bookshelf, books, a panda figurine, trailing pothos vines, postcards, a backpack, and the wall switchboard.

The switchboard might be a strangely specific detail to care about in a 3D model.

But when your entire project exists because someone keeps forgetting a charger, it suddenly becomes important.

I worked on the furniture geometry, materials, lighting, object placement, and camera positioning to make the room feel familiar. I also had to pay attention to the little things that make 3D scenes look wrong: objects clipping into furniture, awkward proportions, and items appearing disconnected from their surroundings.

There's also a 📸 Photo Angle preset that moves the camera to a predefined view designed to match the reference photograph.

It makes the scene easier to recognise and gives the user a familiar perspective of the room.

The goal isn't to make a random 3D bedroom look pretty. It's to make a space feel recognisable enough that the user can relate the interface to their real surroundings.

Clicking on Objects Is Only the Beginning

CheckMate includes interactive belongings such as the MacBook, 65W charger, Hydro Flask, and backpack.

These objects can be selected in the scene, with visual indicators and status badges helping distinguish tracked items. Pulsing radar-style rings draw attention to the objects that matter.

But the more interesting problem is the state of each object.

There are three different questions an assistant needs to distinguish:

Does the item exist?
Where is the item?
Has the item actually been packed?

Knowing that Kanwal owns a charger doesn't mean it's in his bag. Knowing that it usually sits beside his desk doesn't prove that it's still there today.

CheckMate's interactive tracking interface is a starting point for connecting these states. Reliable automatic detection of real-world objects would require additional image-understanding capabilities, sensor input, or user confirmation.

I don't want a packing assistant that confidently tells someone their charger is on the desk when it has actually disappeared into a backpack.

A convincing 3D model is nice. Correct information about the physical world is what would make it genuinely useful.

Giving Packing Some Actual Context with Gemma

Not every trip needs the same things.

A two-day college presentation, a weekend hackathon, and a trip home all have different packing requirements.

That's why I wanted CheckMate's recommendations to depend on more than a fixed list.

The AI reasoning workflow uses Google Gemma to help generate packing suggestions based on the available trip context, tracked belongings, and relevant history.

The inputs can include:

Trip type and duration
Relevant weather information, when available
Items represented in the room
Current tracked packing states
Previous records of forgotten essentials

The idea is to generate a packing manifest that considers what to take, what to leave behind, and what deserves extra attention.

For example, a presentation trip might require the laptop, charger, ID card, and presentation materials, but not a pile of unrelated textbooks.

And if the charger has repeatedly been forgotten before presentations, it should receive more attention than an item Kanwal never forgets.

This is where contextual reasoning becomes useful. A model can help turn the available information into relevant recommendations rather than returning the same generic checklist for every trip.

The important limitation is that AI recommendations are only as reliable as their inputs. The model can reason about reported item states and past events; it cannot independently verify a physical object merely because the object appears in a 3D scene.

Memory: Because Apparently We Need to Learn This Lesson More Than Once

One of the ideas I wanted to incorporate was episodic memory.

People have patterns. Some of us repeatedly forget our ID cards. Some forget umbrellas whenever it rains. And Kanwal seems to have a particularly complicated relationship with his charger.

A useful packing assistant should be able to use relevant history instead of treating every trip as the first one.

For example, a memory record might indicate that the charger was forgotten on previous presentation trips. That information can then be used as a priority signal when generating future recommendations.

Instead of treating all items equally, CheckMate can give recurring problems more attention.

This is an important design goal: the assistant should become more useful by remembering relevant mistakes, not by making the user repeat the same instructions every time.

The memory feature is intended to support this workflow; its usefulness ultimately depends on the records available and how consistently item states and past events are captured.

The Voice Coach: Your Friend, But With Less Patience

I didn't want CheckMate to stop at a list of missing items.

When you're late for a train, you're probably not going to stand in your room and carefully read a dashboard. You need a quick reminder that gets straight to the point.

So I added a voice-coaching workflow using ElevenLabs.

The voice briefing turns the relevant packing information into a short, conversational message. The tone is deliberately Hinglish, because a reminder from a hostel friend shouldn't sound like a corporate notification.

Imagine this:

"Kanwal bhai, sun! Laptop toh bag mein daal liya, par charger check kiya? Nikalne se pehle switchboard dekh le. Delhi jaake phir battery ke bharose presentation mat chalana!"

A little rude? Maybe.

More memorable than "You have three tasks remaining"? I'd like to think so.

The application also includes a browser-based Web Speech API fallback for speech playback when the generated voice service isn't available.

The briefing is designed to use the trip details and critical items that need attention. A future version could make this even better by generating reminders directly from a verified room scan rather than relying on tracked states alone.

How the Pieces Fit Together

Here's the high-level workflow behind CheckMate:

       Room Reference / Spatial Scene
                    |
                    v
          Three.js Room Interface
                    |
                    v
          Interactive Item Tracking
                    |
                    v
       Trip Details + Item States
                    |
                    v
          Gemma Reasoning Workflow
                    |
                    v
       Contextual Packing Manifest
                    |
                    v
          ElevenLabs Voice Coach
                    |
                    v
          Final Exit Check

The 3D interface provides spatial context. The tracking layer represents the state of belongings. The reasoning workflow generates recommendations from the available context, and the voice layer turns important reminders into a spoken briefing.

The final exit check is the natural next step: before leaving, the user should be able to review the critical items and confirm that everything is actually packed.

The Tech Stack

Technology	What it does
React + Vite	Frontend application
Three.js	Interactive 3D room and spatial scene
Google Gemma	AI-assisted packing reasoning
Python + FastAPI	Backend for AI-related processing
ElevenLabs	Generated voice briefings
Web Speech API	Browser speech fallback
Render	Deployment and hosting

I wanted the experience to feel like one application rather than a collection of unrelated demos. The challenge was connecting a visual interface, an AI reasoning workflow, and voice interaction around one practical task.

The Privacy Question

A room photograph is personal.

It can reveal documents, photographs, medication, and details of someone's everyday life that they never intended to share.

That's why privacy matters for an application built around personal spaces.

Rendering a 3D scene in the browser doesn't automatically mean that the original photograph never leaves the device. Image uploads, backend processing, and external AI or voice services all need to be considered separately.

For a more mature version of CheckMate, I'd prioritise local image processing where practical, transparent disclosures about external service calls, and clear controls over what data is retained.

A personal assistant should be useful without making users uncomfortable about the information they're sharing.

What I'd Build Next

CheckMate is a starting point, and there are a few things I'd love to improve.

Automatic room understanding. I'd like users to upload a room photograph and have the application identify furniture and belongings, reducing the amount of manual scene setup.

More reliable spatial tracking. The system should distinguish an item's usual location from its current, verified location and its packing status.

Better long-term memory. Repeatedly forgotten essentials should influence future packing recommendations, while allowing users to correct or remove inaccurate records.

A smarter final scan. Before leaving, the user could review potential omissions identified by image recognition and confirm uncertain items.

More personalised briefings. The voice coach could adapt to trip urgency, preferred language, and the user's most important reminders.

These improvements would move CheckMate closer to the experience I originally imagined: an assistant that connects the room around you with the task you're trying to complete.

Why I Built It

I like the idea behind this challenge because it gives you a reason to build something for a specific person instead of inventing a problem for a portfolio.

CheckMate started with one annoying habit and a fairly simple question: why do our productivity tools keep asking us to remember everything when so much of what we need to do is connected to the physical world?

That question led me to experiment with 3D interaction, contextual AI reasoning, memory, and voice.

And honestly, I enjoyed building something where the technical decisions were connected to a problem I could actually observe.

Will a 3D room magically prevent someone from forgetting their belongings? Of course not. The tracking needs to be accurate, and the reminders need to be based on reliable information.

But I think there's something worth exploring in making digital assistants aware of more than just text on a screen.

For now, I'm hoping CheckMate makes Kanwal pause for five seconds, look at the switchboard, and save himself another charger-related crisis.

Because the real success metric isn't how many technologies I managed to put into one project.

It's whether my friend reaches his destination with his charger.

And whether the presentation survives past slide four.

Prize Categories

These are the categories relevant to the technologies used in CheckMate.

Featured categories — $200 per winner

Render — Best Use of Render: The application is deployed on Render.
Gemma — Best Use of Gemma: Google Gemma is used in the packing-reasoning workflow.

Partner categories — $100 per winner

ElevenLabs — Best Use of ElevenLabs: Voice generation powers the conversational packing-briefing experience.

These are the categories I'm targeting based on the project's integrations. Prize eligibility depends on the challenge's rules and the implementation demonstrated in the submission.

Try CheckMate

🌐 Live app: https://checkmate-ebvi.onrender.com/

💻 Open-source repository: https://github.com/shrutirai29/hacktober

If you could build a digital twin of one space in your life, what would you choose? Your desk? Your kitchen? Your ridiculously messy wardrobe?

And, more importantly, what would you finally stop forgetting?

Hacktoberfest #BuildForAFriend #ThreeJS #Gemma #ElevenLabs #WebDevelopment