TerraQuest: Put the Phone in Your Pocket, Let an Open-Weight Model Name the Animals and Plan

Aniruddha Shirke — Thu, 08 Oct 2026 20:46:18 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Most outdoor apps want your eyes on the screen. TerraQuest wants them on the trail.

TerraQuest is a walking companion that tracks your walk in the background, speaks a small mission to you, and names the plants, birds, insects and other wildlife you photograph. Tap Start, put the phone in your pocket, and walk. When something catches your eye, take one photo. The app identifies it, pins it on the map where you found it, and saves it to a journal you read after the walk, not during it.

How it keeps the screen short

Walks track themselves. GPS distance, active time and estimated calories run without you touching anything.
Missions are read aloud. Prompts like "listen for a bird for 10 seconds" are spoken, and you can answer by voice.
Rewards follow walking, not scrolling. XP and streaks come from walks and finds. A day with no walk doesn't count.
The journal is for later. Each walk saves its route, the photos of what you found, and a short report.
Nearby places are suggested from OpenStreetMap, with walking directions.

Why walking

Regular brisk walking is one of the simplest habits you can build. It is linked to better heart health, mood, sleep and energy, and it needs no gym or equipment. The hard part is keeping it up. TerraQuest gives a walk a reason: a bird to listen for, a plant to find, a streak to keep. Turning "I should walk" into "I wonder what's around the corner" is the whole idea. (Calories are estimates from GPS and time, not medical data.)

Many people walk past plants and birds they've never learned the names of, and learning them usually means stopping to search on a screen. TerraQuest is for anyone curious about nature who doesn't want to look down at a phone, from families on a weekend stroll to people building a daily walking habit.

Demo

Live app: [https://terra-quest-hazel.vercel.app/]

The first scan downloads the on-device model once. After that, scanning works without a connection.

Code

https://github.com/Aniruddhashirke/TerraQuest (MIT licensed)

Aniruddhashirke / TerraQuest

🌿 TerraQuest

A mobile-first outdoor companion. Go for a walk, scan the plants, birds and insects you meet, and watch your journal, map and stats fill up. It's a PWA that installs on your phone and works offline.

Features

Sign in with Google or your phone number (SMS code), or continue as a guest.
Home dashboard: nearest nature spot, Today / Week / All time stats, weekly goal ring with streak, daily mission and recent finds.
Walk tracking: GPS route, distance (Haversine with outlier filtering), active time and estimated calories. Steps show "unavailable" rather than being faked.
Scan and identify: take a photo of a plant, bird or insect. A cloud vision model names it…

View on GitHub

How It Works

start walk
   |
   v
GPS route + timer (runs in the background)
   |
   v
photo of a plant, bird, insect or animal
   |
   +-- online? --> cloud vision model --> name + scientific name
   |                    |
   |                    +-- plant? --> Pl@ntNet second opinion
   |
   +-- offline, slow, or error --> CLIP on the device (Transformers.js)
   |
   v
result card --> saved to the journal with photo, GPS and time
   |
   v
map pins + route --> next mission --> stop walk --> report

1. Identification on the device (open model)

The open-weight model is CLIP ViT-B/32 (Xenova/clip-vit-base-patch32), running in the browser through Transformers.js. It is a zero-shot classifier, so there is no training and no server. The photo is scored against about 70 nature labels, which I tuned to birds, plants and insects common on Indian walks, such as bulbul, myna, banyan and tulsi.

2. A cloud upgrade, and an honest fallback

When the phone is online, the photo also goes to my own serverless function, which asks a cloud vision model for a common and scientific name. The API key never reaches the browser. If the call fails, times out after 25 seconds, or there is no signal, the app drops back to CLIP without an error screen:

if (navigator.onLine) {
  try { list = await identifyOnline(photo); source = 'cloud' } catch { /* fall back */ }
}
if (!list) { list = await classify(photo); source = 'device' }

The result card always tells you which path produced the answer.

3. Missions: rules first, AI second (Gemma)

A deterministic engine picks the next mission from your context: how far you've walked, how many plants and birds you've found, and a cooldown so you don't get the same mission twice. Gemma (open weights, running in the browser over WebGPU, [model variant]) only rewords the text and writes the optional walk note after your walk. If it is offline, slow, or returns something odd, a built-in template is used. The prompt says never to invent a species.

4. Everything is stored on the device

Walks, finds, missions and your profile live in IndexedDB through Dexie, so the app shell and your saved data keep working with no signal.

Other pieces: React 18 and TypeScript, Vite with a PWA service worker, Leaflet with OpenStreetMap and Overpass for nearby places and routes, Wikipedia for species summaries, the browser speech APIs for voice, and Firebase Auth for optional sign-in (guests work fine). Vitest covers the mission rules, geo math, stats, journal reports and identification parsing.

Why Open Matters Here

A trail is the worst place for an app that needs a network.

It works where the walk happens. With CLIP on the device, a scan doesn't depend on cell coverage. A cloud-only identifier stops at the edge of signal.
Photos stay on the phone on that path. That matters for family walks and private land.
It costs nothing per scan. There is no per-image bill for a free walking app.
It's hackable. The model is one constant and the label list is a plain array. Anyone can fork it, swap the model, or add their own region's species.

Where the closed model was better: it gives sharper common and scientific names, while zero-shot CLIP is coarser and will say "kingfisher" where a cloud model might name the exact species.
Where open won: it's the only version that works with no signal, keeps photos private, and costs nothing. That's why the cloud model is an optional upgrade, not the foundation.

What Went Wrong, and What I Changed

npm install failed on a second machine. Transformers.js pulls in sharp and onnxruntime-node, whose native downloads fail on Windows and restricted networks. The browser app never uses them, so I set ignore-scripts=true in .npmrc and stopped copying node_modules between machines.
A zero-shot classifier always has to pick something. Show it a phone screen or a face and it will confidently name a bird. I added decoy labels like "phone screen", "person" and "building" so non-nature photos land there instead.
It can still be wrong. In testing, a colourful coleus leaf came back as "fern" at 69%. Confidence is shown on every result, and the cloud model and Pl@ntNet exist to correct cases like this.
Model output can't be trusted. The cloud answer is parsed as JSON and validated: categories are whitelisted, confidence is clamped between 0 and 1, text lengths are capped, and anything malformed triggers the CLIP fallback.
Plant identification can hurt people. The prompt tells the model never to say whether anything is edible, poisonous or safe to touch.
Adding Gemma broke the production build. Its WebGPU runtime is about 6 MB and the PWA precache limit is 2 MB. Instead of raising the limit, I made it a lazy chunk, cached only the first time someone taps Write a walk note. The main app stayed small.
Sign-in worked locally but failed once deployed with auth/unauthorized-domain. Firebase only allows sign-in from authorized domains, so I added the live domain in the console. Guest mode keeps the app usable regardless.
Browsers have no step sensor. Rather than faking a step count, the app shows steps as unavailable and uses GPS distance for missions.
Public OpenStreetMap servers are fair-use. Nearby results are cached for 30 minutes so the app doesn't hammer them.

What's Next

Bird sound identification with BirdNET, an open model, so you can identify a bird without looking at the screen at all.
Separate local data for each signed-in account.
Restoring an in-progress walk after a reload.

Map data © OpenStreetMap contributors. Species summaries from Wikipedia. Plant second opinions from Pl@ntNet. Built with React, Vite, Dexie, Leaflet, Firebase, Transformers.js and Gemma.

Prize Categories

Week 1 Theme: Touch Grass (primary category)
Best Use of Gemma
General Hacktoberfest Open-Source AI Challenge

DEV Community: Aniruddha Shirke