Touch Grass Quest: a scavenger hunt game powered by on-device AI

Albert Benedict Rustandi — Mon, 12 Oct 2026 03:51:10 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Most apps want you to keep looking at them. I wanted to make one that sends you away. So I turned touching grass into a game: quests, XP, a plant that grows, and an AI that checks your photos without anything leaving your phone.

Touch Grass Quest gives you five outdoor quests every day ("find a pinecone", "spot a puddle") plus one shared quest that everyone gets. You open the in-app camera, take a photo, and an on-device model decides whether you really found it. Finds earn XP that grows a plant from a seedling to a blossoming tree. Streaks multiply your XP, and every find becomes a stamp in your Grass Dex.

The screen is meant to be the shortest part of the experience:

Live camera only. There's no gallery upload, so you actually have to go somewhere.
A cooldown between quests, so you can't farm XP from the couch.
An away bonus: XP for the time the app spent in your pocket.
Collecting, not gambling. Rewards are stamps and a growing plant, with no random pulls, on purpose.

It's for anyone who wants a small, playful reason to leave the house.

Demo

Live app: https://albertbenedict.github.io/touch-grass/

A quest written on-device by Qwen2.5 0.5B, with the XP it's worth.

Taking it outside

A pinecone passed at 90% confidence, a pile of pebbles passed as "a stone" at 93%, and a photo of my laptop screen was rejected. The away bonus showed up after I locked my phone for a few minutes. The app also reopened with airplane mode on after the first load, and that first load took only a few seconds on my Wi-Fi, by my rough count.

Pebbles pass as "a stone" at 93%, for +24 XP (the base reward with my x1.2 streak multiplier). The next quest is already dealt, and the camera is locked for a 3-minute cooldown.

The quest was "find a bush". I photographed leaves up close, and the model said "leaf" at 93%, so it didn't count.

Every find becomes a stamp in the Grass Dex. Unfound ones stay locked.

The check isn't perfect, though. I tried a real tree and it failed twice: once from far away, and once up close. In the failure screenshot above, a close-up of foliage was read as "leaf" instead of "bush". The model seems to want the subject at a medium distance, framed clearly. That was the most useful thing I learned from testing, and it's the first thing I'd fix.

Code

albertbenedict / touch-grass

Touch Grass Quest

A scavenger-hunt game that sends you outside. Every day you get outdoor quests ("find a pinecone", "spot a puddle"), take a photo with the in-app camera, and an AI running on your phone checks whether you found it. Finds earn XP that grows a plant, and every find becomes a stamp in your Grass Dex.

Built for the Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass.

Live app: https://albertbenedict.github.io/touch-grass/

How it works

Daily quests: five a day (2 easy, 2 medium, 1 hard) plus one shared quest that everyone gets on the same day.
Live camera only: there's no gallery upload, so you have to go somewhere. (Append ?dev to the URL to enable file upload for desktop testing.)
Photo check: CLIP compares your photo against the quest target and a set of "none of the above" labels. The target must be the top guess with…

View on GitHub

How I Built It

The whole app is vanilla JavaScript running in the browser with transformers.js. There's no backend, no account, and no API key.

Photo check: CLIP (Xenova/clip-vit-base-patch32) does zero-shot classification. It compares your photo against the quest target plus a set of "none of the above" labels, and the target has to be the top guess with enough confidence.
Quest text: a small open LLM writes each quest sentence. The app tries Gemma 3 270M first, then falls back to Qwen2.5 0.5B. On my setup, Qwen2.5 0.5B is the one that actually loaded.
Offline: a service worker caches the app, and transformers.js caches the model files after the first download.
Progress: XP, streaks, the plant, and the Grass Dex all live in localStorage on your device.

Two things went wrong

CLIP accepted a photo of a tiger as "a leaf." In my test, the app called the tiger "a leaf" at 46% confidence and gave me XP. CLIP always picks the closest label from a short list, so any photo gets forced onto something. I added generic "none of the above" labels (an animal, a person, a screenshot, and more) and a confidence threshold.
The small quest writer wrote quests about "ancient Egyptian trade routes." Now its output is validated, and if it drifts, the app uses a plain template instead. Small models need guardrails.

Limitations

It runs on the honor system. A determined person could still photograph a screen, and real anti-cheat would need a server, which I chose not to add for now.
The photo check is picky about framing. A tree photographed from too far or too close failed in my tests.
A fixed shared quest can be impossible for some people. Not everyone lives near a lake.
The first load downloads large model files.

Why Does Open Innovation Matter?

Photos never leave your phone. The only network traffic is downloading the app and the model files. A closed vision API would mean uploading every picture you take outdoors to someone else's server.
It works with no signal. The place you need this app is a trail, and a cloud API can't follow you there. After the first load, mine doesn't need the internet.
It costs nothing to run. It's a static site, so there's no per-photo bill and no API key to leak.
Models are swappable. When Gemma didn't load for me, falling back to Qwen was just the next line in a list, and I got a working app instead of a broken one.

The honest tradeoff: small open models are less capable than big closed ones, which is exactly why I needed the guardrails above. For a game where the whole point is that nothing leaves your phone, I'd make that trade again.

Try it on your phone: https://albertbenedict.github.io/touch-grass/. Allow the camera, go find a leaf, and tell me in the comments what your first stamp was.

DEV Community: Albert Benedict Rustandi

Touch Grass Quest: a scavenger hunt game powered by on-device AI

What I Built

Demo

Taking it outside

Code

albertbenedict / touch-grass

Touch Grass Quest

Touch Grass Quest

How it works

How I Built It

Two things went wrong

Limitations

Why Does Open Innovation Matter?