My walking app won't count a walk until my phone sees grass (and the photo never leaves it)

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

I keep telling myself I'll go for a walk. Then I don't, and nothing happens. Nobody checks, so skipping costs nothing. When my friends and I say "let's walk every evening", the same thing happens to all of us.

So for this week's theme I built something that checks. It's called Proof of Grass. You promise to walk a few days a week, and the only way to tick a day off is to go outside and take a photo of something real. An open AI model on your phone looks at the photo and decides if it counts.

What I Built

Proof of Grass is a web app you install from the browser, like a normal app.

You start by making a pact: "I walk 4 days a week" (you pick the number). Every day the app gives you three quests, like grass, a cat or the sky. Find any one of them. At least one is always easy: a tree, the sky or grass.

When you're outside, you tap "I'm outside: take the photo". It opens the camera directly, not your gallery. A couple of seconds later you get one of these:

• "✓ Walk counted, 6:56 pm, found: grass"
• "This looks indoors. Step outside!"
• "Looks more like a bird: get closer to the sky, grass or a cat."

That's it. One walk counts per day. Home shows your week as dots (2 of 4, say) and how many weeks in a row you kept the pact.

If you want company, you can make a group and share a 6-letter code. The group board shows everyone's week, who walked today, and whether each person kept or missed last week's pact. It's mostly for me, but this is the part for my friends.

The screen part is meant to be short. Open, snap, done, back to walking.

And the important bit: the photo never leaves your phone. It isn't uploaded and it isn't saved. When a walk counts, the server gets three things: the date, which quest, and the time.

Demo

Live app: https://proof-of-grass.onrender.com (best on a phone; in Chrome tap ⋮ and then "Install app"). It's on free hosting, so the first visit after a quiet spell can take up to a minute to wake up.

(These come from a local test copy of the app with made-up users, taken in a phone-sized browser window on my laptop. The grass photo is "A maple seed on the grass, Salt Lake City Utah" by Bobjgalindo, CC BY 4.0, and the laptop photo is CC0, both from Wikimedia Commons.)

Screenshots from my own phone, from this morning's walk, are further down in "How it went outside".

Code

hermit-01 / proof-of-grass

Proof of Grass

A walking pact you prove with a photo. You promise "I walk 4 days a week". Each day brings three quests (say, a tree, a bird and a dog). Step outside, photograph any one of them, and an open AI model checks the photo on your phone. The photo is never saved or uploaded.

Try it: https://proof-of-grass.onrender.com. It's on free hosting, so the first visit after a quiet spell takes up to a minute to wake the server.

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 1 ("Touch Grass").

How the photo check works

• The phone runs the vision half of OpenAI's CLIP ViT-B/32 (8-bit, about 85 MB, plus a 27 MB WebAssembly runtime) with Transformers.js, inside a Web Worker. It turns the photo into 512 numbers in about a second. After the first download it works offline.
• The text side is computed…

View on GitHub

React and Vite (an installable PWA), Express, MongoDB Atlas, and Transformers.js running CLIP on the phone. MIT licensed, 140 tests, plus an eval script that checks the photo rule against labelled photos.

How I Built It

Why CLIP and not a chatbot

I didn't need an AI that talks. I needed one question answered, fast, on a normal phone: is this photo grass (or a tree, or a cat), and was it taken outside?

For my last project I ran Gemma 3 1B in the browser. It was an 800 MB download and took 54 seconds to start answering on my phone, a POCO M6 Pro. Way too slow for "snap and go".

So this time I used CLIP ViT-B/32, OpenAI's open image-and-text model, through Transformers.js. CLIP doesn't write anything. It turns a photo into 512 numbers, and it turns a sentence like "a photo of grass" into 512 numbers too. If the two lists point the same way, the photo probably shows that thing.

The nice trick: the sentences never change. So I worked out the text side once on my laptop and saved it as a 117 KB JSON file. The phone only downloads and runs the vision half of the model, which is about 85 MB in 8-bit (plus a 27 MB WebAssembly runtime).

On my POCO, the model takes about a second per photo on the CPU. In the installed app, the check takes about 2.3 seconds, timed from sending the photo to getting the answer. The very first check also has to download the model, and on Wi-Fi that added about 10 seconds. On my laptop's browser the model took 1.1 seconds on the CPU and 0.13 seconds on the GPU.

"Is this outdoors?" was harder than "is this grass?"

My first idea was to compare the photo with two sentences, "a photo taken outdoors" and "a photo taken indoors". A photo of a road scored 8% outdoors. A road.

What worked was giving it more to choose from: 7 descriptions of outside ("a street outside", "a photo of a park", "a landscape photo"...) and 9 of inside ("a photo of a desk", "a photo of a computer screen", "a photo of a wall"...). The app adds up how much of the score lands on the outside ones. On my test photos, the outdoor ones scored 37% or more, and the indoor ones 25% or less. So the line is at 35%.

The one indoor photo that got past it: a flower on a windowsill, with trees outside the window. It scored 46%. Honestly, fair enough.

My tree didn't count

The first rule was strict: today's quest has to be CLIP's top guess.

Then I tested it on my phone with a real tree, and it failed. There was a building behind the tree, and CLIP said building 46%, tree 29%.

So I loosened it. Now a walk counts when one of today's quests is in CLIP's top 2 with at least 20%, and the photo looks at least 35% outdoors. My tree passed, and no test photo passed for the wrong quest.

This is the whole rule. It's plain code, not AI:

export function decide({ probs, ranked, pOut }, today, { outdoorMin = OUTDOOR_MIN, questMin = QUEST_MIN } = {}) {
  if (pOut < outdoorMin) return { pass: false, reason: 'indoor' };
  const passing = today.filter((q) => ranked.indexOf(q) < 2 && probs[q] >= questMin);
  if (passing.length === 0) return { pass: false, reason: 'quest', top: ranked[0] };
  return { pass: true, quest: passing.reduce((best, q) => (probs[q] > probs[best] ? q : best)) };
}

My selfie was 33% grass

CLIP's scores have to add up to 100%, split across everything on the list. My first list only had the quests. So when I took a selfie, all that probability had to land somewhere, and grass got 33%.

The fix was to add three things that are never quests: a road, a building and a person. I call them decoys. Their only job is to soak up the points a selfie or a street would otherwise give to "grass".

Then I went on my first real walk. I took a selfie with trees behind me, and the app said "✓ Walk counted, found: grass".

I ran the same photo (cropped from my screenshot) through the checker on my laptop. CLIP gave grass 22% and "a photo of a person" 20%. The decoy was there; it just lost by 2%. "A photo of a person" was too vague for a face filling the frame.

So I tried a few other ways of saying it, and "a selfie of a person" worked best. The same selfie now scores 74% person and 8% grass, and the app says "Looks more like a person: get closer to grass, a cat or the sky."

I also added two public selfies to my exam (next section). With the old wording, at least one of them passed at every setting I tried. With the new one, neither does, and the rest of the exam came out the same.

I gave it an exam

I didn't want to tune numbers by feel. So I collected 41 photos from Wikimedia Commons and labelled each one by hand: what's in it, and whether it's indoors. (Five of the ones I first grabbed turned out to be paintings, so they went. One "water" photo was really a construction site. The last two are the selfies from above.)

npm run eval runs the exact same rule on all of them, with four rules of its own:

• no photo may pass for a quest that isn't in it
• no room, office or screen photo may pass
• no selfie may pass
• at least 18 of the 20 quest photos must pass

main subject passed:        18/20 (bar 18)   missed: sky-2 sky-3
wrong-quest passes:         0 (bar 0)
room/office/screen passes:  0 (bar 0)
selfie passes:              0 (bar 0)
known limit (indoor plant/dog passed): NEG-plantin-1:flower

BAR MET

The two misses are sky photos where CLIP liked the trees or water in them more than the sky. They still pass, just for those quests instead.

I also tried 30 different settings: the outdoor line anywhere from 25% to 50%, and the quest line from 10% to 30%, in 5% steps. The outdoor line made no difference anywhere in that range. The quest line works from 15% to 25%: at 10%, one of the selfies gets through as "sky", and at 30%, one grass photo drops out. My 20% sits in the middle, not on a knife's edge.

No signal in the park

Parks don't always have signal. After the first download, the model is cached, so the check itself works offline. When a walk counts, the app saves it on the phone first, then tries to send it. If there's no signal (or the upload hangs for 15 seconds), you see "No signal right now, so it's saved on this phone", and Home shows "1 walk waiting to sync" until you're back online.

On my real walk I switched on airplane mode before taking the photo. The check still ran, the walk was saved on the phone, and when I switched airplane mode off, it synced.

The backend, and a bug I only found after deploying

The server is Express with MongoDB Atlas, and both the app and the API run from one Render service on the same address. That's on purpose: Brave blocks cross-site cookies, and logging in uses a cookie.

Atlas stores three kinds of things: users, groups and walks. A walk is just {userId, date, quest, checkedAt}, and a unique index on {userId, date} means the database itself refuses a second walk on the same day.

The login has a limit, 10 tries per 15 minutes, so nobody can guess passwords forever. It's supposed to count per visitor. After deploying, I added a temporary page that showed what the server thought my address was. It said 10.31.197.151, which is Render's internal balancer, not me.

Every request comes through three hops: me, then Cloudflare, then Render's balancer. I had told Express to trust one hop, so it took the balancer as the visitor. That meant every user shared one login limit, and one person mistyping their password could have locked everyone out. Trusting three hops fixed it. I wrote a test with the real three-hop header first, watched it fail, then fixed it. A fake header from a visitor sits further left in the list, so it can't get around the limit either.

What doesn't work (yet)

• The server can't check a walk. The photo never reaches it, so someone could fake a walk with a hand-made request. That's fine for a pact with yourself and a few friends. It's not an anti-cheat system.
• CLIP can be fooled. A printed photo of a tree held up outside would pass. The windowsill flower got through the indoor check.
• It catches selfies, not people. A photo of someone standing in a garden can still pass for the flowers or trees around them.
• The sky is tricky. In 2 of my 3 sky test photos, CLIP preferred the trees or water in them.
• The first visit is slow. Free hosting sleeps, so it can take a minute to wake up, and the first check downloads about 110 MB. Use Wi-Fi for that.
• It trusts your phone's clock for the date.

Why Does Open Innovation Matter?

For this project, a few reasons.

The photo stays on the phone. A cloud vision API would need every photo uploaded: photos of my street, the building I live in, sometimes my face. Every single day. With an open model running in the browser, the server learns the date, the quest and the time, and that's all.

It costs nothing per check. No API key and no bill per photo. Render's free plan and Atlas's free M0 cluster run the rest.

It works without signal, once the model is downloaded.

I could look inside it. This is the big one. CLIP gives me scores, not just a yes or no. That's why I could build an exam and pick the thresholds from evidence, try every combination of them, add decoys (they're just three more sentences), and precompute the text half so the phone only runs half the model. A closed "is this grass?" endpoint would give me an answer I couldn't measure, tune or fix.

How it went outside

I took the app on a walk this morning, around my college campus. Today's quests were grass, a cat and the sky.

I wanted to test two things in one go, so I switched on airplane mode and took a selfie with trees behind me. Here's what came back:

The offline part worked. The walk was saved on my phone, and Home showed "1 walk waiting to sync". When I switched airplane mode off, it synced.

The selfie part didn't: it should have said "Looks more like a person". That's the bug from the selfie section above, and it's fixed now.

On my POCO, a check takes about 2.3 seconds from sending the photo to getting the answer. The night before, I tried two photos indoors, and both got "This looks indoors. Step outside!"

Prize Categories

Best Use of MongoDB Atlas. Atlas is the database behind an app built on an open-weight model: users, groups and walks, with a unique index that enforces one walk per day. The server stores only the date, the quest and the time; the photo never gets near it.

Best Use of Render. One Render web service runs the app and its API from a single address, and deploys on every push. The AI itself runs on the phone, so Render is the part that keeps the pact and the group board in one place.

---

The AI checks the photo. Whether you actually go outside is still up to you. I just made skipping a bit harder to ignore.

Full disclosure: I built Proof of Grass with Claude Code as a pair programmer, and this post was written with AI from my notes, test logs and decisions. I directed and reviewed it, and every number in here comes from my own test runs.