GrassHopper

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Grasshopper turns a walk into a tiny side quest. Tap one button and a local Gemma model writes five things to find nearby, from easy ("a bicycle") to hard ("finding a rickshaw with grafiti").
Tell it where you are and the quests get local: in Jhansi, mine included a decorated
auto-rickshaw and a hand-painted shop sign.

Then the phone goes back in your pocket. When you spot something, snap it, and an
open-source vision model checks the photo right on your phone. No upload, no signal needed.
The screen is just a checklist; the walk is the point.

It's for anyone who opens their phone "for a second" on a walk and spent the whole walk glued to their phones. It makes every day walks around your city more enjoyable.

Taking it outside:
Ok that was from claude, i actually tried it, out in my walk, it worked pretty well, it was fun, to actually click photos of things i can actually find in my locality.

The finding a guy in a lungi, cycling around, was a quest i really enjoyed lol i went out with my sister, and we laughed so much finding A photo like that.

Yes, but its not perfect, CLIP got some wrong,, like one time it could really tell what the image uploaded represents, but that was only once.

Demo

grasshopper-ea0p.onrender.com

Code

Siddhi-sahu / Grasshopper

How I Built It

• Gemma 3 (4B), running locally with Ollama. It writes the quests. An Express + TypeScript server asks for exactly five, constrained to a JSON schema, so the output is always valid. Gemma tended to miscount the easy/medium/hard split, so the app assigns difficulty by position and lets Gemma focus on what to find.
• CLIP (clip-vit-base-patch32) via Transformers.js in a Web Worker. It checks photos in the browser. Each photo is compared against the quest, the other quests, and decoys like "a phone screen" or "the inside of a room", so a picture of your ceiling doesn't count.
• IndexedDB keeps quests and photos on the device.
• When CLIP isn't convinced, it says what it thinks it sees ("looks more like a cat to me") and you can count it anyway. The AI suggests; the human decides.

Why Does Open Innovation Matter?

• Your photos stay yours. A walk produces photos of your street, your neighbourhood, maybe your home. With an open model running in the browser, none of them get uploaded.
• It works with no signal. The photo checker downloads once and is cached. Out on a trail or in a dead zone, it still works. A closed vision API would fail exactly when you're outside.
• It costs nothing to run. Both models run on my own hardware: no API key, no per-photo bill, no rate limit.
• I could change the model's behaviour. I could see exactly how Gemma's output was shaped, constrain it to a schema, and swap hosts without rewriting the app.

My Agent Session

Planning Side Quest Walk for Hacktoberfest "Touch Grass"

You

All I need to know about this challenge (Hacktoberfest Open-Source AI Challenge: Week 1, theme 'Touch Grass'), and ideas on what should I build? Like someone built this - https://mymeowfolio.vercel.app; https://github.com/ThunderKhan/meowfolio

Agent

▸ WebFetch https://github.com/ThunderKhan/meowfolio

Input

https://github.com/ThunderKhan/meowfolio

Output

Meowfolio: private street-cat scrapbook. YOLOS-tiny detection + DINOv2-small embeddings via Transformers.js in the browser, IndexedDB storage, static Vercel hosting, no backend. Auto identity matching disabled after false positives; humans confirm identity.

Summary: build a new project with open-source AI at its core that gets people outside. Due Oct 11. Writing quality is weighted most. Explain why open matters (offline, private, free, swappable). Partner categories: Gemma, Render, ElevenLabs, Entire, etc.

What Meowfolio got right: fully in-browser local models, a clear privacy story, strong visual identity, honesty about limits.

Ideas: 1) Side Quest Walk: Gemma writes 5 small things to find on a walk, a local CLIP model verifies your photos. 2) Proof of Grass: a screen-time blocker that unlocks on a verified outdoor photo. 3) Leaf Ledger. 4) Trail Ears (BirdNET offline). 5) Run Club Hype Coach (Gemma + ElevenLabs). 6) Cloud Atlas Journal.

Recommendation: Side Quest Walk. Spend half the time on the build and half on the write-up, and test it outside.

Show more

You

I like Side Quest Walk. Later: use the user's location so Gemma suggests local things to find, easy to hard (e.g. in Jhansi, India: find a rickshaw). Save this session to DEV. I'll configure Entire myself, so tell me how. Initialize the project in ~/projects/hackathons after we pick a name; suggest some. Stack: TypeScript, Node, Express. Suggest the tech we need.

Agent

Saving this session to DEV, then suggesting names, a tech stack, and Entire setup steps.

4 of 4 messages

Building Grasshopper: local Gemma quests + on-device CLIP photo checks

You

I like the name Grasshopper, I've initialized it. What do you mean by 'plan with Gemma, check photos offline'? I like the stack. Running locally is the first priority. Frontend inspired by meowfolio: green, cutesy, Y2K. Web and phone. Does the Gemma host matter for deployment?

Agent

▸ Bash ollama list

Input

$ ollama list

Output

gemma3:4b (3.3 GB) already pulled

Scaffolding an npm workspace: server/ (Express 5 + TypeScript) and web/ (Vite + vanilla TypeScript).

▸ Write server/src/gemma.ts, quests.ts, index.ts

Input

server/src/gemma.ts, quests.ts, index.ts

Output

GemmaClient interface with an Ollama implementation; POST /api/quests asks gemma3:4b for 5 quests using a zod schema converted to JSON Schema and passed as Ollama's structured-output format.

▸ Bash curl -X POST localhost:3001/api/quests -d '{"place":"Jhansi, India"}'

Input

$ curl -X POST localhost:3001/api/quests -d '{"place":"Jhansi, India"}'

Output

5 quests in ~13s, including 'a yellow rickshaw reflected in water' (easy). Too contrived for 'easy'.

Tightening the prompt: one thing per quest, no reflections or combinations, and difficulty comes from rarity. The 'visual' field must be 2-5 words for the image classifier.

▸ Bash retest Jhansi + no place

Input

$ retest Jhansi + no place

Output

Jhansi: a decorated auto-rickshaw, a pile of ripe mangoes, a hand-painted shop sign... Generic run came back with a 1/2/2 split instead of 2/2/1.

Gemma orders quests well but miscounts the split, so I assign difficulty by position: easy, easy, medium, medium, hard.

▸ Write web/src/clip.worker.ts, verify.ts, store.ts, main.ts, style.css

Input

web/src/clip.worker.ts, verify.ts, store.ts, main.ts, style.css

Output

Transformers.js CLIP (Xenova/clip-vit-base-patch32) zero-shot in a Web Worker. Candidate labels = target + the other quests + decoys like 'a phone or computer screen' and 'the inside of a room'. Pass if it's the top label or scores >= 0.3. Photos are downscaled to 1024px and stored in IndexedDB. Green Y2K UI with pixel-font title bars. If CLIP isn't convinced, the walker can choose 'count it anyway'.

▸ Bash vite build

Input

$ vite build

Output

built; clip worker 546 kB + onnxruntime wasm

Plan with Gemma, check offline: quest generation needs the server once, but CLIP runs in the phone's browser and is cached after the first download. Photo checks work with no signal, and photos never leave the device. The Gemma host mostly doesn't matter; it sits behind an interface so it can be swapped.

▸ Bash node test.mjs (same CLIP model + pass logic)

Input

$ node test.mjs (same CLIP model + pass logic)

Output

cats/a cat: pass 0.897 | cats/a bicycle: fail, saw 'a cat' | tiger/a tiger: pass 0.999 | tiger/a patch of moss: fail, saw 'a tiger'

4/4 correct on sample images. Real street photos still need testing outside.

You

The title wraps weirdly: 'Grasshoppe' then 'r'. Put Grass on one line and hopper on the next.

Agent

▸ Edit web/src/main.ts + style.css

Input

web/src/main.ts + style.css

Output

<h1><span>Grass</span><span>hopper</span></h1> with each span on its own line

4 of 4 messages

Grasshopper part 3: location, hosted Gemma 4, and shipping to Render

You

Do auto-detect location and hosted Gemma; I have an API key. I'll deploy both backend and frontend to Render.

Agent

▸ Bash list Gemini API models containing 'gemma'

Input

$ list Gemini API models containing 'gemma'

Output

models/gemma-4-26b-a4b-it, models/gemma-4-31b-it

▸ Write server/src/gemma.ts (geminiGemma), server/src/place.ts, web 📍 button

Input

server/src/gemma.ts (geminiGemma), server/src/place.ts, web 📍 button

Output

GEMMA_PROVIDER switches between local Ollama and Google-hosted Gemma 4. GET /api/place rounds coordinates to ~1km and reverse-geocodes with OpenStreetMap Nominatim; only the city goes to Gemma. Express also serves web/dist so Render runs a single service.

▸ Bash curl /api/place?lat=25.45&lon=78.57

Input

$ curl /api/place?lat=25.45&lon=78.57

Output

{
  "place": "Jhansi, Uttar Pradesh, India"
}

▸ Bash POST /api/quests via hosted gemma-4-26b-a4b-it

Input

$ POST /api/quests via hosted gemma-4-26b-a4b-it

Output

First run: rejected because Gemma 4 wrote titles longer than the 80-char limit. Later runs: Gemini 500 'Internal error' and 90s timeouts.

Debugging hosted Gemma 4: it emits 'thought' parts before answering. thinkingBudget: 0 isn't supported, and thinkingLevel returns 400/500. A plain JSON-mode request with no schema answers in ~18s. Fixes: trim long strings instead of rejecting them, move the JSON schema and system prompt into the user message, filter out thought parts, and retry once.

▸ Bash POST /api/quests (Jhansi) after fixes

Input

$ POST /api/quests (Jhansi) after fixes

Output

200 in 45s (attempt 1 hit a 500; the retry succeeded): yellow auto rickshaw, chai stall, marigold garland, colorful sari, carved stone pillar

You

Deployed at https://grasshopper-ea0p.onrender.com. Looks like it's not working.

Agent

▸ Bash probe Render: /, assets, /api/health, /api/place, /api/quests

Input

$ probe Render: /, assets, /api/health, /api/place, /api/quests

Output

page 200, assets 200, health {provider: gemini}, place 'Jhansi, Uttar Pradesh, India', quests 502 after 46s

▸ Bash run generateQuests against both hosted models side by side

Input

$ run generateQuests against both hosted models side by side

Output

gemma-4-31b-it: Gemini 500 after 19s | gemma-4-26b-a4b-it: timeout after 90s

Hosted Gemma is failing on Google's side, minutes before the deadline. I added a hand-written starter pack, labelled in the UI as not AI-written, so a walk is never blocked. Each Gemma try is capped at 30s.

▸ Bash server with a bogus key → POST /api/quests

Input

$ server with a bogus key → POST /api/quests

Output

source: starter, 5 quests returned instantly

4 of 4 messages

Prize Categories

• Best Use of Gemma
• Best Use of Entire