This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 1 — Touch Grass.
What I Built
Pocket Naturalist is an offline field guide for walks. You photograph a plant, bird or bug, and Gemma 4 12B, running on my own laptop, tells you what it probably is. It also points out one detail to look at more closely, gives you a small quest to find next ("find a plant with compound leaves"), and writes a journal of your walk at the end.
There's no cloud and no account, and it works with no mobile signal: the phone talks to the laptop over a hotspot. You can snap photos live, or, if you don't want to carry a laptop around a garden (I didn't), take photos with your normal camera and add them from the gallery afterwards.
The rule I gave myself: the screen should be the shortest part of the walk. The answer is read aloud, and the phone goes back in your pocket.
What it does:
- 👀 Shows its reasoning: the features it saw, its best guess, and what else it could be.
- 🔁 Takes a second look when it's unsure: it describes the plant's structure first, then names it.
- 🔍 Mystery finds: when it's still unsure, it gives you one thing to check on the real plant ("blue flowers mean butterfly pea…") instead of a confident wrong answer.
- ✏️ "I know what this is": correct it, and the journal gives you the credit.
- 🌿 An Indian field guide: the fix that made it actually useful where I live. More on that below.
Demo
Two weeds Gemma got wrong before the field guide. Notice the "look closer" tips: they only describe what's actually in the photo.
It was "Sago Palm" before the field guide.
When Gemma isn't sure, it doesn't guess. It tells you what to check on the real plant.
The walk journal, written by Gemma from my 17 finds. It uses Gemma's own names, mistakes included (that's not teak), which is exactly why the app shows "could also be" and lets you correct it.
On my real 17-photo walk, here's what changed after the fix:
| Plant | Stock Gemma 4 | With the field guide |
|---|---|---|
| Castor | Virginia Creeper | ✅ Castor |
| Curry leaf | Chinese Fringe Tree | ✅ Curry leaf |
| Congress grass | Common Ragweed | ✅ Parthenium |
| Four o'clock flower | "Mexicaner" (not a plant) | ✅ Four o'clock flower |
| Black nightshade | Wild Jasmine | ✅ Black nightshade |
| Araucaria | Sago Palm | ✅ Norfolk Island / Cook pine |
| Red rose | Adenium | ✅ Rose |
Overall: 4 → 10 correct out of 15 photos where I know the right answer.
Code
gajanand27-05
/
pocket-naturalist
Offline field guide for walks: snap a plant or bird, local Gemma 4 identifies it and sends you on the next quest. Hacktoberfest 2026 Touch Grass.
🌿 Pocket Naturalist
An offline field guide for walks. Snap a plant, bird or bug on your phone. Gemma 4, running on a laptop in your backpack, tells you what it probably is, what to look at more closely, and gives you a small quest to find next. When you finish, it writes a journal of your walk.
You don't need mobile signal, a cloud API or an account. Your photos never leave your own devices.
Built for the Hacktoberfest 2026 Open-Source AI Challenge, Week 1: "Touch Grass".
How it works
phone (browser) ──same Wi-Fi or hotspot, no internet──▶ laptop: server.py ──▶ Ollama ──▶ gemma4:12b (vision)
- Start a walk. You get your first quest, read aloud by the phone's built-in offline voice.
- 📷 Snap a find, or 🖼️ add photos from your gallery later. You don't have to carry the laptop: take photos in the garden with…
It's one standard-library Python file plus one HTML page, with nothing to pip install:
ollama pull gemma4:12b
python server.py # prints a URL to open on your phone
How I Built It
The setup: phone browser → Wi-Fi hotspot (no internet) → server.py on my laptop → Ollama → gemma4:12b. The laptop is nothing special: an RTX 3050 with 6 GB of GPU memory and 16 GB of RAM. Gemma 12B doesn't fully fit, so it runs partly on the GPU and partly on the CPU, at about 30–40 seconds per photo. Every answer is forced into a fixed JSON format with Ollama's format option, so the app never has to guess what the model meant.
Day 1: making it honest
The first thing I learned is that a vision model can be confidently wrong. A house sparrow came back as "Jungle Babbler, 95% sure." A "fun fact" about hibiscus was made up, and so were some of the scientific names. So I:
- made Gemma list what it sees first, then rank three candidates, then choose;
- removed fun facts and scientific names and replaced them with "look closer" tips that may only describe what's visible in the photo;
- told the journal to use only the walk data. Its first draft described "the afternoon sun hanging low" on a walk it knew nothing about.
Then a vine on my railing came back as "Creeping Charlie". I went outside to look: it had bright blue flowers. Butterfly pea. That moment became a feature: when Gemma isn't sure, the app now gives you a clue to check on the real plant instead of a guess. That sends you back to the plant instead of back to the screen, which is the whole point of "Touch Grass".
Day 2: the real walk, and a bias I didn't expect
I walked my neighbourhood and took 17 photos of ordinary plants: castor, moringa, curry leaf, a four o'clock bush, roadside weeds. Stock Gemma got 4 of 15 right.
The interesting part was how it was wrong. For the castor plant, Gemma described "palmately lobed leaves, deeply serrated edges, large star-shaped leaf". That's a perfect description of castor. Then it named it "Virginia Creeper", an American vine. It did the same everywhere: moringa became "Purslane", curry leaf became "Chinese Fringe Tree", and a four o'clock flower became "Mexicaner", which isn't a plant at all.
Its eyes were fine. Its vocabulary was American.
Fix 1: a field guide in plain text
Real field guides are regional, so I wrote one: field_guide.csv, a list of 107 common Indian plants from gardens, kitchens, roadsides, waste ground and parks. Each entry has the features that tell it apart. For example:
Castor: shrub; very large star-shaped palmate leaves with 7–11 toothed lobes; reddish stems; spiny seed clusters
It goes into Gemma's prompt with three rules I learned the hard way:
- Sort it A–Z. In my first version, "Holy basil" was the first entry, and Gemma started calling half my photos holy basil.
- Plants only. Without that rule, the house sparrow became a "lark".
- "Never force a match." If nothing fits, give your own best name with lower confidence.
I also had to keep the prompt short enough to fit Ollama's default 4,096-token context. When I raised it, my 16 GB laptop ran out of memory and the app server was killed. Dropping the local-language names from the prompt (they stay in the CSV for people) fixed that.
Measured on 28 labelled photos (22 of mine, 6 from the internet):
| Correct (first answer) | My garden photos | |
|---|---|---|
| Stock Gemma 4 12B | 13/28 | 8/22 |
| + field guide | 20/28 | 16/22 |
I stopped tuning the list after one revision, so I wasn't just fitting it to my own test photos.
Fix 2: fine-tuning on a DGX Spark (and why it lost)
I also had access to an NVIDIA DGX Spark, so I tried the "proper" fix too: a LoRA fine-tune of Gemma 4 12B on 1,193 photos from DIMPSAR, an open dataset of Indian plant photos from Karnataka and Kerala (Pushpa B.R. & N. Shobha Rani, Data in Brief 2023, CC BY 4.0).
To keep the app's output format, each training answer was Gemma's own answer with only the species name corrected. I trained a LoRA on the language layers (rank 16, one epoch, about 70 minutes), merged it, and patched the weights into Ollama's official gemma4:12b file so it kept working with images.
| (measured on the DGX) | Stock | Fine-tuned |
|---|---|---|
| 28 test photos | 12/28 | 13/28 |
| DIMPSAR validation | 16/100 | 20/100 |
| Confidence | 0.6–0.95 | 1.0 on every photo |
+1 photo, and it became 100% sure of everything, which would switch off the app's "second look" and Mystery finds. Two lessons I'm taking with me:
-
Check the scale of numbers you copy into training data. 4% of Gemma's own answers said
"confidence": 85instead of0.85. The app had quietly clamped those, so nobody noticed until the fine-tuned model said 1.0 for everything. - In structured output, the loss is mostly the long text. Validation loss fell from 1.30 to 0.33, but name accuracy only went from 16% to 20%. The species name is a few tokens next to four descriptive sentences.
So the app ships with stock Gemma 4 plus the field guide. A 107-line text file beat 70 minutes on a DGX Spark.
Other models I measured
On my first 13 test photos: Gemma 4 12B 10/13, Qwen3.5 9B 9/13, MiniCPM-V 4.5 8/13, Qwen2.5-VL 3B 8/13 (it said "100% sure" about almost everything), and Gemma 4 12B QAT on llama.cpp 7/13. That last one was faster, but adding Gemma 4's new MTP drafter gave no speed-up on a 6 GB GPU, because when the model is split between GPU and CPU, checking the draft costs as much as it saves. Every model did fine on clean internet photos. Real, messy garden photos are where Gemma pulled ahead.
What still doesn't work
- Some plants are still wrong: moringa (called curry leaf; both have small leaflets), a ZZ plant, an Indian almond tree and one rose plant. That's why the app always shows "could also be", Mystery finds and the ✏️ correction.
- Birds and insects: the field guide doesn't cover them yet. My one bird test is still a miss.
- Its confidence is a hint, not a measurement. It says 85% for right and wrong answers alike.
- My test labels are careful judgements, not a botanist's. A couple of photos I left out of the score because I couldn't identify them myself.
I built this with Claude Code as my pair programmer, including the scoring scripts that made every decision above a measured one.
Why Does Open Innovation Matter?
For this project, open isn't a nice-to-have. Every fix above depended on it.
- It works where there's no signal. Gardens and parks are where mobile data is weakest. The phone only needs a local link to the laptop.
- My photos stay mine. Photos of my street and garden, with times attached, are location data. They never leave my phone and laptop.
- It costs nothing per photo. I ran hundreds of test photos through Gemma while building this, with no API bill and no rate limit. A closed API would have made that kind of measuring expensive.
- I could look inside and choose the fix. With an open model I could try prompts, swap models, quantize, add an MTP drafter, and fine-tune the weights myself, then measure which actually worked. The answer (a text file beat a fine-tune) is only something you can learn when you're allowed to try both.
-
Anyone can make it local.
field_guide.csvis a plain file. Someone in Kerala, Kenya or Kansas can replace it with their own region's plants, and the same open model becomes their field guide. That's the part I'm most excited about: open models plus open, editable local knowledge.
Prize Categories
- Best Use of Gemma. Gemma 4 12B (open weights) runs locally through Ollama and does all of the vision identification, the second-look reasoning, the quests and the walk journal. I also fine-tuned it (LoRA on a DGX Spark) and tested its MTP drafter and QAT version along the way.
Now close the laptop and go and see what's growing by your gate. 🌿
Top comments (0)