DEV Community

Cover image for Grassdex: Building a 100% Offline Nature Dex & Touch-Grass Bingo with Local Gemma
Farhan
Farhan

Posted on

Grassdex: Building a 100% Offline Nature Dex & Touch-Grass Bingo with Local Gemma

#devchallenge #hf26challenge #gemma #opensource

TL;DR: Grassdex is a zero-cloud local web app where your phone camera snaps nature over a private hotspot to your laptop, where an open-weight vision model (Gemma 4 via Ollama) names the species and checks off a model-generated 3x3 touch-grass bingo card. The screen is designed to be the shortest part of the walk (under 10 seconds).

The Itch

Most outdoor and nature identification apps turn you into a screen-gazer: you walk into the woods, take a photo, wait on a cloud server to respond, get bombarded by notifications or social feeds, and spend ten minutes staring at your phone instead of the trees.

I wanted the exact opposite: an app that gets you outside, does its job in 10 seconds flat, and immediately tells you to put your phone away. And it had to work deep in the park or on a trail with zero cell reception, keeping all personal photos strictly private.

What I Built

Grassdex combines two things:

  1. A Personal Nature Dex: A phone camera snap sent over local Wi-Fi/hotspot to a laptop running local vision inference. It classifies plants, insects, birds, and fungi, returning common names, confidence scores, and concise fun facts.
  2. Touch-Grass Bingo: A dynamic 3x3 scavenger-hunt card generated by Gemma based on your setting (park, trail, campus, garden, or street). Instead of running multiple expensive passes, Grassdex uses a single model call to identify the subject and verify whether it fulfills your selected bingo square.

When you score a line of three, the screen displays: "Bingo! Now put the phone away."

In Action

Touch Grass Bingo Mobile UI

Nature Dex Log View

Prize Categories

Best Use of Gemma ($200)

Grassdex runs Google's open-weight Gemma 4 (gemma4:e4b, 7.5B Q4_K_M with CLIP vision projector) entirely on local hardware using Ollama:

  • Vision Identification: Processes raw multipart camera images locally without external cloud APIs.
  • Structured JSON Synthesis: Employs constrained JSON schema parsing to reliably extract structured common names, classification categories, and facts.
  • Creative Scavenger Generation: Generates balanced 9-square outdoor bingo prompts with strict safety boundaries (no touching, picking, or approaching wildlife).
  • Single-Call Verification: Merges identification and verification reasoning into a single inference pass, cutting laptop latency in half.

Overall Hacktoberfest Week 1 Submission (Theme: Touch Grass)

Grassdex is built deliberately to maximize outdoor presence and minimize screen attachment. Every interaction is designed to take less than 10 seconds before returning the user's attention to the physical world.

How It Works

The architecture is intentionally minimal: Flask + a single vanilla HTML page + local JSON files.

Phone Camera (Browser) ──LAN HTTP──> Flask App (Laptop) ──localhost:11434──> Ollama (gemma4:e4b)
                                             │
                                             └──> data/grassdex.json, data/bingo.json, data/photos/
Enter fullscreen mode Exit fullscreen mode

The 10-Line Local Call

By combining the identification prompt and the bingo verification prompt into a single payload, a single call evaluates both:

res = ollama.chat(
    model="gemma4:e4b",
    messages=[{"role": "user", "content": prompt, "images": [photo]}],
    format="json",
    keep_alive="30m",
    options={"temperature": 0.1},
)
Enter fullscreen mode Exit fullscreen mode

Keeping the model warm (keep_alive="30m") and running with 100% GPU offload on an NVIDIA RTX 3050 (allocating 4.77 GB VRAM) delivers warm response latencies of 1.4s to 5.6s for identification, and 10s to 16s for full combined bingo verification.

Why Open Mattered

Four concrete reasons why open-weight models beat closed APIs for this project:

  1. Zero Signal, Full Functionality: When you walk under heavy tree canopy or through a nature preserve, cellular data drops. Cloud-hosted AI stops working. Because Gemma runs on the laptop in your backpack, Grassdex never loses connection.
  2. True Location & Photo Privacy: Raw phone photos often contain EXIF location metadata, family members, or private surroundings. In Grassdex, photos never traverse the public internet. They stay on the laptop disk.
  3. Zero Cost Per Snap: Cloud multimodal API calls cost money with every upload. Running local open-weight models means 1 snap or 1,000 snaps costs exactly $0.00.
  4. Complete Local Control: The model, prompt parameters, and fallback pools are entirely customizable. If another vision model is released, it can be swapped with a single line change in app.py.

Where Closed Would Have Won

Closed flagship APIs (like Gemini 1.5 Pro or GPT-4o) still have an edge in fine-grained botanical taxonomy—such as distinguishing between subtle subspecies of lichen or obscure caterpillars without specialized prompting. But for an interactive outdoor companion that encourages you to touch grass, local speed and offline resilience far outweigh fractional taxonomy advantages.

Taking It Outside (Field Test & Honest Numbers)

I took Grassdex out for an autumn park walk with my laptop in my backpack and my phone on a local hotspot. Over a 45-minute outing, screen time was strictly bounded: pull out the phone, frame a subject, tap snap, pocket the phone while the laptop runs inference over LAN, check the result, and keep walking.

Total screen-on time was under 4 minutes across the entire 45-minute walk (< 9% of total time).

The Numbers

All figures are traced to docs/metrics/field-test.md generated by scripts/analyze_log.py:

  • Snaps logged: 29 (17 rated across plants, bugs, birds, fungi, and controls)
  • Strict accuracy (exact species/subject): 65% (11/17, Wilson 95% CI: 41% – 83%)
  • Lenient accuracy (right genus or close subject): 94% (16/17, Wilson 95% CI: 73% – 99%)
  • JSON Parse Failures: 0 of 29 (0%)
  • Abstentions: 1 of 29 (a ceramic mug control, where Gemma reported 10% confidence and noted "No living thing present").
  • Inference Latency: Median 16.1s, p90 18.4s, min 9.7s, max 36.3s.
  • Touch-Grass Bingo: Completed 3 squares in a vertical column ([1, 4, 7]), triggering the "Bingo! 3 In A Row" celebration.

The Big Discovery: Uncalibrated Confidence

One critical finding: Gemma's raw confidence scores are uncalibrated.

  • Mean reported confidence when right: 96%
  • Mean reported confidence when close: 93%
  • Mean reported confidence when completely wrong: 95%

When Gemma 4 was dead wrong, it was just as confident as when it was right. This proved why having a human-in-the-loop review system (Right / Close / Wrong) and a player Manual Override button on the Bingo card are absolutely essential for real-world use.

The Best Moment

The vision reasoning on negative bingo checks was surprisingly perceptive. When I snapped an autumnal red tree branch for the square "Green foliage under dappled sunlight", Gemma didn't just reject it blindly—it explained:

"The dominant foliage colors are red and orange, indicating autumn rather than green summer growth."

The Worst Failure

Scale and small organisms. When photographing tiny wood ants on a fallen branch, the ants were only a few dozen pixels wide. Gemma completely bypassed the ants and proudly identified:

"Forest Trees (plant, 95% sure)"

It wasn't a hallucination—there were indeed trees in the background—but it highlights the limitation of wide-angle phone cameras when identifying miniature macro wildlife.

What Broke & Lessons Learned

  • Interactive TTY Pitfall: When scripting Ollama CLI commands in PowerShell, non-piped calls lingered in interactive chat mode. Switching to piped streams or direct Python API calls solved it cleanly.
  • Hallucinated Petal Counts: Small models struggle with precise counting ("find a flower with five petals"). We tuned the bingo card prompt to focus on recognizable colors, shapes, textures, and broad living categories.
  • Thermal & Battery Balance: Running local vision inference on an RTX 3050 pulls ~45W on battery. Setting keep_alive prevents repeated reload overhead while conserving power during the walk.

Safety and Limits

  • Grassdex is NOT a foraging guide and should NEVER be used for edibility, medical, or wildlife-safety decisions.
  • Do not touch, pick, disturb, or consume any plant, insect, or fungus.
  • Always observe wildlife from a respectful distance.

Try It

Four commands to run it locally:

git clone https://github.com/YTxFSGAMERz/hacktoberfest-Grassdex.git
cd hacktoberfest-Grassdex
python -m venv .venv; .\.venv\Scripts\activate
pip install -r requirements.txt
python app.py
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:5000 (or set $env:HOST = "0.0.0.0" for private phone hotspot access).

Credits & Disclosure

  • Model: Google Gemma 4 (gemma4:e4b)
  • Runtime: Ollama
  • Backend: Flask
  • Commits: Started Oct 6, 2026. All commits fall strictly inside the Hacktoberfest Week 1 challenge window.

I used an AI coding agent for boilerplate assistance and line edits. The project conception, architecture, field testing, and numbers are authentically mine.

Top comments (0)