Foliage Hunter: a Colorado leaf guide on a fine-tuned Gemma 3

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Foliage Hunter is a pocket leaf guide for peeping fantastic fall foliage. You walk, you find a leaf, you snap it. A Gemma 3 model fine-tuned on 18 Colorado species names the tree, Magpie TTS speaks a short guideline, and the find gets pinned to a winding trail of everything you have spotted this season. The screen is meant to be the shortest part of the walk.

It is for anyone on the Front Range who wants to know what they are looking at without stopping to read. The loop is deliberately short. One tap to shoot, a few seconds of a leaf wreath spinning, and the guide starts talking before the whole reply has even been synthesized. You keep walking. Asking the guide free-form questions about the current find (mic in, spoken answer out) is planned for a follow-up; this release identifies and speaks the find, and the mic path only works against the built-in demo guide.

The part that makes it a Colorado guide and not a generic plant app is a LoRA adapter I trained this week on iNaturalist photos of the species you actually meet here: quaking aspen, narrowleaf cottonwood, Gambel oak, boxelder, Rocky Mountain maple, chokecherry, serviceberry, peachleaf willow, plus the urban Front Range imports (green ash, Siberian elm, silver maple) and the fall-color classics (sugar maple, northern red oak, American sweetgum, ginkgo), and the three conifers that frame several trails (ponderosa, Douglas-fir, blue spruce).

Demo

Trail landing	Identification	Past finds

Below are six test finds. The clips are Magpie TTS output the phone played. DEV (like GitHub READMEs) does not allow inline <audio>, so the players live on the project Pages site: Hear the guide.

Find	Listen	What the guide said
Quaking aspen	Play 16 s	That looks like quaking aspen. Nice find. Leaf colors include ember, bronze, gold, and rust. Famous for fluttering leaves and Colorado gold in fall. Bark and buds feed wildlife, not a human snack.
Sugar maple	Play 14 s	That looks like sugar maple. Nice find. Leaf colors include amber, ember, rust, and gold. Famous for tapped sap made into maple syrup. Leaves and raw parts are not snacks.
American sweetgum	Play 16 s	That looks like American sweetgum. Nice find. Leaf colors include bronze, amber, rust, and ember. Famous for star-shaped leaves and spiky gum balls. Not a food plant.
Northern red oak	Play 16 s	That looks like northern red oak. Nice find. Leaf colors include scarlet, burgundy, rust, and amber. Famous for pointed-lobed leaves and strong fall red. Acorns are high-tannin and need processing.
Ginkgo	Play 16 s	That looks like ginkgo. Nice find. Leaf colors include amber, bronze, ember, and burgundy. Famous as a living fossil with fan-shaped leaves. Seeds need careful prep; the fleshy coat is foul and irritating.
Boxelder	Play 13 s	That looks like boxelder. Nice find. Leaf colors include amber, olive, bronze, and burgundy. A maple relative common in yards and washes. Grown for shade, not fruit.

There is no hosted URL on purpose. The app runs in Expo Go on your phone and talks to a model you host. The README's "Run it" and "Taking it outside" sections get you from clone to trail.

I took it outside

I took it out for about ten minutes in Castle Rock, Colorado, on the afternoon of October 10. Phone on cellular, Mac at home on the tailnet, guide server in the lab. Each find now also stores an on-device place name (city and state from the phone's geocoder, no Google Maps API), so the drawer can show where I was standing when I snapped the leaf. Two finds that stuck with me: American sweetgum (star leaves turning hard) and green ash on a street strip. Both pinned as Castle Rock, CO.

American sweetgum	Green ash

The first attempt failed in the field. Every photo came back as a 500 from the gateway, and the server log said PIL.UnidentifiedImageError. Real iPhones shoot HEIC and the simulator never did, so the guide server had never seen one. The fix was to re-encode on the phone with expo-image-manipulator (resize to 1280 px, JPEG at 0.82) before upload, which also cut the payload to a fraction of the original. After that the finds in the table above came in one after another, and the trail filled in as I went.

Code

EdyVision / foliage-hunter

Hacktoberfest Challenge Week 1 - Outdoor Activity Application with Open Weight Models Used

Foliage Hunter

Foliage Hunter is a pocket leaf guide for Colorado's fall. You walk, you find a leaf, you snap it. A Gemma 3 model fine-tuned on Colorado species names the tree, a guide voice tells you what you are looking at, and the find is pinned to a winding trail of everything you have spotted this season. The screen is meant to be the shortest part of the walk.

Why this exists

Built for the Hacktoberfest Open-Source AI Challenge: Week 1 on DEV (October 5–11, 2026). The week's theme was Touch Grass: build something with open-source AI at its core that gets people off the screen and into the world. Fall foliage was literally on the prompt's list of examples, and the Front Range was peaking. Entered under #hf26challenge and the Best Use of Gemma category.

Why open models matter here

• A regional model is only possible with…

View on GitHub

MIT. The adapter, the split manifests, and the held-out results are public on Hugging Face:

• Adapter: EdyVision/foliage-hunter-gemma3-12b
• Experiments: EdyVision/foliage-hunter-experiments

How I Built It

Two open models are in the live loop. Everything else is on-device code or a curated catalog.

Role	What runs
Name the leaf	Gemma 3 12B IT + LoRA (vision identify only)
Speak the guide line	NVIDIA Magpie TTS 357M + NanoCodec (GGUF)
Species notes	Curated catalog on the phone
Leaf colors	On-device JPEG palette
Place on the find	Phone GPS + platform reverse geocode
Free mic conversation	Planned (/v1/voice/turn); demo guide only in this release

  phone (Expo Go)                       home lab
 ┌───────────────────┐   Tailscale    ┌──────────────────────────────────────┐
 │ camera / library  │ ─────────────▶ │ LiteLLM  (alias: gemma-foliage)      │
 │ jpeg palette      │                │   └─▶ guide-server (FastAPI + PEFT)  │
 │ species traits    │ ◀───────────── │        Gemma 3 12B IT + LoRA r=16    │
 │ Magpie playback   │   audio/wav    │ nemo-speech (GGUF)                   │
 │ GPS + place name  │                │   Magpie TTS 357M                    │
 │ trail + history   │                │                                      │
 └───────────────────┘                └──────────────────────────────────────┘

The eyes: Gemma 3 12B IT plus a LoRA. I pulled 80 research-grade iNaturalist photos per species through the public API (Colorado place filter for the local tiers), 1,440 photos from 729 observations, and split them 80/20 stratified per species. Training is PEFT LoRA on all-linear modules, r=16, alpha=32, 600 steps, lr 2e-4, with the loss masked to the answer tokens only. The prompt is closed-set on purpose. A general VLM will happily say "maple" for a boxelder; the task I actually have is "which of these 18 is it":

"Identify the plant in the photo.\n"
f"Choose exactly one common name from this list: {choices}.\n"
"Reply with that common name only. No punctuation, no extra words."

The job runs through the Ray Jobs API on a DGX Spark in my home lab. Evaluation is generate-and-match on the 288 held-out photos: 85.1% (245/288). The adapter is 284 MB, base weights are not re-uploaded, and the model card carries the Gemma terms. Gemma is the specialist here: it names the leaf and nothing else. When the Gemma 4 training path on this GPU is unblocked, the LoRA job takes a new BASE_MODEL_ID and the app will not know the difference.

The voice: NVIDIA Magpie TTS, as GGUF. Speech out is Magpie TTS Multilingual 357M under NeMo-Speech.cpp on the same box. After identify, the app constructs a short spoken line from the species name, on-device palette colors, and a curated trait note, then streams it sentence by sentence so the first clip plays while the rest is still being synthesized. Clips save next to the find; reopening a past find replays with no network. A free conversation about the current find (speech in, spoken answer out) is next: that needs a real /v1/voice/turn on the guide path. This release does not ship that route, so the mic only answers through the built-in demo guide.

The app: Expo SDK 57, hexagonal. ports/ holds the interfaces (GuideClient, Speech, Camera, PaletteReader, FindHistory, Locator), adapters/ the Expo and HTTP implementations, application/ the use-cases, and one file (composition/wire.ts) reads env vars and assembles everything. That is what makes the models swappable; the identify flow never learns which TTS or which VLM it is talking to. Leaf colors are extracted on the phone from the JPEG bytes, place comes from the phone's location APIs, and without a server URL the app falls back to a built-in demo guide so the UI still works offline.

export interface Speech {
  speak(text: string, audio?: SpokenAudio, options?: SpeakOptions): Promise<void>;
  stop(): void;
  discardSaved(uris: string[]): Promise<void>;
}

The path outside: Tailscale. The phone and the Mac sit on the same tailnet. make start-ts makes Metro advertise the Mac's 100.x address so the Expo Go QR code works on cellular, and the guide URL points at the lab's tailnet host. No port forwarding, no public endpoint.

Things that bit me

• HEIC. Covered above. Conversion was necessary.
• License filters leak. My corpus builder accepts cc0, cc-by, cc-by-sa, cc-by-nc, and the manifests still contain 72 -nd and -nc-sa photos because the match was too loose. Images were private training only and are not redistributed, and the real per-photo breakdown is now in the README and the dataset card.

What is next

Free conversation with the guide about the leaf you are holding: mic audio in, a spoken answer that can describe what the model sees, with the current find as context. The UI affordance is already there; the server route and ASR wiring are the missing piece.

Why Does Open Innovation Matter?

A regional model only exists because the base is open. No hosted vision API is going to learn the difference between a Gambel oak and a northern red oak on a Colorado trail for me. With google/gemma-3-12b-it as an open base, a 284 MB adapter and 1,440 photos turn a general model into a local one, and I can publish the adapter and every split manifest so someone in Vermont or Oregon can retrain it for their species in an afternoon. An API with training capabilities can get us to this point, but for self-funded researchers and students, that can become costly as they learn their edge cases or restrictions they may need to set.

Photos and location never leave hardware I control. Every leaf photo is taken somewhere, usually near where I live. Inference runs in my own lab and the phone reaches it over a private tailnet. Place names for finds are resolved on the phone. No third-party Maps key, no per-image bill, no photo or location on a server I do not own. For more private applications such as health and wellness apps, this is everything to researchers. Before we publish sanitized datasets to the cloud, our data must live somewhere and be processed somehow.

Everything is swappable. The app speaks plain OpenAI-compatible HTTP. Point EXPO_PUBLIC_GUIDE_MODEL at a different alias and nothing in the client changes. When the device TTS sounded like a GPS unit, I swapped in Magpie behind the same Speech port in an afternoon. When the Gemma 4 training path on this GPU is unblocked, the LoRA job takes a new BASE_MODEL_ID and the app will not know the difference.

It costs nothing to run and it degrades gracefully. Marginal cost per identification is electricity. Color extraction is on-device, saved clips replay offline, and the demo guide keeps the UI alive with no network. The screen stays short because nothing in the loop is waiting on a vendor.

Prize Categories

• Best Use of Gemma. Gemma 3 12B IT is the model that identifies the leaf: closed-set vision over 18 Colorado species, LoRA published, served from my own hardware. Magpie handles speech. The short screen time on the trail is the product of that split.

Built solo. Thanks to the iNaturalist observers whose photos taught the model what a Colorado leaf looks like; every one of them is credited by observation ID in the experiments dataset.