Leaf & Gill: an offline plant and mushroom identifier that knows when it's guessing

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

The best places to hike have one thing in common: no signal. That's exactly where you find the plant you want to name, or the mushroom you really shouldn't touch. It's also exactly where every identification app stops working, because they all send your photo to a server.

So I built one that doesn't.

What I Built

Leaf & Gill identifies wild plants, trees and mushrooms offline, anywhere in the world. You open it once with Wi-Fi, and it saves an open-source AI model to your phone, like downloading offline maps. After that it works in airplane mode, at the top of a mountain or at the bottom of a forest.

You snap a photo, glance at the answer and put the phone back in your pocket. That's the whole interaction, so the screen is the shortest part of the walk.

For every photo you get:

• The top 5 matches out of 11,896 of the most-observed plants and fungi on Earth. Never a single verdict.
• A trust meter (Very likely, Likely or Not sure) that has learned when the model is guessing.
• Red warnings for deadly look-alikes. The warning appears when a toxic genus is among the matches, and also when it's a known look-alike of the top match. In testing, a photo of poison hemlock was once identified as coriander. The red banner still warned about hemlock.
• "Often confused with", computed from the model itself rather than written by hand.
• A short description from Wikipedia, with every sentence about eating removed.
• A life list of your finds, which never leaves the phone.

It never tells you something is edible. Every mushroom result says: never eat a wild mushroom based on an app. That isn't just a disclaimer. In a 2023 study of mushroom-poisoning cases in Australia, the best identification app got the species right only 49% of the time. Safety shaped every design decision below.

Demo

Try it: https://mohamedaminehamdi.github.io/leaf-and-gill/
On an iPhone, open it in Safari, choose Share → Add to Home Screen and open it from the icon. The first launch downloads the model (609 MB) in the background. After that, turn on airplane mode and it still works.

No plant photo handy? Tap one of the example photos (fly agaric, poison hemlock, chanterelle, dandelion). They answer instantly, even while the model is still downloading, so you can see the warnings and the trust meter in action.

Deadly look-alikes	A classic mix-up

On the right, a chanterelle is only rated Likely, not certain, and the app warns about the toxic jack-o'-lantern that chanterelle hunters confuse it with.

Code

mohamedaminehamdi / leaf-and-gill

Identify wild plants, trees and mushrooms offline, anywhere. Open-source AI (BioCLIP 2) running on your phone.

Leaf & Gill

Identify wild plants, trees and mushrooms offline, anywhere in the world. An open-source AI model runs inside your phone's browser. No signal, no account, no install, no photo ever uploaded.

Try it: https://mohamedaminehamdi.github.io/leaf-and-gill/ On an iPhone, open it in Safari and choose Share → Add to Home Screen The model is downloaded once (~600 MB, like offline maps) and works in airplane mode after that.

Snap	Deadly look-alikes	Honest uncertainty

What it does

Point the camera at a leaf, a tree or a mushroom, then put the phone back in your pocket.

• Top 5 matches from 11,896 of the most-observed plants and fungi on Earth, never a single verdict.
• A trust meter (Very likely / Likely / Not sure) that knows when the model is guessing.
• Red warnings for deadly look-alikes, from a hand-written list of toxic genera
  • When one is among the matches.
  • When…

View on GitHub

How I Built It

The model: BioCLIP 2, inside the browser

BioCLIP 2 is an open model from the Imageomics Institute, released under the MIT licence. It was trained on 200 million biology images across the tree of life. It works like CLIP: it turns a photo and a species name into vectors, and a photo lands closest to its own name. That means I don't need a classifier for a fixed list. I can match photos against any list of names.

1. Export. I exported BioCLIP 2's image half to ONNX and converted it to fp16, which comes to 609 MB. Before uploading it to Hugging Face, I checked it against the original PyTorch model on real iNaturalist photos: the minimum cosine similarity was 1.00000.
2. Species list. I pulled the 11,896 most-observed plants and fungi worldwide from GBIF, counting human observations only. The deadly species are force-included however rare they are, because the app must always be able to warn about a death cap.
3. Names. I turned every name into a vector once, at build time, using BioCLIP 2's text half. The result is an 18 MB file the phone downloads with the app.
4. On the phone. onnxruntime-web runs the image half on the GPU through WebGPU, falling back to WebAssembly. One photo becomes one vector, which is compared against 11,896 names. That takes about 150–300 ms on a laptop GPU.

The app itself is one HTML file and one JavaScript module. There's no framework and no build step. A service worker caches the app, and the model sits in the browser's Cache Storage.

No generated text, anywhere

Writing field-guide cards with an LLM was tempting. For a tool that talks about poisonous mushrooms, it was also the wrong call. So nothing in the app is generated:

• Names come from GBIF.
• Descriptions come from Wikipedia, credited. A filter drops every sentence that mentions eating, taste, cooking or foraging.
• "Often confused with" comes from the model itself. For each species, I find the most similar species from a different genus, because cross-genus mix-ups are the ones that hurt people. Some of what fell out:
  • Wild carrot → poison hemlock
  • Morel → false morel
  • Chanterelle → false chanterelle
  • Wild garlic → lily of the valley, autumn crocus
• Danger warnings come from a hand-written list of toxic genera, with no AI involved.

A small test suite runs in CI and fails the build if edible, tasty or delicious appears anywhere in the shipped data. It caught three things I'd never have spotted by eye:

1. GBIF's common name for Musa acuminata: "Edible banana".
2. Wikipedia's "the delicious milk cap".
3. My own warning text, which said one plant "resembles edible members of the carrot family".

The trust meter: TabPFN learns when to doubt

A raw model score is a bad guide: it says "100%" far too often. So I measured how often it's actually right.

The test set. I collected 3,000 iNaturalist research-grade photos observed after June 2025, newer than BioCLIP 2's training data, so it can't have memorised them.

	Plants	Fungi
Right species as the top match	85.3%	85.4%
Right species in the top 5	97.4%	96.6%

Figures are for photos of species in the pack.

Then the real-world catch: 22% of the photos were species outside the pack. On a real trail you'll photograph things the app doesn't know, and a good app should say "not sure" when that happens.

The meter. TabPFN is an open tabular foundation model. It learns from a small table in one pass, with no training run. I gave it four numbers per photo, labelled with whether the top match was right:

• the top match's probability
• its lead over the runner-up
• how close the photo is to the best name at all
• whether it's a plant or a fungus

Results. I tested it on 1,219 held-out photos:

	Raw model score	TabPFN trust meter
Separates right from wrong (AUROC)	0.817	0.843
Calibration (Brier, lower is better)	0.167	0.148
Answers it dares to call "Very likely"	1.1%	23.6%

The miss. I aimed for top-band answers to be right at least 95% of the time. On held-out photos it reached 92.4%. So the top band is labelled "Very likely", not "Confident", and every result shows the calibrated "% likely right" next to it.

On the phone. TabPFN can't run in a phone browser, so I evaluate it on a 21×21×21 grid for each kingdom and ship the predictions as JSON. The phone interpolates between grid points, offline.

Why Does Open Innovation Matter?

Because the trail has no signal, and closed apps need one.

• It runs with no internet. That's only possible because the weights are open. I could download BioCLIP 2, convert it, shrink it and run it in Safari. With a closed API, the model stays on someone else's server, and so does your photo.
• Your photos and location stay yours. Nothing is uploaded, ever. There's no account, no analytics, no server.
• I could measure it honestly. Because I own the whole pipeline, I could test the exact model the phone runs on photos it has never seen, publish the numbers, and build a trust meter on top. With a closed model I'd be trusting a marketing page.
• It costs nothing per photo. No API key, no quota, no bill. It runs on GitHub Pages for free.
• Anyone can fix it. Every piece is open: BioCLIP 2, GBIF, Wikipedia, onnxruntime, TabPFN, and this repo (MIT). Anyone can extend the danger list, rebuild the species pack for their region, or swap in a better model next year.

An open model from a biology institute, open species data from GBIF, and an open tabular model from Prior Labs, all running offline in a browser on a hiking trail. That combination isn't available from any closed API.

My Agent Session

I built this with Claude Code as my pair programmer, from the first research question ("can a 609 MB biology model run in iPhone Safari?") to the evaluation and the safety tests.

Prize Categories

• Best Use of TabPFN: TabPFN is the app's trust meter. It learned from 1,781 labelled photos when BioCLIP 2's answer can be trusted, and beats the raw model score on held-out photos (AUROC 0.843 vs 0.817, Brier 0.148 vs 0.167). Its predictions ship as a grid the phone reads offline.
• Best Use of GitHub Copilot: the project is automated with GitHub Actions:
  • CI runs the offline safety tests on every push: no edibility wording, every deadly species present and flagged, classic deadly mix-ups raise the alarm, and the pack and trust grid are consistent.
  • A second workflow deploys the app to GitHub Pages.

Not a substitute for an expert. Never eat anything based on an app. Go outside anyway. 🌿🍄