Name it, then look up: a field guide where the phone's job ends after one photo

Olabowale Babalola Israel — Mon, 12 Oct 2026 00:18:32 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

Most nature apps have the same quiet problem. You see a bird, you lift your phone, and by the time the app has told you its name, the bird has gone and you're still reading about it.

I wanted the opposite. In Spotted, getting the name is the quick bit. Then the app asks for something harder: put the phone down for one minute and actually look at the thing.

What I Built

Spotted is a pocket field guide that runs on your phone, in the browser. You take or upload one photo of a bird, lizard, insect, tree or plant. An open model suggests a name from the field guide. Then a button called Look up asks a second open model, Gemma, for one thing to notice. The screen dims to a 60-second timer. When the minute ends there's a soft chime and one question: What did you notice? Your answer is saved into the sighting.

That minute is the whole point. The name, the facts, the feed, the map: all of that is there to get you to the minute and then get out of the way.

A few details I cared about:

It admits when it doesn't know. If the top match scores under 0.12, or is less than 0.04 ahead of the runner-up, Spotted doesn't name anything. It says Not in my guide yet, shows the closest two or three as "maybes", and lets you save the sighting under your own label. A confident wrong name is worse than no name, especially for someone just starting to learn.
The guide is local. It has 114 species. Many are things you actually meet in West African compounds, streets and parks: pied crows, laughing doves, rainbow agamas, neem, flamboyant, bitter leaf, the village weaver. It also keeps some familiar species from elsewhere so it isn't lost on another continent. Habitat, diet, behaviour and conservation status are written into the app, and every species links to the source of those facts.
Rare animals don't get exact pins. Sharing a location is off until you turn it on. Threatened species, plus a few timber trees marked sensitive, are rounded to a cell about 11 km across before anything reaches the server.
It works offline. Once the models are cached, naming and Look up run without signal. Notes you save offline wait on the phone and sync later.
A small social trail. There's a feed, comments, reactions ("wonder", "seen it", "protect"), a map and a life list, so a sighting can be shared. It's deliberately light, though. Comments appear straight away, and then there's nothing to scroll.

It's for anyone who has wondered "what is that?" about a bird on a fence or a lizard on a wall, and wants an answer that ends with them looking up instead of down.

Demo

Live app: https://spotted-91b2.onrender.com

It's on Render's free plan, so if nobody has visited for a while the first load takes about a minute to wake up. After that it's quick.

The video is real footage of the live site in a phone-sized browser. The model download, Gemma's thinking time and the 60-second timer are sped up, and labelled as sped up on screen.

Try it in 60 seconds:

Open the app and tap Open the field.
Tap a reference photo (the pied kingfisher is a clear one), or upload your own.
On the result, press Look up. The first time, Gemma downloads into your browser (about 308 MB, so use Wi-Fi). Then the screen dims and gives you one thing to notice.
When the minute ends, or when you tap I looked, answer What did you notice? and save the note.

Code

Esimuda / spotted

Spotted: snap nature, an open-weight model names it, share it with your community

Spotted

See nature. Share it. You photograph a bird, animal, insect, or plant. An open model suggests the name. Then you share the sighting, or you just look up again.

The product name is one constant, APP_NAME in src/config/brand.ts.

The demo trail is fictional and West African: Lekki Conservation Centre, Okomu National Park, Obudu Mountain Resort, and Yankari Game Reserve. The guide also includes species from elsewhere, so a walk outside Nigeria still gets a name. Reference photos are credited in public/seed/credits.json. The people in the feed are not real.

Verify in 60 seconds

Live app: https://spotted-91b2.onrender.com

Open the field tab and run a reference photo (pied kingfisher is a clear one).
On the result, press Look up. The first time, Gemma 3 270M downloads in the browser. The screen then dims to a 60-second timer and one thing to notice.
When the minute ends, or when…

View on GitHub

MIT licensed. If you want to look at the interesting parts first:

src/lib/identify/rank.ts     the "not sure" rule
src/lib/identify/look.ts     Gemma in the browser: one thing to notice
src/components/look-up.tsx   the dimmed minute, the chime and the question
src/data/more-species.ts     the West African additions to the guide
src/lib/privacy.ts           opt-in pins, softened cells for sensitive species

How I Built It

Spotted is a Next.js PWA with a small SQLite-backed API for the trail. The two jobs that need AI both run on open models in the browser through Transformers.js.

Naming: CLIP ViT-B/32. I use Xenova/clip-vit-base-patch32 (MIT), an 8-bit ONNX build of OpenAI's CLIP, as a zero-shot classifier. Every species in the guide becomes a text prompt. The photo is compared with all of them, and the scores are a softmax across the whole guide. The guide works as an allow-list: CLIP can only answer with a species that has written facts behind it, so it can't invent an animal.

The scores are spread across 114 options, so the numbers look modest. On my reference pied kingfisher photo, the top match was pied kingfisher at 25%, with the next name at 10%. That lead is what the rule in rank.ts checks for. Below 0.12, or with a gap under 0.04, you get maybes instead of a name.

Look up: Gemma 3 270M. This is the heart of the app. When you press Look up, the page loads onnx-community/gemma-3-270m-it-ONNX and runs it right there in the browser: WebGPU with the q4f16 build where it's available, and WASM with q4 where it isn't. Getting the official weights to run on the WASM runtime took one small change. A gather index in the graph needed casting to int32, and that patched graph ships in public/models/gemma.

Gemma's job here is narrow on purpose. It gets the species name and the guide's behaviour sentence, and has to answer in one sentence that starts with "Watch". For the pied kingfisher, the sentence it gave on my test run was "Watch it hunt this way." Simple, and it points you at the right thing, because pied kingfishers hover over the water before they dive. If Gemma's answer is the wrong shape, or the phone can't run it, the app falls back to the guide's own behaviour line and labels it Field guide. The minute still happens either way.

I kept Gemma away from the facts deliberately. A 270M-parameter model is small enough to run on a phone, and too small to trust with a conservation status. So the facts come from the guide and its sources, and Gemma only gives the nudge.

Some real numbers from testing:

What	Measured
Gemma download, first time only	about 308 MB, then cached
Gemma load from cache (WASM, q4)	1,679 ms
Gemma generating the one sentence (WASM)	about 20 s
CLIP naming a photo once cached	about 7 s in a headless phone-sized browser
Render free instance waking from sleep	about 55 s

About 20 seconds for one sentence on WASM is slow, so the app shows a calm "Finding one thing" screen while it works. I decided that was an acceptable trade for now: a short wait in exchange for a full minute of looking up, with nothing leaving the phone.

Where it gets things wrong. Here's what I saw testing with Creative Commons photos at my desk:

Pied kingfisher: named correctly.
A giant land snail, which isn't in the guide: Not in my guide yet, with gecko, sun squirrel and Nile monitor as low maybes. That's the behaviour I wanted.
Village weavers: the weak spot. One male came back as "Looks like little bee-eater", another as "not sure", and a nest-building male as yellow-billed kite. General-purpose CLIP doesn't know African weavers well.
A termite mound: called an African forest elephant at 35%, which is funny and wrong. Big brown shape in a field.

That's why the "not sure" state exists and why "Looks like" is worded more softly than "This looks like". It's also why the next item below is a better vision model.

Hosting. The app runs on Render's free web service from a render.yaml blueprint, with a cheap /api/health check. The free instance has about 512 MB of RAM, which isn't enough to host CLIP and Gemma on the server, and I like that constraint. It pushes the AI onto the phone, which is where it should be for something you use outdoors. Render serves the app, the trail and the model files. The visitor's phone does the thinking.

Why Does Open Innovation Matter?

For this project, the closed version would have been a worse product, not just a more expensive one.

No signal, still works. The places worth looking up are often the places where the signal drops. Once the open weights are cached, CLIP still names things and Gemma still finds you something to notice, with no network at all. A closed API stops working right where the walk starts getting good.
The photo stays on the phone. Naming doesn't need your picture to leave your device. For a rare animal, a photo plus GPS is a map to it. Open models running locally, plus the softened pins, mean that data doesn't have to sit on someone else's server.
No meter on curiosity. There's no bill for each name. A community trail that charges per question is a trail that teaches people to ask fewer questions.
I could change the model. When Gemma's WASM run failed on an index type, I could open the graph and fix it. When CLIP turned out to be weak on weavers, I could see why, and I know exactly what replaces it: BioCLIP (MIT), a CLIP trained on the tree of life. The guide is just a list of prompts, so a better open model means swapping a file, not signing a new vendor contract.

What's Next

To be straight about where it stands:

A proper walk. I've tested Spotted at my desk with reference photos, not yet on a real walk with my own pictures. That's the next thing, and it'll show whether 114 species is enough for a Benin City street or a Lagos compound.
BioCLIP. It's the specialist model I wanted from the start. It isn't bundled because there's no Transformers.js ONNX export of its weights yet. Once there is one, it replaces CLIP and the weaver problem should mostly go away.
More of the guide. More West African birds, insects and trees, with sources, and local names next to the English ones.
Faster Look up on older phones. Twenty seconds on WASM is too long. I want to try a shorter prompt and streaming the first words as they come.

Credits and licences

Code: MIT, https://github.com/Esimuda/spotted
CLIP ViT-B/32 via Xenova/clip-vit-base-patch32: MIT
Gemma 3 270M Instruct via onnx-community/gemma-3-270m-it-ONNX: Gemma Terms of Use
Transformers.js (@huggingface/transformers): Apache-2.0
Reference photos in the app: Wikimedia Commons, each credited in public/seed/credits.json
Photos in the demo video: "Pied kingfisher (Ceryle rudis rudis) male" and "Giant tiger land snail (Achatina achatina) Ghana", both by Charles J. Sharp, CC BY-SA 4.0, via Wikimedia Commons
The people on the demo trail are fictional. Their photos are openly licensed reference images, not their own.

DEV Community: Olabowale Babalola Israel