DEV Community

David Anton Alvarez
David Anton Alvarez

Posted on

NaturDex: a Pokédex for real nature that runs offline on a recycled phone

Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass Submission 🌿

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

We live surrounded by plants and animals we never look at. The robin on the fence, the heather on the hillside, the butterfly crossing the path. Kids walk past all of them with their eyes on a screen.

So I flipped it around and used the screen to send them outside. NaturDex is a Pokédex for real nature. You photograph a plant or an animal, and an AI model running on the phone tells you what it is. If it's new, it goes into your NaturDex with its number, its photo and the date you found it. The goal is to fill it, and the only way to do that is to go out and look.

The scenario I designed it for is a school field trip to the forest. Schools could hand out recycled phones (old phones everyone has in a drawer) with NaturDex installed, and the activity is to fill the NaturDex. That scenario set three hard rules:

  1. No signal. Forests don't have coverage, so everything has to work offline.
  2. Kids' privacy. No accounts, no uploads. Photos and GPS positions of children never leave the phone.
  3. Old hardware, zero cost. It has to run on a phone that's a few years old, with no app store and no API bill.

And, since it's for children outdoors, a fourth one: safety first. NaturDex warns before it entertains.

How a trip works:

  • Before leaving, on wifi: tap Prepare trip. It downloads the model (87 MB) and the list of 1,944 species that live in the region (Asturias, in northern Spain).
  • In the forest: tap the big camera button. In a few seconds NaturDex answers in one of three ways: "It's a European Robin!", "Which one is it?" (and the kid compares the photo with three options), or "Hmm, I'm not sure…" with a tip to take a better photo.
  • Every species found gets its NaturDex number (No. 0046 for the robin) and a coloured badge for its kind: bird, plant, butterfly, fungus…

Why only Asturias? Because it's where I live, and where I could actually use it. The AI model knows living things from all over the world, but the species list and, above all, the safety rules have to be right for the place where the kids are. Which vipers live here, which plants are poisonous, which species are protected by law: I can only vouch for that in a region I know. So I chose to do one region well instead of the whole world badly. The last section explains how other regions could join.

If you don't live in Asturias, you can still try it: on a computer, the camera button opens a file picker, so you can give it any photo of a common European species, like a robin, a fly agaric or a peacock butterfly.

Demo

Try it: virien84.github.io/naturdex. Open it on a phone, tap Prepare trip on wifi, then go outside. You can also install it from the browser menu (Install app / Add to Home screen).

The design is a handheld field device: a red shell, a big blue lens that glows while the AI is thinking, a leaf-green LCD screen and chunky buttons that press down.

Ready to go A confident answer Added to the NaturDex
Camera screen It's a European Robin New species

That robin is a small bird far away, between branches, and NaturDex is still 100% sure. The shield means it's a protected species: look, but don't disturb it.

Safety comes first. Dangerous species get a black-and-yellow hazard warning above everything else. On the right, the model isn't sure which snake it is, but one of the candidates is a snake, and that's enough to warn.

Deadly mushroom "Could it be…?"
Fly agaric warning Snake warning

When the model hesitates, the kid decides. It's a learning moment: compare your photo with the options. And when NaturDex really doesn't know, it says so instead of guessing.

Which one is it? Not sure My NaturDex
Which one is it Not sure Album

At night the screen switches to a backlit LCD:

Dark mode

The photos in these screenshots are openly licensed iNaturalist observations from Spain (CC BY / CC0). Credits are in screenshots/CREDITS.md.

I took it outside

I installed it on my phone, a mid-range Android (vivo V50 Lite), and went out to photograph trees and plants. All three answers showed up: direct identifications, "Which one is it?" with the right species among the options, and "not sure". It worked with no errors.

The "not sure" one taught me something. I had photographed a whole tree standing among other trees. The model looks at a square crop of 224×224 pixels, and in that square there were several crowns and a lot of sky, so it couldn't tell which tree I meant. It was right not to guess. So now the camera says "For trees, photograph one leaf, flower or the bark", and the "not sure" screen repeats it.

The second lesson was about patience. After taking a photo, the app looked frozen. The message "Looking closely…" existed, but the screen never got the chance to show it (more on that below). Kids would tap again, or give up. Now the lens glows and a spinner turns from the very first instant.

Code

NaturDex

A nature "Pokémon Snap" for school field trips: photograph plants and animals and fill your NaturDex, fully offline on recycled phones.

Try it: https://virien84.github.io/naturdex/ (open it on a phone, tap Prepare trip on wifi, then go outside).

  • app/ — the PWA (React + Vite + vite-plugin-pwa + onnxruntime-web).
  • embeddings/ — Python scripts (uv) that prepare the model and the species list on a laptop.
  • SAFETY.md — how NaturDex keeps kids and protected species safe.

How it is hosted

What Where
The app GitHub Pages, rebuilt on every push to main by .github/workflows/deploy.yml
The BioCLIP model (87 MB, too big for GitHub) Hugging Face: Virien/naturdex-bioclip-onnx
Species list + embeddings (~5 MB) In this repo, app/public/species/

Everything runs on the phone: photos and locations never leave it.

Credits




  • app/: the PWA (React + Vite + vite-plugin-pwa + onnxruntime-web).
  • embeddings/: Python scripts that prepare the model and the species list on a laptop.
  • SAFETY.md: every safety rule and where the data behind it comes from.

MIT licensed. The model is on Hugging Face: Virien/naturdex-bioclip-onnx.

How I Built It

The trick that makes it fit on a phone is splitting the AI work in two: the heavy, one-off part runs once, on a laptop, and the phone only does what it has to do for each photo.

LAPTOP, once                                 PHONE, every photo (offline)

GBIF ─► 1,944 species of Asturias            Photo ─► BioCLIP image encoder
          │                                          (ONNX int8, 87 MB, WASM)
          ▼                                              │
BioCLIP text encoder ─► 1 "fingerprint"                 ▼
per species (512 numbers)  ──────────────────►  compare with the 1,944
                                                fingerprints ─► top 5 + %
Enter fullscreen mode Exit fullscreen mode

1. The model: BioCLIP. BioCLIP is an open CLIP model (MIT) trained by the Imageomics Institute on the Tree of Life: millions of images of organisms with their full taxonomy. Like any CLIP model, it puts images and texts in the same space, so it can do zero-shot classification: no training on my side, just compare the photo with a text description of each species.

2. The species list, on the laptop. A script downloads from GBIF every species with at least 20 human observations in Asturias since 2000 (1,933), and a safety script adds 11 dangerous species that are too rare to make the cut, because the app can only warn about species it knows. Then BioCLIP's text encoder turns each species into a "fingerprint": its full taxonomy and common name, averaged over 15 photo prompts ("a photo of a…", "a blurry photo of a…"). That's a 4 MB file.

3. The model, on the phone. I exported only BioCLIP's image encoder to ONNX and quantized its weights to int8: from 345 MB to 87 MB. It's too big for GitHub Pages, so it lives on Hugging Face, which serves it with CORS. The phone runs it with onnxruntime-web (WebAssembly), compares the photo's embedding with the 1,944 fingerprints, and picks a verdict:

const top = candidates[0].probability
const verdict = top >= CONFIDENT ? 'confident' : top >= UNSURE ? 'maybe' : 'unsure'
return { candidates, verdict, warnings: precautionWarnings(candidates) }
Enter fullscreen mode Exit fullscreen mode

Those thresholds (70% and 40%) aren't guesses. I evaluated the int8 model, running exactly as the phone does, on 244 research-grade iNaturalist photos from Spain (82 species, including every dangerous one):

top-1 top-3 top-5
66% 81% 86%

In the evaluation (an earlier run, with 232 of those photos), above 70% the top answer was right 84% of the time, so NaturDex says "It's a…!". Between 40% and 70% it was right only ~45%, but the right species was usually in the top 3, so NaturDex shows three and lets the kid choose. Below 40% it was right 7% of the time, so it says "not sure".

4. Safety rules (SAFETY.md). This is the part that matters most, because a wrong answer here isn't a bug, it's a child touching something.

  • Look, don't touch. Every species can carry a warning: danger (vipers, the Asian hornet, foxglove, death cap…) or caution (wasps, nettles…). They come from a hand-curated list with a source for each, plus group rules: never eat wild mushrooms, don't touch amphibians' skin.
  • Precaution rule: if any of the top 5 candidates with at least 5% probability is dangerous, the warning shows, even when it's not the top answer. That's the "Could it be a Viperine Snake?" screenshot. On the 124 test photos of hazardous species, 118 got a warning.
  • Every snake is "danger". In the test, the model identified a venomous Baskian viper as a harmless viperine snake with 96% confidence. Kids can't tell them apart either, so NaturDex never relies on that difference.
  • NaturDex never says that anything is edible.
  • Hide where threatened species live. The 283 species protected in Spain or Asturias show a "look, but don't disturb it" note, and for threatened ones the saved GPS position is rounded to about 10 km, like iNaturalist does, so a shared album can't lead anyone to a nest.

5. Offline for real. It's a PWA: a service worker caches the app shell, the fonts and onnxruntime's WebAssembly; the model and species list go to Cache Storage when you tap Prepare trip; the album lives in IndexedDB. Once prepared, it works with no connection: I tested it by preparing the trip, cutting the network, reloading the page and identifying a butterfly. If you're building something similar, this post on offline PWA pitfalls by ohugonnot covers the Vite PWA and Workbox traps well.

Three bugs worth sharing

The app looked frozen, because the browser never got to paint. Decoding a full-size phone photo, checking whether it's blurry and running the model all happen on the page's single thread. React had already switched to the "thinking" screen, but the browser had no free moment to draw it. The fix is to wait for a paint before the heavy work:

// Waits until the browser has drawn the screen. Decoding the photo and running
// the model block the page, so without this the "thinking" screen never shows.
function nextPaint() {
  return new Promise((resolve) => requestAnimationFrame(() => setTimeout(resolve, 0)))
}
Enter fullscreen mode Exit fullscreen mode

The spinner and the glowing lens are CSS transform/opacity animations, which the GPU keeps running even while the model has the page busy. Measured inside the page with the CPU throttled 4×: the spinner is painted 0.05 s after the photo arrives, and the result comes ~13 s later.

Firefox got stuck at 99%. After downloading, the app asks the browser not to delete its data (navigator.storage.persist()). Chrome, Brave and Safari answer at once. Firefox asks the user first, and the app waited for that answer. If the kid didn't notice the prompt, the download looked stuck forever. Now it waits at most 3 seconds and moves on. I found it by running the full flow (download + identify) in Chromium, Brave, Firefox and WebKit with Playwright, and all four pass now.

"It's an European Robin!" In English it's a European. Small, but the app talks to kids.

I built it with Claude Code as my coding agent: it did the heavy lifting on the ONNX export, the evaluation script, the PWA plumbing and the design, while I made the decisions and tested it outside.

Why Does Open Innovation Matter?

For this project, open isn't a nice-to-have. A closed API would have broken every one of the rules:

  • It works in the forest. A cloud model needs coverage. BioCLIP runs on the phone, in the browser, in airplane mode. That's only possible because the weights are open: I could export them to ONNX, quantize them to int8 and host the result myself.
  • Children's photos and locations stay on the phone. With a cloud API, every photo a child takes (and where they took it) would go to a server the school doesn't control. Here there are no accounts, no uploads and no analytics. Nothing to consent to, nothing to leak.
  • It costs nothing per photo. A class of 25 kids taking photos all morning would be thousands of API calls. With an open model running locally, the cost is zero, forever, and the school doesn't need a credit card or an API key.
  • It runs on old phones. It's just a web page: no app store, no installation process, no minimum Android version beyond a modern browser. Recycled phones are enough.
  • A specialist beats a generalist here. BioCLIP was trained by researchers specifically on the Tree of Life, with taxonomy. It knows thousands of European species, from beetles to orchids. And being open, I could measure exactly how good it is on photos from my region, and set the thresholds from that data instead of trusting a black box.
  • It's adaptable. The model doesn't change from one place to another; only the species list and the safety rules do. Another region is a matter of re-running the Python scripts with another GBIF area. No one has to ask for permission.

The data is open too: species from GBIF, test photos from iNaturalist, protected species from public Spanish catalogues. Open models plus open data made a tool that a school can own.

What's next: regions made by the people who live there

Asturias is a starting point, not the limit. The final idea, and the reason it had to be open source, is that anyone could add their own region:

  • A region pack is small: the species list and its fingerprints (about 5 MB, generated with the scripts in the repo from GBIF data), plus the region's safety files. The model stays the same for everyone.
  • The safety files are the valuable part, and the part I can't write for other places: which species are dangerous there, which ones are protected, and where to hide locations. Today safety.py is built around Spanish and Asturian catalogues. The plan is to turn it into one folder per region (dangerous.csv, sensitive.csv, the local protected-species list), so the people who know each place are the ones who feed it: a biology teacher, a park ranger, a local naturalists' group.
  • New regions would arrive as pull requests to a shared repository, where others can review the sources before a pack goes live, and NaturDex would list them in Prepare trip so a school picks its own before the trip.

That's the kind of collaboration a closed app can't have: the AI is open, the data is open, and the knowledge that keeps kids safe comes from the community itself.

My Agent Session

This is the session from the last day: testing it outside on my phone, fixing the "frozen" screen, the Pokédex-style redesign, the install icons, the cross-browser test that caught the Firefox bug, and preparing this post (including swapping out a photo of a run-over fox for a living one). It's in Spanish, my native language.

NaturDex: field-testing, a Pokédex-style redesign and cross-browser fixes
You

He hecho una pequeña prueba solo con arboles y plantas. He tenido los tres supuestos: no detección, detección con opción de que el usuario elija y la detección directa.

Agent

¡Buena señal! Los tres resultados que has visto son justo los que diseñamos. Salen de classifier.js, que mira lo seguro que está el modelo de su primera opción:

Lo que viste Confianza de la 1ª opción Qué tal acertó en las pruebas del PC (244 fotos)
Detección directa ("It's a …!") ≥ 70 % Acierta ~84 % de las veces
Elegir entre opciones ("Which one is it?") entre 40 % y 70 % La 1ª acierta ~45 %, pero la buena suele estar entre las 3 primeras
No detección ("Not sure, take another photo") < 40 % Solo acertaría ~7 %, así que es mejor no …
You

1-Si. 2-No estoy seguro pero creo que si. 3-Igusl que la 2. 4-un árbol entero cerca de otros. 5-no

Agent

¡Perfecto! Con esto la prueba en el vivo sale bien: funciona en un móvil real, sin errores, y aparecen los tres casos.

Sobre el caso que no detectó (un árbol entero junto a otros): era lo esperable. En el recorte de 224×224 se mezclan las copas de varios árboles con el cielo, y el modelo no sabe en cuál fijarse. La app se comportó bien: prefirió decir "no estoy seguro" antes que inventarse una especie.

He hecho un cambio pequeño para guiar a los niños (todo en inglés, como el resto de la app):

  • Cámara (app/src/screens/CameraScreen.jsx): el consejo que sale debajo del botón ah …
You

Antes de subir y desplegar hay algo que no me gusta: después de sacar la foto, no da muestras de hacer nada y pienso que se ha colgado. Reparalo poniendo un indicador de espera

Agent

Ya veo la causa. Te lo explico y lo arreglo.

Ahora hago los cambios:

Compila bien. Ahora lo pruebo en un navegador automático: le paso una foto y compruebo que la ruedecita aparece antes de que salga el resultado.

Me he equivocado: he dejado en el comando una llamada suelta a python3, que se queda esperando para siempre (la memoria ya avisaba de esto). La detengo y lo repito bien.

Ya está arreglado: la ruedecita de espera aparece en cuanto haces la foto. En una prueba con el procesador ralentizado 4 veces para imitar un móvil modesto, se pintó a los 0,05 s de llegar la foto, y el resultado salió a los ~13 s.

Por qué parecía colgada. JavaScript en una página web hace todo con un solo "trabajador". Después de la foto le tocaban tres tareas pesadas seguidas:
1. Abrir la foto.
2. Comprobar si está borrosa u oscura.
3. Pasar el modelo de IA.

Para redibujar la pantalla, ese mismo trabajador tiene que estar libre. Como no paraba ni un momento, el mensaje "Looking clo …

You

Haz el commit y el push

Agent

Ya está subido y publicado. La web https://virien84.github.io/naturdex/ ya tiene la ruedecita de espera y los consejos para fotografiar árboles.

  • Commit: d8c635b, "Show a spinner while identifying, and tips for photographing trees".
  • Despliegue: GitHub Actions lo terminó sin errores y he comprobado que la web carga bien. …
You

Mientras tengo una pregunta de Claude code: la gente recomienda instalar los plugins omni-route, frontend-design, headroom, mem-palace y playwright. Quiero instalarlos, que se puedan usar en todos mis proyectos y que me digas como se usan

Agent

Antes de instalar nada miro qué es cada uno y de dónde sale. Algunos son oficiales de Anthropic y otros son de terceros, y un plugin puede ejecutar código en tu PC.

Ya tienes instalados para todos tus proyectos los dos plugins oficiales, frontend-design y playwright. Los otros tres no los he instalado todavía: al investigarlos he visto cosas que conviene que sepas antes de decidir. …

You

Vamos a trabajar en el frontend, ya que es de noche y no voy a salir a sacar fotos, pero se puede trabajar en eso. Usa frontend-design para que la aplicación tenga un diseño similar a una pokedex, centrada en el público juvenil

Agent

Esta sesión empezó antes de instalar el plugin, así que leo directamente las instrucciones de la skill frontend-design y las aplico. Luego reviso cómo está hecha la app ahora.

Ahora el código. Primero, una función para numerar las especies como en una Pokédex:

Ahora la carcasa del aparato (App.jsx), la ficha de resultado, que pasa a parecer una entrada de la Pokédex, y la pantalla del álbum (My NaturDex).

Ahora la hoja de estilos completa: colores, carcasa, lente, pantalla LCD, botones físicos y etiquetas de tipo.

Compila bien. Ahora hago capturas en tamaño móvil para revisar el diseño con mis propios ojos, como pide la skill:

En modo oscuro, el fondo de la foto sale verde claro y deslumbra. Lo cambio a oscuro fijo, cierro el servidor que se había quedado abierto y compilo:

NaturDex ya tiene un diseño tipo Pokédex: la app parece un aparato de explorador. …

Qué ha cambiado
- Carcasa roja con la gran lente azul y tres LEDs. Mientras la IA piensa, la lente brilla y …

You

Si, haz commit y push

Agent

El nuevo diseño ya está en la web: https://virien84.github.io/naturdex/. …

Recuerda el plazo: la entrega es mañana lunes 12 a las 08:59. Lo que queda, por orden de importancia:
1. Post de DEV.
2. Iconos para instalar la app.
3. Logros (gamificación).

You

Vamos con los iconos

Agent

Creo los iconos a partir de la lente del favicon. Hacen falta tres tamaños:
- 192 y 512 px: los que Android exige para poder instalar la app.
- Versión "maskable" de 512 px: Android recorta el icono en círculo, gota u otra forma según el móvil. Esta versión va a sangre, con el fondo rojo hasta los bordes, para que el recorte nunca deje esquinas raras.
- 180 px: para iPhone.

Los iconos están hechos y Chrome ya considera NaturDex instalable: su comprobación interna no devuelve ningún error. …

You

Antes del commit y push: yo uso Brave, y habrá gente que use otros navegadores que no sean Chrome. ¿Funcionará igual?

Agent

Buena pregunta. En vez de responder de memoria, lo pruebo: Playwright puede usar los motores de Firefox y de Safari (WebKit), además de Chrome, y miro si tienes Brave instalado en el PC para probarlo también.

Brave, Chrome y WebKit (Safari) funcionan. Firefox se queda atascado en la descarga, así que miro qué pasa:

De paso, en las pruebas salió "It's an European Robin!". En inglés es "a European", porque "European" suena con /j/. Lo arreglo también:

Funciona igual en Brave, Firefox y Safari, después de arreglar un fallo que dejaba la app colgada en Firefox.

Lo he probado de verdad, no de memoria. En cada navegador hice el recorrido completo con la versión de producción: descargar el modelo desde Hugging Face, hacer una foto de un petirrojo e identificarla. …

You

Si, haz commit y push

Agent

Los iconos y los dos arreglos ya están publicados en https://virien84.github.io/naturdex/. …

You

Empecemos

Agent

Primero cargo las instrucciones para publicar en DEV y consulto las bases del reto: plantilla obligatoria, etiquetas y criterios de valoración.

Perfecto. Hago las capturas desde la web publicada, en tamaño móvil y en JPEG para que pesen poco:

El script tenía un fallo: después de guardar una especie, pulsaba el botón de cámara de abajo, pero si ya estás en la cámara ese botón no hace nada. Eso también le pasaría a un niño. Arreglo la app para que pulsar un botón siempre lleve al principio de esa pantalla.

Antes de seguir, dos problemas con las fotos:
1. La del zorro es un zorro atropellado: no es lo que queremos enseñar a niños ni a los jueces.
2. Las fotos de prueba vienen de iNaturalist, y cada una tiene su propia licencia y su autor. Si salen en el post y en el repo hay que citarlos.

Fotos con licencia libre descargadas: 32, cada una con su autor. Ahora paso todas por la app para elegir las que dan cada caso (seguro, aviso de peligro, "Which one is it?"):

You

Por favor, que no aparezca el zorro atropellado

Agent

Entendido: el zorro atropellado no saldrá. Era una foto del primer lote y ya la he descartado. Antes de elegir sustituto, miro las fotos de zorro nuevas para asegurarme de que son de un zorro vivo:

En la culebra parece verse la tripa hacia arriba, lo que podría indicar que está muerta. Con la petición que me has hecho, prefiero no arriesgar: miro las otras fotos de culebra.

Las dos están vivas. Uso la del pedregal, de Martiño Cabana Otero (CC BY), que también da "Could it be a Viperine Snake?":

Las capturas están bien. Hasta el petirrojo, que sale pequeño y entre ramas, lo reconoce al 100 %. Ahora creo el archivo de créditos de las fotos para el repositorio:

Ahora guardo esta sesión en DEV para incrustarla en el post, como hiciste en el anterior.

Prize Categories

None. NaturDex doesn't use any partner technology, so I'm entering only the overall prize.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.