Wallko: walk to water your garden, scan real plants to grow it (all on-device, open models)

Sahil Singh — Sun, 11 Oct 2026 14:36:33 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Wallko is a cosy Android game that only works if you go outside.

You go for a walk. Walking fills a water tank, measured in drops. On the way you point the camera at a real plant, and Wallko names it, turns it into a collectable card and plants it in your little garden as a hand-painted potted sprite. Your plants drink from the tank on their own every day. Stop walking and the tank runs dry: after 3 days without water a plant wilts, after 7 it withers.

Illustration built from real on-device results on my phone: a confident answer (left) and an unsure one where you pick (right).

The numbers that drive it:

Steps: 1 drop per 100 steps from Health Connect, capped at 15,000 steps a day.
GPS walks: 2 drops per 100 m. Points faster than 12 km/h don't count, so driving doesn't pay.
Exploring: the map is split into 100 m cells under a fog of war. Every cell you enter for the first time clears the fog and gives 3 drops.
Elixir: every 500 m walked inside never-visited cells gives 1 elixir, and elixir is the only way to bring a withered plant back. So the game pushes you down streets you haven't walked yet.
Dawn bonus: walk drops count double between 4:00 and 7:00.
Rarity: cards are Common to Legendary based on how often the species is observed worldwide (iNaturalist and GBIF counts).

The screen part is short on purpose: open the camera, tap a guess, put the phone away. The garden and the map are things you check after the walk, not during it.

Everything runs on the phone. There is no account and no backend. Walks, photos, plants and explored cells never leave the device.

Taking it outside

My father and I go for a walk every day, and we often talk about the wild flowers we pass. I've always found them beautiful, and I think knowing more about them makes it easier to enjoy them. That's where Wallko came from. I didn't plan it: one day on a walk I saw a wild flower and thought, "let's find out what this is." So far I've only taken Wallko outside once, so this is just the start.

Demo

Video: youtube.com/shorts/GPJSu5oyzt0

APK: wallko-v0.1.0.apk (187 MB, Android 10+, ARM phones). Open it on your phone and allow installing from unknown sources; it's a debug-signed preview build.

You can also build the APK yourself from the repo: ./gradlew assembleDebug (JDK 21, Android SDK 37, no API keys needed). The README has the full steps.

My Tulsi card: rarity, family, thirst, sun and growth, and the photo I took (place name blurred).

Code

25sahilsingh / wallko

Wallko

A cosy Android game that gets you outside. Go for a walk, scan real plants with the camera, collect them as cards and grow them in a little virtual garden. Walking fills your water tank and the plants drink from it on their own. Stop walking and they wilt, then wither. Walking into places you have never been earns elixir, the only thing that can bring a withered plant back.

Everything runs on the phone. There is no account and no backend. Walks, plants and explored areas never leave the device. The only network use is map tiles (OpenFreeMap), the ML Kit segmentation model (fetched by Google Play services).

Scan: live camera, then pick from the top three guesses. The plant is cut out (ML Kit subject segmentation) and repainted in one of three styles (Magenta style transfer) as a potted sprite.
Garden, Collection: a meadow with snap-to-grid plots…

View on GitHub

The BioCLIP model files are in the bioclip-v1 release. The scripts I used to test, export and quantise the models are in tools/ml-research/, and the results are written up in docs/ml-research.md.

How I Built It

Wallko is Kotlin and Jetpack Compose, with Room for the garden and walks, CameraX for the scanner, and WorkManager for the slow jobs. The interesting part is how a photo becomes a named plant and a sprite:

Step 1: my plants weren't in the model

I started with Google's AIY Plants V1 classifier (Apache 2.0, 5 MB LiteRT model, 2,101 species). It's small and fast, and it was right about wild plants: a red campion scored 0.96. Then I tried it on the plants in my own home, and it failed on almost all of them. AIY knows mostly wild North American species, and my plants are Indian and house plants:

My plant	AIY said
Lucky bamboo	top guess at 0.05
Tulsi	Chickweed, 0.11
Areca palm	Nikau palm, 0.58 (wrong, and confident)

A threshold can't fix a confident wrong answer, so I needed a different model.

Step 2: looking for a better namer

I checked several open options before settling:

dima806 houseplant ViT (Apache 2.0): picked the right plant but only at 6–8 %. Rejected.
MobileViT houseplants: no licence, and confidently wrong. Rejected.
PlantNet-300K: has none of my plants. Rejected.
iNaturalist's full model: not public.
BioCLIP (imageomics/bioclip, ViT-B/16, MIT licence, trained on TreeOfLife-10M): the best by far.

Step 3: BioCLIP, zero-shot, on the phone

BioCLIP is a CLIP model for the tree of life. I use it zero-shot: the app never trains anything.

On my PC, every species name becomes a 512-number text embedding, stored in a table. The prompt a photo of {latin}. worked best. Adding common names made it worse.
On the phone, the photo becomes a 512-number image embedding, and the best match is the row with the highest dot product.
So adding a species means adding a row to the table. No retraining. My list has 2,162 rows: AIY's 2,101 species plus 60 houseplants and Indian plants I added.

I exported the image encoder to ONNX and quantised it to int8 with ONNX Runtime, which took it from 345 MB to 87 MB. On the phone it runs with ONNX Runtime Android (I gave up on a LiteRT conversion because the converter pulled gigabytes of CUDA libraries).

On my Redmi phone (8 cores, 4 threads) the model loads in about 0.8 s and takes about 620–720 ms per photo. Results on the phone:

My plant	BioCLIP said
Lucky bamboo	correct, 0.98
Tulsi	correct, 1.00
Red campion	correct, 0.99
Elephant bush (Portulacaria afra)	correct (1.00 on PC, named correctly in the app)

It's not perfect, and I'd rather say so:

Areca palm is borderline. In the full app scan it comes out first at 0.40 with Nikau palm right behind at 0.37, so the app shows both as "could be" instead of picking one.
A tight crop of my money plant came out as Butcher's-broom; money plant was second.
My bean plant came out as Kudzu, because beans aren't in my list yet.

Step 4: AIY as the gate, BioCLIP as the namer

BioCLIP always picks some species, even for a photo of my room. AIY has a background class, and rooms and walls score 0.66–0.79 on it. So AIY stayed, with a new job:

AIY looks first. If the background score is 0.30 or more, it's not a plant: no guesses, just "retake".
If it is a plant, BioCLIP names it from my 2,162-row list.
Only guesses scoring at least 0.30 are shown. The top one counts as confident at 0.35+ with a 0.10 lead; otherwise the guesses are shown as "could be".

If the BioCLIP files aren't on the phone, the app falls back to AIY alone.

I tested the whole chain on the phone with 4 room photos and 6 plant photos. All 4 rooms were stopped at the gate (background 0.44–0.89), in about 0.1 s each. All 6 plants got through (background 0.00) and were named in about 0.9 s.

A lesson: a bigger list made it worse

The BioCLIP authors publish precomputed embeddings for the whole TreeOfLife set. I tried 109k plant names instead of my 2,162. Coverage went up (it found my bean), but accuracy didn't: 3 of 7 test photos right, against 4 of 7 with my curated list. With 109k names, a rare look-alike relative wins too often. For a game where people scan the plants around them, a curated list is better.

Step 5: the potted sprite

ML Kit subject segmentation cuts the plant out of the photo.
Magenta arbitrary style transfer (Apache 2.0, LiteRT) repaints it in one of three styles (watercolour, flat pastel, crayon). The style images are my own, made from the app's palette.
The cut-out goes into a pot and into the garden.

Pure style transfer washed every plant in the style image's pink and blue blobs. The fix was to blend: 0.4 of the art style with 0.6 of the photo's own style vector. Plants now look painted but still green.

Getting the model onto the phone

The 87 MB model is too big to put in the APK, so it lives in the GitHub release. On first run, the app downloads the three BioCLIP files over Wi-Fi in the background, checks each one's size and SHA-256, and only then moves it into place. Until they arrive, scanning works with AIY alone. Building only for ARM phones also brought the APK down from 313 MB to 187 MB.

The other open pieces

ONNX Runtime and LiteRT for on-device inference.
MapLibre with OpenFreeMap tiles for the fog-of-war map (no API key). With no tiles, the app draws a plain map from the same cell data.
Card data (names, rarity) built once on my PC from iNaturalist, GBIF and Wikidata, and shipped inside the APK.

One piece is not open: ML Kit subject segmentation comes from Google Play services. It's the one closed model in the pipeline, and it only makes the sprite, not the name. It could be swapped for an open segmentation model later.

Why Does Open Innovation Matter?

For this app, open models weren't a nice extra. They're the reason it works at all.

It works on the trail with no signal. Naming a plant happens entirely on the phone. Parks and hills are exactly where the mobile signal drops, and that's where people scan plants.
Photos and location stay on the phone. A plant photo taken on a walk says where you were and when. With no server, I don't have to ask anyone to trust me with that.
It costs nothing per scan. No API bill, no rate limit, no key in the app. If a thousand people walk with it, it costs me the same as one.
I could fix it myself. This is where open beat closed for me. When AIY didn't know tulsi or lucky bamboo, I could test other models, export BioCLIP to ONNX, quantise it to a quarter of its size, measure it on my own phone, and add the plants of my region as rows in a table. A closed plant-ID API would have given me its own species list and its own answers, and no way to change either.
I can ship it. AIY and Magenta are Apache 2.0 and BioCLIP is MIT, so I can redistribute the models in the APK and in a GitHub release.

My Agent Session

I built Wallko with Claude Code (Anthropic). To be clear, that's a closed agent harness, not an open one. The open parts of this project are the models and the runtimes on the phone.

How I used it: I wrote the plan and game rules first (they're in docs/), then worked through it in phases. I used subagents for separate jobs: one rebuilt the BioCLIP export and int8 model after a reboot wiped my working files, one ran the on-device benchmark on my phone, one wired BioCLIP into the scan flow behind the AIY gate, and one audited the repo before I published it. I tested every model result myself on my own plants.

DEV Community: Sahil Singh