Walktober

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Walktober is a small reason to go outside. Photograph what you notice, complete quests, and fill your scrapbook with the season. Then put your phone away.

Take a pic of something notable on your autumnal ramblings. It is then converted into a sticker and added into your scrapbook.

Demo

🍂 walktober.eoinmcgrath.com

Code

💻️ github.com/eoinmcg/walktober

How I Built It

Everything runs in the browser, with no server and no API calls, and it works offline once the files are cached. All the AI is open-weight and runs through ONNX Runtime.

Cutout and sticker: U²-Net-P
The image is separated from its background by U²-Net-P, a small (about 4.5 MB, Apache-2.0) salient-object-detection network. It runs through onnxruntime-web (MIT) on WebAssembly inside a Web Worker, so the UI stays responsive. The worker decodes the photo, resizes it to 320×320, normalises it, runs the model, and sends back a soft mask. The main thread turns that mask into alpha. The sticker (rounded white border, thin black outline, drop shadow) is plain canvas code on top of that alpha. It involves no extra model.

Classification and matching: CLIP via transformers.js
In order to make things interesting I decided to add a number of quests. Basically, take a pic of an acorn, mushroom etc. This requires a way of matching the sticker to the quest. I first tried MobileNetV2, which is lightweight but only has around 1000 ImageNet classes. This simply didn't cut it when attempting to classify images. After some experimentation, I settled on CLIP (Xenova/clip-vit-base-patch32, an ONNX conversion of OpenAI’s MIT-licensed model) through @huggingface/transformers. CLIP embeds images and text into the same vector space, so the cutout can be compared against any text description by similarity, without retraining.

Text embeddings: precomputed at build time
scripts/build-text-embeddings.ts loads CLIP’s text encoder in Node (AutoTokenizer plus CLIPTextModelWithProjection) and embeds every quest from quests.json once. It writes the vectors to public/embeddings/text-embeddings.json, rounded to 5 decimals to keep the file small. At runtime the browser loads only the vision encoder and compares the cutout’s embedding with those stored vectors. The text encoder never ships to the phone, which saves download size and memory. The build script and the runtime must use the same CLIP model, or the vectors live in different spaces and the similarities mean nothing.

Why Does Open Innovation Matter?

Purely Local & Offline-First

The core mandate of Walktober was 100% on-device execution. A closed-source API couldn't satisfy this requirement. Because model files and WebAssembly runtimes are served as standard static assets, the app operates without reliance on external servers. Photos are processed locally - ensuring privacy, instant response times, and uninterrupted outdoor use without requiring cell coverage.

Architectural Freedom

Open weights allow developers to reshape system architecture rather than merely call remote endpoints. By leveraging CLIP's decoupled vision and text encoders, I offloaded text encoding to build time. Furthermore, using open weights meant being able to swap U²-Net-P (4.7 MB) for larger alternatives or replace MobileNet with CLIP freely. Such decisions aren't possible with black-box APIs.

Debuggability & Inspectability

Open source makes edge cases solvable. When the classifier threw an unexpected JSON parse error, I diagnosed it directly by inspecting the transformers.js source code. I discovered that the library treated image classification as a multimodal task by looking for a tokenizer file; my development server was catching missing requests and returning an HTML fallback page with a 200 HTTP status code, triggering the JSON error. Understanding this permitted a clean workaround building the pipeline directly from its individual component parts.

Seamless Composition

U²-Net, CLIP, ONNX Runtime, transformers.js, and Nuxt originate from distinct maintainers, yet seamlessly interoperate through open standards (like ONNX) and permissive open-source licenses. Each piece remains independently swappable—such as introducing a plant-specific classifier later—without rewriting the core application.

Prize Categories

General Category