Tuft: a scavenger hunt that checks your photos on your phone, offline

Shishir Shrivastava — Wed, 07 Oct 2026 18:17:18 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Tuft is a scavenger hunt you mostly listen to. You pick a hunt ("Neighbourhood Basics", "Autumn Walk"), put the phone in your pocket, and a spoken clue tells you what to find: "Find a tree. Put your hand on the bark, then take a photo of the trunk." When you find it, you take one photo. An open-weight vision model running on your phone decides whether the photo really shows what the clue asked for. Then you get the next clue.

The problem: "go outside" is easy advice, and a game gives a walk a purpose. But a game that trusts everyone has nothing to prove, and one that verifies photos in the cloud sends pictures of your street, your kids and your dog to someone else's server. Tuft verifies locally, so it can be honest and private at once. The screen is the short part: a clue is a sentence you hear, a photo takes a few seconds, and the rest is walking.

Demo

Deployed app: https://shishir2405.github.io/tuft/

Code

Shishir2405 / tuft

On-device scavenger hunt: an open-weight vision model checks your photos on your phone. No server, works offline.

Tuft

A scavenger hunt that talks to you, checks your photos on your phone, and works with no signal.

You pick a hunt, pocket the phone, and listen. Each clue is spoken ("Find a tree. Put your hand on the bark…"). When you find the thing, you take one photo; an open-weight vision model running on the device decides whether it really is that thing. No account, no server, no photo ever uploaded.

The screen is the shortest part of the walk: a clue is a sentence you hear, a photo is a few seconds, and the rest is outdoors.

Built for the Hacktober Open-Source AI Challenge, Week 1: "Touch Grass".

Why it exists

"Go outside" is easy advice and hard to follow. A scavenger hunt gives a walk a purpose, but the usual app versions either trust you (no fun, nothing checked) or send every photo to a cloud…

View on GitHub

Apache-2.0.

How I Built It

Stack. React + TypeScript + Vite, a service-worker PWA, no backend at all. Inference uses transformers.js and ONNX Runtime Web in a Web Worker.

Model. SigLIP base (google/siglip-base-patch16-224, Apache-2.0), 8-bit ONNX weights. A hunt target is just a sentence, so there is no label set to train: the photo and the text "a photo of a mushroom" are embedded into the same space and compared.

Split encoders. The full model is about 200 MB of image tower plus text tower. I run them as separate graphs and precompute text embeddings for the built-in hunts at build time (npm run embed, 35 vectors, 144 KB). Players only download the ~100 MB image tower. The text tower is fetched lazily if someone imports a hunt with new labels. The scale and bias that SigLIP applies after the towers are read from the upstream checkpoint, since a split export drops them.

Deciding "found". My first rule, an absolute probability threshold, rejected every correct photo: SigLIP's per-pair probabilities for ordinary photos are tiny (a clear dog photo scored 3e-4). The working rule has three parts. The target must be the best of all candidate prompts. It must hold at least half the softmax share against its look-alikes ("a cat" for "a dog") and a fixed list of cheat scenes (a screen, a room, a hand). And it needs a very small absolute probability floor, because without it several wrong pairs reached a high share by default when nothing matched.

Measured, honestly. On 19 Wikipedia lead images (15 true, 4 unrelated), 13/15 true photos counted as finds, the other 2 came back "close, try again", and 0 of 175 wrong photo/target pairs were accepted. The thresholds were tuned on those same images and they are encyclopedic rather than phone photos, so treat the result as a smoke test. Image embedding took 143 to 240 ms in Node on an M1 across two runs. I have not tested outdoors or on real phones.

Engineering decisions.

Model behind an InferenceEngine interface; UI and core never import the ML library. A deterministic hash-embedding engine makes CI fast, and a build-time e2e mode swaps it in for browser tests.
Untrusted input (imported hunt files, localStorage) is validated; corrupt state is dropped, not trusted.
ONNX runtime files are served from our own origin. The library defaults to a CDN, which would have been a hidden online dependency. A test records that no CDN request happens.
The native camera opens through a file input instead of getUserMedia: simpler and more accessible.
WebGPU failed in headless Chromium ("Failed to get GPU adapter") and killed model loading, so WASM is the default and WebGPU is opt-in.

What did not work first time. The absolute threshold above; WebGPU auto-detection; and my first offline test, which only passed once I stopped waiting for a "model ready" label that the hunt screen deliberately hides.

Tests. 58 unit and component tests, 2 mock-engine browser tests in CI, and one real-model browser test that downloads the model, goes offline, reloads, and verifies a real photo (passed once, headless Chromium, first verification about 3.1 s on WASM).

Why Does Open Innovation Matter?

What the open approach actually bought, as implemented:

Privacy by construction. The photos never leave the device because there is nowhere to send them. A closed vision API would have received every photo.
Offline. Once the model is cached, a hunt works in a field with no signal. That is verified by the offline test, in Chromium only.
Open vocabulary without a server. Anyone can write a hunt in JSON and the model scores it. No retraining, no per-request fee.
Control over the pipeline. Splitting the encoders to save a 112 MB download, reading the calibration constants, precomputing embeddings and tagging them by model id are only possible with the weights in hand.
Replaceable. A different CLIP-style model needs one interface implementation and a re-run of the embed script.

What it did not buy: better accuracy than a large hosted model (it almost certainly isn't), a small first download (about 100 MB), or fine-tuning, which I did not do.

My Agent Session

No DevRelay session was saved for this project, so I'm not including one.

DEV Community: Shishir Shrivastava