This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Postcards from Outside is a fall scavenger hunt for getting outside together, built for a walk with my mom and my service dog, Tucker.

You tell it who's coming along (kids, walking the dog, an easy pace, or just adults), add your names, and if you want, type in where you are. It builds a short list of things to find that actually make sense for your group, your area, and the time of year. Red leaves and pumpkins in New York in October, but not in Florida.

Then the phone goes in your pocket. When you find something, snap a photo (or snap them all as you walk and add them later), and an open-source AI model running right on the phone checks it off. If it isn't sure, you can try again or just count it anyway, because nobody should end up arguing with their phone in the middle of a walk.

At the end, your photos become a postcard: a collage with your own title, the date, your names, and a 🐾 stamp if the dog came. Save it, or share it through your own apps, on your terms.

We took it on a real walk. Mom had Tucker's leash and I had the phone. Her verdict: she thought it was cool. Here's our postcard:

Demo

Try it here. Works best on a phone, outside.

Code

melissa-24 / postcards-from-outside

postcards-from-outside

View on GitHub

How I Built It

The stack: Vite + React, Transformers.js running an open-weight CLIP model (Xenova/clip-vit-base-patch32) on the phone's CPU inside a Web Worker, OpenStreetMap's Nominatim for the optional location lookup, and a plain HTML canvas for the postcard. Hosted as a static site on DigitalOcean. No backend, no accounts, no API keys.

Starting from last week's lessons

Last week I built a restaurant picker for my mom, and running AI on the GPU crashed my laptop's Intel graphics over and over. So this time I started with what I knew worked: CPU only, with Transformers.js. I also picked the right kind of model. Last week a tiny text model struggled to write a sensible sentence. This week the AI's only job is to look at a photo and answer "is this the thing?", which is exactly what CLIP is good at.

Testing before building

Before writing any app code, I tested CLIP on real Wikimedia photos of my hunt items:

• The model is a 154 MB download and takes about 8 seconds to get ready, so it loads once on the "Getting ready" screen and stays loaded for the whole hunt.
• Each photo check takes about 0.15–0.3 seconds on my laptop. (Last week a single restaurant pick took around 30 seconds.)

Teaching it not to be fooled

CLIP compares a photo against a list of labels: the item you're looking for, plus a few decoys. My first test had a funny problem: a maple leaf passed as a fire hydrant. When none of the labels fit, CLIP still has to spread its confidence across them, so a wrong answer can win with a weak score.

The fix took a few rounds:

1. Add background labels like "grass," "pavement," and "the sky" to every check, so a photo of nothing in particular has somewhere to go. That fixed both false passes.
2. But then a real park bench lost to "pavement," because benches sit next to paths. So items can now opt out of competing with their own surroundings (skipBackground: ["pavement"] for the bench, "the sky" for clouds).
3. Require a minimum confidence: the item has to be CLIP's top answer and score at least 0.4.

The item list is hand-written, not AI

The AI never invents what to look for. I wrote the list myself: each item has who it suits, which regions, which months, and what the AI should compare it against. A small function builds each hunt by season and region, and treats "who's coming" as a preference, not a filter: items that suit everyone first, then items that suit someone, then anything else in season. That way the hunt always fills up.

Then we went outside

On our real walk, every photo came back "Hmm, not sure," even a very clear mailbox. Looking at the photos afterward, the reason was obvious: they were wide shots. The mailbox was a small part of the picture surrounded by driveway and lawn, the squirrel was a speck on a big lawn, and my "smooth rock" was really a bed of jagged landscaping rocks (so honestly, CLIP had a point).

So I changed two things:

• A tip on the hunt screen: get close so the thing fills the photo.
• Take a photo or choose one from your camera roll. That turned out to be a better way to play anyway: look at the list once, put the phone away, snap pictures with your normal camera as you walk, and check everything off afterward. The screen becomes the shortest part of the walk, which is the whole point.

Re-testing with my own close-up photos, most items passed. One acorn was called "a black walnut," which, fair: I'd made the decoy too similar, so I swapped it for something less confusable.

Working with a coding agent

I built this with Claude Code in VS Code, with a few ground rules: I wrote the screens, the item list, and the hunt builder; Claude Code wrote the trickier library pieces (the vision worker, location lookup, postcard drawing, sharing) and the experiments; and I ran every git command myself and tested everything on my own. I gave it a CLAUDE.md brief up front with the plan and last week's lessons, so it didn't repeat my GPU mistakes.

Why Does Open Innovation Matter?

Your photos never leave your phone. The AI runs on the device, so pictures of your walk, your yard, and your family aren't uploaded anywhere. They only leave when you save or share the postcard.

It costs nothing per photo. A cloud vision API charges per image. Here, every check runs on the phone, so it's free no matter how many people play or how many retakes it takes.

It works where signal is bad. Once the model is loaded, checking photos doesn't need the internet, which matters on trails. (Making the whole app work fully offline is next on my list.)

I could see inside it and tune it. Because the model runs locally, I could look at every score for every label and adjust the decoys and confidence threshold until it behaved. That's how I figured out the maple-leaf-as-fire-hydrant problem and the park bench problem. With a closed API, I'd have gotten a yes or no and no way to understand why.

Location is optional and minimal. If you type a place, it's looked up once with OpenStreetMap just to pick a rough region. Skip it, and nothing leaves the phone at all.

What's Next

• Challenge a friend: a link that gives a friend the same list to go find. It's built, but not quite working yet.
• Full offline mode as an installable app, so it opens on the trail with no signal.
• Saving a hunt in progress, so a long walk or a closed tab doesn't lose your list.
• Re-tuning with real walk photos, especially the misses.

Prize Categories

• Best Use of DigitalOcean: deployed on DigitalOcean App Platform as a static site.