Look Up: an open-weight model on your phone reads the clouds, then tells you to put the phone away

Kamelyoul — Tue, 06 Oct 2026 23:13:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Look Up is a tiny web app for cloud spotting.

You point your phone at the sky and take one photo. An open-weight vision model, running on the phone itself, tells you which of the ten cloud families you're looking at, what that cloud usually means for the next few hours, and gives you one small thing to watch for. Then it asks you to put the phone away.

That last part is the whole point. The best thing about clouds is that they move: a cumulus grows or evaporates in five minutes, a contrail vanishes or spreads, a halo appears around the sun. You can only see that if you stop looking at a screen. So every reading ends with a mission:

Cumulus: Pick one cloud and watch it for five minutes. Is it growing upward, or fraying at the edges and evaporating?

Cirrus: Pick one wisp and watch which way its tail is combed. That is the wind direction kilometres above you.

You tap Put the phone away. The screen goes black and counts the time until you come back (you can lock the phone). When you return, you tap what the cloud did ("It grew", "It faded", "It moved on"), and the app keeps a small sky log with one number I care about: how many seconds you spent looking up for every second you spent looking at the screen.

It's for anyone who has ever looked up and wondered "what is that?", but especially for people who reach for the phone out of habit. This time, the phone sends you back to the sky.

And when it sees a cumulonimbus, there is no timer. The mission is "go indoors".

Demo

👉 https://look-up-sky-smoky.vercel.app

Open it on your phone, tap Get ready for no signal once on Wi-Fi (it downloads the model, about 90 MB), then go outside and tap Read the sky. No account and no install (you can add it to your home screen if you like).

Code

Kamelyoul / look-up

Look Up — point your phone at the sky, learn the cloud, then put the phone away. On-device CLIP, works offline.

Look Up

Point your phone at the sky. An open-weight model running on your phone names the clouds, tells you what they usually mean, and gives you a small thing to watch for. Then you put the phone away.

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass (October 5 to 11, 2026). The project and this repository were created during the challenge window.

What it does

Read the sky: take a photo of the sky (or pick one you already took).
CLIP ViT-B/16 (open weights, MIT) runs in the browser through Transformers.js and ONNX Runtime Web. It classifies the photo zero-shot into the 10 WMO cloud genera, plus contrail, clear sky and not the sky.
You get a card with what to check with your own eyes, what that cloud usually means for the next hours (hedged weather lore), and a mission…

View on GitHub

Vanilla TypeScript and Vite, with no framework and no backend. MIT licensed. npm test runs the 34 unit tests; npm run eval reproduces the accuracy numbers below.

How I Built It

The model: CLIP, zero-shot, in the browser

The core is OpenAI's CLIP ViT-B/16, an open-weight (MIT) image–text model. I run it with Transformers.js on top of ONNX Runtime Web (WebAssembly), using Xenova's ONNX export at 8-bit.

CLIP was never trained to classify clouds. It doesn't need to be: it maps images and sentences into the same space, so I describe each class in plain English and pick the closest description. Each of the 13 classes (the ten WMO cloud genera, plus contrail, clear sky and not the sky) gets 3 to 6 prompts like:

prompts: [
  'a photo of cumulus clouds',
  'fluffy white cotton-ball cumulus clouds with flat bases in a blue sky',
  'puffy cauliflower-shaped white clouds on a sunny day',
],

The descriptive prompts matter. CLIP has seen "fluffy white clouds" millions of times and "stratocumulus" far less.

The trick: the text half never ships

CLIP has two towers: one for text and one for images. The prompts are fixed, so I run the text tower once, at build time, in Node, and ship only the 43 resulting vectors: a 120 KB JSON file. The phone downloads only the vision tower. That saves a download of tens of MB and makes classification a dot product:

// score of a class = its best-matching prompt (max, not mean)
for (let r = 0; r < bank.classes.length; r++) {
  let dot = 0;
  for (let i = 0; i < dim; i++) dot += img[i] * bank.matrix[r * dim + i];
  best.set(id, Math.max(best.get(id) ?? -1, dot));
}
// then softmax(100 * score) over the 13 classes

A few design decisions:

Max, not mean. The "not the sky" class mixes very different things (grass, a room, a screen, a street). Averaging over them gives a blurry "average non-sky" that matches nothing well. So a class scores as its best-matching prompt. On my test set, that rejected 22 of 24 non-sky photos, against 18 with averaging. Point it at your feet and it tells you to point it up.
It's allowed to be unsure. If the top class is under 45%, the card says "Altocumulus, or maybe Cirrocumulus — I'm torn". Real skies often contain several genera at once, and I'd rather the app say so than bluff.
Storms get special treatment. If "cumulonimbus" is the top guess, the runner-up, or simply above 10%, the card adds a red line: "This could also be a thunderstorm cloud. If it is tall and dark, or you hear thunder, go indoors instead."

Did it actually work? I measured it.

Zero-shot demos are easy to cherry-pick, so I built a small, reproducible test set from Wikimedia Commons: the first 12 JPEG photos in each cloud category whose file name names only that genus (132 sky photos), plus 24 photos of lawns and living rooms that the app should refuse. The evaluation script uses the exact code path the browser uses: same weights, same prompts, same scoring.

Metric (CLIP ViT-B/16, 8-bit)	Result
Top-1 accuracy, 13 classes (chance ≈ 7.7%)	43% (57/132)
Correct class in the top 3	68% (90/132)
Accuracy when the app says it's confident	69% (25/36)
Contrails	11/12
Thunderstorm photos that get the storm warning	9/12 (10 false alarms on 120 other skies)
Sky photos wrongly rejected as "not the sky"	0/132
Lawns and living rooms correctly rejected	22/24

It's good at the distinctive skies (contrails, cumulus, the altocumulus "sheep"). It's honestly bad at the subtle ones: cirrostratus (2/12) and stratocumulus (1/12) mostly come back as something else. That's why the app shows its top guesses and phrases everything as "check with your own eyes". It's a field guide that nudges you to look, not an oracle. (Caveat: Commons labels are crowd-sourced, and many photos contain more than one genus.)

Choosing the model with the numbers. I also ran the smaller-patch sibling, CLIP ViT-B/32. It's twice as fast and more precise when confident (84% vs 69%). But it caught only 5 of 12 thunderstorm photos where B/16 caught 9, and it raised more false alarms (16 vs 10). For an app that tells people when to go indoors, that settled it. I shipped B/16 and accepted the extra ~200 ms. The full per-image tables are in the repo (eval/).

The rest

Offline: a small service worker caches the app shell. Transformers.js stores the weights and the WASM runtime in the browser's Cache Storage, so after the first load it works in airplane mode.
"Phone away" screen: it's black, has a big counter and nothing else. Time is measured from timestamps, so it keeps counting if you lock the phone.
Sky log: localStorage only. Nothing leaves the device.
Tests: 34 unit tests (Vitest) cover the scoring and the base64 embedding format. A test fails if someone edits the prompts without regenerating the embeddings. Another checks that the thunderstorm card can never offer a "watch it" timer. A GitHub Actions workflow runs the tests and the build on every push.

Why Does Open Innovation Matter?

For this app, open weights aren't a nice-to-have. Three things only work because of them.

1. The sky is where the signal isn't. Hills, beaches, fields and trails are exactly where you have one bar or none. A cloud app that calls a hosted vision API fails in the very place it's meant to be used. With open weights, the model is a file. The browser caches it and the app works in airplane mode.

2. Your photos stay yours. Sky photos contain more than sky: rooftops, your street, sometimes your kids at the edge of the frame. Here, nothing is uploaded, because there's no server to upload to. I didn't have to write a privacy policy for a server I'd have to trust; there just isn't one.

3. It costs nothing to run, so it can stay free. There is no API key, no per-request bill and no rate limit. The whole thing is a static site. If a thousand people use it on a sunny Saturday, my cost doesn't change.

There's a fourth, more personal reason: I could open it up and change it. Because the model is open, I could split it in two, run half of it at build time, and ship only the half the phone needs. You can't do that with a closed endpoint. The classes are just sentences in a TypeScript file too. Want to add mammatus or noctilucent clouds? Add a line, run npm run embeddings, and redeploy. No retraining, no permission needed.

Where closed models would win: a large hosted multimodal model would likely be more accurate on the hard pairs (altocumulus vs. cirrocumulus). But it would need a signal, see your photos, and cost money per look. For an app whose job is to get you off the screen and out where there's no Wi-Fi, a small model you can carry beats a big one you can't.

My Agent Session

Full disclosure: I built this with an AI coding agent (Claude Code). It wrote most of the code, the tests and the evaluation scripts, and helped draft this post. Every number above comes from scripts in the repo that you can rerun (npm test, npm run eval), not from the agent's say-so. I don't have a DevRelay recording of the session.

DEV Community: Kamelyoul