Hoard: an AI that tidies the screenshots you were never going to tidy

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Screenshots are the laziest note-taking system ever invented, and I mean that as a compliment. Wi-Fi password at your mum's? Screenshot. Where you parked? Screenshot. Boarding pass, the address someone texted, the recipe, the error message you'll Google later: screenshot, screenshot, screenshot.

The catch is the other end. Finding one again means scrolling through thousands of near-identical white rectangles, and the pile only grows, because nobody has ever sat down on a Sunday to organise their screenshots. Nobody ever will.

Hoard is for people like that, and it's built so they never have to.

• Point it at your screenshots folder once. It reads every screenshot slowly and steadily in a background tab, newest first. It picks up where it stopped if you close it, and catches new screenshots each time you come back.
• Search by what you remember. "wifi password", "where did I park", "that sunset photo". Hoard has read every word on every screenshot and looked at the picture, so it finds things by their text, their meaning or their looks.
• Let it sweep. Duplicates, one-time codes that died five minutes after they arrived, boarding passes for flights you already took, and accidental screenshots of your home screen, gathered into one pile. One click moves them into a Swept by Hoard folder. Nothing is ever deleted, and there's an undo.
• Ask it. "when is rent due?" Gemma 3 reads the screenshots that match best and answers in one sentence, with the screenshot underneath so you can check it.

All of it runs inside the browser tab, on open models. No account, no upload, no server. After the first visit it works with Wi-Fi off, which matters more than you'd think: the moment you most need your boarding pass is at the gate, with flight mode on.

Demo

Try it: hoard-905r.onrender.com in Chrome or Edge on a laptop.

You don't need your own screenshots: click Try a sample pile and Hoard draws 40 made-up screenshots of a fictional person's phone (chats, receipts, a boarding pass, OTP texts, a map, a sunset, duplicates, a blank screen) and indexes them in a minute or two (after a one-time ~250 MB model download). Then try:

• where did I park (the note says "Level 3, Bay C14", and never the word "park")
• sunset photo (a picture with no words in it)
• the Sweep tab: 11 screenshots it thinks you can let go of, and why
• when is rent due? then Ask (downloads Gemma once, ~790 MB)

Code

Eternity0207 / hoard

Search and sweep your screenshot pile with open AI models running in your browser.

Hoard

Search and sweep your screenshot pile with open AI models that run in your browser.

You take screenshots so you don't have to remember things: the Wi‑Fi password, where you parked, a boarding pass, an address someone texted you. Then you have 4,000 of them and you can't find any of it.

Hoard reads the folder for you, slowly and steadily, in a background tab:

• Search by memory. "wifi password", "where did I park", "that sunset photo". Each screenshot is OCR'd and embedded twice (its text and its picture), and search mixes keywords, meaning and looks.
• Sweep the junk. Duplicates, one-time codes that died minutes after you got them, boarding passes for flights you already took, and accidental screenshots of your home screen. One click moves them into a Swept by Hoard subfolder. Nothing is ever deleted.
• Ask. "when is rent due?" Gemma 3 reads the best-matching screenshots and…

View on GitHub

How I Built It

Hoard is static files: Vite and vanilla JavaScript, no framework. The plumbing for running models offline in a browser (a Web Worker with Transformers.js and ONNX Runtime Web, an Origin Private File System model cache, a service worker, and the WASM runtimes served from the app's own origin) is reused from my Week 1 and Week 2 projects, so this week went into the parts that are actually about screenshots.

Four open models, one job each

Job	Model	Licence	Download
Read the words	Tesseract 5 LSTM, via tesseract.js	Apache-2.0	5 MB
Understand the picture	SigLIP base (int8)	Apache-2.0	210 MB
Understand the words	gte-small (int8)	MIT	34 MB
Answer questions (optional)	Gemma 3 1B (4-bit, WebGPU)	Gemma terms	790 MB

For each screenshot, Hoard makes a thumbnail, a perceptual hash and a "how blank is this?" score (no AI needed for those), then runs OCR, embeds the picture with SigLIP and embeds the OCR text with gte-small. A few strict text rules decide its kind (a boarding pass says "boarding pass"), and SigLIP decides the rest by comparing the picture to short descriptions ("a screenshot of a chat conversation", "a street map"...). Everything goes into IndexedDB, one record at a time, which is what makes indexing resumable.

I measured before I tuned

Before building the UI, I wrote the sample pile as a canvas drawing script, so the demo and the tests need no real data and no image files in the repo. Then I wrote 30 search queries to tune on and, before looking at a single result, 20 more held-out queries that I promised myself I'd only score at the end.

Search mixes three signals, and the eval shows why you need all three:

Search by	Tuning hit@1	Held-out hit@1
Keywords (BM25 over the OCR text)	70%	75%
Meaning (gte-small over the OCR text)	83%	85%
Picture (SigLIP)	90%	85%
All three	93%	100%

Each signal fails in its own way. Keywords alone matched sunset photo to the home screen, because of the little "Photos" label under an app icon, and where did I park to the dentist, because the clinic is on "Park Road". Meaning gets "park" ≈ "Car, Level 3, Bay C14" but can't see a sunset. SigLIP can see the sunset, and it surprised me by reading large text too, but it only gets a 224-pixel squashed thumbnail, so small print is invisible to it.

The trick that made mixing them work was to stop comparing raw scores. On the sample pile, gte-small's similarities sit in a narrow band around 0.76 (give or take 0.035), while SigLIP's hover around zero and a great match is 0.15, so adding them up lets one model shout over the other. Hoard converts each to a z-score across your whole pile ("how unusually close is this screenshot?"), ignores anything below average, and adds them with weights picked on the tuning queries: 1.0 × keywords + 0.75 × meaning + 1.5 × picture.

An honest caveat: I drew the sample pile myself, with big clean text. Real screenshots are messier, so read these numbers as "the design works", not as a benchmark. The eval page ships in the repo (npm run dev, then open /eval.html), so anyone can run it on their own changes.

The bugs the eval caught

Tesseract was skipping the most important line. The first OCR run read everything on the boarding pass except the words "BOARDING PASS", and missed the band name on the concert ticket. Both were white text on a coloured bar. It turns out tesseract.js defaults to page segmentation mode 6, which treats the image as one uniform block of text. A screenshot is a page of scattered blocks; switching to automatic segmentation (mode 3) found both headers.

A line break hid a one-time code. G-552019 is your verification code. wasn't flagged as a stale code, because the phone had wrapped it as verification / code. and my regex wanted a space. Now whitespace is normalised first.

A receipt looked like a ticket. "Paid to Priya, for: concert tickets" has a date in the past and the word "tickets", so Sweep called it an expired ticket. The expired rule now needs words that only appear on a ticket (boarding pass, gate and seat, "show this at the door").

After those fixes, Sweep flags 11 of the 11 screenshots it should on the sample pile, and nothing else.

Gemma: what changed when I measured it

Ask hands the OCR text of the best matches to Gemma 3 1B and asks for the exact detail in one sentence. My first prompt showed it the top three screenshots. On eight questions, the better of two wordings got 5 right, and between them they invented things: one "paid Priya €47" (it was ₹2,500; OCR had lost the ₹ sign and turned a 🎶 emoji into "47"), and the other decided I'd parked at a concert hall, because a map was in the context.

A 1B model is easily distracted, so I gave it fewer distractions: only the best match, plus up to two others if they scored within 70% of it, each labelled with its kind ("a note", "a receipt") and date, and an instruction to copy the detail rather than paraphrase it. On the same eight questions it got 6 right and stopped inventing answers. With two more questions added, ten in all:

• 7 right: Wi-Fi password, payment amount, hotel confirmation number, dentist time, tracking number, rent due date, party address
• 1 half right: the gate number is there, garbled with the seat
• 1 declined: "where did I park?" got "I couldn't find that", although search had put the right note first
• 1 wrong: "who is bringing snacks?" OCR can't tell which chat bubbles are yours, so it credited the other person

Median answer time also dropped, from 13 to 8 seconds on my laptop's integrated GPU, because there's less to read. It's good enough to keep, but not good enough to trust blindly, so every answer shows the screenshot it came from, and the label says so. Search is the product; Ask is the shortcut on top of it.

Sweeping without fear

A tool that deletes your files had better be right, so Hoard doesn't delete anything. Every rule explains itself in the UI, every suggestion has a checkbox, "Keep forever" excludes a screenshot permanently, and sweeping just moves files into a Swept by Hoard subfolder using the File System Access API. Undo moves them back to wherever they came from, subfolders included. I tested the whole flow (scan, sweep, undo, rescan) against a fake folder in the browser's private file system, which speaks the same API as a real one.

The duplicate rule needed two signals. A perceptual hash alone thinks every chat screenshot from the same app is a duplicate (white background, bubbles); text alone would merge two different photos with no words. So two screenshots are duplicates only if they look alike and say the same thing, and Hoard keeps the copy with the most text.

Slow and steady, on purpose

Indexing has three paces, under Settings:

• Sloth: one screenshot at a time, with a nap in between. The fans stay quiet and you forget it's running.
• Steady (default): two at a time, so OCR and the embeddings overlap. About 1.7 seconds per screenshot on my laptop, so a thousand screenshots take roughly half an hour in a background tab.
• Sprint: four at a time with two OCR workers, for when you're impatient.

It works newest first, because last week's screenshots are the ones you go looking for, and search works on whatever has been read so far.

The offline bug

The first time I tested offline mode properly (load the app, stop the server, reload), it showed a blank page. Two bugs were stacked on top of each other:

1. The first visit fetches the app's JavaScript before the service worker is in control, so it was never cached. The build now writes the list of hashed bundle files into the service worker, which precaches them on install.
2. Even then, the cached files didn't match. The server sends Vary: Origin, module scripts are requested with an Origin header, and the precache requests weren't, so caches.match politely said "no". ignoreVary: true fixed it.

Now the app loads with the server gone, the models come off disk, and where did I park still finds the car.

Hosting

Hoard is a free static site on Render, set up by a render.yaml Blueprint. The setting that matters is the pair of cross-origin isolation headers (COOP: same-origin, COEP: credentialless), which unlock SharedArrayBuffer, so ONNX Runtime can use every CPU core.

Why Does Open Innovation Matter?

Screenshots are the most private files on your phone. Look at the sample pile: bank login codes, private chats, a home address, a boarding pass with a name on it, a prescription, someone's mum's Wi-Fi password. Real folders are worse. A cloud screenshot organiser has to see all of that to be useful. Hoard runs open models in the browser tab, so the screenshots never leave the device. There's no upload code to trust; open DevTools → Network while it indexes and see for yourself.

Slow is only affordable when it's free. With a paid vision API, indexing 4,000 screenshots is 4,000 billed calls, and so is re-indexing them after every improvement. That pushes you to process as little as possible as fast as possible. With open weights on your own laptop, the marginal cost is zero, so Hoard can take its time: read everything, at sloth pace, in the background, and do it again whenever the pipeline improves. The sloth approach only works because nobody is charging per leaf.

Open let me look inside and fix things. The OCR bug that cost me "BOARDING PASS" was a configuration default I could read in the source and change in one line. The z-score idea came from looking at the raw similarity scores of two models side by side. I ran the exact same weights in an eval page and in the app, so the numbers in this post are the numbers you get. With closed OCR and embedding APIs, I'd have been tuning a black box.

It works where you need it. No signal on the plane, no signal in the car park, roaming abroad: the screenshots you need most are often needed exactly where the internet isn't. A local model doesn't care.

Prize Categories

• Best Use of Gemma: Gemma 3 1B runs on-device over WebGPU (or the 270M model on CPU) to answer questions from the user's own screenshots, with the prompt redesigned after measuring it: from 5 to 6 right on the same eight questions, no invented answers, and a median of 8 seconds instead of 13.
• Best Use of Render: the app is hosted on Render as a static site, with a Blueprint that sets the cross-origin isolation headers multithreaded in-browser inference needs.