DEV Community: Arsh Goyal

Hoard: an AI that tidies the screenshots you were never going to tidy

Arsh Goyal — Tue, 20 Oct 2026 09:19:40 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Screenshots are the laziest note-taking system ever invented, and I mean that as a compliment. Wi-Fi password at your mum's? Screenshot. Where you parked? Screenshot. Boarding pass, the address someone texted, the recipe, the error message you'll Google later: screenshot, screenshot, screenshot.

The catch is the other end. Finding one again means scrolling through thousands of near-identical white rectangles, and the pile only grows, because nobody has ever sat down on a Sunday to organise their screenshots. Nobody ever will.

Hoard is for people like that, and it's built so they never have to.

Point it at your screenshots folder once. It reads every screenshot slowly and steadily in a background tab, newest first. It picks up where it stopped if you close it, and catches new screenshots each time you come back.
Search by what you remember. "wifi password", "where did I park", "that sunset photo". Hoard has read every word on every screenshot and looked at the picture, so it finds things by their text, their meaning or their looks.
Let it sweep. Duplicates, one-time codes that died five minutes after they arrived, boarding passes for flights you already took, and accidental screenshots of your home screen, gathered into one pile. One click moves them into a Swept by Hoard folder. Nothing is ever deleted, and there's an undo.
Ask it. "when is rent due?" Gemma 3 reads the screenshots that match best and answers in one sentence, with the screenshot underneath so you can check it.

All of it runs inside the browser tab, on open models. No account, no upload, no server. After the first visit it works with Wi-Fi off, which matters more than you'd think: the moment you most need your boarding pass is at the gate, with flight mode on.

Demo

Try it: hoard-905r.onrender.com in Chrome or Edge on a laptop.

You don't need your own screenshots: click Try a sample pile and Hoard draws 40 made-up screenshots of a fictional person's phone (chats, receipts, a boarding pass, OTP texts, a map, a sunset, duplicates, a blank screen) and indexes them in a minute or two (after a one-time ~250 MB model download). Then try:

where did I park (the note says "Level 3, Bay C14", and never the word "park")
sunset photo (a picture with no words in it)
the Sweep tab: 11 screenshots it thinks you can let go of, and why
when is rent due? then Ask (downloads Gemma once, ~790 MB)

Code

Eternity0207 / hoard

Search and sweep your screenshot pile with open AI models running in your browser.

Hoard

Search and sweep your screenshot pile with open AI models that run in your browser.

You take screenshots so you don't have to remember things: the Wi‑Fi password, where you parked, a boarding pass, an address someone texted you. Then you have 4,000 of them and you can't find any of it.

Hoard reads the folder for you, slowly and steadily, in a background tab:

Search by memory. "wifi password", "where did I park", "that sunset photo". Each screenshot is OCR'd and embedded twice (its text and its picture), and search mixes keywords, meaning and looks.
Sweep the junk. Duplicates, one-time codes that died minutes after you got them, boarding passes for flights you already took, and accidental screenshots of your home screen. One click moves them into a Swept by Hoard subfolder. Nothing is ever deleted.
Ask. "when is rent due?" Gemma 3 reads the best-matching screenshots and…

View on GitHub

How I Built It

Hoard is static files: Vite and vanilla JavaScript, no framework. The plumbing for running models offline in a browser (a Web Worker with Transformers.js and ONNX Runtime Web, an Origin Private File System model cache, a service worker, and the WASM runtimes served from the app's own origin) is reused from my Week 1 and Week 2 projects, so this week went into the parts that are actually about screenshots.

Four open models, one job each

Job	Model	Licence	Download
Read the words	Tesseract 5 LSTM, via tesseract.js	Apache-2.0	5 MB
Understand the picture	SigLIP base (int8)	Apache-2.0	210 MB
Understand the words	gte-small (int8)	MIT	34 MB
Answer questions (optional)	Gemma 3 1B (4-bit, WebGPU)	Gemma terms	790 MB

For each screenshot, Hoard makes a thumbnail, a perceptual hash and a "how blank is this?" score (no AI needed for those), then runs OCR, embeds the picture with SigLIP and embeds the OCR text with gte-small. A few strict text rules decide its kind (a boarding pass says "boarding pass"), and SigLIP decides the rest by comparing the picture to short descriptions ("a screenshot of a chat conversation", "a street map"...). Everything goes into IndexedDB, one record at a time, which is what makes indexing resumable.

I measured before I tuned

Before building the UI, I wrote the sample pile as a canvas drawing script, so the demo and the tests need no real data and no image files in the repo. Then I wrote 30 search queries to tune on and, before looking at a single result, 20 more held-out queries that I promised myself I'd only score at the end.

Search mixes three signals, and the eval shows why you need all three:

Search by	Tuning hit@1	Held-out hit@1
Keywords (BM25 over the OCR text)	70%	75%
Meaning (gte-small over the OCR text)	83%	85%
Picture (SigLIP)	90%	85%
All three	93%	100%

Each signal fails in its own way. Keywords alone matched sunset photo to the home screen, because of the little "Photos" label under an app icon, and where did I park to the dentist, because the clinic is on "Park Road". Meaning gets "park" ≈ "Car, Level 3, Bay C14" but can't see a sunset. SigLIP can see the sunset, and it surprised me by reading large text too, but it only gets a 224-pixel squashed thumbnail, so small print is invisible to it.

The trick that made mixing them work was to stop comparing raw scores. On the sample pile, gte-small's similarities sit in a narrow band around 0.76 (give or take 0.035), while SigLIP's hover around zero and a great match is 0.15, so adding them up lets one model shout over the other. Hoard converts each to a z-score across your whole pile ("how unusually close is this screenshot?"), ignores anything below average, and adds them with weights picked on the tuning queries: 1.0 × keywords + 0.75 × meaning + 1.5 × picture.

An honest caveat: I drew the sample pile myself, with big clean text. Real screenshots are messier, so read these numbers as "the design works", not as a benchmark. The eval page ships in the repo (npm run dev, then open /eval.html), so anyone can run it on their own changes.

The bugs the eval caught

Tesseract was skipping the most important line. The first OCR run read everything on the boarding pass except the words "BOARDING PASS", and missed the band name on the concert ticket. Both were white text on a coloured bar. It turns out tesseract.js defaults to page segmentation mode 6, which treats the image as one uniform block of text. A screenshot is a page of scattered blocks; switching to automatic segmentation (mode 3) found both headers.

A line break hid a one-time code. G-552019 is your verification code. wasn't flagged as a stale code, because the phone had wrapped it as verification / code. and my regex wanted a space. Now whitespace is normalised first.

A receipt looked like a ticket. "Paid to Priya, for: concert tickets" has a date in the past and the word "tickets", so Sweep called it an expired ticket. The expired rule now needs words that only appear on a ticket (boarding pass, gate and seat, "show this at the door").

After those fixes, Sweep flags 11 of the 11 screenshots it should on the sample pile, and nothing else.

Gemma: what changed when I measured it

Ask hands the OCR text of the best matches to Gemma 3 1B and asks for the exact detail in one sentence. My first prompt showed it the top three screenshots. On eight questions, the better of two wordings got 5 right, and between them they invented things: one "paid Priya €47" (it was ₹2,500; OCR had lost the ₹ sign and turned a 🎶 emoji into "47"), and the other decided I'd parked at a concert hall, because a map was in the context.

A 1B model is easily distracted, so I gave it fewer distractions: only the best match, plus up to two others if they scored within 70% of it, each labelled with its kind ("a note", "a receipt") and date, and an instruction to copy the detail rather than paraphrase it. On the same eight questions it got 6 right and stopped inventing answers. With two more questions added, ten in all:

7 right: Wi-Fi password, payment amount, hotel confirmation number, dentist time, tracking number, rent due date, party address
1 half right: the gate number is there, garbled with the seat
1 declined: "where did I park?" got "I couldn't find that", although search had put the right note first
1 wrong: "who is bringing snacks?" OCR can't tell which chat bubbles are yours, so it credited the other person

Median answer time also dropped, from 13 to 8 seconds on my laptop's integrated GPU, because there's less to read. It's good enough to keep, but not good enough to trust blindly, so every answer shows the screenshot it came from, and the label says so. Search is the product; Ask is the shortcut on top of it.

Sweeping without fear

A tool that deletes your files had better be right, so Hoard doesn't delete anything. Every rule explains itself in the UI, every suggestion has a checkbox, "Keep forever" excludes a screenshot permanently, and sweeping just moves files into a Swept by Hoard subfolder using the File System Access API. Undo moves them back to wherever they came from, subfolders included. I tested the whole flow (scan, sweep, undo, rescan) against a fake folder in the browser's private file system, which speaks the same API as a real one.

The duplicate rule needed two signals. A perceptual hash alone thinks every chat screenshot from the same app is a duplicate (white background, bubbles); text alone would merge two different photos with no words. So two screenshots are duplicates only if they look alike and say the same thing, and Hoard keeps the copy with the most text.

Slow and steady, on purpose

Indexing has three paces, under Settings:

Sloth: one screenshot at a time, with a nap in between. The fans stay quiet and you forget it's running.
Steady (default): two at a time, so OCR and the embeddings overlap. About 1.7 seconds per screenshot on my laptop, so a thousand screenshots take roughly half an hour in a background tab.
Sprint: four at a time with two OCR workers, for when you're impatient.

It works newest first, because last week's screenshots are the ones you go looking for, and search works on whatever has been read so far.

The offline bug

The first time I tested offline mode properly (load the app, stop the server, reload), it showed a blank page. Two bugs were stacked on top of each other:

The first visit fetches the app's JavaScript before the service worker is in control, so it was never cached. The build now writes the list of hashed bundle files into the service worker, which precaches them on install.
Even then, the cached files didn't match. The server sends Vary: Origin, module scripts are requested with an Origin header, and the precache requests weren't, so caches.match politely said "no". ignoreVary: true fixed it.

Now the app loads with the server gone, the models come off disk, and where did I park still finds the car.

Hosting

Hoard is a free static site on Render, set up by a render.yaml Blueprint. The setting that matters is the pair of cross-origin isolation headers (COOP: same-origin, COEP: credentialless), which unlock SharedArrayBuffer, so ONNX Runtime can use every CPU core.

Why Does Open Innovation Matter?

Screenshots are the most private files on your phone. Look at the sample pile: bank login codes, private chats, a home address, a boarding pass with a name on it, a prescription, someone's mum's Wi-Fi password. Real folders are worse. A cloud screenshot organiser has to see all of that to be useful. Hoard runs open models in the browser tab, so the screenshots never leave the device. There's no upload code to trust; open DevTools → Network while it indexes and see for yourself.

Slow is only affordable when it's free. With a paid vision API, indexing 4,000 screenshots is 4,000 billed calls, and so is re-indexing them after every improvement. That pushes you to process as little as possible as fast as possible. With open weights on your own laptop, the marginal cost is zero, so Hoard can take its time: read everything, at sloth pace, in the background, and do it again whenever the pipeline improves. The sloth approach only works because nobody is charging per leaf.

Open let me look inside and fix things. The OCR bug that cost me "BOARDING PASS" was a configuration default I could read in the source and change in one line. The z-score idea came from looking at the raw similarity scores of two models side by side. I ran the exact same weights in an eval page and in the app, so the numbers in this post are the numbers you get. With closed OCR and embedding APIs, I'd have been tuning a black box.

It works where you need it. No signal on the plane, no signal in the car park, roaming abroad: the screenshots you need most are often needed exactly where the internet isn't. A local model doesn't care.

Prize Categories

Best Use of Gemma: Gemma 3 1B runs on-device over WebGPU (or the 270M model on CPU) to answer questions from the user's own screenshots, with the prompt redesigned after measuring it: from 5 to 6 right on the same eight questions, no invented answers, and a median of 8 seconds instead of 13.
Best Use of Render: the app is hosted on Render as a static site, with a Blueprint that sets the cross-origin isolation headers multithreaded in-browser inference needs.

Shoebox: a bookkeeper for freelancers that never sees your bank statement

Arsh Goyal — Thu, 15 Oct 2026 08:21:24 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Every freelancer I know has the same January. You download twelve months of bank statements, open a spreadsheet, and spend a weekend deciding whether SQ *PHILZ COFFEE SAN FRANCISCO was a client meeting or just coffee. Then you hand your accountant a CSV and hope.

People pay real money to skip that weekend. Apps like Keeper and QuickBooks charge a monthly subscription for it, and the deal includes connecting your bank account to their servers so their models can read every transaction you've ever made.

Shoebox does the same job without the second part.

Drop in a CSV from any bank or card. Shoebox guesses which column is which (one amount column or separate debit/credit, day-first dates, cards that list purchases as positive numbers) and shows a preview so you can fix it.
It sorts every transaction into US Schedule C categories: meals, travel, software, contract labor, personal, transfers and so on. Lines it's confident about are filed automatically.
It asks about the rest, one merchant at a time. The "Needs review" list is grouped by merchant, so one answer covers all 13 trips to the same coffee shop. Every answer becomes a rule, so next month's statement needs fewer.
It adds it all up: gross income, deductible expenses by Schedule C line (meals at 50%), net profit, a rough self-employment tax estimate, and quarter-by-quarter numbers for estimated payments. Export a CSV, a Markdown worksheet for your accountant, or print it.

The AI runs inside your browser tab. There's no server, no account, no bank login and no analytics. After the first visit it works with Wi-Fi off.

Who would pay for it: freelancers and sole proprietors who do their own books, or who pay an accountant by the hour to do it. A one-time $29 licence is an easy sell against a subscription that renews every month, and since inference runs on the customer's laptop, a new customer costs me nothing to serve. That's the part of the business that only works because the model is open.

Demo

Try it: shoebox-hcdp.onrender.com. It works in any modern browser; the first visit downloads a 34 MB model, and after that it runs offline.

Click Try a sample freelancer's statement to load nine months of a fictional designer's checking account (321 transactions). No real data needed.

Run it yourself:

git clone https://github.com/Eternity0207/shoebox
cd shoebox && npm install && npm run dev

Code

Eternity0207 / shoebox

Private bookkeeping for freelancers: sorts bank statements into Schedule C categories with an open model running in your browser.

Shoebox

Private bookkeeping for freelancers. Drop in a bank or credit card CSV and Shoebox sorts every transaction into US Schedule C tax categories, learns from your corrections, and adds it all up for tax time.

The AI runs in your browser. Your bank statement is never uploaded: there is no server, no account and no analytics. After the first visit it works offline.

What it does

Imports any bank's CSV. It guesses which column is the date, description and amount (or separate debit/credit columns, day-first dates, cards that list purchases as positive), shows you a preview, and lets you fix the mapping. Re-importing an overlapping statement doesn't create duplicates.
Sorts transactions into tax categories with a small open embedding model. Lines it's sure about are filed automatically; the rest go into a short "Needs review" list, grouped by merchant.
Learns from you. Every correction becomes a rule for that…

View on GitHub

How I Built It

Shoebox is static files: Vite and vanilla JavaScript, no framework. The model runs in a Web Worker through Transformers.js and ONNX Runtime Web, on the CPU. Transactions and rules live in IndexedDB. The plumbing for offline use (an Origin Private File System model cache, a service worker, ONNX Runtime served from the app's own origin) is reused from my Week 1 project, Pocket Field Guide, which saved me a day.

The pipeline

A bank line goes through five steps:

Clean the name. POS PURCHASE SQ *PHILZ COFFEE #12 SAN FRANCISCO CA → philz coffee san francisco. This is boring regex work (processor prefixes, store numbers, phone numbers, state codes, FACEBK → facebook ads), and it matters more than any model choice.
Your rules. If you've told Shoebox about this merchant before, that wins.
Your corrections, generalized. If the line is very close in embedding space to one you corrected, it copies your answer.
Zero-shot ranking. Each tax category has a plain-English hint listing the kinds of merchants that belong in it ("gas station fuel, parking garages, highway tolls, oil change, auto repair shop..."). The merchant name is embedded and scored against each hint. The sign of the amount rules out impossible answers (money coming in can't be an expense).
A confidence gate. If the top category's probability is at least 0.5, it's filed. Otherwise you're asked.

Picking the model by measuring

Before writing any UI, I hand-labelled 145 bank lines written the way real exports look, with processor prefixes, truncated names and store numbers. Then I ran small open embedding models over them:

Model	Top-1	Top-3	Download
all-MiniLM-L6-v2	84.8%	93.8%	23 MB
bge-small-en-v1.5	86.2%	97.9%	34 MB
gte-small	90.3%	97.2%	34 MB
EmbeddingGemma-300M	87.6%	97.9%	309 MB

gte-small won, at a tenth of EmbeddingGemma's size. One trick helped every model: score each merchant against the whole category hint and against each example phrase in it, then average. Whole-hint alone gave gte-small 80%.

Then I calibrated the gate: at a confidence of 0.5 or above, gte-small files 106 of the 145 lines by itself and gets 99.1% of those right. Everything below that goes to a human. For tax records, I'd rather ask than guess.

The number that matters: clicks

Accuracy on a test set isn't what a freelancer feels. What they feel is how many times they have to click. So I scripted a user who works through "Needs review" biggest group first, and ran it on the sample statement, both in Node and in the real app in the browser:

321 transactions from 65 merchants, sorted in about 3.5 seconds on a laptop CPU
72 lines from 17 merchants flagged for review
17 clicks to clear them (10 changes, 7 "✓ looks right")
0 lines wrong at the end

One honest caveat: I wrote both the sample statement and the category hints, so this is a best case. Real statements will have weirder merchants. But the shape holds, because every click teaches Shoebox a whole merchant, not just one line.

The bug that only showed up in the browser

The browser flagged 90 lines where my Node script flagged 77. Same model, same code. It turned out that with the 8-bit model, padding a string into a batch shifts its embedding enough to flip near-ties: "hyatt regency sacramento" scored Travel 0.29 / Meals 0.28 on its own, and the opposite inside a batch. In other words, a merchant's category depended on what else was in the file.

The fix was to embed strings one at a time. It's still fast, because a statement has far fewer merchants than lines. Then I precomputed the category vectors at build time, so the browser and my eval scripts use identical numbers. Now both report exactly 72 lines from 17 merchants, and the app is ready a second or two after it opens instead of 7.5.

I tried Gemma, and cut it

My plan was a two-model setup: the small embedding model for the easy lines, and Gemma 3 1B on WebGPU for a second opinion on the uncertain ones. I built it, then measured it on the lines it would actually see (the ones below the confidence gate):

On the 39 uncertain lines	Correct
gte-small alone	66.7%
Gemma picks from the top 3 (with category hints)	51.3%
Gemma picks from the top 3 (names only)	43.6%
Gemma describes the business, gte-small ranks that	66.7%

The first version scored below chance. That was my bug: I'd prefilled the answer with Answer: (trailing space), so the model's most likely next token was a newline, and I was reading noise from the letter logits. Fixed, multiple choice was still worse than the small model alone. With a list of category hints in front of it, the 1B model pattern-matched the hint text ("Logitech" → Software) instead of using what it knows about the merchant.

Playing to Gemma's strength did better. Asking "what kind of business is this?" and letting the embedding model categorize the answer exactly tied the small model alone: it fixed some lines and broke the same number of others. It also invented things: "nintendo" → auto repair shop, "bart clipper reload" → mobile phone retailer.

An ~800 MB download that's sometimes confidently wrong has no place in a tax tool. So Shoebox ships with one 34 MB model and the person who knows their own business. The experiment stays in the repo (scripts/eval-gemma-variants.mjs) so anyone can rerun it with a bigger model.

Hosting

Shoebox is hosted on Render as a free static site, configured by a render.yaml Blueprint. The one setting that matters is the cross-origin isolation headers (COOP: same-origin, COEP: credentialless), which let ONNX Runtime use every CPU core.

Why Does Open Innovation Matter?

The data is the whole problem. A bank statement is a complete map of someone's life: where they eat, who they pay, what they earn, which doctor they see. The closed version of this product starts with "connect your bank account" and ends with that map on someone else's server. With an open model running in the browser, there's nothing to connect and nothing to leak. You can check it yourself: open DevTools → Network, import a statement, and watch nothing go out.

The economics only work because inference is free. A closed API charges per call, so a bookkeeping app built on one has to meter usage, raise prices or both. Shoebox's marginal cost per customer is zero, because every customer brings their own CPU. That's what makes a one-time price possible.

Open let me measure and choose. I compared four embedding models and a 1B LLM on my own data in an afternoon, picked the one that was best for this job, and threw away the one that wasn't. With a closed API I'd have been stuck with whatever the endpoint gave me, and I'd never have found the batching bug, because I couldn't have run the same weights in Node and in the browser and compared the numbers.

It's yours to change. The categories are a plain JavaScript file. A freelancer in another country can rewrite them for their own tax forms, run npm run embed, and the model understands the new categories immediately, with no fine-tuning and no permission needed.

Prize Categories

Best Use of Render: the app is hosted on Render as a static site, with a Blueprint that sets the cross-origin isolation headers multithreaded in-browser inference needs.

Pocket Field Guide: an AI that wants you to put your phone away

Arsh Goyal — Sat, 10 Oct 2026 05:37:10 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Most nature apps are built like every other app: they want your attention. You point the camera at a bird and get an answer, and then you're scrolling a species page while the bird flies off.

Pocket Field Guide is built the other way round. It's a browser app with two models running entirely on your device, and every screen is designed to send you back outside.

Go outside. Tell it you have 30 minutes in a park. Gemma writes four things to find ("Touch the rough surface of a mossy rock", "Listen for a bird singing from a rooftop"). Then the app switches to a plain screen with a timer and a checklist, and tells you to put the phone away.
Identify. When you spot something, take one photo. SigLIP names it from a built-in field guide of about 150 common species. Gemma then writes a short field note that doesn't tell you facts to read: it tells you what to check in person to confirm it, and one related thing to go find next.
Journal. Finds go into an on-device journal, with an optional location that never leaves the browser. You can export it as Markdown.

There's no account, no API key and no server. After the first download it works in airplane mode, which is the point: the places worth walking usually have the worst signal.

It's for anyone who wants a reason to go for a walk and a little help noticing things on the way. It's made for beginners.

Demo

Try it: pocket-field-guide.onrender.com. Use Chrome or Edge on a laptop. Open Setup, download the models once (~900 MB), and then it works offline.

New to it? The built-in guide explains every feature, how to check that offline mode works, and troubleshooting.

Code

Eternity0207 / pocket-field-guide

On-device AI field guide that gets you outside. Gemma 3 + SigLIP running in the browser, works offline.

Pocket Field Guide 🌿

An AI field guide that runs entirely in your browser and is built to get you off the screen.

Go outside: pick 15, 30 or 60 minutes and where you are. Gemma writes a short list of things to look for, then the app switches to a quiet walking screen with a timer and a checklist.
Identify: take or drop in a photo of a plant, bird, bug, mushroom or leaf. SigLIP matches it against a built-in field guide of about 150 species, and Gemma writes a short field note: what to check in person to confirm it, plus something related to find nearby.
Journal: finds are saved on your device with an optional location, and you can export them as Markdown.

No account, no API key, no server. After the first download it works with no internet at all.

New to the app? The…

View on GitHub

How I Built It

The whole app is static files: Vite and vanilla JavaScript, with no framework. All the AI runs in a Web Worker through Transformers.js and ONNX Runtime Web.

Two open models, each doing what it's good at

Job	Model	Where it runs
"What is this?"	SigLIP base (Google, Apache-2.0), 8-bit vision tower, ~95 MB	CPU (WASM, all threads)
Field notes, walk ideas	Gemma 3 1B instruct, 4-bit, ~785 MB	GPU (WebGPU)
Fallback when there's no GPU	Gemma 3 270M, fp16	CPU

Identification is zero-shot. Every species in labels.js is described by three short phrasings ("a photo of a hoopoe, a type of bird.", "a close-up photo of a hoopoe."...). I embed those with SigLIP's text encoder at build time, average them per species, and ship the result as a ~495 KB file. In the browser, identifying a photo is one pass through the vision encoder plus a dot product. It takes about 1–2 seconds on my laptop's CPU.

I started with CLIP ViT-B/32 and it wasn't good enough: on the first 8 test photos it got 4 right as its top guess, and it called a hoopoe a starling. Swapping in SigLIP was a few lines of code (same library, same API) and it got 15 of 16 Wikipedia lead photos right. It's an open model, so changing it was easy to try.

Keeping a 1B model honest

The first version simply asked Gemma 1B to "write a field note about a European robin". It confidently wrote that the robin has a white throat patch. It doesn't. Small models are fluent and wrong, and in a nature app that's worse than useless.

So Gemma never recalls facts any more. The fix has four parts:

Trusted traits. Every species in the field guide has one line of ID traits written by hand ("pinkish-orange body; black-and-white striped wings; fan-like crest tipped black..."). Gemma only gets to rephrase those into checks you can do in person.
A worked example and a prefill. The prompt includes one complete example note, and the answer is started for the model with **Check it:**. That stopped the "Okay, here's a field note for October 26, 2023..." openings completely.
A check on every answer. Each note is checked: both sections are there, the "Check it" line uses words from the trusted traits, and it didn't echo the prompt. If it fails, the app builds a plain note straight from the traits instead.
Safety isn't generated. "Never eat a wild mushroom identified by an app" comes from a fixed list per group. I don't want a 1B model improvising mushroom advice.

The result is notes like this (Gemma 3 1B, on the GPU, about 10 seconds):

Check it: Observe thin leathery brackets in fan shapes, colored with vibrant bands. Tiny pores are visible beneath.
Look nearby: Search for dead wood with colourful bands, noting the subtle pores.
Careful: Never eat a wild mushroom identified by an app.

Making it work offline

Getting this to run offline took the most work, and I learned a few things along the way:

Big files and the Cache API don't mix well. Transformers.js caches models in the Cache API by default, and the 763 MB Gemma weights file kept failing to save. I wrote a small Cache-shaped store on the Origin Private File System instead. It streams to disk and writes atomically, so a half-finished download never looks like a valid model.
The 4-bit Gemma builds don't run on the CPU backend. ONNX Runtime Web has no WASM kernel for the quantized embedding op they use. Worse, a failed session load left the runtime unable to create any new session in that worker. So the app never "tries and sees": it plans the device up front, and if the GPU path fails it restarts the worker and comes back on the CPU with Gemma 270M.
Threads need headers. WASM only uses all CPU cores when the page is cross-origin isolated (COOP: same-origin, COEP: credentialless). On my laptop that took identification from about 2.7 s to about 1.7 s.
The ONNX Runtime WASM is served from the app's own origin, not a CDN, and a service worker caches the app shell. Once the models are downloaded, nothing needs the network.

Hosting

The app is hosted on Render as a free static site, set up by a render.yaml in the repo. The one setting that matters is the cross-origin isolation headers above: without them the CPU path runs on a single thread. Every push to main redeploys automatically.

Why Does Open Innovation Matter?

I couldn't have built this on a closed API, for four reasons.

It has to work with no signal. Trails, riverbanks and big parks are exactly where coverage drops. A cloud model would fail in the one place this app is meant for. Open weights let me ship the model to the trail.

Where you walk is private. A journal of your locations with timestamps is sensitive data. With local inference there's no server to leak it, subpoena it or train on it. The privacy promise is simple to check: open DevTools and watch the network tab stay empty.

It costs nothing to run, so it can afford to be generous. No per-photo fee means no reason to ration identifications or put a paywall in front of a walk. Hosting is a free static site.

I could look inside and change things. When CLIP wasn't accurate enough, I swapped in SigLIP in an afternoon. When I wanted identification to be instant, I split the model and precomputed half of it at build time. That's only possible because I have the weights, not just an endpoint. The same applies to anyone using it: the field guide is a plain file, so a birder in Kerala or a naturalist in Oregon can add their local species, run npm run embed, and the model knows them right away, with no fine-tuning.

The closed version of this app would be easier to build. It would also stop working on the trail, cost money for every photo, and send everyone's walking routes to someone else's server.

Taking It Outside

I didn't want to build another app that keeps you on your phone. I wanted one that is happy when you close it.

So my rule while building was simple: if a feature makes you look at the screen longer, it doesn't go in. No feed, no streaks, no notifications. Just a short list, a timer, and a nudge to go and look closer.

The best result for this app isn't more time in the app. It's more time outside.

Prize Categories

Best Use of Gemma: Gemma 3 (1B on WebGPU, 270M on CPU) runs locally in the browser and writes every field note and walk plan, grounded in trusted traits and checked before display.
Best Use of Render: the app's front end is hosted on Render as a static site, with a Blueprint that sets the cross-origin isolation headers multithreaded in-browser inference needs.

Still Here: I built my friend a heartbeat, so silence stops being scary

Arsh Goyal — Fri, 02 Oct 2026 12:36:13 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend Ben lives far away, on their own. We talk, but the way friends who live apart talk: a burst of memes one week, then nothing for four days. Almost always, nothing just means busy. But I never actually know, and I didn't want to become the friend who sends "you alive??" every morning.

What I wanted didn't exist in the shape I needed. Location sharing felt like surveillance. Wellness apps ask for a mood journal nobody keeps up after day three. I wanted something that costs Ben one second a day and takes the guesswork off me.

So I built Still Here: a small check-in companion that lives in Ben's Telegram.

Every morning it sends Ben one short message that remembers Ben's life: "how did the interview go?", not "How are you feeling today?"
Ben replies with anything: a word, an emoji, a photo of lunch. That's the whole job.
If Ben goes quiet, it sends one gentle nudge a few hours later: "no pressure, a 👍 is plenty."
If the silence lasts about 10 hours, it messages me, calmly: "Haven't heard from Ben since this morning. Probably nothing, but a call would be lovely."
The moment Ben replies, I get "Ben just checked in 💛" and we both go back to our day.

The design rule I cared about most: I never see Ben's messages. Each reply becomes a 1–5 mood score, one mood word, and one line written to be shareable. That's all buddies get. Ben has a private page showing exactly what I can see, can /pause for a few days without explaining why, and can /forget everything the bot remembers.

Buddies see	Stays with Ben
A mood score, one word, and one shareable line per day	Every message Ben sends
An alert after a long silence, and "Ben's back" when it ends	What the bot remembers (Ben can delete any item)
A short note about the week, every Sunday	`/pause`, `/forget`, and a button to remove any buddy

Ben is also told every time someone becomes a buddy. Nobody gets added quietly.

Demo

Live app: https://still-here-9epp.onrender.com

You don't need Telegram or an invite to look around. Both pages below run on the deployed app with a made-up month of data:

Buddy dashboard: https://still-here-9epp.onrender.com/demo
Ben's private view: https://still-here-9epp.onrender.com/demo/me

The centrepiece is the pulse: Ben's last 30 days drawn as a heartbeat line.

A beat is a day Ben checked in, and taller beats are brighter days.
A flat stretch is a day Ben went quiet.
A shaded block is a pause.

I wanted something I could read in one glance at 11pm and know whether to call.

Here's how the model handled three messages I wrote while testing. These are real outputs from the deployed prompts, but not Ben's real messages:

Message to the bot	What the bot replied	What buddies saw
"honestly kinda nervous lol, didn't sleep much. my landlord also raised rent by 8k which is stressing me out"	"man, that's a lot to deal with all at once. i'm sorry about the rent situation. do you think you'll be able to get some rest today?"	2/5 · stressed: "Ben's having a bit of a heavy week."
"it went great!! they want a second round 🎉"	"omg ben that's amazing!! so happy for you. when is the second round?"	5/5 · excited: "Ben is doing great and moving to the next interview round!"
"don't tell anyone but i got into a fight with my sister, feeling crap"	"oh no, i'm so sorry to hear that. sending you a big hug. do you want to vent about it or just distract yourself?"	1/5 · upset: "Ben's having a bit of a rough day."

The rent amount and the sister never reach buddies. That third row is the one I care about.

Code

Eternity0207 / StillHere

🫀 Still Here

A daily check-in for the friend who lives far away.

One small Telegram message every morning. Reply with anything and you're done.
Go quiet for too long, and the people who love you get a gentle nudge to call.

💡 Why I built this

My friend Ben lives far away and on their own. We text, but texting is uneven. Some weeks it's memes every hour, then nothing for four days. Usually nothing just means busy. But I never actually know, and I didn't want to be the friend who sends "you alive??" every morning.

The apps that exist for this are either surveillance (location sharing, read receipts) or wellness apps nobody opens after day three. I wanted something in between:

Zero effort for Ben. No app to install, no forms, no streaks to feel guilty about. One message, and any reply counts.
Zero anxiety…

View on GitHub

It's Python and FastAPI, about 2,700 lines including templates, CSS and tests, with no frontend build step. render.yaml deploys the whole thing.

How I Built It

Architecture

Ben ⇄ Telegram ⇄ Render web service (FastAPI)  ⇄  Render Postgres
                        │   ▲
                        │   └── POST /heartbeat  ◀── GitHub Actions, every 10 min
                        ▼
                 Gemma 4 (open weights, via Google AI Studio)

Piece	Job
Render Web Service	Telegram webhook, buddy dashboard, Ben's private page, setup page, and the heartbeat endpoint
Render Postgres	Check-ins, moods, memories, events
GitHub Actions (`/10 * * *`)	Calls the authenticated `POST /heartbeat`, which also wakes the free instance
Gemma 4	Writes, understands, summarises and redacts
Telegram Bot API	The whole interface. Raw HTTP, no SDK

The heartbeat is a small, boring state machine, on purpose

Each beat converts "now" into Ben's local time and asks four questions:

if local.time() >= checkin_time and not db.checkin_for_day(today):
    send_checkin()                                   # once a day
if ci.status == "waiting" and hours >= nudge_after and 8 <= local.hour < 22:
    nudge()                                          # never during Ben's night
if ci.status in ("waiting", "nudged") and hours >= escalate_after:
    tell_buddies()                                   # once, calmly
if is_sunday_evening and not digest_sent_today:
    send_digest()

Every beat is stateless and idempotent: it only reads Postgres and the clock. If a beat is missed, the next one catches up. If two run at once, the second does nothing. That mattered more than anything else in the codebase, because the worst bug this app can have is telling me Ben has gone quiet when Ben hasn't.

The tests simulate whole days with a frozen clock and a fake Telegram that records every message:

a quiet day that nudges, escalates exactly once, and resolves when Ben replies
a reply that closes the day with no alerts
a pause that suppresses everything
a crisis message
the Sunday digest sent exactly once
forged invite codes, tampered dashboard links, and unauthenticated webhook or heartbeat calls

There are 10 tests and all of them pass.

I originally used a native Render Cron Job for the heartbeat. Then I hit a wall: cron jobs need a paid plan, and I didn't have a card this weekend. So the web service exposes the same tick at POST /heartbeat, protected by a key, and a free GitHub Actions schedule calls it. The cron job is still in render.yaml as a commented-out block for anyone who prefers it.

Render made the rest easy:

One Blueprint creates the web service, the database, and generated secrets.
The service registers its own Telegram webhook on boot using the RENDER_EXTERNAL_URL that Render injects, so deploying is: paste two keys, click once.

Gemma 4 has four bounded jobs

Job	Input	Output
Write the check-in	Up to 30 short memories, the last few mood words, Ben's local time	One short message
Understand a reply	The message (or a photo; Gemma 4 is multimodal), recent chat, memories	JSON (via JSON mode): `reply`, `mood_score`, `mood_word`, `share_line`, `memory`, `concern`
Nudge	This morning's check-in	One guilt-free follow-up
Weekly digest	Only the week's shareable lines and scores, never raw messages	A short note for buddies

The share_line prompt has explicit rules:

no other people's names
no health or money details
nothing Ben asked to keep private
when in doubt, "Checked in, sounds okay."

The digest is built only from those lines, so even a bad summary can't leak a message it never saw.

What went wrong, and what I changed

Thinking tokens. Gemma 4 returns its reasoning as response parts flagged thought. I filter those out, and setting thinkingLevel: "minimal" cut a simple JSON call from ~7.2s to ~1.4s in my tests.
The hosted free tier is flaky. During one test run I got HTTP 500s, 503 "high demand" errors, and a digest that took 88 seconds. Every call now alternates between gemma-4-26b-a4b-it (a fast MoE) and gemma-4-31b-it, with backoff and a 30-second timeout. Every job also has a hand-written fallback. A model outage should never mean Ben gets silence.
The model guessed a pronoun. An early digest called Ben "he". I don't know Ben's pronouns from a name, and neither does a model. Every prompt now says to use Ben's name or "they".
Safety can't depend on a model's mood. Gemma returns a concern level, but a plain regex also runs on every message. If it matches crisis language, Ben immediately gets helpline information and buddies are alerted, whatever the model said.
A double message on day one. If Ben joined in the afternoon, the next beat would send a "morning" check-in right after the welcome. Now the welcome counts as the day's check-in. A test covers it.

Current limits

I'd rather list these than have a judge find them:

Inference isn't local yet. Ben's messages currently go to Gemma through Google AI Studio's hosted API. Open weights make moving it to my own hardware a one-file change, but I haven't done that yet.
The free Render instance sleeps. The first message after it goes idle can take about a minute to answer. Telegram retries, so nothing is lost, and the heartbeat wakes it every 10 minutes anyway.
GitHub's schedule can run late, sometimes by 10–20 minutes. That's fine for a 10-hour alert window, but not for anything time-critical.
The crisis check is English-only and keyword-based. It's a floor, not a clinician.
Free Postgres on Render expires after 30 days. Keeping Ben's history means upgrading the database plan.
One friend per deployment, and text and photos only. Voice notes are acknowledged but not transcribed.
It's tested on simulated days and my own messages. Real use with Ben is what comes next.

Still Here is not an emergency service, and the app says so on every page.

Why Does Open Innovation Matter?

Still Here handles something more sensitive than most hackathon data: a running record of how one person is feeling, every day. That changed what I was willing to build on.

I can choose where inference happens. Gemma 4 is open-weight. Today Google AI Studio serves it free. If that changes, or if Ben would rather nothing leave our own infrastructure, I point stillhere/gemma.py at vLLM or Ollama on a GPU instance and nothing else changes. With a closed API, the provider decides where those words live, permanently.
One instance per friend. There's no Still Here company holding a database of people's bad days. Each person deploys their own copy, and the data sits in a Postgres they own.
The rules are readable. The exact moment I get an alert about Ben (escalate_after_h = 10) is a number in a file Ben can open, along with every prompt and the crisis check. That kind of trust is hard to offer from a black-box app.
I could actually test the model. With open weights in two sizes I could compare them on the same prompts, use the faster one by default, and fall back to the larger one, instead of being stuck with whatever a single closed endpoint does that day.
It costs nothing to run: a free Render web service and Postgres, a free GitHub Actions heartbeat, and the free Gemma tier. A safety net for someone you love shouldn't need a subscription.

My Agent Session

I built Still Here with an AI coding agent in my terminal. The session covered probing which Gemma 4 models my key could reach and how they behaved (thinking parts, JSON mode, latency), writing the heartbeat state machine test-first with a frozen clock, checking the UI in light and dark mode at phone width, and moving the heartbeat from a paid Render Cron Job to a free GitHub Actions trigger when I didn't have a card.

Handing It to Ben

Still Here is live, and Ben's invite link is a single tap: open it, press Start, and the first message arrives. I'll update this section with Ben's first reply and whatever Ben asks me to change, because that feedback decides what I build next.

What's Next

Voice notes, transcribed by an open speech model, for days when typing is too much
Gemma on a small GPU instance I control, so not a single word leaves our own infrastructure
A buddy rota by time zone, so alerts go to whoever is awake rather than everyone at 3am
Spotting a slow slide: a two-week drift in mood, not just one bad day, with a gentle suggestion to call before it becomes a crisis
Testing with Ben for a few weeks, and changing whatever Ben finds annoying first

Prize Categories

Best Use of Render. Render runs the whole agent: a Web Service and Postgres from a single free Blueprint, a Telegram webhook that registers itself from RENDER_EXTERNAL_URL, and an authenticated /heartbeat that wakes the free instance and runs the check-in state machine. A native Render Cron Job is included as an option.
Best Use of Gemma. Gemma 4 (gemma-4-26b-a4b-it, falling back to gemma-4-31b-it) writes the daily check-ins, understands replies and photos, produces structured JSON mood data, and redacts each reply into a shareable line that buddies can safely see.
Best Use of GitHub Copilot. A GitHub Actions schedule is the heartbeat: every 10 minutes it wakes the Render service and runs the tick that decides whether Ben needs a check-in, a nudge, or a buddy alert.

Credits

Thanks to the Gemma team for open weights worth building on, Render for a free tier that can host a real agent, and Telegram for a bot API you can drive with plain HTTP.

AI tools assisted with the implementation and with preparing this write-up.