SnapDone: I stopped screenshotting things I'd forget, and let an on-device Gemma do the remembering

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

My camera roll is a graveyard of good intentions: an electricity bill due "sometime next week", a flight confirmation, a dentist appointment from a chat, a conference schedule. I screenshot them so I won't forget, and then I forget the screenshot.

SnapDone turns those screenshots into finished actions. Pick a screenshot, and the app:

1. reads the text on the phone with OCR,
2. asks Gemma 3 1B, running locally through llama.cpp, to pull out what matters (title, date, time, amount, notes),
3. shows you the result to review and edit,
4. and only after you confirm, creates a reminder (local notifications the day before and on the day), a calendar event, or a task.

No account, no cloud and no API key. The screenshot, the extracted text and the history never leave the phone.

Demo

Code

https://github.com/Thilagavathi1/snapdone/tree/main

How I Built It

Stack: Expo SDK 58, React Native 0.88, TypeScript, Expo Router, Zustand, expo-sqlite, expo-notifications, expo-calendar, Reanimated, Zod, llama.rn and ML Kit text recognition.

The pipeline is four small services, each behind an interface so it can be swapped or mocked:

OCR service ──► AI service ──► Action service ──► SQLite history

A few things I learned along the way:

• Small models need structure, not freedom. I give llama.rn a JSON schema, which it turns into a grammar, so Gemma can only produce the shape I expect. Then I validate again with Zod. If anything fails, the app shows Needs review and switches to manual entry, instead of guessing.
• OCR order matters more than the model. My first test on a Hacktoberfest schedule screenshot came back as a dateless "task". The OCR was accurate, but it read the page column by column, so each card's date box ended up far from its title. Rebuilding the text row by row from the text block positions put "MON 26 OCT · Build an Open-Source AI Assistant · 10:30 PM" on one line. After that, the same screenshot became a proper calendar event.
• Memory is the real constraint on a phone. On my mid-range MediaTek phone, every JavaScript reload left another 800 MB copy of the model in native memory until the phone started swapping. Freeing old contexts before loading, plus 4 threads and a smaller context window, made it usable again.
• Never act without consent. The model proposes; you decide. Nothing is scheduled until you tap the button.

Why Does Open Innovation Matter?

Screenshots are some of the most private data on a phone: bills with account numbers, medical appointments, travel bookings, family chats. Sending them to a closed cloud API to turn them into reminders is a bad trade.

With an open-weight model running locally:

• Privacy by architecture. The screenshot is processed in the phone's memory. There's no server to trust, because there's no server at all.
• It works offline. After a one-time model download, it works in airplane mode.
• No recurring cost. No API key, no per-request bill, and no rate limits on a feature meant to be used a little every day.
• Swappable. The model sits behind a small LocalModel interface. Any GGUF that llama.cpp can run can replace Gemma 3 1B, from a smaller model for older phones to a bigger one for flagships.

Where the open approach lost: speed and raw accuracy. A cloud model would read a busy schedule faster and better than a 1B model on a mid-range phone. For a "do the boring thing for me, quietly" app, I'll take private and free over fast.

Licensing note: Gemma is open-weight under Google's Gemma Terms of Use. ML Kit is free and runs on the device, but it isn't open source.

My Agent Session

The Lazy Payoff

Before: screenshot a bill → forget it → late fee.
Now: screenshot a bill → open SnapDone → tap once to confirm → get reminded the day before. That's the whole interaction.

Prize Categories