Screenshot Séance: I Built a Local AI to Find the Screenshots That Still Need Me

Dhanush Koyi — Mon, 26 Oct 2026 07:00:04 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

Taking a screenshot is easy. Coming back to it is apparently a separate hobby.

A return label, a concert ticket, a recipe, a discount code. Each one comes with a tiny promise: “I’ll deal with this later.”

Then later arrives, and the useful thing is somewhere between a meme and three screenshots of the same conversation.

I built Screenshot Séance, or séance., around that unfinished business.

Find the few screenshots that still need you; let the rest go.

Séance uses Gemma 4 running locally through Ollama to read a screenshot folder and sort it into five buckets:

Bucket	What it means
Needs you	Something to do: return an item, attend an appointment, use a code
Let go	Memes, duplicates, finished things, or expired deadlines
Keep	Something worth keeping for its own sake
Sensitive	Something flagged as containing passwords, card details, IDs, or recovery codes
Not sure	Something that needs a human glance

Choose a folder, let it process, and come back to a smaller set of decisions.

The “Do it all for me” button puts actionable items on your list, keeps the keepsakes, and moves eligible clutter into a release folder. Anything marked as sensitive or uncertain awaits you.

Letting go is reversible. Files move into _released_by_seance beside the source folder; the app never permanently deletes them. Undo brings them back.

There is also Old Ambrose, an elderly sloth tarot reader who introduces the experience through captions. Your screenshots are the spirits, and some of them have unfinished business.

Yes, I gave the screenshot folder a haunted-house theme.

Every spooky label has a plain-language counterpart. The Daylight skin keeps things straightforward if you prefer your file management without the séance.

Demo

Try Screenshot Séance →

Start a sorting run, explore the morning results, and try “Do it all for me.” You can also review individual items, explore the list, and try the release and undo options.

The hosted demo uses 12 synthetic screenshots and recorded classifications from the actual local pipeline. Sorting is an accelerated replay, and actions are simulated in browser memory.

It does not access your screenshot folder, move your files, or run Gemma on Vercel.

The full application runs on your own computer with Ollama. The public demo lets you explore the experience without installing the model or sharing personal screenshots.

Code

View Screenshot Séance on GitHub

Read the README and setup instructions

The application code is MIT licensed. The repository includes the backend, frontend, synthetic screenshots, benchmark script, tests, and browser demo implementation.

The README explains how to install the prerequisites, pull gemma4:e4b through Ollama, and launch the local application.

How I Built It

The frontend uses React, TypeScript, Vite, Tailwind CSS, and Motion. A local FastAPI backend handles the screenshot workflow, SQLite stores progress, and Ollama serves the open-weight Gemma 4 model.

The pipeline works like this:

Scan the folder. Record image metadata, hashes, and local thumbnails.
Catch near-duplicates. Use perceptual hashes to identify copies before spending another model call.
Read one image at a time. Send a resized screenshot to Gemma through Ollama’s /api/chat endpoint.
Validate the result. Request structured JSON, check the response, and retry invalid output once.
Apply local rules. Resolve sensitivity, uncertainty, and deadlines before assigning the final bucket.
Save progress. Persist the work so an interrupted run can resume.

Gemma interprets the screenshot: what it contains, whether it suggests an action, and whether there is a deadline or sensitive information.

Ordinary Python code handles the rules around those predictions. A sensitivity flag takes priority over an actionable label. Low-confidence results go to “Not sure.” If the model flags something as actionable but cannot identify an action, it also gets held for review.

Those checks still depend on what the model notices. Sensitivity detection can miss things, which is one reason reversible file moves and human review matter.

The unattended part needs to survive interruptions.

The worker records progress in SQLite and can resume after a restart. After the initial sort, a watcher can detect newly added screenshots.

The computer needs to stay awake to keep processing. If it sleeps or the app closes, saved progress resumes the work afterward.

The useful results become a list. Calendar export produces an .ics file with all-day events for dated items and to-dos for undated ones.

Choosing the model involved a tradeoff.

In an initial benchmark of ten real screenshots on a laptop with an RTX 2060, 6 GB VRAM, and 32 GB RAM, I compared two models with thinking disabled and images capped at a 1024-pixel longest side:

Model	Median per image	Valid JSON on first attempt	Card-number examples caught
`gemma4:e4b`	3.39 seconds	9/10	2/2
`gemma4:e2b`	2.16 seconds	10/10	1/2

The smaller model was faster, but it missed one of the two sensitive card/bank examples. I chose e4b as the default.

This is a small local benchmark, not a general accuracy claim. A separate run on the 12 synthetic samples recorded a median API latency of 11.64 seconds, so the earlier timing is not a speed promise.

For this workflow, processing time can be flexible. I care more about reducing the attention the folder asks of me.

Validation covered the model pipeline and the surrounding application.

The live synthetic run finished with 11 images classified by the model, one duplicate identified without a model call, and zero processing errors. That confirms the run completed; it does not establish perfect classification accuracy.

The backend has 24 passing tests, including fake-model pipeline tests for listing, calendar export, release and restore, autopilot, and interrupted runs. The normal frontend build and hosted demo build also passed.

I used GitHub Copilot during development. For transparency about this article, I also used ChatGPT to draft and revise the write-up from my project notes and repository. The benchmark figures describe the documented local runs; they are separate from the AI assistance used to prepare this post.

Why Does Open Innovation Matter?

Screenshots are an awkward thing to send to a hosted model.

A folder can contain jokes, payment details, recovery codes, private messages, and something you forgot was there. Asking someone to upload that collection would change the kind of product I was building.

Running Gemma locally through Ollama lets screenshot classification happen on the same machine as the files. Once the dependencies and model are installed, the local application can work offline.

There is no inference account to create and no per-image API bill. There are still hardware and electricity costs, but a long sorting run does not accumulate usage charges from a hosted model provider.

Open weights also give me a practical set of models to choose from. I can compare them on this task, change the configured Ollama model, and keep the surrounding application under my control.

The application code is available too. Someone can inspect the rules, change the confidence threshold, adjust how deadlines are handled, or see exactly what “Let go” does before trusting it with a folder.

That combination made the idea practical: local inference for private images, inspectable behavior, and the freedom to let the machine work at its own pace.

The challenge’s Slow and Steady theme fits the workflow. Sorting can take time while I do something else. The part that needs my attention should be smaller by the time I return.

I still have to return the shoes myself. I would just like to find the label before the deadline.

Prize Categories

Best Use of Gemma — Gemma 4 runs locally through Ollama and performs the screenshot understanding at the core of the application.
Best Use of GitHub Copilot — I used GitHub Copilot during development of Screenshot Séance.

What’s hiding in your screenshot folder: forgotten to-dos, memes, or a little of everything?

I’d love to hear what you would want a tool like this to handle—and what you would always want to review yourself. Questions about the local setup or how it works are welcome too.