Your Screenshots Have a Memory. So Why Can't You Find Them?

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

What I Built

There is a particular kind of digital ghost that every developer knows.

You capture a screenshot of an error you just fixed. Maybe it contains a PostgreSQL query, a useful command, a UI reference, an invoice, or an important piece of documentation.

You promise yourself that you'll find it when you need it.

A few weeks later, you have hundreds of screenshots scattered across your computer.

Their names tell you almost nothing:

• Screenshot_20261012_184205.png
• Screenshot_20261014_102134.png
• Screenshot_20261018_230512.png

The information is still there. You just can't find it.

For Hacktoberfest's final challenge, Ghost in the Machine, I wanted to build something that brings these forgotten digital memories back to life.

That idea became SiftShot.

SiftShot is a privacy-first, local-first screenshot management application for Windows.

Its central idea is simple:

Search screenshots by what they contain, not just by what they're named.

Instead of uploading screenshots to a cloud service or manually opening hundreds of files, SiftShot uses locally executed neural LSTM OCR to extract readable text and make screenshot collections searchable.

What can SiftShot do?

• Recursively discover screenshots in configured directories.
• Extract text locally using Tesseract's neural LSTM OCR.
• Store screenshot metadata and extracted text in SQLite.
• Search screenshots by OCR text and filenames.
• Browse screenshots in a modern dashboard.
• Preview original images and inspect extracted text.
• Avoid unnecessarily reprocessing unchanged files.
• Restrict image access to configured directories.

Original screenshots are never silently renamed, moved or deleted.

SiftShot in action

Dashboard overview

The dashboard provides an overview of indexed screenshots, OCR availability, processing statistics and the screenshot collection.

Screenshot gallery

Browse discovered screenshots with their filenames, thumbnails, file sizes and OCR status.

OCR preview

Inspect the original image alongside its locally extracted text and filesystem metadata.

Search by screenshot contents

Search for a phrase contained inside an image and retrieve the screenshot without remembering its original filename.

Demo

Watch the SiftShot walkthrough:

▶ Watch SiftShot on YouTube

The 81-second demonstration covers the actual application workflow:

1. Open the SiftShot dashboard.
2. Scan a configured screenshot directory.
3. Browse the discovered screenshots.
4. Open a screenshot and inspect its extracted OCR text.
5. Search for a phrase contained inside an image.
6. Open the matching screenshot preview.

The demonstration uses synthetic screenshots processed by the actual local Tesseract installation. There is no mocked frontend data.

Code

GitHub repository:

adarsh-singh106/SiftShot

SiftShot uses a modular architecture:

SiftShot/
├── apps/
│   └── desktop/
│       └── React + TypeScript
│
├── engine/
│   └── src/siftshot/
│       ├── domain/
│       ├── config/
│       ├── ingestion/
│       ├── ocr/
│       ├── storage/
│       └── api/
│
├── docs/
│   ├── ARCHITECTURE.md
│   ├── SETUP-GUIDE.md
│   ├── DEMO-SCRIPT.md
│   └── FINAL-RELEASE-REPORT.md
│
└── README.md

The repository includes the project architecture, dependency lockfiles, automated tests, setup instructions, security documentation and contribution guidelines.

How I Built It

I didn't want SiftShot to be another OCR script with a frontend attached.

I wanted independent, understandable components that could be tested and maintained.

1. Screenshot discovery

The Python ingestion engine recursively scans configured directories using standard filesystem APIs.

It handles:

• Multiple watch directories.
• Overlapping directory paths.
• Case-insensitive extension matching.
• Duplicate path prevention.
• Metadata extraction.
• Symbolic-link restrictions.
• Missing and inaccessible files.

The discovery process does not load entire image files into memory.

2. Local neural LSTM OCR

This is the intelligence layer of SiftShot.

I chose Tesseract OCR, an open-source OCR engine with an LSTM-based neural recognition architecture.

The final implementation explicitly selects the LSTM engine:

cmd = [
    str(binary),
    str(target),
    "stdout",
    "--oem",
    "1",
]

The engine runs locally through a controlled subprocess invocation.

It uses:

• Explicit command arguments.
• shell=False.
• A 10-second default execution timeout.
• Structured success and failure results.
• Text normalization.
• Graceful handling when Tesseract is unavailable.

The application does not send screenshot contents to an external OCR API.

3. SQLite persistence

Extracted text and screenshot metadata are stored locally using SQLite.

The repository layer handles:

• Transactional persistence.
• Parameterized SQL queries.
• Write-Ahead Logging (WAL).
• Duplicate path prevention.
• Updating changed screenshots.
• Reusing OCR results for unchanged files.
• Case-insensitive substring search across OCR text and filenames.

Image binaries remain ordinary files on the user's filesystem.

4. API and desktop dashboard

The backend uses FastAPI and communicates through a loopback-only HTTP service.

The frontend uses React, TypeScript and Vite.

The dashboard provides:

• Screenshot statistics.
• Scanning controls.
• Screenshot gallery.
• OCR status indicators.
• Search results.
• Full-resolution image previews.
• Extracted text inspection.

The current release uses a browser-hosted dashboard connected to the local FastAPI service. Native Tauri packaging remains a future milestone.

5. Privacy and security

A screenshot can contain much more than a picture.

It can contain source code, database information, receipts, personal documents or internal application data.

SiftShot therefore follows several safety principles:

• Local screenshot processing.
• Loopback-only API binding.
• Restricted CORS origins.
• Canonical path validation.
• Protection against path traversal.
• No unrestricted image-serving endpoint.
• Parameterized database queries.
• Subprocess execution without shell commands.
• OCR execution timeouts.
• No automatic modification of original images.

The project has automated security tests, but has not undergone a formal third-party penetration test.

6. Engineering and testing

One of the most interesting parts of building SiftShot was its development workflow.

I coordinated six AI coding instances:

Instance	Responsibility
0	Architecture and integration
1	OCR engine
2	SQLite storage
3	FastAPI and security
4	React frontend
5	Documentation

The five specialist instances worked in isolated Git worktrees and feature branches.

Their changes were integrated and reviewed by the engineering lead.

The final release verification recorded:

• 225 passing backend tests.
• Clean Ruff linting.
• 49 files passing the formatting check.
• No mypy errors across 25 source files.
• Successful TypeScript compilation.
• Successful Vite production build.
• A real Windows OCR and search integration test.
• SQLite persistence verified after a backend restart.
• SHA-256 confirmation that the synthetic test image remained unchanged.

One issue discovered during final Windows verification was UTF-8 BOM handling in TOML configuration files. Some Windows tools save UTF-8 files with a Byte Order Mark, which initially caused configuration parsing failures.

I fixed the loader and added a regression test.

This was a useful reminder that production-minded software is not just about making the happy path work. It is also about handling the environments in which real users run it.

Why Does Open Innovation Matter?

This is the most important design decision behind SiftShot.

Screenshots often contain information people wouldn't casually upload to an external service.

Using a closed, cloud-hosted OCR API would introduce an unnecessary dependency for a problem that can be solved locally.

With open-source Tesseract and local inference:

Privacy: Screenshots can be processed without sending their contents to an external OCR provider.

Offline capability: Once the application, Tesseract and compatible language data are installed, the core processing workflow can run without an internet connection.

Transparency: The recognition engine and its invocation are inspectable. Its behavior is not hidden behind an inaccessible hosted service.

Cost: There are no per-image cloud inference charges or mandatory API subscriptions.

Control: The OCR engine can be replaced or extended without redesigning the entire application.

Open innovation gives users more control over their data and software dependencies.

There are limitations. OCR accuracy depends on image resolution, typography, language data and hardware. During our LSTM verification, the engine correctly recognized most of a synthetic technical screenshot but confused a few similar-looking characters and punctuation marks.

SiftShot currently provides OCR substring search, not semantic vector search or visual reasoning.

I deliberately chose an open-source neural OCR engine that solves the actual problem rather than adding a generative model simply to call the project an AI application.

The Spirit of It

A screenshot is often a tiny record of a moment.

A bug we solved.

A command we needed.

A design that inspired us.

A receipt we might need months later.

But when those fragments disappear into folders, their usefulness disappears with them.

The idea behind SiftShot is to give those forgotten fragments an afterlife.

The machine already has the memories. We just needed to make them searchable.

SiftShot: Capture less chaos. Find anything.

Thank you to everyone building, maintaining and contributing to open-source software this Hacktoberfest.