Ghost Hand: A Local Handwriting Reader That Flags Its Own Doubts

Devanshi Bhatt — Thu, 29 Oct 2026 19:02:49 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

Ghost Hand is a local web app that reads handwritten family recipes and letters, in English and Spanish, and refuses to guess quietly.
Vision models fail on handwriting in an ugly way. They don't say "I can't read this." They hand you a fluent, confident, wrong word. In a family archive that is the worst possible failure: imagine a teaspoon silently becoming a tablespoon in the only copy of a recipe, or a name changing in a letter nobody can check anymore.

So Ghost Hand treats disagreement as doubt. It reads each page several times, compares the readings word by word, and flags every word the model isn't consistent about. Flagged words are shown next to the cropped original handwriting, and a family member fixes only those. Human corrections are stored separately from the model's output and from the original image. Corrected cards are placed on an ofrenda (the Día de los Muertos altar) and exported as plain Markdown, JSON and images that open without the app.

How I read the theme

The ghost in the machine is the person whose hand wrote the card. The machine's only job is to carry their words forward without putting its own words in their mouth. Día de los Muertos is why the app has an ofrenda, and why Spanish sits beside English.

Demo

The video shows: Wi-Fi switched off on screen, a handwritten card uploaded, flagged words highlighted beside the cropped original, one word corrected, and the card placed on the ofrenda.

Code

devanshibhatt23 / Ghost-Hand

Ghost Hand 🕯️🌼

Hacktoberfest Open-Source AI Challenge: Week 4 — Ghost in the Machine
Transcribing handwritten ancestral recipes and letters with local open-weight vision AI and uncertainty guarantees.

Ghost Hand is a local, 100% offline-capable application built to transcribe handwritten family cards, recipes, and letters in English and Spanish. Centered around the spirit of the Mexican Día de los Muertos ofrenda (altar), Ghost Hand is designed to honor the past while ensuring private family memories never leave your local device.

🎯 The Core Problem & The "Uncertainty-by-Disagreement" Engine

Standard closed and open multimodal vision models hallucinate quietly when confronted with messy handwriting, cursive loops, or faded ink. In family archives, silent misreadings corrupt recipes and distort ancestral words.

Ghost Hand implements a code-enforced guardrail around the model:

Multi-Pass Inference with Controlled Seeds: Each segmented line is read once at greedy temperature ($T = 0$), followed…

View on GitHub

MIT licensed. It runs locally on macOS, Linux and Windows, or with Docker (docker compose up --build).

How I Built It

The Open-source AI
• Model: Gemma 4 E4B (gemma4:e4b), an open-weight multimodal model, running locally through Ollama.
• No cloud calls. Every read is local inference. There is no closed API anywhere in the pipeline.

The idea: Uncertainty by Disagreement

I don't ask the model how sure it is, because models are bad at reporting that. Instead:

Read the page once at temperature 0 (the greedy reading, used as the reference).
Read it 2 more times at temperature about 0.7 with different seeds.
Align every sampled reading against the reference with Python's difflib.SequenceMatcher.
A word's agreement score is votes / N. Below 0.8, the word is flagged.
A line is also flagged when the sampled readings disagree about how many words it has.

The prompt asks the model to transcribe exactly as written, not to correct spelling, not to guess unclear words, and to keep Spanish accents exact.
The model proposes. The code doubts it.

The pipeline
photo -> resize + grayscale + contrast -> line crops (OpenCV)
-> 1 greedy read + N sampled reads (Gemma 4 via Ollama)
-> word alignment + agreement scores -> flags
-> review page: flagged words next to the cropped original
-> human corrections -> ofrenda page + plain-text export

• Preprocessing: Pillow resizes to at most 1600 px, converts to grayscale and boosts contrast.

• Line splitting: OpenCV with Otsu thresholding and a horizontal projection profile, so each flagged word can be shown against the line it came from. Pages are read as a whole and mapped back to the line crops, which kept it responsive on a laptop.

• Three layers on disk: original image, raw model output, human corrections. Nothing the model said is ever overwritten.

• Front end: plain HTML, CSS and JavaScript with all fonts and assets bundled locally. No CDN.

• Tests: word-agreement scoring, the evaluation metrics, and a test that checks the front end makes no external network references.
Does it work? I tested various images - with both English and Spanish text, and both well-written and illegible handwriting. The model showed more errors while reading illegible handwriting, and showed nearly zero errors while reading

One limit I want to be upfront about

Disagreement catches a model that is inconsistent. It cannot catch a model that is consistently wrong. That is exactly why I report flag recall instead of claiming the flags are a guarantee.

Why Does Open Innovation Matter?

• Family papers stay on the family's machine. Letters and recipes are private. With open weights running locally, no photo of a loved one's handwriting is uploaded anywhere. The test suite checks the front end for external references, and the demo video runs with Wi-Fi off.

• The archive must outlive the app. A tool that preserves memories can't depend on a service that might change its terms, raise its price or disappear. Local weights plus plain-text export mean the transcriptions survive with no vendor involved.

• The method needs many cheap reads. Uncertainty by disagreement reads every page 3 times, with controlled seeds. Locally that costs nothing per read and is exactly repeatable. Through a metered API, every page would cost more and the exact model could change underneath the results.

• Anyone can reproduce it. The seeds, the threshold, the prompt and the evaluation code are all in the repo. A reader can rerun the evaluation on their own laptop.

Where a closed model would probably be better

A much larger closed model may well read messy handwriting more accurately than a small local one. For this use, I would still choose the open one: the cost of a slightly worse first guess is small, because every doubtful word goes in front of a human, while the cost of uploading a family's private papers is not small.

Prize Categories

• Gemma: Gemma 4 E4B is the core of the project and does all the reading.
• Github

DEV Community: Devanshi Bhatt

Ghost Hand: A Local Handwriting Reader That Flags Its Own Doubts

devanshibhatt23 / Ghost-Hand

Ghost Hand 🕯️🌼

🎯 The Core Problem & The "Uncertainty-by-Disagreement" Engine