This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
What I Built
For Día de los Muertos, families build an ofrenda: an altar with photos of the people they've lost, their favourite foods, marigolds to guide them home, and the stories that keep them close.
Marigold helps you build one on your own computer:
- Add someone and drop in a few old photos.
- Write down what you remember: what they always said, what they cooked, the song they hummed on Saturday afternoons.
- A local open-weight model (Gemma 3 4B via Ollama) looks at each photo and suggests a gentle caption. Then it writes a short remembrance using only what you gave it.
- Marigold builds the ofrenda: papel picado, their photos, your words, their favourite things as the offerings, and candles. It's saved as one HTML file you can keep, print or share.
The photos are public-domain archive photos and the memories are fictional; with your own family photos, it's their faces on the ofrenda.
My take on "Ghost in the Machine": the scariest ghost in AI isn't a spooky one. It's the confident little invention. Ask a model to write about your grandmother and it may "remember" a town she never lived in, a year that's off by one, or a sister who never existed. In most apps that's a bug. In a remembrance, it's a ghost that rewrites someone's life.
So Marigold has a ghost check. It doesn't just tell the model to stick to the facts; it checks. Every name, place, number, month and weekday in the written text is compared against everything you wrote and everything the model saw in the photos. Anything that came from nowhere is flagged as a 👻 and highlighted, so you can remove it, or add it to your memories if it's true. Here it is catching a city that was planted in a test:
Demo
Try it: download marigold-win-x64.zip (or the macOS/Linux build) from Releases, install Ollama, run ollama pull gemma3:4b, and double-click marigold. It opens in your browser at 127.0.0.1:5180.
Code
Marigold 🌼
A private, offline ofrenda for the people you remember.
Add a few old photos and write down what you remember: what they always said, what they cooked, the song they hummed. A local open-weight model (Gemma 3 4B via Ollama) looks at each photo and writes a short remembrance using only what you gave it. Marigold then builds an ofrenda page with their photos, your words and their favourite things, saved as one HTML file you can keep, print or share.
The ghost check 👻
A language model writing about someone you loved will sometimes "remember" things that never happened: a town she never lived in, a year that's wrong, a name nobody had. In a remembrance, that's the worst kind of mistake.
So Marigold doesn't just ask the model to stick to the facts. It checks. Every name, place, number, month and weekday…
How I Built It
Stack: .NET 8 (an ASP.NET Core minimal API) serving one offline HTML page, with Ollama running Gemma 3 4B, which reads images as well as text. There are zero third-party NuGet packages, no CDN scripts and no web fonts.
Two jobs for the model. Both go through Ollama's /api/chat:
-
Looking at a photo: a JSON schema in
formatmakes the model returncaption,descriptionandera. The prompt says to describe only what's visible and never guess a name, age, relationship or city. Captions are suggestions; you can change any of them. - Writing the remembrance: the prompt lists the facts (name, years, your memories, their favourite things, what's in the photos) under a STRICT RULE: use only these, add no name, place, date, year, age, job, food, song or event. "It is better to be short than to invent." A lesson from the real model. I first used a JSON schema for the remembrance too. It worked with my test stand-in, but on the real Gemma 3 4B it failed twice in CI: wrapped in forced JSON, the model kept writing until it hit the length limit and never closed the string. Plain text ("title on the first line, then the remembrance") fixed it. Now a cut-off answer is detected and retried once with a different seed. Structured output is great for short fields, but for a few paragraphs of prose, plain text is more reliable on a small model.
The ghost check is plain C#, not another model. A prompt is a request, not a guarantee, so I check the output with about 120 lines of string matching (Ghosts.cs):
- It builds a bag of words from everything you wrote and everything the model saw.
- It flags capitalized words mid-sentence (names, places), numbers (years, ages), and months and weekdays that aren't in that bag.
- It avoids false alarms: sentence-start words don't count, "May she rest" isn't a month but "married in May" is, family words like Abuela or Nani are fine, and plurals match ("Halloween costumes" covers "costume").
- It runs again every time you edit, so adding "She was born in Guadalajara" to your memories turns the warning off. It's predictable, it's explainable, and it's tested.
The ofrenda outlives the app. The page is a single file with photos embedded as data URIs, no scripts, no links and no fonts to download. It prints to PDF. Photos are resized in the browser before saving (canvas → JPEG), so a 12 MB scan becomes a few hundred KB without any image library.
Private by construction. The server binds to 127.0.0.1, rejects any other host name (blocking DNS-rebinding), and needs a custom header for every change, so another website in your browser can't read or edit anything. Uploaded files are checked by their first bytes (JPEG/PNG only), and photos are only ever served by ID from Marigold's own folder.
Tests. There are 33 built-in self-tests for the ghost check, photo validation, storage and the ofrenda page, run on Windows, macOS and Linux. Then CI does the real thing: it installs Ollama and gemma3:4b, downloads a few public-domain photos from Wikimedia Commons (licence checked, credits recorded), and a Playwright test uses the app like a person would. It adds a (fictional) grandmother, uploads the photos, waits for Gemma, writes the remembrance, checks the ghost check, then opens and downloads the ofrenda.
Here's the real Gemma 3 4B, on a GitHub runner with no GPU:
| Result | |
|---|---|
| Photos described | 4 / 4, no names guessed |
| Era guesses vs. the archive's dates | 1944 sewing photo → "1930s–1950s" ✅, c. 1942 radio photo → "1950s" (close), 1922 homestead → "1930s–1950s" (a bit late) |
| Time per photo / for the remembrance | ~67 s / ~42 s |
| Invented details in the real remembrance | 0 (in both the browser run and the command-line run) |
And this is what it wrote, from the family's memories, with a clean ghost check:
Why Does Open Innovation Matter?
Family photos and memories of someone who has died are about as private as data gets. To describe a photo, a model has to see it. With a closed API, your grandmother's face, her kitchen and your memories of her would be uploaded to a server you don't control, under terms that can change. With an open-weight model, Gemma looks at them on your own laptop. That isn't a promise in a privacy policy; it's how the app is built. Turn off the Wi-Fi and it still works.
Open also matters for time, and remembrance is about time. An ofrenda should still open in twenty years. A closed service can shut down, change its prices or retire the model. Marigold's output is a plain HTML file, the model weights sit on your disk, and the code is MIT. Nothing here needs anyone's permission to keep working.
And open let me build the safety I wanted instead of hoping for it. I could pin the exact model, set temperature and seed, choose when to force a JSON schema (and when not to), and put my own check on the output. I can swap in gemma3:12b or qwen2.5vl:7b with one flag if a bigger model writes better. With a closed API I'd get whatever the model does that day.
Prize Categories
Best Use of Gemma. Gemma 3 4B runs locally through Ollama and does both halves of the work: its vision input describes the photos, and it writes the remembrance in English, Spanish, French, Portuguese, Punjabi or Hindi. A GitHub Actions job runs the real model end to end in a browser on every push.
Top comments (0)