This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
TL;DR: Gravespeaker turns a headstone photo into a researched ghost story, spoken aloud by ElevenLabs. Gemma reads the inscription and writes the monologue; Mastra orchestrates the research and storytelling, with SerpApi supplying the search results. The hard part was keeping the theater without inventing the person. Try it here.
What I Built
I used to make up their stories 🪦
Growing up, I was always intrigued by the paranormal and graveyards especially. I would go hang out in the cemetery just for the fun of it, coming up with wild stories about each person buried there: who they were, where they lived, and how they died. I wanted to know everything and what I didn't know, I made up.
That idea is where this app comes from: go to a graveyard, snap a picture of a tombstone (or upload a test copy), and the app will give you facts transformed into story mode by Gemma 4. ElevenLabs is responsible for the voice, which can be either male or female.
When you put it all together, it's just like the stories I used to imagine walking through the cemetery, except this time I have actual records to work from. Apparently, making up the entire biography was the easy version.
Why it needs the theater 🎭
My primary objective was to turn this into a theatrical story without diluting the person who lived behind the gravestone. That's why the search engine and fact engine were so important to this build: without raw facts, the theatrics have nothing to play with, and without theatrics the whole app turns into another obituary in the daily paper (for those of you old enough to remember those).
It's for the person who stops to read a name and wonders what happened between those two dates. That was me long before I had an app for it.
AKA Cool Things to Watch For
There's a lot going on behind the scenes in this application that you'd never notice unless you know to look for them. Here's a short list:
- Gemma reads the inscription, but ordinary code decides whether the search results match that person.
- Proposed facts have to carry quotes that can be checked against their sources.
- A narrator is selected from cues in the records, with a voice selector if you'd rather hear the other one.
- When the research can't support a biography, the story stays with the stone.
Demo
One stone, one life, one story 🪦
Turn on your sound for this one. The tunnel effect is intended to make the voice sound like it's coming from below the stone, which is a perfectly reasonable thing to want your phone to do in a cemetery.
- Take or upload a photo and let Gemma read the inscription. If the stone names more than one person, choose whose story to tell.
- Read the generated story, then press play beside Hear their story.
- For a researched story, open Facts for the inscription and Sources for its records.
The app selects its research candidate automatically and labels generated stories as AI reconstructions. A story drawn only from the inscription says so, without research tabs pretending there's more to see.
You can try a photo from the reviewed fixture collection. The manifest records the original photographers, source links, and licenses.
Code
anchildress1
/
gravespeaker
Photograph a headstone. Discover a life. Hear the story the records left behind.
🪦 Gravespeaker
Photograph a headstone. Discover a life. Hear the story the records left behind.
- About
- Features
- Tech Stack
- Architecture
- Project Structure
- Getting Started
- Configuration
- Deploying
- Benchmarks
- Security
- License
- Author
About
Gravespeaker is a mobile-friendly web app. Upload or snap a photo of a gravestone, check what the AI read from the stone, and hear a short theatrical story built only from facts checked against public records you can click through.
The story is openly fiction, labelled "Story generated with AI", but it states only facts that appear in a cited source. If the records can't establish who the person was, the app says so instead of making something up.
Built for Hacktoberfest 2026, Week 4: Ghost in the Machine. The PRD has the full scope, schedule and definition of done.
Features
Step
What you get
Read
Name and life dates from a photo, shown read-only. Maiden names are split
⚖️ The project uses Polyform Shield 1.0.0 with supplemental terms. The application is source-available; Gemma is open-weight and Mastra is the open-source framework at its core.
The app is built with SvelteKit and TypeScript, deployed on Cloud Run, with Gemma 4 26B hosted through the Gemini API. The source and saved experiments include the inscription benchmarks, recorded search results, and story evaluation harness.
The local unit suite passes 379 tests, covering namesakes, conflicting dates, failed searches, and unsupported story claims. The generated stories have their own evaluation harness, because a mocked model is remarkably cooperative about saying exactly what the test expects.
How I Built It
Reading the stone is more than reading the words 🪨
First, the image is captured (or uploaded) and sent to Gemma 4 26B for analysis to read the gravestone. I thought about doing this deterministically instead, but there are so many variants to account for plus wear and tear on the gravestone itself.
Plus I didn't just want to read the results, I needed to know which part was the name, whether it's a maiden or married name, whether it's a dual burial plot and if so are there two people or just one, when did they die, is the full date present or just a year, does the gravestone define a birthday. Technically speaking, this is the sort of problem AI handles exceptionally well.
The saved reading run scored 57/58 fields across 20 photos, with no extra people or request/JSON errors. That's a small smoke test with one call per image, including the app's normalization, and the remaining mistake matters: a worn 1818 still came back as 1848.
An earlier run also produced a person literally named "[]". I can reject a name with no letters in ordinary code. A plausible but wrong year is harder, and it can send the search after the wrong person. The benchmark notes keep the prompts, failures, and scoring details available for inspection.
Challenge one: finding the right person 🧭
As it turns out, searching for people isn't as easy as it sounds. I guess that's why there's specialized sites for this kind of thing, but I didn't have that. So I made my own instead.
There were two primary challenges I faced in this build and one of those was getting the search criteria just right. I didn't want AI in charge of a search when it makes more sense to call SerpAPI with deterministic data points. When I first read the specs, I thought that I'd have everything I possibly needed to make this whole idea come to life.
I was wrong...
SerpApi: five search engines and a disappearing credit balance 🪙
I tried a variety of different search combinations, including the person's name, date of birth, date of death, and burial location.
One Google search did not return enough reliable data about a person given their name, date of birth, or death date. So I attempted to correct that by adding in other search providers with the same search lookup — which worked right up until Claude helpfully pointed out that I'd run out of credits before I finished testing the app at the rate I was using them.
So one search provider went to five and back down to two. Finding more results helped, but I still needed to know whether they belonged to the person on the stone.
Saving the results so I could afford to test 🗂️
I used Mastra to orchestrate the search pipeline and spent my free credits running tests to figure out which sources actually belonged to the person on the stone. This required up to 100 credits per test run until I started saving SerpAPI results for reuse. Yes, there's a cache, but that doesn't give me a fixed set of results to test against. Storing results was my solution.
Google added another complication: some result links went through redirects that didn't reliably return a destination while recording. The replay harness preserves those resolved links and rejects incomplete recordings. I wanted to change the matching logic and see what changed, not wonder whether Google had handed me a different experiment. Replaying the saved responses runs the research workflow without spending more search credits.
Mastra: which search to try next—and when to stop 🧭
Mastra runs this as a workflow with explicit inputs and outputs: plan the searches, run the next one, resolve the candidates. There is no model deciding which query to try or whether a birth year matches. That's ordinary code, and I can replay it against the same responses without asking an LLM to reconsider arithmetic.
The app starts with the name and fully readable birth and death years, trying plain queries and then obituary, grave, or cemetery terms. Yandex goes first, then Google if it doesn't find a dated match. Later queries can add a town from browser location, photo GPS, or the inscription. I'm working from the assumption that most people are buried close to where they lived.
Mastra handles retries, and the workflow stops when it finds a medium- or high-confidence candidate, runs out of queries, or encounters repeated provider failures.
The stopping rule made a measurable difference on the same nine-person replay set:
| Search strategy | Search attempts, including retries | Cases with a dated match |
|---|---|---|
| Wait for three sources | 136 | 5/9 |
| Try each query plain, then with keywords; stop at the first dated match | 72 | 5/9 |
Requiring more sources sounded reassuring until it cost almost twice as many attempts without finding another dated match. In replay 25, only two cases stop after the first search. A dated match is the engine's assessment, not independent proof that the entire biography is correct.
For example, Ethan John Chartre takes four attempts in the saved replay: the plain name-and-dates query on both engines, then the version with obituary terms. The fourth returns a funeral-home result that the matcher grades high. Same person, same dates, different search wording—and finally something useful to work with.
What makes a match convincing enough? 🪪
High confidence means the name and every available birth or death year match, with at least one year present. Medium means the name and one year match while another year remains unconfirmed. A name alone is useful as a lead and pretty terrible as permission to start narrating somebody's life, so it needs date evidence from the page or fallback snippet before it can support a researched story.
The identity tests cover middle initials, generational suffixes, and pages that mention the right person inside somebody else's obituary. Jr. and Sr. are doing quite a lot of work when the alternative is giving a son his father's life story.
The gravestone didn't come with a Facebook username 🪧
I also wanted to be able to search social media for people who passed away since 2010 — a completely arbitrary date meant to tune appropriately in the future after proper benchmarks and data integrity checks are complete. However, SerpAPI's Facebook profile endpoint required a profile identifier—a username or numeric ID—and I was unable to find a way to link a gravestone with a profile identifier by the challenge deadline. General web searches can still turn up social links; the dedicated profile lookup didn't make it into this build.
Challenge two: keeping the theater without inventing a life 🎭
Finding a plausible search result is only the beginning. The app fetches the source pages, checks their date evidence, and drops pages that contradict the inscription. Gemma extracts proposed facts with supporting quotes, and code checks those quotes against the source text before they reach the storyteller. If a page can't be fetched, the available search snippet is the fallback evidence.
Gemma turns those supported facts into a first-person monologue. If the research doesn't supply enough evidence, it works from the stone; if no generated sentence survives the checks, the UI shows a plain epitaph. Running out of records doesn't mean the app gets to invent some.
The evaluator was taking the theater out 🧪
The evaluator needed to understand the difference too. If someone died at 96, calling it a long life is reasonable. If they died at 45, the same line needs another look. A ghost talking about the wind around the grave is atmosphere. A ghost watching their children grow has just acquired children, and I need a source for that.
The evaluator originally pushed too far in the other direction and started taking the theater out with the unsupported facts. I pushed back. I wasn't building all of this just to have a second model reduce the story to a list of dates with dramatic punctuation.
The story evaluation harness now has two kinds of input: recorded research cases and deliberately constructed cases that tempt specific mistakes. The written cases include good and bad reference stories, so the evaluator has to demonstrate that it can tell the difference before its scores mean much.
Mastra's createScorer runs a Gemini Flash judge that separates life claims from atmosphere and identifies supporting evidence. Code then checks that the quoted evidence actually appears in the supplied records. An unsure verdict counts as a failure, and a separate check catches years outside the person's lifespan. Each generated case gets three attempts, because one lucky paragraph is a very convenient way to fool myself.
The production checks catch new names, numbers, and certain contradictions; the evaluator tackles what the sentences actually mean. That evaluation work is still in progress, and I'm not claiming a complete passing score for generated stories. Getting the judge to recognize the problem is part of solving it.
ElevenLabs: a voice from below the stone 🎙️
ElevenLabs reads the finished paragraph using Callum or Matilda. Callum is slowed to 0.85 speed; Matilda stays at 1.0. Both use a tunnel effect with distance added, because the intention is for the voice to come from somewhere below the stone instead of sounding like a cheerful assistant reading an obituary.
The narration endpoint streams MP3 audio using Flash v2.5 when the visitor presses play. The starting voice comes from pronouns and relationship cues in verified quotes, then a name-based guess. It's a narrator, not verified gender or a recreation of the person's actual voice, and the visitor can switch it.
I also wanted narration to change based on when the person died, but that hasn't made it into the build. The speed and effects are fixed. Getting the facts right took priority over giving every century its own dramatic entrance.
Why Does Open Innovation Matter?
What choosing Gemma let me change 🪛
I didn't start by assuming the open model would win. I tested hosted Gemma against Gemini Flash, and smaller Gemma variants locally through Ollama and LiteRT-LM. The local models struggled with worn inscriptions, including inventing surnames, so I kept the larger hosted model for this build.
That ability to try local inference matters for this project. A cemetery is a fairly inconvenient place to discover that the app needs a better signal, and open weights give me a route toward running the model myself instead of making the hosted endpoint a permanent requirement. This version uses hosted services; the local experiments show both the possibility and how much work remains.
Mastra gives me the same room to work on the orchestration. Its open-source workflows and scorer let me put ordinary code around the model, replay the research, and define what a good story means for this particular app. Someone else can inspect those choices, reproduce the experiments, or decide my ghost needs a different set of rules. They don't have to start by rebuilding the framework.
My Agent Session
Entire: keeping the reasoning with the code 🧵
Entire records the agent sessions and ties checkpoints to commits. That matters when the finished code tells me what a rule does but doesn't tell me which argument, failed experiment, or particularly convincing wrong answer put it there.
The recorded session preserves the disagreement over atmosphere in the story evals, including the decision to undo a prompt change that was stripping out valid language. Without that context, tightening the prompt again would look like progress. It would also put me right back in the argument I'd already had.
There's a concrete example in the confidence-label fix. Codex caught a case where a source page was rejected for contradicting the stone, but its search confidence could still help label the final story as matching every date. Claude added a regression test and changed the grade to use the remaining evidence when a page gets dropped.
That commit carries Entire checkpoint 01M68EC8YQA3S5V7J8WWQ9ARF3, including the tradeoff behind the more conservative grade. It's useful context to keep beside the code, especially when another helpful AI decides the stricter rule looks unnecessary.
With the repository and checkpoint data available locally, the session can be inspected with:
entire checkpoint explain e003abf
The Spirit of It
Why I wanted the stone to answer 🕯️
I wanted the experience to feel a little unsettling, like you've stopped to read a gravestone and someone on the other side has decided to answer. That's where the first-person story and the voice come together. Reading a name and hearing a story addressed to you are two very different ways to spend a few minutes in a cemetery.
That's my interpretation of Ghost in the Machine: give someone a reason to stop at a stone they might otherwise walk past, let the theater pull them into the story, and leave them with something about the person who actually lived.
Prize Categories
| Category | Its role in Gravespeaker |
|---|---|
| Best Use of Gemma | Reads the inscription, extracts proposed facts, and writes the theatrical monologue; the reading comparisons and story eval harness are linked above. |
| Best Use of Mastra | Runs the research and story workflows, owns search-step retries and stopping, and supplies the custom story scorer. |
| Best Use of Entire | Preserves the agent sessions and checkpoints behind the implementation, including the evaluation decisions and confidence-label fix. |
| Best Use of SerpApi | Supplies Google and Yandex results for identity matching and research; saved responses make the workflow experiments repeatable. |
| Best Use of ElevenLabs | Streams the selected narrator with the pacing and tunnel effect described above. |
Still curious about the person behind the stone 🪦
Growing up, I wanted to know everything, and what I didn't know, I made up. That was half the fun. Building this means figuring out how to keep that curiosity and theater without making up somebody else's life.
🛡️ The Ghostwriter Has Been Identified
Codex helped draft this post from the project files and composed this footer, apparently unwilling to remain an anonymous apparition.
The saved benchmark results are available for inspection; the ghosts have declined peer review.
Top comments (0)