Paper Ghosts: I screened 370 million old newspaper articles for ghosts. One was a magic lantern

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 4, Ghost in the Machine.

Why I built this. I've always wondered about ghost stories. American horror movies like The Conjuring say they're based on true events, and they felt real to me, so I wanted to see the real ones for myself. Then this challenge introduced me to Día de los Muertos, and while I was researching it, it hit me that the people in old ghost reports were real too: named in their own papers, then forgotten. The Library of Congress has scanned those papers, but the OCR is so broken that the stories are almost impossible to search. So I went looking for the original reports, in the exact words people printed at the time.

In 1870 an empty mansion stood in the middle of Springfield, Ohio. People who passed it at midnight "saw strange lights and heard stranger sounds". Two young men from town spent a night inside to settle it.

Nine years later a Pennsylvania paper, the Cambria Freeman, printed how it ended. A gang of robbers had been hiding behind a secret door. When curious people came to see the ghosts, the robbers drove them off "by means of a magic lantern".

A projector faking a haunting, printed in 1879. That's a ghost in the machine, and it's one of the 78 reports in Paper Ghosts.

Paper Ghosts is a morgue for ghosts. (A newspaper's "morgue" is its library of old clippings.) It holds ghost reports printed in American newspapers, and it shows each one with the scan it came from, the exact words behind every fact, and a link to the page at the Library of Congress. It never retells a story, and it never makes one up.

Open it: paper-ghosts.onrender.com. There's no login and nothing to install.

The short version:

• I screened all 370,313,913 articles in the archive for ghost words, on one laptop, in just under two hours.
• Twenty-seven minutes in, I killed the worker on purpose. Temporal lost under two minutes of reading.
• About 1 in 100 flagged articles turned out to be a real ghost report.
• Among 150 labelled articles holding 32 ghost reports, an open model on my laptop found 24, and a 397B model found 23. That's a tie, and the laptop one runs free.
• The site holds 78 reports. Every fact on every card is a quote from the scan, or it says Not stated.

What I Built

The problem it solves

Ghost stories get retold until nobody knows where they came from. The original reports still exist, on newspaper pages the Library of Congress has scanned. But they're very hard to find, for three reasons:

1. There's too much to read. The archive holds 370,313,913 articles, and even a screen for ghost words flags 1,236,122 of them.
2. Almost all of those hits are something else. In the answer key, only about 1 in 100 flagged articles is a real report of a haunting. The other 99 just use the words.
3. The text is broken. OCR turns "apparition" into "apparilion" and "Appar'tIon", so an exact search misses real reports. And papers reprinted each other so often that one report can turn up again and again.

Here's what Paper Ghosts does about each one. Temporal streams the whole archive through the screen in under two hours. Chaining the reprints leaves 974,399 distinct stories. An answer key and a strict quote rule define what counts as a real report, and two open models were tested against it. The small reader that will apply them to the whole archive comes next. Atlas's fuzzy search finds the misspelled reports. And every report on the site is shown with its scan and the exact words behind each fact, so you can check it yourself instead of trusting a retelling.

The theme was Halloween or Día de los Muertos. Paper Ghosts is both. The ghosts are Halloween. The people are Día de los Muertos: every report names real people, in their own paper's words. Detective John Burnside, who caught the Springfield gang. Miss Parsons, the parish clerk's daughter at Cock Lane, twelve years old, who heard the knocking.

It has four parts:

1. The screen. All 370,313,913 articles on 16,887,460 pages of AmericanStories, Harvard's OCR of the Library of Congress's Chronicling America scans, went through a ghost-word screen. It flagged 1,236,122 articles. Papers reprinted each other's stories all the time, so once the reprints are chained together, that's 974,399 distinct stories.
2. The answer key. 450 articles, labelled one at a time against a written policy, then labelled again blind. It says about 1 in 100 flagged articles is a real ghost report.
3. The readers. Two open-weight models tested as the "teacher" that will label training data for a small reader: Qwen3.5-397B on Tinker, and gemma4:26b on my laptop.
4. The Morgue, the site. It holds the 78 ghost reports in the answer key. Each one shows the clipping beside its transcription, with the quoted words marked, and an index card that says Not stated wherever the paper didn't say.

What it isn't yet: the small trained reader that would file the whole archive. That comes next, for reasons I explain under open innovation below. The Morgue shows labelled reports, not machine output, and it says so on the page.

Demo

Live: paper-ghosts.onrender.com

The drawer, lights on. Search by town, state or word, and filter by decade, by how it ended, and by where the paper was printed.

Paper samples, lights off: the clearest scan for each way a report can end, captioned only with its own quoted words.

How we know: every number in this post, with its range, read from the repo's numbers sheet.

On a phone, lights off.

It's a static site with no server, so you can also run it yourself:

git clone https://github.com/Hotragn/Paperghosts && cd Paperghosts
pnpm --dir web install && pnpm --dir web build && pnpm --dir web preview   # http://localhost:4173

Code

Hotragn / Paperghosts

Paper Ghosts

A morgue of ghost reports from American newspapers, 1774–1963. Paper Ghosts reads the AmericanStories OCR of the Library of Congress's page scans, picks out the articles that report a ghost, and files each one with the exact words of the scan that state its place, what was seen and how it ended. The site never writes a ghost story: every value on it is a quote, and every report links to its clipping.

The demo: the Morgue

The 78 ghost reports among the 450 frozen evaluation articles, each with its clipping beside its transcription, the quotes marked in the text, and a typed index card that says "Not stated" wherever the paper didn't say. There's no server and no database behind it, and it works offline.

make setup                     # once: the Python environment and the web packages
pnpm --dir web build
pnpm --dir web preview         # http://localhost:4173

uv…

View on GitHub

How I Built It

Reading 340 GB of newspapers on a laptop, then killing it on purpose (Temporal)

AmericanStories ships each year as one big compressed file: 180 files, about 340 GB. I didn't want to store them, so a Temporal workflow streams each year straight from Hugging Face, four at a time, and runs every page through the ghost-word screen. Each year is one activity, and it sends Temporal a heartbeat every 200 pages.

The full read started at 04:43 UTC on October 30. Twenty-seven minutes in, I sent SIGKILL to the worker's whole process group: nine processes, including the four years that were in the middle of streaming.

After the kill: each year that was mid-stream timed out on its heartbeat and came back as attempt 2.

Here's what happened next:

• A new worker was polling within seconds.
• Temporal timed out each killed year two minutes after its last heartbeat, and started it again five seconds later.
• The 96 finished years weren't touched. Their manifests were byte-identical before and after.
• The four killed years started over from the top, because you can't seek in a gzip stream. All four were reading again 111 seconds after the kill.
• The read finished all 180 years at 06:41 UTC, just under two hours after it started. The kill cost under two minutes, plus re-reading about 4 GB.

The same night, one year's download died by itself partway through a page (tarfile.ReadError: unexpected end of data). Temporal retried it, and nobody had to do anything.

The kill test was also built to catch a nastier bug. If only the worker's main process dies, a year process it started can live on as an orphan and keep writing files. Now each year process checks at every heartbeat whether its parent has changed, and stops before it writes anything. In a test on 1850, the orphan stopped 0.18 seconds later, and no file was written or changed. The full kill test write-up has the event numbers and screenshots.

Then the reprints. Papers copied each other constantly, so the 1,236,122 flagged articles are mostly the same stories printed again and again. My first try at chaining them grew chains through near-matches (A looked like B, B looked like C, so A joined C). Now a printing joins a story only if it matches that story's first printing directly. That leaves 974,399 distinct stories.

An answer key first, and one rule: show me the words

Most flagged articles aren't ghost reports. So before any model read anything, I made an answer key:

• 450 articles: 300 to test on and 150 to tune on, drawn from 17 slices of the archive.
• Each one labelled one at a time against a written policy.
• A second, blind pass relabelled all 300 test articles and agreed on what kind of article it was for 271 of them (κ 0.88).
• Fresh judges, who couldn't see who said what, re-ruled every dispute.

The answer key holds 46 ghost reports among the 300 test articles. Weighted back to the whole screen, that's about 1 in 100 flagged articles (0.97%, with a 95% range of 0.38–1.81%).

Then one rule shapes everything else: a reader that paraphrases makes up ghosts. So every fact a model pulls out (the place, what was seen, how it ended, who saw it) has to come with the exact words that say it. A quote checker throws out any fact whose quote isn't in the article. I tuned its strictness on the 150 tuning articles, to the point where no quote borrowed from a different article gets through.

A laptop model kept up with a 397B one

Two open-weight models took the teacher test on the 150 tuning articles, scored after the quote checker:

	Qwen3.5-397B (on Tinker)	gemma4:26b (on my laptop)
Report or not	139 of 150 (87–96%)	134 of 150 (83–93%)
What kind of article	112 of 150 (67–81%)	100 of 150 (59–74%)
Reports found	23 of 32 (55–84%)	24 of 32 (58–87%)
The story's state	13 of 23 (37–74%)	22 of 24 (74–98%)
How it ended, as scored	16 of 23 (49–84%)	5 of 24 (9–40%)
How it ended, format fixed (scoring only)	16 of 23	20 of 24 (64–93%)

Ranges are 95% Wilson intervals.

The most useful finding was about format, not judgment. gemma4's 5 of 24 on endings made it look like it couldn't tell how a story ended. It could. 18 of its 19 misses wrote the ending as a bare word with no quote, and the checker rightly threw those out. Read as plain answers, it got 20 of 24. A training label still needs its quote, so the fix belongs in the prompt, never in the scorer.

The 397B's misses on the state were the other kind: judgment. It filled in a state from the paper's name or a bare town name, which the policy forbids. That matters here. The Springfield, Ohio story ran in a Pennsylvania paper.

OCR is cruel, so the search forgives it (MongoDB Atlas)

A century-old scan turns "apparition" into "apparilion". The 78 reports live in Atlas as schema-validated documents (140 KB in all), and fuzzy Atlas Search forgives the misreads: a search for "apparition" finds the Cock Lane report through its scan's "apparilion", and a Memphis college report through "Appar'tIon" (the query and its results).

• Ghosts like this one: Atlas Search's moreLikeThis finds each report's five nearest reports, in 68 ms at the median.
• Near this town: the 46 towns where the papers were printed are geocoded from GeoNames into paperghosts.places, behind a 2dsphere index. A $geoNear finds the papers printed within 250 km of Philadelphia, or 600 km of Kansas City, in about half a second. It's labelled as the papers' towns, never the stories'. A story's place is only what its text says, and only 14 of the 78 reports name a state.
• The live site is static, so it carries Atlas's answers with it. The API runs the same queries against Atlas live.
• Vector Search isn't on yet. The index is built for 384-dimension vectors, but the free Hugging Face credits for an open embedding model ran out, and I chose not to pay. Until then, moreLikeThis does "ghosts like this one".

The trace that cut a request in half (Sentry)

Every Atlas query in the API and the scripts runs inside a Sentry span, db.atlas.query. The span records the query's kind and how long it took, never the words searched for. Each release is a commit.

The traces showed every "ghosts like this one" request paying for two round trips: a find for the report's text (41 ms at the median), then the moreLikeThis (66 ms). A report never changes within a release, so the API now reads each one once. A repeat request went from 110 ms to 61 ms at the median, and from 123 to 68 ms at p95.

Sentry also watches the site itself: page loads, Web Vitals, errors, and source maps for each release.

To be plain about scope: the model calls in this project ran before the tracing existed, so there are no LLM spans to show, and I didn't make any up after the fact.

Live on Render

render.yaml builds web/ and publishes it as a static site at paper-ghosts.onrender.com. No secrets are in the repo; the Sentry settings live in Render's environment.

Why Does Open Innovation Matter?

Because this project hit the closed-service trap halfway through, and open weights got it out.

The teacher test ran the 397B on Tinker. Then I found the account's credit balance was $0, and that day's runs ($3.19–14.45, by their recorded tokens) had gone to my card. I stopped all Tinker spending, and sampling is now switched off in code. At the measured rate, the 397B would cost $65–297 per 10,000 articles, depending on how much of the prompt is cached.

Because the weights are open, the reading didn't stop. It moved to my laptop:

• gemma4 on a 24 GB Mac found as many of the test's reports as the 397B, and got the story's state right far more often.
• Nothing about the archive left the machine to make those answers. They ran through Ollama, on my desk.
• Nothing is locked in. The labels, the prompt, the policy and the scores are files in the repo, and the next reader can be any open model that fits.

The cost of local is time and memory. With thinking off, gemma4 reads 244 articles an hour, so a 3,000-article training draw takes 12.3 hours. With thinking on, six measured articles took 2.4–8.1 minutes each. And running gemma4 next to a training job once froze the laptop. That's why the small reader comes next, one model at a time.

The whole project stands on open work: the Library of Congress's free scans, Harvard's open OCR of them, open-weight models, and open-source tools. Take away any one of them, and these ghosts go back in the drawer.

My Agent Session

I built this with Claude Code, and Entire recorded every session. Each commit carries an Entire-Checkpoint trailer that points to the session that made it, so you can go from any line of code to the conversation behind it. The session index lists them with their real token counts. Two show the problem solving best:

• The reprint-chaining fix (59,983,120 tokens over 73 API calls): chains were growing through near-matches, so a printing now joins a story only by matching its first printing directly.
• A judge ruling on its own case, caught (26,688,512 tokens over 34 calls): the first rulings on disputed labels came from the same labeller whose labels were in dispute. Fresh judges re-ruled all 69 disputes blind.

Entire redacts secrets by itself, and our scan of all history and every checkpoint found no keys. It did find something else: an early session's process listing named another account on this Mac. I backed up all the checkpoints, rewrote the 121 affected ones so each listing became a single redaction line, recomputed Entire's content hashes, and checked them with entire checkpoint explain and entire doctor before pushing.

The Spirit of It

Día de los Muertos isn't about being scared. It's about saying the names of the dead so they stay remembered.

Every report in the Morgue has names in it. Detective John Burnside, who caught the Springfield gang. Miss Parsons, the parish clerk's daughter at Cock Lane, twelve years old, who heard the knocking. Their papers printed their names, and then the pages sat in archives for a century and more.

Here's the moment it lands. When a machine scanned the Cock Lane page, it misread "apparition" as "apparilion". A typo like that can bury a story for good, because nobody searches for the misspelling. Atlas's fuzzy search found it anyway, and her name came back with it.

Paper Ghosts can't tell you what Miss Parsons heard. It shows what was printed about her, in the words that were printed, next to the page they came from, and it adds nothing. Where the paper didn't say, the card says Not stated. A blank is more honest than a guess, and I think it's the right way to remember someone you never met.

What It Can't Do (Yet)

• The trained reader isn't built. The Morgue shows the 78 labelled reports, not a model's output. Filing the whole archive needs the small reader that the teacher test is for.
• It's 78 reports, not thousands. They're the ghost reports in the 450-article answer key.
• The map shows where papers were printed, not where ghosts were seen. Only 14 of the 78 reports name a state.
• Vector Search is waiting on an embedding model I won't pay for yet.
• Claude made the labels. The 450 answer-key labels were made one article at a time by Claude, against the written policy. The blind second pass and the re-rulings came from separate Claude judges that couldn't see the first labels. Claude also wrote the code, under my direction. Training labels will only ever come from an open model run locally, and the answer key stays for testing only.

Data: article text from AmericanStories (Harvard's Dell Research, CC-BY-4.0). Page scans from the Library of Congress's Chronicling America (no known copyright restrictions).

Prize Categories

• Temporal: the whole 340 GB read as one workflow, with one heartbeating activity per year. A SIGKILL of the worker's whole process group 27 minutes in cost under two minutes, and the orphan fix stops a stray process from writing. Screenshots and event numbers are in the repo.
• MongoDB Atlas: fuzzy Atlas Search that forgives OCR, moreLikeThis, $geoNear on a 2dsphere index, and schema-validated documents.
• Sentry Agent Tracing: spans on every Atlas query, the fix they found (110 ms to 61 ms), and the site's tracing and source maps. No LLM spans, as stated above.
• Gemma: gemma4:26b as the local teacher on a 24 GB Mac, scored against the answer key, and the format-versus-judgment finding.
• Render: the live site, deployed from render.yaml.
• Entire: the session trail, its token counts, and the redaction.
• Tinker: the 397B teacher test, its measured cost, and the spend stop. There's no fine-tune yet.

The code is MIT licensed: github.com/Hotragn/Paperghosts.

-30-