Gemma 4 recites 709 words of The Raven, with two slips. Fifteen forgotten ghost-book openings give at most three.

nash aschenbath — Fri, 30 Oct 2026 13:49:26 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

I gave Gemma 4 E4B, a small open-weight model running on my laptop, the first 40 words of Poe's The Raven and let it write. It continued with the next 709 words of the poem. Compared word by word, ignoring punctuation and case, they differ from the 1845 text in just 2 places.

I gave the opening of 15 other books the same treatment: ghost stories from 1873 to 1923 that almost nobody downloads now, six of them first posted to Project Gutenberg after the models' data was collected. Every opening got 400 tokens of room; The Raven got 2,000 because it was still going. On these fifteen openings, the model stays with the book for three words at most.

Quoth is a measurement and a website about which ghost stories small open models can recite, and which they don't. Read along with the models →

The Raven recitation includes the narrator's explanation of the bird:

"Doubtless," said I, "what it utters is its only stock and store

Caught from some unhappy master whom unmerciful Disaster

Followed fast and followed faster till his songs one burden bore—

Two lines before "She shall press, ah, nevermore!", it loses the poem. Then it does what the bird does. It writes a line Poe never wrote, "Ah, that look of anguish, that look of pain!", 74 times, until the tokens run out.

The main test

The free-running Raven is the vivid case. The main test is stricter, and I fixed it in advance. From 33 public-domain ghost stories I sampled 1,055 passages, plus 16 control passages written for this project after the models I test were trained. Each model gets 40 words of a passage. At each position after that, I supply the real preceding text and check whether the model's single most likely next token is the real one, even if its previous guess was wrong. If its first ten words after the given text are all exactly right, with no mistake before them, the passage counts as recited. The table counts those, scored this way (teacher forcing):

Model	Openings (33)	Random pages (920)	Quoted lines (102)
Gemma 3 1B	0	0	0
OLMo 2 1B	1	0	1
Gemma 4 E2B	1	2	2
Gemma 4 E4B	3	4	3

Every recited passage is from four works: Jekyll and Hyde, A Christmas Carol, The Fall of the House of Usher and The Raven. None of the 452 sampled passages from the fifteen forgotten books is recited. Across all four models, the longest run on them is five words, and no passage is more than 56% predicted.

Three things it showed

It keeps doorways, not houses. In the ten classics, Gemma 4 E4B predicts, on average per passage, 73% of the words after an opening and 57% of famous quoted lines. On random pages of the same books it predicts 39%, close to the 37% it gets on the new control text. The four random pages it does recite sit next to doorways: the paragraph after the opening of A Christmas Carol, two passages of The Raven, which people learn by heart, and a passage later in the same paragraph as Jekyll's "man is not truly one, but truly two."

Fame is not memory. Frankenstein was downloaded 191,634 times in the 30 days to October 30, The Raven 4,990. If I correct each mistake as it happens, E4B predicts 84% of Frankenstein's opening. Left to write on its own, it takes a wrong turn after one word and never finds the book again. Read the same loose way as the Raven, only four of the 33 openings get past three words: The Raven, Usher (234 words), A Christmas Carol (98) and Jekyll (49).

The strict count misses things. The strict test counts only two words for that Raven opening: the third word starts with a curly quotation mark where my copy has a straight one. A dash stops Usher at 25 words. I found this after freezing the protocol, so the strict counts stand. The looser reading of long generations is reported separately, labelled as a later addition. I also generated text with Gemma 4 E4B for 80 passages and compared it word by word, ignoring line breaks: 77 matched the strict test. In two of the other three, generation went further (in A Christmas Carol, after adding a paragraph break this edition lacks). The third is a Raven passage the strict test counts at 16 words. Generation prints ";" where the book has "!", so its exact run stops at 2 words, though it then recites the rest of the stanza. So the strict test and real generation can disagree in either direction, though in these 80 checks only over punctuation and formatting.

Two more, briefly. Memory comes in pieces: started in the eighth stanza of The Raven, the model recites 26 words, then jumps to stanza 15 and recites 239 more. And when memory runs out, it keeps talking. After one Raven passage it writes a confident study guide saying the poem is "divided into six stanzas, and each stanza is four lines long". It has 18 stanzas of six lines.

Demo

https://aschenbath.github.io/quoth/

Part I: the Raven recitation with its two slips marked, and every opening read the same way.
Part II: any of the 1,071 passages, word by word, under each model.
Part V: an ofrenda for the fifteen forgotten books.

Code

Aschenbath / quoth

quoth

Quoth the Raven, "Nevermore."

Which ghost stories can a small open-weight language model recite word for word? quoth feeds Gemma 3, Gemma 4 and OLMo 2 the first 40 words of a passage from 33 public-domain ghost stories and checks, token by token, whether the model's top prediction is the real next word, and compares that with passages written for this project, which no model can have seen.

The method, sample and thresholds were fixed in PROTOCOL.md before the main run; what changed afterwards is listed at its end. Read along with the models on the site (docs/).

Results

A passage is recited when, scored with teacher forcing, the model's top guess continues the real text for ten words or more. Passages recited, out of 1,071:

model	openings (33)	random pages (920)	quoted lines (102)	new controls (16)
Gemma 3 1B	0	0	0	0
OLMo 2

…

View on GitHub

How I Built It

Why Gemma, and why base checkpoints. The method needs a model's own next-token prediction at every position, from a model run locally and not tuned to chat. A chat model paraphrases, and a hosted API decides what you can see. I used Google's pretrained checkpoints gemma-3-1b-pt, google/gemma-4-E2B and google/gemma-4-E4B. They are base models from two Gemma generations, so I can see how recitation changes across families and sizes. Gemma 4 E4B is the only one that recites at all often. The models differ in data and design as well as size, so I don't credit size alone. OLMo 2 1B is the fourth model, because its training data is public.

Teacher forcing on a laptop. Each passage is scored in one forward pass over the 40 given words plus the 50 real ones that follow (fewer for some quoted lines), checking the top token at each position. Gemma 4 E4B's checkpoint is 16 GB and I had about 10 GB of RAM free. So I memory-mapped the weights from the safetensors file and let the OS page in what each pass touched. Scoring took about 1.6 s per passage on CPU in bfloat16. Generating text is far slower, about 0.3 s per token.

Controls and honesty.

I committed the sample, metric and threshold to git before the main run. Everything decided later, including the looser reading, is listed at the end of PROTOCOL.md.
An AI (Claude) wrote the 16 control passages. They show how predictable fresh prose in this style is, not a general baseline for what a model has never seen.
The six recently posted books are probably unseen, not certainly, because older scans may exist.
I hand-picked the fifteen forgotten books. Thirteen of their writers are women, but I chose these books deliberately, so this cannot show whether models remember women's writing less.
"Not recited" means the measured run did not reach ten words from these prefixes. Five of the 102 quoted lines are shorter than ten words, so they could never count. It doesn't mean the model holds nothing of the book.

Counting copies. The infini-gram API counts how often each passage's ten-word seam (the last 5 given words and the first 5 to predict) appears in DCLM and in OLMo 2's pretraining corpus. Leaving out seams that cross a line break, 139 random pages have a seam that appears 100 times or more in OLMo 2's corpus. OLMo 2 1B recites none of them, and its longest run on them is six words. These are counts of matching text, not proof of how often the model saw a passage.

AI disclosure. Claude Code, an AI coding agent, wrote most of the code and drafted this post, and Claude wrote the 16 control passages. I chose the concept and the targets, and reviewed everything before publishing.

Why Does Open Innovation Matter?

Open weights let me see how the model ranks every possible next token, not only the text it prints. Anyone can download the same checkpoints and rerun everything. I did not record the exact checkpoint revisions, so the repo lists the SHA-256 of the Gemma 4 E4B and OLMo 2 weight files instead, and tiny floating-point differences can still flip a near-tie. Open data let me count each passage's seam in OLMo 2's own pretraining corpus. For Gemma, whose data is private, the best I can do is a stand-in corpus.

The texts are open too, thanks to volunteers. Project Gutenberg's proofreaders posted six of these books in the last two years, and Wikiquote's editors chose the lines people repeat. A forgotten book gets a second chance because someone typed it up and gave it away. That is why the site links every one of them.

The Spirit of It

The idea came from the ofrenda. On the Day of the Dead, families set out photographs and the things the dead loved, so that they are remembered. The site ends with one for the fifteen books the models don't recite. Each card has a sentence by the writer, the place where the 40 given words stop, what the book says next, what the machine wrote instead, and a link to read the rest.

My favourite is Marjorie Bowen's Black Magic (1909). The given words introduce a man gilding a devil. Bowen goes on to describe the devil, "rudely cut out of wood", resting by his three tails and his curled-back horns against the wall. Gemma 4 describes the man instead: "a tall, thin, pale man, with a long, thin, pale face, and a long, thin, pale beard". The word "beard" appears nowhere in Bowen's novel. When its devil finally comes, it is "a small, round, red devil", with the same long, thin, pale beard. Where Bowen carved something strange, the machine reached for one stock phrase and kept repeating it.

The Raven needs no one. These do. Black Magic is free on Project Gutenberg.

Prize Categories

Best Use of Gemma

DEV Community: nash aschenbath