DEV Community

soy
soy

Posted on • Originally published at media.patentllm.org

My RAG's Model Had Already Read the Books - Per-Book Verdicts, a Fake Regression, and Catching a 9B Leaking Prior Knowledge

#rag #llm #ai #reproducibility

My evaluation told me my RAG had gotten worse. PASS dropped from 9/20 to 5/20
between May and July. Same database, same prompts, same vLLM version, same
model. I eventually diffed the stored request payloads byte by byte: 63,430
characters, identical. Every input bit was the same. The "regression" was
noise — and proving that took a two-month forensic reconstruction that a
five-line metadata stamp would have made unnecessary.

That's one of three stories from this project. The other two: a retrieval
budget that silently stopped working when the corpus grew 14×, and a 9B model
that keeps smuggling pretraining knowledge into answers that are supposed to
use only the retrieved excerpts — in a way you can catch mechanically, with
no GPU and no judge model.

This is the readable version of my technical report. Everything runs on one
consumer machine (RTX 5090, 32 GB VRAM) with
NVIDIA Nemotron Nano 9B v2 served by vLLM.

The setup: a corpus the model already knows

The corpus is all seven Harry Potter novels: 1.36M words, chunked into 5,579
overlapping 250-word pieces, stored in a single 40 MB SQLite file (FTS5 index
plus a sqlite-vec table of BGE-large embeddings). Query embedding runs on
CPU so the GPU stays dedicated to generation.

Why this corpus? Because it's adversarial for faithfulness. Berkeley's
"Speak, Memory" study showed
GPT-class models memorized Harry Potter about as thoroughly as anything in
their training data, and Microsoft's
"Who's Harry Potter?" unlearning paper
chose the series precisely because that knowledge is so hard to remove. So
when I tell my model "answer using ONLY the provided excerpts," it has a
parametric shortcut available for every single question. If it ever leans on
that shortcut, the corpus makes the slip detectable — the model can produce
entities the context withheld. (Whether my particular model actually
memorized the text is a question I test directly later in this post — the
answer surprised me.)

Per-book verdicts, not merged answers

The founding measurement: I asked "What gift did Dumbledore give Harry for
Christmas?" against Book 1 alone (400 chunks), and retrieval nailed it. The
same query against all 7 books (5,579 chunks) with the same top-15 budget:

Scope Method cloak chunk note chunk Ron's ID chunk recall
Book 1 (400 chunks) trigram rank 323 13 15 3/3
7 books (5,579) trigram 4,488 123 125 0/3
7 books (5,579) BM25 3,161 90 814 0/3

Nothing changed except the size of the competing pool. The winners were
later-book Christmas/Dumbledore/Harry passages — high keyword overlap, zero
relevance. A retrieval budget tuned at 400 chunks silently stops working at
5,579.

My fix is unfashionable: run the whole pipeline once per book. Each book
gets its own retrieval and its own model call under a strict prompt — answer
in 1–4 sentences with chapter citations, or output exactly
NOT_FOUND_IN_THIS_BOOK. Seven independent verdicts, displayed side by side.

There is deliberately no merge step. A merged answer launders a fabricated
verdict together with grounded ones. When I ask "How did Voldemort split his
soul?", Book 6 contributes the mechanism and Book 7 contributes the
accidental-Horcrux coda, and Books 1–5 correctly abstain. Seven cards show you
that. One synthesized paragraph would hide which book said what — and any
aggregator smart enough to assemble the pieces would be re-introducing exactly
the prior-knowledge shortcut I'm trying to exclude. (Recent papers like
SPD-RAG and PAGE-RAG decompose per-document too, but they aggregate at the
end; the no-merge display is the part I'd defend hardest for research use.)

Cost: 7× the model calls. On a local box that's latency, not money — a full
20-question × 7-book evaluation takes 11–15 minutes at concurrency 3.

The regression that wasn't

In May I ran the eval across four retrieval backends. Best result: PASS 9/20.
In July I re-ran the exact stored configuration: PASS 5/20.

Before concluding anything, I compared the stored May request payloads against
freshly generated ones. Byte-identical — same 63,430-character context,
same system prompt, same sampling parameters, same model revision, same vLLM.
So the only remaining variable was sampling at temperature 0.3. The
decomposition across 140 book-verdicts (20 questions × 7 books):

  • 123/140 (88%) reproduced exactly, two months apart
  • 17 flips: +9 new false-positive books, −4 dropped FPs, +3 newly-found expected books, −1 lost
  • True-positive coverage actually improved (19 → 21 expected books found)

The PASS drop was entirely "one extra spurious book demoted PASS to PARTIAL."
Exact-set scoring means a single flipped book changes the verdict. Single
unseeded runs at temp 0.3 are a noisy instrument with a ±4 swing on a
20-question suite — comparing two of them "detects" regressions that do not
exist.

The kicker: my May logs also contained a real collapse (PASS 9 → 0 within
four days) that I had mentally filed as model weirdness. Payload forensics
showed it was an unrecorded config change — someone (me) had widened the
context ~2.4× and doubled max_tokens. More context, worse precision: the wide
runs produced up to 91 false-positive book verdicts in one run. Both
failure modes — seeing regressions that aren't there, and blaming the model
for config drift — come from the same root cause: nothing was stamped.

What fixed it

  1. Seed every request. vLLM accepts per-request seed. Measured: two seeded runs at concurrency 3 → 15/15 identical verdicts, 14/15 byte-identical answer contents. The residual wobble is batch-composition nondeterminism — kernel shapes change with serving load — so bit-exactness additionally requires concurrency 1.
  2. Stamp every run. Window, top-K, max_tokens, seed, prompt version and hash, model and resolved revision, repo commit — into the results JSON.
  3. Pin the model revision. I caught my Hugging Face cache re-resolving refs/main on every cold start, with three different snapshots of the "same" model accumulated over two months. An upstream push would have silently changed my weights mid-study. vllm serve --revision <sha>.
  4. One prompt source. My UI and my eval harness had drifted to different prompt texts — the eval was measuring a prompt nobody shipped.

None of this is novel. What I can offer is the measured cost of not doing
it: one fabricated regression, one mis-attributed collapse, and a forensic
reconstruction that only worked because the harness happened to store raw
payloads.

Catching the leaks

The strict prompt says: every claim must trace to an excerpt. Does the model
comply?

Here's a Book 4 verdict for "How did Voldemort split his soul?":

Voldemort split his soul through the creation of Horcruxes — objects
containing soul fragments... [Book 4 Ch.32 §5]

The word "Horcrux" appears zero times in all 995 Book 4 chunks. It exists
only in Books 6 (58×) and 7 (118×). The model answered from pretraining and
wrapped it in a fabricated citation.

That case suggested a detector so cheap it feels like cheating: for every
FOUND answer, flag terms that appear nowhere in the context the model was
given. No judge model, no GPU, purely lexical (with naive stemming), and —
because the harness stores every payload — retroactively applicable to every
run I've ever done. I audited 671 stored FOUND answers across 15 runs.

Two tiers of signal:

  • Entity leaks — ungrounded terms capitalized mid-sentence. 43% of FOUND answers contain at least one. The champions:
Leaked entity Count Why it's damning
horcrux(es) 41 exists only in Books 6–7; every leak is into a Book 1–5 verdict
albus 20 context says "Dumbledore"; model supplies the first name
peter / pettigrew 18 / 16 context says "Wormtail" or "Scabbers"; model reveals the true identity
remus 15 context says "Lupin"
salazar / helga 14 / 12 founders' first names, absent from retrieved chunks
  • Dense leaks — answers with ≥8 ungrounded terms. These are the fabricated essays; the record holder had 175 ungrounded terms in a single "cited" answer.

The dominant pattern deserves a name: entity completion leakage. The model
mostly doesn't invent events — it completes entity references from
parametric memory. First names, true identities, book titles. Under a
citation-discipline prompt this is a real violation (a Book 3 verdict naming
"Peter Pettigrew" asserts an identity the excerpts deliberately withhold at
that point — an unsupported spoiler presented as cited fact). It's also, in
effect, the name-cloze memorization probe from "Speak, Memory" occurring
spontaneously inside a downstream task — countable for free.

The audit's fringe benefits: it surfaced a misspelled entity ("Siris" for
Sirius) and whitespace-collapsed tokens ("apersoncouldbesplit") inside
otherwise fluent answers — sampling artifacts that verdict-level scoring would
never show you. It also confirmed the config coupling: the wide-context runs
that destroyed precision were also the leak-rate outliers. More tangential
context doesn't just cause more answers; it causes less grounded ones.

Two honest caveats: the rate is a lower bound (paraphrase leakage is invisible
to a lexical check), and flagged terms are audit candidates, not verdicts —
the point is that the candidate set is small enough to eyeball.

So had it actually read the books? I tested it

Everything above leans on "the model knows Harry Potter." My evidence for
that was circumstantial: GPT-4 memorized these books (Berkeley), Llama-2
needed surgery to forget them (Microsoft), and my model keeps producing first
names the context withheld. But those first two facts are about other
models. So I ran the established memorization probe directly on mine:
name cloze (Chang et al., EMNLP 2023) —
show the model a short passage from the books with one character name masked,
give it no other context, ask for the name. A model with no exposure to the
text or its surrounding discussion should score near zero.

Fifteen distinctive names, one passage each, temperature 0, seed fixed:

Masked name Nemotron's guess
Snape Snape ✓
Draco Draco ✓
Luna Luna ✓
Umbridge Dolores ✗ — but that's her first name
Hagrid Sirius ✗ (in-franchise)
Bellatrix Lily ✗ (in-franchise)
Lupin Murdock ✗ — the only non-HP guess of the night

Exact match: 3/15 (20%). For calibration: GPT-4 reportedly scores ~76%
on these books, and genuinely unseen books land around 0–10%.

But look at the misses: 14 of 15 guesses are Harry Potter characters.
The model recognized the universe in essentially every passage — it just
couldn't recall the exact token. And the "Dolores" miss is my favorite
datapoint of the night: asked for a masked surname, the model volunteered
the character's first name. That is not passage memory. That is franchise
knowledge.

Let me be careful about what this does and doesn't establish — none of it
implies anything improper about how the model was built:

  • The verbatim-memorization signal is weak for this 9B at pilot scale. And 20% is an upper bound — an entity-aware model can sometimes infer the right name from the passage alone, no memorization needed.
  • The entity knowledge is near-total, and this probe cannot tell you where it came from: the novels themselves, or the web's enormous derivative layer — wikis, reviews, three decades of discussion. I make no claim about what was or wasn't in the training set. Encountering Harry Potter somewhere is close to unavoidable for any web-scale corpus, and NVIDIA is unusually transparent here — they published the majority of the pretraining data.
  • n=15 is a pilot. The full version (more items, more models, public-domain control books) is queued.

Here's why I like this result better than the one I expected: the shape of
what the model knows matches the shape of what leaks. The grounding audit
found entity completions — first names, identities, titles — not fabricated
passages. The cloze probe found entity knowledge — near-total franchise
recognition — not passage recall. Two independent behavioral measurements,
one picture: this model doesn't smuggle memorized text into context-only
answers. It smuggles who's who.

The checklist

If you evaluate a local RAG and want your numbers to mean something next
month:

  1. Send a seed with every request; use serving concurrency 1 when you need bit-exact runs.
  2. Write window / top-K / max_tokens / seed / prompt-hash / model revision / repo commit into every results file.
  3. Pin the model revision at the server (--revision <sha>); your HF cache re-resolves main more often than you think.
  4. Keep exactly one prompt source for the thing you ship and the thing you measure.
  5. Store the raw payloads. Every claim in this post is re-checkable because the harness saved every request and response. Storage is the cheapest reproducibility technology that exists.

Limits, and what's next

This is a case study: one model (9B hybrid Mamba-Transformer), one corpus,
20 adversarially-designed questions. The memorized corpus cuts both ways — it
makes leakage detectable, but abstention behavior may differ on corpora the
model has never seen. The books are copyrighted, so the corpus can't ship;
the pipeline is corpus-agnostic (any book/chapter/text CSV builds the whole
index) and the harness code is slated for an MIT release once the repo is
scrubbed for publication.

Next experiments: reasoning on/off ablation (the chat-template thinking
toggle changes short-answer latency by an order of magnitude), seed-ensemble
confidence intervals on PASS, a prompt-version A/B under seeds — all one
env-var away in the current harness — and the full name-cloze study (≥50
items, multiple models, public-domain memorized controls like Sherlock
Holmes, which would also make the whole study replicable without a
copyrighted corpus).

Related posts from the same machine:
one RTX 5090 vs a 12-GPU cluster
and
the CPU-bound sequel.

References

Drafted with AI assistance (Claude). Every number traces to stored run
artifacts (payloads, results, and metadata stamps) from 2026-05-24 through
2026-07-25.

Top comments (0)