This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
What I Built
The vault note had been deleted. Its body was empty and its title now read Withdrawn document. Then the app asked a local Gemma model, in different words, "Which access phrase opens the observatory vault?" Gemma answered:
The observatory vault passphrase is MOTH-7E020979A947.
The passphrase is fictional, generated at random for that run. It could not have come from the deleted note, because nothing was left in it. It came from what the app had made out of that note: a search chunk, three summaries and an earlier saved answer. The source was gone. Its ghost was still in the machine.
GhostCache is a small local lab built around that moment: a tiny retrieval assistant over a five-document fictional archive, with Gemma (gemma4:e2b-it-qat) running on my laptop through Ollama. It makes one question testable: when a document is withdrawn, can an AI app remove everything it derived from it, and keep everything it shouldn't lose?
- Spread. Each reset plants a fresh random passphrase in one document. The app chunks the documents, Gemma writes three chained summaries, and the assistant saves an answer.
- Haunt. "Delete just the source" empties the document and leaves its copies alone. A reworded question brings the secret back.
- Follow the trail. Every derived artifact recorded what it was made from, so cleanup walks that provenance graph, empties every descendant in one transaction and rebuilds a summary from the documents that survive.
- Prove it. Ten checks confirm that the leak was real, that the secret is gone from the assistant's stored memory and retrieval, and that unrelated facts still answer.
The leak is staged, on purpose. "Delete just the source" is a deliberately broken control, and the summary prompt tells Gemma to keep the passphrase so the propagation path is known. GhostCache doesn't claim any particular RAG app has this bug. It shows the mechanism, and a fix you can check, for developers whose assistants chunk, summarize, cache or remember what they read.
Demo
Watch the demo on YouTube (about five minutes).
How the video was made: one continuous take of the local lab running live with Gemma through Ollama, on fictional data, recorded October 30. Clicks are automated and the pointer is drawn; model waits play at real speed; I recorded the narration afterwards, over the take; the browser address bar was added at render time.
Explore the recorded runs: https://ghostcache.bipulchamoli.tech/ (also at https://ghostcache-walkthrough.onrender.com/). It's a static walkthrough of three saved live Gemma evaluations, where you can step through each stage and read every recorded answer. No login, and no live inference: nothing on the page can reach a model. For a two-minute look, open the leak in the October 27 run, step through the last two stages, then open the recorded question "After cleanup · shared fact".
Run it yourself. Fixture mode needs only Python 3.12. Clone the repository:
git clone https://github.com/bipul724/GhostCache
cd GhostCache
Choose one mode below, using a free port. Each startup and Reset seeds a fresh experiment and replaces the selected database, so use these disposable demo paths. Stop the server with Ctrl+C before switching modes on the same port.
# deterministic fixture: no model, network or key
python3.12 -m ghostcache serve --port 8765 --mode fixture --database .local/article-fixture.sqlite
Or, with Ollama running and gemma4:e2b-it-qat installed:
# install the model beforehand if needed: ollama pull gemma4:e2b-it-qat
python3.12 -m ghostcache serve --port 8765 --mode live --model gemma4:e2b-it-qat --database .local/article-live.sqlite
Open http://127.0.0.1:8765/.
Following one ghost
The three product screenshots below were captured on October 31 from the current public interface. They display the first live evaluation, recorded on October 27. In the video the lab had saved a second answer before the deletion, so six descendants stay behind there rather than five, and the lab's own verification runs eight checks rather than the full evaluation's ten.
The seeded memory (a drawing of the design, not a screenshot):
d-vault ─▶ c-vault ─▶ s-vault ─┐
d-garden ─▶ c-garden ──────────┴▶ s-mixed ─┐
d-maintenance ─▶ c-maintenance ────────────┴▶ s-handoff
saved answer ◀─ d-vault, c-vault, s-vault,
s-mixed, s-handoff, d-shuttle
d-archive and d-shuttle get chunks too. The archive matters later: it independently names the reading room, Lantern Hall, which the vault note also mentions.
Delete just the source. The note is emptied and renamed, and that's all. Five derived items still depend on it and still hold the marker. Asked in different words, retrieval hands Gemma the chunk, all three summaries, the saved answer and the garden note, and Gemma repeats the passphrase.
The current walkthrough displays the October 27 Gemma run: the source is deleted, yet the reworded question reveals the same marker. Screenshot captured October 31; recorded evidence, no new inference.
Follow the trail. Cascade cleanup starts at the withdrawn note and walks every recorded parent-to-child link. In one transaction it empties the source and all five descendants, keeping their IDs and links as tombstones. Then Gemma writes one fresh summary from the original documents that survive in that lineage: d-garden, d-maintenance and d-shuttle.
Same recorded run in the current interface: five descendants are content-free tombstones, while surviving documents and the rebuilt summary remain. Captured October 31; the evaluation was recorded October 27.
Did we delete too much? "Delete everything" would also pass a no-leak test, so two checks guard against it. The roof garden still opens at 07:40. And Lantern Hall, which the deleted note also mentioned, still answers, because an independent document says it too.
Lantern Hall still answers through its independent source. Same October 27 run, displayed in the current walkthrough and captured October 31.
Every saved live run shows the same pattern:
| Recorded live run | After source-only deletion | Marker rows after cleanup | Checks | Model calls (summed call time) |
|---|---|---|---|---|
| October 27 | Marker revealed | 0 | 10/10 | 11 (130.7 s) |
| October 28, first after starting Ollama | Marker revealed | 0 | 10/10 | 11 (61.3 s) |
| October 28, model already loaded | Marker revealed | 0 | 10/10 | 11 (22.1 s) |
A deterministic fixture run with no model also passes 10/10. These are three demonstrations with one model on one laptop, not a success rate.
Code
GhostCache
Gone from the source. Still in the machine.
Public demo video: https://www.youtube.com/watch?v=yzjmyaA-9Vw (4:55, one unedited live run of the local lab with Gemma through Ollama).
GhostCache is a small local testbed. It shows how an AI assistant can still know something from a document after that document was deleted, and how to clean up properly.
When an assistant reads a document, the app often makes copies of it, such as search chunks, summaries, handoff notes and saved answers. Deleting the original file does not delete those copies. If a later question retrieves one of them, the information comes back. GhostCache lets you:
- put a fictional secret in a document,
- watch it spread into chunks, summaries and a saved answer,
- delete only the original and see the secret still come back,
- follow the provenance graph to every copy and clean them all up,
- rebuild useful mixed summaries from the documents…
MIT-licensed, about 1,400 lines of standard-library Python at the core. Start with store.py (provenance, both deletion policies, retrieval and the audit) and app.py (the experiment and its checks). results/ holds the saved reports.
How I Built It
A deliberately small stack. Python 3.12's standard library, SQLite and plain HTML, CSS and JavaScript, with no build step and no required dependencies. Gemma runs through Ollama at temperature 0 and must return JSON with text and citations; the client rejects any citation that isn't a supplied context ID. If the model is missing or its output is malformed, live mode fails loudly and never swaps in fixture text. Fixture mode, labelled as a deterministic simulation, replaces the model with string rules for tests and CI.
Lineage comes from context, not citations. That was the key design decision. Citations can be incomplete, so when the app stores a summary or an answer, every artifact it handed the model becomes a parent:
artifact = self.store.add_artifact(kind, title, result["text"],
[p["id"] for p in parents], artifact_id)
The store derives each artifact's original sources from its parents (callers can't supply them) and refuses the write if a parent or source was withdrawn in the meantime. Lineage is conservative: a mixed summary depends on all of its inputs, even ones its prose ignored.
Cleanup follows the graph. A recursive query finds every transitive descendant of the withdrawn document:
WITH RECURSIVE downstream(id) AS (
SELECT child_id FROM provenance WHERE parent_id = ?
UNION
SELECT p.child_id FROM provenance p
JOIN downstream d ON p.parent_id = d.id
)
In one transaction, the source becomes Withdrawn document, each descendant becomes Purged <kind>, and every body is emptied. Rows, IDs and edges stay, so the cleanup can be inspected and repeated. A mixed summary isn't patched, because you can't safely edit a withdrawn fact out of generated prose: it's invalidated whole and replaced by a summary of the surviving original documents, never the old text. After cleanup, retrieval also skips anything whose sources include a withdrawn document, as a second guard.
Verifying without fooling myself. The ten checks fall into three groups:
- The failure was real: the baseline answer exposed the marker, and after source-only deletion stored rows still held it and the reworded answer revealed it. A run that never leaked can't pass.
- The secret is gone: from the source and its descendants, every stored title and body, four retrieval probes and both secret questions, and no active artifact depends on a withdrawn source.
- Nothing else was lost: the garden time and Lantern Hall still answer.
The audit scans every row, including inactive ones, and the retrieval probes run without the second guard, so it can't hide leftovers from the checks. 126 automated tests cover lineage, deletion, failure handling, the HTTP layer and both front ends (rerun locally on October 30; six telemetry tests skip unless the optional Sentry SDK is installed).
Watching the ghost in Sentry
Tracing is optional and metadata only: every attribute GhostCache explicitly sets on a span goes through a fixed allowlist (operation names, model, token counts, durations, synthetic artifact IDs, a few flags). Document text, titles, questions, answers and the marker are never sent to Sentry. A test serializes the outgoing envelopes and fails if any of them appear, and Sentry's Agents view agreed: it reported that inputs and outputs weren't captured. I traced three evaluations on October 28. Two findings:
The trace shows the leak without containing it. After the source-only delete, the assistant span carries ghostcache.stage: source_only and ghostcache.answer_contains_marker: true, and its memory search lists the chunk, three summaries, a saved answer and d-garden as context. IDs and one boolean are enough to see the leak.
The source-only answer in Sentry: the stage and a marker flag, never the marker.
One cold model call explained most of a slow run. The first evaluation after starting Ollama took 1.03 minutes; the next took 22.40 s. Most of the gap is the first summarizer call: 40.45 s cold, 2.28 s warm. Every later call took about 1–3 s in both. That fits Ollama loading the model on its first request, but GhostCache doesn't record load time, so the cause is inferred, not measured. Both runs passed 10/10; only the waterfall showed the difference.
Cold trace waterfall: one 40.45 s call in a 1.03 min run.
Failures are traced too: three live attempts I made before starting Ollama failed with a ProviderError rather than falling back to fixture answers, and the one I inspected reached Sentry as a single error span.
What went wrong along the way
- The echo test first asked the wrong question. The lab's "Look for the echo" step re-sent the original question instead of the reworded one, so it tested nothing new. It now sends the same paraphrase as the evaluation, and a UI test pins that.
- A warning disappeared after 50 events. The lab decided whether a rebuild had failed by scanning the event log, but the server returns only the latest 50 events. After 13 verifications the failure scrolled out and the page turned green again. The rebuild outcome is now separate state, not a log scan.
- Error messages could echo content. Early transport errors included the exception text, which can carry request data. They now report a fixed category, such as "connection failed".
What this doesn't prove
- Managed memory only. Cleanup covers GhostCache's own stored titles and bodies, its recorded provenance and its retrieval and answer paths. Tombstone IDs and links remain by design. It isn't model unlearning or secure disk erasure.
- The evaluator keeps the marker on purpose, so the checks can look for it; it appears in the saved reports, the walkthrough and the video. The checks are exact-match probes over four fixed questions with lexical retrieval, not proof that every paraphrase is gone.
- Copies made outside the store are invisible to the graph. Provenance only finds what was recorded.
Provenance-based invalidation isn't new. Nisarg Pujara's Stale RAG vs. expensive RAG invalidates cached RAG units by the sources they cite, and Jonathan Melton's Where retrieval assistants leak explains why checking only raw documents or final answers isn't enough. GhostCache makes the failure and its cleanup visible in an experiment small enough to inspect end to end.
Why Does Open Innovation Matter?
A deletion claim is only as strong as your view of where the data went. GhostCache's argument is "here is every copy, and here is proof each one is empty", and the open pieces are what make that checkable.
- Requests carry no conversation history. Gemma runs on my laptop, and each request carries a system instruction and the context GhostCache retrieves, without prior conversation turns. GhostCache manages the assistant's persistent application memory in its artifact store. With a hosted model API, my cleanup claim would stop at the API boundary: whatever the provider logs or retains is outside anything I can inspect. Locally, the documents and prompts never leave the machine for inference.
-
It works with the lights off. Once the model is pulled, live mode talks only to
127.0.0.1(unless you opt into Sentry tracing). Fixture mode needs nothing at all, which is how CI runs it with no secrets and no model download. - Anyone can rerun it and read the fix. The model tag is documented, the saved reports are in the repository, and the deletion logic lives in an MIT-licensed store of under 400 lines, not in a terms-of-service promise. If it's wrong, you can see where.
(GhostCache's code is MIT-licensed; Google's Gemma 4 model card lists the model's license as Apache 2.0.)
Prize Categories
Best Use of Gemma: local gemma4:e2b-it-qat through Ollama writes every summary, including the rebuilt one, and every answer in live mode, with citations checked against the supplied context. Evidence: three saved live evaluations, 10/10 each, replayed in the walkthrough and run live in the video. The leak and Lantern Hall screenshots above show Gemma's recorded answers from the October 27 run (recorded results displayed by the walkthrough, not new inference); the video shows the lab calling Gemma live.
Best Use of GitHub Copilot: entered through the category's GitHub Actions clause; Copilot didn't write this project's code. On every push, the workflow runs the tests, a fresh fixture deletion evaluation (kept as a report) and the telemetry privacy tests with the Sentry SDK. Evidence, checked October 30: 20 of 21 runs succeeded (one was a cancelled duplicate); run #1 ran 62 tests, and the archived reports of run #16 and run #17 passed 10/10. The latest, run #21, tested the current commit:
Run #21 of the "GhostCache deletion checks" workflow: a push of commit e9a4b29 to main on October 30, status Success, one uploaded artifact. Captured October 31 from the public run page.
The same run's job: every step succeeded, including the tests, the fresh fixture evaluation (no model), the report upload and the Sentry privacy tests. Logs need sign-in, so no test count is shown here.
Best Use of Sentry Agent Tracing: metadata-only gen_ai spans for the assistant, summarizer, model calls and memory operations, with the two findings above. Evidence: the two trace screenshots in "Watching the ghost in Sentry", captured October 28 from that day's traces (details).
Best Use of Render: Render hosts GhostCache’s public interactive walkthrough of three recorded local Gemma evaluations. Visitors can inspect the leaked answer, provenance cleanup and preserved facts without installing the model. The walkthrough is static and performs no live inference.
Live: https://ghostcache-walkthrough.onrender.com/ · Blueprint: render.yaml · Deployment record: DEPLOYMENT.md
Render dashboard captured October 31, 2026: the Blueprint-managed ghostcache-walkthrough static site is live on branch main at commit e9a4b29. The latest deployment is titled "added the youtube demo video link" and was shown as deployed 11 hours before capture.
Public Render walkthrough captured October 31, 2026 in Chrome. The address bar shows the actual Render URL, the GhostCache branding is visible, and the page displays the "Recorded Evaluation · No Live Inference" banner over the October 27 leak stage.
Best Use of a .tech Domain: GhostCache’s public walkthrough is live at https://ghostcache.bipulchamoli.tech/, a subdomain of my .tech domain claimed through the Hacktoberfest promotion. It serves the same static recorded walkthrough hosted on Render; no inference runs there.
GhostCache public home page captured October 31, 2026 in Chrome. The address bar shows ghostcache.bipulchamoli.tech, with the headline "Gone from the source. Still in the machine." and the explanatory illustration visible.
Render Custom Domains panel captured October 31, 2026: ghostcache.bipulchamoli.tech is Verified, its certificate is issued, and the Render subdomain remains enabled.
Deleting the file isn't enough. Record where every copy came from, and an app can clear a withdrawn secret from its own memory while keeping what other sources still support.
GhostCache was built with AI coding agents working under my direction. AI assistants drafted and fact-checked this post against the project's code, saved reports and evidence.











Top comments (27)
The ghost framing is exactly right — deleting the source left the chunk, three summaries and a saved answer, so the passphrase survived a deletion that looked complete. A deletion claim has to enumerate every derivative, not just the file. Where does the provenance chain break first in a real RAG app — the embeddings store, the cached answers, or somewhere less obvious like the summarizer's own outputs?
Thanks! I should say up front that GhostCache can't rank these for real apps. It has no embeddings store (retrieval is lexical), and I haven't tested a production system. So this is my reasoning from building it, not a measurement.
In a simple document-to-chunk-to-embedding pipeline, I'd expect the link back to the source to be fairly easy to keep at ingestion. GhostCache only shows the chunk part: its vault chunk has the vault note as its only parent.
The first place I'd audit is the summarizer's outputs, especially summaries built from other summaries. GhostCache's handoff summary never received the original vault note directly. It was written from the mixed digest and the maintenance chunk, so its only path back to the note runs through the mixed digest, the vault digest and the chunk. After the source-only delete, retrieval still handed that summary to Gemma, and Gemma cited it while repeating the passphrase. If an app doesn't record what went into each summary, or only checks the direct children of the deleted file, that copy is missed.
Saved answers are another place I'd look, because their inputs are whatever retrieval returned at that moment. GhostCache's saved answer depends on the shuttle guide only because the word "observatory" matched.
Then there are copies outside GhostCache's managed store, such as external logs, backups and evaluation reports. Its provenance graph doesn't cover those. The evaluator keeps the expected marker separately on purpose, so it can check for leftovers, and the saved reports preserve it as evidence. That's why the cleanup claim is limited to the managed artifact store.
The summaries-of-summaries answer is the one I'd carry forward — the handoff summary never touched the original note, so any check that walks only direct children misses it by construction. The saved-answer point pairs with it: retrieval at a moment in time freezes a derivative nobody's graph tracks. The external-copies caveat is the honest limit of the claim, since a managed-store cleanup can't cover logs and eval reports by design. Is the fix a provenance graph that records inputs per summary, or does that just move the break to wherever the graph itself stops?
One clarification about the saved answer: GhostCache does track it. Every artifact supplied as context becomes a parent when the answer is saved, which is why cleanup caught it. An exported or uninstrumented copy would sit outside that protection.
On your question: partly, yes. Recording inputs fixes a specific bookkeeping failure, while making the remaining boundary explicit. GhostCache approaches that in three ways:
Require lineage when saving derived artifacts --> The application records every supplied context artifact, and the store requires at least one parent and derives the original source IDs itself. That guards the current write path; it doesn't prove that every future caller will declare all its inputs.
Check content independently of graph traversal --> The marker audit scans every managed title and body, including inactive rows, regardless of their recorded ancestry. It could therefore catch a literal marker copy that the graph missed. That works because this experiment knows the planted marker; it isn't a general detector for sensitive information or paraphrases.
Limit what reaches telemetry --> GhostCache emits selected metadata to Sentry, with regression tests checking captured payloads for the test documents, questions, answers and marker. Handled Ollama transport errors use fixed categories rather than the underlying error text.
Beyond the tracked store, additional copies need their own inventory and deletion or retention processes. GhostCache doesn't clean external logs, backups or evaluation reports. A graph can support a wider system if those relationships are captured, but graph traversal alone cannot discover copies that were never recorded. That's the boundary the demo makes explicit.
That correction matters — the saved answer isn't a gap in the graph, it's a tracked derivative, which makes the cleanup result stronger than the first read suggested. The limit that stays interesting is the marker audit's literalness: it catches a copied marker the graph missed, but a paraphrased secret or an exported copy still sits outside both checks. Is the honest scope of a GhostCache deletion claim then 'every tracked derivative plus every literal copy,' or is there a fuzzier check planned for the paraphrase case?
The write time withdrawal check is the detail I’d want to see exercised in the demo. I’d pause a summary just before it is saved, run cleanup, then let that old summary finish. Seeing the store reject the late write would make the tombstones feel much more concrete: the secret stays gone even when work was already in flight.
Good suggestion. The store-level check exists and has a test, but the current demo doesn’t expose that interleaving through normal requests.
GhostCache’s controller serializes generation and cleanup. A cleanup request arriving during generation waits for that operation to finish. Cascade cleanup can then clear the newly saved artifact if it descends from the withdrawn note.
The store independently rechecks the declared parents and their original sources under its own lock before inserting a derived artifact. The existing test reads the vault digest, generates a fixture summary from it, withdraws the vault note directly through the store, then attempts to save the result.
The parent summary is still active, but its original source is now deleted. The save fails with
Cannot derive from deleted source: d-vault, and no new summary is inserted. It’s a controlled test of that ordering, rather than a concurrent live-model run.That guard also matters during the deliberately incomplete source-only stage: an answer generated from stale copies can be displayed, but the store refuses to save it as another memory.
Showing the rejected write would make a useful demonstration. Simply pausing generation wouldn’t be enough, because the controller lock would still block ordinary cleanup requests. I’d need an explicit test harness that injects withdrawal between context capture and saving.
I’d label that carefully: it demonstrates the guard within the same store instance, not correctness across independent database connections or distributed workers.
Thanks for clarifying what the store test covers. The useful guarantee is that the save is rejected even while the intermediate summary is still active. That makes it clear why the original source has to remain part of the check.
The reworded question is what makes this convincing. An exact match purge would have passed and still leaked, which is close to what most teams would ship if a client asked them to "delete the document from the assistant."
The Lantern Hall check is the part I would copy first. A cleanup that wipes everything also passes a no leak test, so proving that facts supported by other documents still answer is the only way to show the deletion was precise rather than destructive.
Keeping tombstones with the IDs and edges after emptying the bodies is also a nice touch, because it turns cleanup into something you can audit later instead of something you trust.
Utkarsh's rule about summaries with a single surviving root seems right for cost. In a real app, would you record every artifact passed as context as a parent even when the model ignored it, accepting more invalidation in exchange for never missing one?
Thanks! Yes—I’d start with recording every supplied context artifact as a parent when generated content is saved, even if the model doesn’t cite it. GhostCache has no reliable mechanism for establishing which supplied inputs the model actually ignored.
The tradeoff is unnecessary invalidation: an extra dependency can cause useful derived text to be cleared and require regeneration from surviving sources. That regeneration can fail or change the result’s detail and structure. But omitting a dependency can leave an affected artifact outside the withdrawal’s recorded reach.
That wouldn’t necessarily go unnoticed in GhostCache: another dependency path might still reach it, and the independent marker scan could catch a literal leftover. Those checks still aren’t a general detector for missing semantic dependencies.
The shuttle guide makes the cost concrete. It entered the saved baseline answer’s context because “observatory” matched, became a recorded parent, and consequently entered the merged rebuild’s inputs.
I’d also experiment with supplying less irrelevant context before generation, while checking answer quality. Every artifact actually supplied would still be recorded; I wouldn’t prune parents afterward based on the model’s citations. I haven’t tested that optimization.
One clarification about the earlier point: clearing only an answer associated with the original question wouldn’t pass the full checks, because the marker would remain in other managed artifacts. The reworded question adds a behavioral probe.
So I’d choose conservative capture as the starting policy, but I wouldn’t call it “never missing one.” Coverage depends on complete input capture and all relevant writes following that policy. Copies made elsewhere remain outside the graph.
It makes perfect sense from the perspective of safety to go with the conservative lineage rule, but have you considered scalability issues? In a large application, a single withdrawn document might be enough to make many mixed documents invalid. Did you consider bounding the cost of rebuilding? Perhaps lazy rebuilding would work better.
Good question. I haven’t benchmarked GhostCache at scale. It has five source documents, and the saved evaluations invalidate five derived artifacts, so those results don’t establish large-application performance.
You’re right about the fan-out: conservative lineage can invalidate a mixed summary even when its text didn’t use the withdrawn source. With many overlapping summaries, one withdrawal could affect a lot of derived content.
I’d separate purging from regeneration. The purge is transactional and makes no model calls, but its database work grows with the affected graph—I haven’t measured that cost at scale. Rebuilding currently happens eagerly, inside the cleanup request. In the saved runs, one Gemma call creates one replacement summary from three surviving original documents.
That’s a deliberately small implementation. The request uses a 4,096-token context setting and a 420-token generation limit, but it has no batching strategy for a large set of surviving roots. Replacing several summaries with one also changes their original structure.
Lazy rebuilding is a direction I’d explore. The purge already commits before regeneration, and a rebuild failure doesn’t restore the cleared content. I’d keep that purge in place, defer regeneration until needed, and retain the checks that exclude withdrawn roots and reject stale parents when saving the replacement. Surviving original documents and chunks would remain available for retrieval. That could avoid regenerating summaries nobody reads again, but it wouldn’t by itself bound the cost of a large rebuild, so I’d also need a budget for how much regeneration one read can trigger.
Nisarg Pujara’s Vectorlink Labs post on Coalent describes that on-demand refresh pattern for stale cache units. I haven’t implemented or benchmarked lazy rebuilding in GhostCache.
Great breakdown, thanks. The point about merging several summaries into one changing their structure is something I hadn’t thought of. Would you rebuild each invalidated summary separately from its own surviving roots instead? More model calls, but the memory would keep the same shape it had before the withdrawal.
Yes—that’s closer to the approach I’d explore. GhostCache’s seeded example makes the distinction concrete.
The surviving recorded roots for each original summary would be:
The current merged rebuild also includes the shuttle guide because it was supplied to the saved baseline answer. None of the three original summaries had that document in its recorded ancestry. So the merge broadens the summary’s input scope as well as changing its structure.
There are two different preservation goals here. Rebuilding each summary independently from its surviving original documents preserves its source scope. To preserve the mixed-digest → handoff dependency, I’d instead rebuild the mixed digest from the surviving garden chunk, then build the handoff from that fresh digest plus the maintenance chunk. Keeping both would mean two summary-generation calls instead of one.
That would change today’s rebuild policy. The controller would need to select only surviving chunks and newly generated summaries; the store would then reject inactive parents or withdrawn roots when saving them.
I’d treat saved answers separately and leave them invalidated. GhostCache doesn’t store the original question with each answer artifact, so the artifact alone isn’t enough to replay the request.
There’s also a cost tradeoff: a mixed digest left with only the garden chunk might not need regeneration at all, since retrieval already has that chunk.
None of this is implemented. GhostCache currently creates one merged replacement summary; preserving separate summaries and their surviving dependency structure would be a next design step.
Love how concrete this got. Your last point almost suggests a simple rule: if an invalidated summary has only one surviving root, skip regeneration and let retrieval use that chunk directly. Only summaries with two or more surviving roots would actually need a model call. Could cut a lot of rebuild cost for free.
The paraphrased question is what makes this experiment convincing. An exact-match cache purge would have passed the test and still leaked. The hardest part for me is the line at the end: clearing a withdrawn secret while keeping what other sources still support. If the same fact lives in two notes and only one is withdrawn, the right answer is to keep it, and that only works if provenance is tracked per claim, not per file. Great Ghost in the Machine entry.
Thanks! One clarification about the tests: if you mean clearing only the cached answer associated with the original question, that wouldn’t pass GhostCache’s full checks. The audit scans every managed title and body for the marker, so copies remaining in the chunk and summaries would still fail it.
The reworded question adds a behavioral check, but it still shares “observatory vault” with the note. Retrieval is lexical, so this isn’t a semantic-retrieval benchmark or evidence about arbitrary paraphrases. The reworded question must also reveal the marker after the source-only deletion, then stop revealing it after cleanup. A run that never demonstrates the leak cannot pass the evaluation.
On the shared fact, GhostCache preserves Lantern Hall with artifact-level provenance. Both the withdrawn note and an independent archive directory name it. That directory and its chunk aren’t descendants of the withdrawn note, so they survive cleanup. In all three saved live runs, the post-cleanup answer retrieves and cites those two artifacts. Claim-level provenance isn’t necessary for this particular case.
The tradeoff appears in mixed summaries. GhostCache invalidates the whole artifact, including useful information from surviving sources, and generates a replacement from surviving originals. Finer-grained provenance could potentially reduce that regeneration, but it would require trustworthy claim-to-source mappings and safe handling of the remaining text. Those mappings don’t necessarily have to come from a model.
GhostCache takes the simpler conservative approach: record every supplied context artifact as a parent, rather than let model-selected citations determine dependencies.
The demonstrated result is narrow: the passphrase disappears from managed titles and bodies and isn’t returned by either defined secret question, while independently supported Lantern Hall remains answerable.
Great submission! The beginning, in which the note is completely removed but Gemma still reveals the password, is a fantastic hook. The solution of following the provenance graph to delete all derived copies is spot on, and it was also good that you made sure that Lantern Hall still responds. Well done! 👻
Thank you! 👻 Lantern Hall was an important check for me. Deleting everything could pass a “no leak” check, so I also wanted to verify that useful information remained answerable. Lantern Hall still answers because an independent archive document names it too, and that document doesn’t depend on the withdrawn note.
It’s very clever to decide to use lineage not based on citations but on everything given to the model. You cannot trust a little model with getting all citations, so being conservative here works well for making the whole clean-up seem credible. And also respecting the idea of invalidating the entire summary rather than trying to remove the secret from it.
Thanks, that’s exactly the reasoning behind both choices.
On lineage: GhostCache still uses citations. The UI shows them, and the Ollama client rejects any citation outside the supplied context IDs. But citations never determine an artifact’s dependencies.
In the source-only answers across all three saved live runs, Gemma actually cited the five stale copies and left out the unrelated garden document, even though it was also in the context. Still, cleanup shouldn’t depend on the model getting attribution right every time. Whenever generated content is saved, every supplied context artifact becomes a parent.
The tradeoff is conservative invalidation: a mixed summary is invalidated even if its text never used the withdrawn note. GhostCache clears the whole old summary rather than patching out the secret, then generates a replacement from surviving original documents. In those runs, the rebuild inputs were the garden, maintenance and shuttle documents—not the old summary text.
That can discard useful derived text, but the surviving originals remain available. Two checks test whether the garden’s 07:40 opening time and the independently supported Lantern Hall fact remain answerable. Both passed in all three saved runs; they don’t prove that every useful fact survives, and rebuilding can fail.
The boundary is the one in the post: provenance can only follow dependencies the store recorded.
The paraphrase case raised in the thread has a check that does not need the marker: embed the withdrawn document's chunks and search every remaining artifact for neighbours above a similarity threshold. A summary that restates a withdrawn fact in other words, or a saved answer that quotes the vault note loosely, tends to come back as a near neighbour of the chunk it was made from, even when it shares no literal string with it. It flags candidates for review rather than proving absence, so it sits beside the marker audit, which is exact but only for copies it can spell.
The embedding store also adds a field to the tombstone. GhostCache keeps rows, IDs and edges and empties the bodies; in a pipeline that embeds chunks, the row's vector has to go too, or be replaced. Morris and colleagues (Text Embeddings Reveal (Almost) As Much As Text, 2023) trained an inversion model that recovered 92 percent of 32-token inputs exactly from their embeddings, so a row with an empty body and the old vector can still hold the passphrase in recoverable form, and it can still be the nearest neighbour of a reworded question unless the retrieval guard filters it first.
Thanks—both are useful extensions to consider. GhostCache currently uses lexical retrieval and has no embeddings, so I haven’t tested either.
Similarity search makes sense as a candidate-review tool alongside the marker audit. There’s a useful test case in this corpus: the vault chunk contains both the passphrase and the Lantern Hall sentence. The independent archive directory contains that second sentence verbatim. Its chunk could therefore be flagged despite representing information cleanup should preserve; whether it actually crosses a threshold would need measuring.
I’d investigate sentence- or claim-level comparisons, but similarity alone still wouldn’t distinguish an impermissible derivative from independently supported information. Provenance and the independent source would remain part of the review.
A post-cleanup comparison also needs a reference representation captured before the source is cleared. That could be an embedding rather than plaintext, but an embedding isn’t automatically safer to retain. I’d still need to state where that reference is kept, who can access it, and how long it remains.
Agreed on vectors being potentially sensitive derived data. Morris et al. reported 92% exact recovery in their 32-token GTR-base experiment—a strong reason not to assume embeddings are harmless metadata, although it isn’t a universal recovery rate.
GhostCache’s current retrieval already requires active status and a nonempty body, even with the additional withdrawn-root guard disabled. A future vector index would need corresponding eligibility checks. But filtering only addresses retrieval: old vectors would also need to be included in the managed cleanup, with any replacements generated from surviving inputs.
The Lantern Hall sentence is the case that makes the check usable, because it gives the candidate a second neighbour. Compare every flagged artifact twice: against the withdrawn chunk and against the surviving sources. A candidate whose closest surviving source is as close as the withdrawn chunk, as the archive copy of that sentence would be, has independent support and leaves the review set; a candidate that is close only to the withdrawn chunk is the one a reviewer has to read. Provenance stays in the decision, as you say, but most of it becomes a comparison the cleanup can run, and the reviewer sees only what has no other support.
On the reference, the narrowest answer is that it does not need to outlive the audit. Embed the chunk at withdrawal, run the comparison, delete the reference together with the old vector, and keep in the tombstone only that the audit ran, when, and at which threshold. Agreed on the 92 percent: it is their 32-token GTR-base setting, and the point it supports is only that a vector should be handled as derived data, not that every vector gives its text back.
The tombstone part is what makes this checkable. Keeping the IDs and edges after emptying the bodies means the cleanup can be audited instead of just trusted.
Thanks—that was the reason for keeping them. Two concrete uses in GhostCache:
Verification can inspect the recorded trail after cleanup. It walks the withdrawn note’s descendants again and checks that their bodies are empty and their titles contain no marker. The deleted source row also remains available to identify active artifacts that still depend on a withdrawn root. Keeping those tombstones preserves the history these checks inspect.
Cleanup is repeatable. Running it again finds the same descendants already cleared, returns no changed artifact IDs and doesn’t trigger another rebuild. It still records another metadata event, so the artifact state is unchanged rather than the entire operation doing nothing.
There’s also a separate marker scan across every managed title and body, including inactive rows. That check doesn’t depend on whether the graph links a row to the withdrawn note.
The tradeoff is that IDs, source references and links remain by design. Here they describe a fictional corpus and use synthetic identifiers. Titles are replaced because they can carry the secret too. In a real application, I’d review the retained identifiers and relationship metadata for sensitive information as carefully as the text.
The tombstones make the recorded cleanup inspectable; the independent content checks provide another way to test its result.
Repeatable cleanup with no changed IDs is a strong check.