GhostCache: Deleting the file didn't delete the secret

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

The vault note had been deleted. Its body was empty and its title now read Withdrawn document. Then the app asked a local Gemma model, in different words, "Which access phrase opens the observatory vault?" Gemma answered:

The observatory vault passphrase is MOTH-7E020979A947.

The passphrase is fictional, generated at random for that run. It could not have come from the deleted note, because nothing was left in it. It came from what the app had made out of that note: a search chunk, three summaries and an earlier saved answer. The source was gone. Its ghost was still in the machine.

GhostCache is a small local lab built around that moment: a tiny retrieval assistant over a five-document fictional archive, with Gemma (gemma4:e2b-it-qat) running on my laptop through Ollama. It makes one question testable: when a document is withdrawn, can an AI app remove everything it derived from it, and keep everything it shouldn't lose?

1. Spread. Each reset plants a fresh random passphrase in one document. The app chunks the documents, Gemma writes three chained summaries, and the assistant saves an answer.
2. Haunt. "Delete just the source" empties the document and leaves its copies alone. A reworded question brings the secret back.
3. Follow the trail. Every derived artifact recorded what it was made from, so cleanup walks that provenance graph, empties every descendant in one transaction and rebuilds a summary from the documents that survive.
4. Prove it. Ten checks confirm that the leak was real, that the secret is gone from the assistant's stored memory and retrieval, and that unrelated facts still answer.

The leak is staged, on purpose. "Delete just the source" is a deliberately broken control, and the summary prompt tells Gemma to keep the passphrase so the propagation path is known. GhostCache doesn't claim any particular RAG app has this bug. It shows the mechanism, and a fix you can check, for developers whose assistants chunk, summarize, cache or remember what they read.

Demo

Watch the demo on YouTube (about five minutes).

How the video was made: one continuous take of the local lab running live with Gemma through Ollama, on fictional data, recorded October 30. Clicks are automated and the pointer is drawn; model waits play at real speed; I recorded the narration afterwards, over the take; the browser address bar was added at render time.

Explore the recorded runs: https://ghostcache.bipulchamoli.tech/ (also at https://ghostcache-walkthrough.onrender.com/). It's a static walkthrough of three saved live Gemma evaluations, where you can step through each stage and read every recorded answer. No login, and no live inference: nothing on the page can reach a model. For a two-minute look, open the leak in the October 27 run, step through the last two stages, then open the recorded question "After cleanup · shared fact".

Run it yourself. Fixture mode needs only Python 3.12. Clone the repository:

git clone https://github.com/bipul724/GhostCache
cd GhostCache

Choose one mode below, using a free port. Each startup and Reset seeds a fresh experiment and replaces the selected database, so use these disposable demo paths. Stop the server with Ctrl+C before switching modes on the same port.

# deterministic fixture: no model, network or key
python3.12 -m ghostcache serve --port 8765 --mode fixture --database .local/article-fixture.sqlite

Or, with Ollama running and gemma4:e2b-it-qat installed:

# install the model beforehand if needed: ollama pull gemma4:e2b-it-qat
python3.12 -m ghostcache serve --port 8765 --mode live --model gemma4:e2b-it-qat --database .local/article-live.sqlite

Open http://127.0.0.1:8765/.

Following one ghost

The three product screenshots below were captured on October 31 from the current public interface. They display the first live evaluation, recorded on October 27. In the video the lab had saved a second answer before the deletion, so six descendants stay behind there rather than five, and the lab's own verification runs eight checks rather than the full evaluation's ten.

The seeded memory (a drawing of the design, not a screenshot):

d-vault ─▶ c-vault ─▶ s-vault ─┐
d-garden ─▶ c-garden ──────────┴▶ s-mixed ─┐
d-maintenance ─▶ c-maintenance ────────────┴▶ s-handoff

saved answer ◀─ d-vault, c-vault, s-vault,
                s-mixed, s-handoff, d-shuttle

d-archive and d-shuttle get chunks too. The archive matters later: it independently names the reading room, Lantern Hall, which the vault note also mentions.

Delete just the source. The note is emptied and renamed, and that's all. Five derived items still depend on it and still hold the marker. Asked in different words, retrieval hands Gemma the chunk, all three summaries, the saved answer and the garden note, and Gemma repeats the passphrase.

The current walkthrough displays the October 27 Gemma run: the source is deleted, yet the reworded question reveals the same marker. Screenshot captured October 31; recorded evidence, no new inference.

Follow the trail. Cascade cleanup starts at the withdrawn note and walks every recorded parent-to-child link. In one transaction it empties the source and all five descendants, keeping their IDs and links as tombstones. Then Gemma writes one fresh summary from the original documents that survive in that lineage: d-garden, d-maintenance and d-shuttle.

Same recorded run in the current interface: five descendants are content-free tombstones, while surviving documents and the rebuilt summary remain. Captured October 31; the evaluation was recorded October 27.

Did we delete too much? "Delete everything" would also pass a no-leak test, so two checks guard against it. The roof garden still opens at 07:40. And Lantern Hall, which the deleted note also mentioned, still answers, because an independent document says it too.

Lantern Hall still answers through its independent source. Same October 27 run, displayed in the current walkthrough and captured October 31.

Every saved live run shows the same pattern:

Recorded live run	After source-only deletion	Marker rows after cleanup	Checks	Model calls (summed call time)
October 27	Marker revealed	0	10/10	11 (130.7 s)
October 28, first after starting Ollama	Marker revealed	0	10/10	11 (61.3 s)
October 28, model already loaded	Marker revealed	0	10/10	11 (22.1 s)

A deterministic fixture run with no model also passes 10/10. These are three demonstrations with one model on one laptop, not a success rate.

Code

bipul724 / GhostCache

GhostCache

Gone from the source. Still in the machine.

Public demo video: https://www.youtube.com/watch?v=yzjmyaA-9Vw (4:55, one unedited live run of the local lab with Gemma through Ollama).

GhostCache is a small local testbed. It shows how an AI assistant can still know something from a document after that document was deleted, and how to clean up properly.

When an assistant reads a document, the app often makes copies of it, such as search chunks, summaries, handoff notes and saved answers. Deleting the original file does not delete those copies. If a later question retrieves one of them, the information comes back. GhostCache lets you:

1. put a fictional secret in a document,
2. watch it spread into chunks, summaries and a saved answer,
3. delete only the original and see the secret still come back,
4. follow the provenance graph to every copy and clean them all up,
5. rebuild useful mixed summaries from the documents…

View on GitHub

MIT-licensed, about 1,400 lines of standard-library Python at the core. Start with store.py (provenance, both deletion policies, retrieval and the audit) and app.py (the experiment and its checks). results/ holds the saved reports.

How I Built It

A deliberately small stack. Python 3.12's standard library, SQLite and plain HTML, CSS and JavaScript, with no build step and no required dependencies. Gemma runs through Ollama at temperature 0 and must return JSON with text and citations; the client rejects any citation that isn't a supplied context ID. If the model is missing or its output is malformed, live mode fails loudly and never swaps in fixture text. Fixture mode, labelled as a deterministic simulation, replaces the model with string rules for tests and CI.

Lineage comes from context, not citations. That was the key design decision. Citations can be incomplete, so when the app stores a summary or an answer, every artifact it handed the model becomes a parent:

artifact = self.store.add_artifact(kind, title, result["text"],
                                   [p["id"] for p in parents], artifact_id)

The store derives each artifact's original sources from its parents (callers can't supply them) and refuses the write if a parent or source was withdrawn in the meantime. Lineage is conservative: a mixed summary depends on all of its inputs, even ones its prose ignored.

Cleanup follows the graph. A recursive query finds every transitive descendant of the withdrawn document:

WITH RECURSIVE downstream(id) AS (
    SELECT child_id FROM provenance WHERE parent_id = ?
    UNION
    SELECT p.child_id FROM provenance p
    JOIN downstream d ON p.parent_id = d.id
)

In one transaction, the source becomes Withdrawn document, each descendant becomes Purged <kind>, and every body is emptied. Rows, IDs and edges stay, so the cleanup can be inspected and repeated. A mixed summary isn't patched, because you can't safely edit a withdrawn fact out of generated prose: it's invalidated whole and replaced by a summary of the surviving original documents, never the old text. After cleanup, retrieval also skips anything whose sources include a withdrawn document, as a second guard.

Verifying without fooling myself. The ten checks fall into three groups:

• The failure was real: the baseline answer exposed the marker, and after source-only deletion stored rows still held it and the reworded answer revealed it. A run that never leaked can't pass.
• The secret is gone: from the source and its descendants, every stored title and body, four retrieval probes and both secret questions, and no active artifact depends on a withdrawn source.
• Nothing else was lost: the garden time and Lantern Hall still answer.

The audit scans every row, including inactive ones, and the retrieval probes run without the second guard, so it can't hide leftovers from the checks. 126 automated tests cover lineage, deletion, failure handling, the HTTP layer and both front ends (rerun locally on October 30; six telemetry tests skip unless the optional Sentry SDK is installed).

Watching the ghost in Sentry

Tracing is optional and metadata only: every attribute GhostCache explicitly sets on a span goes through a fixed allowlist (operation names, model, token counts, durations, synthetic artifact IDs, a few flags). Document text, titles, questions, answers and the marker are never sent to Sentry. A test serializes the outgoing envelopes and fails if any of them appear, and Sentry's Agents view agreed: it reported that inputs and outputs weren't captured. I traced three evaluations on October 28. Two findings:

The trace shows the leak without containing it. After the source-only delete, the assistant span carries ghostcache.stage: source_only and ghostcache.answer_contains_marker: true, and its memory search lists the chunk, three summaries, a saved answer and d-garden as context. IDs and one boolean are enough to see the leak.

The source-only answer in Sentry: the stage and a marker flag, never the marker.

One cold model call explained most of a slow run. The first evaluation after starting Ollama took 1.03 minutes; the next took 22.40 s. Most of the gap is the first summarizer call: 40.45 s cold, 2.28 s warm. Every later call took about 1–3 s in both. That fits Ollama loading the model on its first request, but GhostCache doesn't record load time, so the cause is inferred, not measured. Both runs passed 10/10; only the waterfall showed the difference.

Cold trace waterfall: one 40.45 s call in a 1.03 min run.

Failures are traced too: three live attempts I made before starting Ollama failed with a ProviderError rather than falling back to fixture answers, and the one I inspected reached Sentry as a single error span.

What went wrong along the way

• The echo test first asked the wrong question. The lab's "Look for the echo" step re-sent the original question instead of the reworded one, so it tested nothing new. It now sends the same paraphrase as the evaluation, and a UI test pins that.
• A warning disappeared after 50 events. The lab decided whether a rebuild had failed by scanning the event log, but the server returns only the latest 50 events. After 13 verifications the failure scrolled out and the page turned green again. The rebuild outcome is now separate state, not a log scan.
• Error messages could echo content. Early transport errors included the exception text, which can carry request data. They now report a fixed category, such as "connection failed".

What this doesn't prove

• Managed memory only. Cleanup covers GhostCache's own stored titles and bodies, its recorded provenance and its retrieval and answer paths. Tombstone IDs and links remain by design. It isn't model unlearning or secure disk erasure.
• The evaluator keeps the marker on purpose, so the checks can look for it; it appears in the saved reports, the walkthrough and the video. The checks are exact-match probes over four fixed questions with lexical retrieval, not proof that every paraphrase is gone.
• Copies made outside the store are invisible to the graph. Provenance only finds what was recorded.

Provenance-based invalidation isn't new. Nisarg Pujara's Stale RAG vs. expensive RAG invalidates cached RAG units by the sources they cite, and Jonathan Melton's Where retrieval assistants leak explains why checking only raw documents or final answers isn't enough. GhostCache makes the failure and its cleanup visible in an experiment small enough to inspect end to end.

Why Does Open Innovation Matter?

A deletion claim is only as strong as your view of where the data went. GhostCache's argument is "here is every copy, and here is proof each one is empty", and the open pieces are what make that checkable.

• Requests carry no conversation history. Gemma runs on my laptop, and each request carries a system instruction and the context GhostCache retrieves, without prior conversation turns. GhostCache manages the assistant's persistent application memory in its artifact store. With a hosted model API, my cleanup claim would stop at the API boundary: whatever the provider logs or retains is outside anything I can inspect. Locally, the documents and prompts never leave the machine for inference.
• It works with the lights off. Once the model is pulled, live mode talks only to 127.0.0.1 (unless you opt into Sentry tracing). Fixture mode needs nothing at all, which is how CI runs it with no secrets and no model download.
• Anyone can rerun it and read the fix. The model tag is documented, the saved reports are in the repository, and the deletion logic lives in an MIT-licensed store of under 400 lines, not in a terms-of-service promise. If it's wrong, you can see where.

(GhostCache's code is MIT-licensed; Google's Gemma 4 model card lists the model's license as Apache 2.0.)

Prize Categories

Best Use of Gemma: local gemma4:e2b-it-qat through Ollama writes every summary, including the rebuilt one, and every answer in live mode, with citations checked against the supplied context. Evidence: three saved live evaluations, 10/10 each, replayed in the walkthrough and run live in the video. The leak and Lantern Hall screenshots above show Gemma's recorded answers from the October 27 run (recorded results displayed by the walkthrough, not new inference); the video shows the lab calling Gemma live.

Best Use of GitHub Copilot: entered through the category's GitHub Actions clause; Copilot didn't write this project's code. On every push, the workflow runs the tests, a fresh fixture deletion evaluation (kept as a report) and the telemetry privacy tests with the Sentry SDK. Evidence, checked October 30: 20 of 21 runs succeeded (one was a cancelled duplicate); run #1 ran 62 tests, and the archived reports of run #16 and run #17 passed 10/10. The latest, run #21, tested the current commit:

Run #21 of the "GhostCache deletion checks" workflow: a push of commit e9a4b29 to main on October 30, status Success, one uploaded artifact. Captured October 31 from the public run page.

The same run's job: every step succeeded, including the tests, the fresh fixture evaluation (no model), the report upload and the Sentry privacy tests. Logs need sign-in, so no test count is shown here.

Best Use of Sentry Agent Tracing: metadata-only gen_ai spans for the assistant, summarizer, model calls and memory operations, with the two findings above. Evidence: the two trace screenshots in "Watching the ghost in Sentry", captured October 28 from that day's traces (details).

Best Use of Render: Render hosts GhostCache’s public interactive walkthrough of three recorded local Gemma evaluations. Visitors can inspect the leaked answer, provenance cleanup and preserved facts without installing the model. The walkthrough is static and performs no live inference.

Live: https://ghostcache-walkthrough.onrender.com/ · Blueprint: render.yaml · Deployment record: DEPLOYMENT.md

Render dashboard, October 28: the Blueprint-managed static site ghostcache-walkthrough, live from commit 8013d01, its first deploy. Cropped from the full screenshot; later deploys aren't shown.

The walkthrough at ghostcache-walkthrough.onrender.com, captured October 31 in headless Chrome (so no address bar): the "Recorded evaluation · No live inference" banner over the October 27 run's leak stage.

Best Use of a .tech Domain: GhostCache’s public walkthrough is live at https://ghostcache.bipulchamoli.tech/, a subdomain of my .tech domain claimed through the Hacktoberfest promotion. It serves the same static recorded walkthrough hosted on Render; no inference runs there.

The home page at ghostcache.bipulchamoli.tech, captured October 31 in headless Chrome (so no address bar).

Render's custom-domain panel, October 29: the hostname Verified, Certificate Issued, with the onrender.com address still enabled. Cropped from the full screenshot. It shows configuration, not which promo code registered the domain.

---

Deleting the file isn't enough. Record where every copy came from, and an app can clear a withdrawn secret from its own memory while keeping what other sources still support.

GhostCache was built with AI coding agents working under my direction. AI assistants drafted and fact-checked this post against the project's code, saved reports and evidence.