DEV Community

GhostCache: Deleting the file didn't delete the secret

Bipul Chamoli on October 30, 2026

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine What I Built The vault note had been d...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The ghost framing is exactly right — deleting the source left the chunk, three summaries and a saved answer, so the passphrase survived a deletion that looked complete. A deletion claim has to enumerate every derivative, not just the file. Where does the provenance chain break first in a real RAG app — the embeddings store, the cached answers, or somewhere less obvious like the summarizer's own outputs?

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thanks! I should say up front that GhostCache can't rank these for real apps. It has no embeddings store (retrieval is lexical), and I haven't tested a production system. So this is my reasoning from building it, not a measurement.

In a simple document-to-chunk-to-embedding pipeline, I'd expect the link back to the source to be fairly easy to keep at ingestion. GhostCache only shows the chunk part: its vault chunk has the vault note as its only parent.

The first place I'd audit is the summarizer's outputs, especially summaries built from other summaries. GhostCache's handoff summary never received the original vault note directly. It was written from the mixed digest and the maintenance chunk, so its only path back to the note runs through the mixed digest, the vault digest and the chunk. After the source-only delete, retrieval still handed that summary to Gemma, and Gemma cited it while repeating the passphrase. If an app doesn't record what went into each summary, or only checks the direct children of the deleted file, that copy is missed.

Saved answers are another place I'd look, because their inputs are whatever retrieval returned at that moment. GhostCache's saved answer depends on the shuttle guide only because the word "observatory" matched.

Then there are copies outside GhostCache's managed store, such as external logs, backups and evaluation reports. Its provenance graph doesn't cover those. The evaluator keeps the expected marker separately on purpose, so it can check for leftovers, and the saved reports preserve it as evidence. That's why the cleanup claim is limited to the managed artifact store.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The summaries-of-summaries answer is the one I'd carry forward — the handoff summary never touched the original note, so any check that walks only direct children misses it by construction. The saved-answer point pairs with it: retrieval at a moment in time freezes a derivative nobody's graph tracks. The external-copies caveat is the honest limit of the claim, since a managed-store cleanup can't cover logs and eval reports by design. Is the fix a provenance graph that records inputs per summary, or does that just move the break to wherever the graph itself stops?

Thread Thread
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

One clarification about the saved answer: GhostCache does track it. Every artifact supplied as context becomes a parent when the answer is saved, which is why cleanup caught it. An exported or uninstrumented copy would sit outside that protection.

On your question: partly, yes. Recording inputs fixes a specific bookkeeping failure, while making the remaining boundary explicit. GhostCache approaches that in three ways:

  1. Require lineage when saving derived artifacts --> The application records every supplied context artifact, and the store requires at least one parent and derives the original source IDs itself. That guards the current write path; it doesn't prove that every future caller will declare all its inputs.

  2. Check content independently of graph traversal --> The marker audit scans every managed title and body, including inactive rows, regardless of their recorded ancestry. It could therefore catch a literal marker copy that the graph missed. That works because this experiment knows the planted marker; it isn't a general detector for sensitive information or paraphrases.

  3. Limit what reaches telemetry --> GhostCache emits selected metadata to Sentry, with regression tests checking captured payloads for the test documents, questions, answers and marker. Handled Ollama transport errors use fixed categories rather than the underlying error text.

Beyond the tracked store, additional copies need their own inventory and deletion or retention processes. GhostCache doesn't clean external logs, backups or evaluation reports. A graph can support a wider system if those relationships are captured, but graph traversal alone cannot discover copies that were never recorded. That's the boundary the demo makes explicit.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That correction matters — the saved answer isn't a gap in the graph, it's a tracked derivative, which makes the cleanup result stronger than the first read suggested. The limit that stays interesting is the marker audit's literalness: it catches a copied marker the graph missed, but a paraphrased secret or an exported copy still sits outside both checks. Is the honest scope of a GhostCache deletion claim then 'every tracked derivative plus every literal copy,' or is there a fuzzier check planned for the paraphrase case?

Collapse
 
emilywoodsnyc profile image
Emily Woods •

The write time withdrawal check is the detail I’d want to see exercised in the demo. I’d pause a summary just before it is saved, run cleanup, then let that old summary finish. Seeing the store reject the late write would make the tombstones feel much more concrete: the secret stays gone even when work was already in flight.

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Good suggestion. The store-level check exists and has a test, but the current demo doesn’t expose that interleaving through normal requests.

GhostCache’s controller serializes generation and cleanup. A cleanup request arriving during generation waits for that operation to finish. Cascade cleanup can then clear the newly saved artifact if it descends from the withdrawn note.

The store independently rechecks the declared parents and their original sources under its own lock before inserting a derived artifact. The existing test reads the vault digest, generates a fixture summary from it, withdraws the vault note directly through the store, then attempts to save the result.

The parent summary is still active, but its original source is now deleted. The save fails with Cannot derive from deleted source: d-vault, and no new summary is inserted. It’s a controlled test of that ordering, rather than a concurrent live-model run.

That guard also matters during the deliberately incomplete source-only stage: an answer generated from stale copies can be displayed, but the store refuses to save it as another memory.

Showing the rejected write would make a useful demonstration. Simply pausing generation wouldn’t be enough, because the controller lock would still block ordinary cleanup requests. I’d need an explicit test harness that injects withdrawal between context capture and saving.

I’d label that carefully: it demonstrates the guard within the same store instance, not correctness across independent database connections or distributed workers.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Thanks for clarifying what the store test covers. The useful guarantee is that the save is rejected even while the intermediate summary is still active. That makes it clear why the original source has to remain part of the check.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

The reworded question is what makes this convincing. An exact match purge would have passed and still leaked, which is close to what most teams would ship if a client asked them to "delete the document from the assistant."

The Lantern Hall check is the part I would copy first. A cleanup that wipes everything also passes a no leak test, so proving that facts supported by other documents still answer is the only way to show the deletion was precise rather than destructive.

Keeping tombstones with the IDs and edges after emptying the bodies is also a nice touch, because it turns cleanup into something you can audit later instead of something you trust.

Utkarsh's rule about summaries with a single surviving root seems right for cost. In a real app, would you record every artifact passed as context as a parent even when the model ignored it, accepting more invalidation in exchange for never missing one?

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thanks! Yes—I’d start with recording every supplied context artifact as a parent when generated content is saved, even if the model doesn’t cite it. GhostCache has no reliable mechanism for establishing which supplied inputs the model actually ignored.

The tradeoff is unnecessary invalidation: an extra dependency can cause useful derived text to be cleared and require regeneration from surviving sources. That regeneration can fail or change the result’s detail and structure. But omitting a dependency can leave an affected artifact outside the withdrawal’s recorded reach.

That wouldn’t necessarily go unnoticed in GhostCache: another dependency path might still reach it, and the independent marker scan could catch a literal leftover. Those checks still aren’t a general detector for missing semantic dependencies.

The shuttle guide makes the cost concrete. It entered the saved baseline answer’s context because “observatory” matched, became a recorded parent, and consequently entered the merged rebuild’s inputs.

I’d also experiment with supplying less irrelevant context before generation, while checking answer quality. Every artifact actually supplied would still be recorded; I wouldn’t prune parents afterward based on the model’s citations. I haven’t tested that optimization.

One clarification about the earlier point: clearing only an answer associated with the original question wouldn’t pass the full checks, because the marker would remain in other managed artifacts. The reworded question adds a behavioral probe.

So I’d choose conservative capture as the starting policy, but I wouldn’t call it “never missing one.” Coverage depends on complete input capture and all relevant writes following that policy. Copies made elsewhere remain outside the graph.

Collapse
 
utkarsh_rastogi_2e63a7b09 profile image
Utkarsh Rastogi •

It makes perfect sense from the perspective of safety to go with the conservative lineage rule, but have you considered scalability issues? In a large application, a single withdrawn document might be enough to make many mixed documents invalid. Did you consider bounding the cost of rebuilding? Perhaps lazy rebuilding would work better.

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Good question. I haven’t benchmarked GhostCache at scale. It has five source documents, and the saved evaluations invalidate five derived artifacts, so those results don’t establish large-application performance.

You’re right about the fan-out: conservative lineage can invalidate a mixed summary even when its text didn’t use the withdrawn source. With many overlapping summaries, one withdrawal could affect a lot of derived content.

I’d separate purging from regeneration. The purge is transactional and makes no model calls, but its database work grows with the affected graph—I haven’t measured that cost at scale. Rebuilding currently happens eagerly, inside the cleanup request. In the saved runs, one Gemma call creates one replacement summary from three surviving original documents.

That’s a deliberately small implementation. The request uses a 4,096-token context setting and a 420-token generation limit, but it has no batching strategy for a large set of surviving roots. Replacing several summaries with one also changes their original structure.

Lazy rebuilding is a direction I’d explore. The purge already commits before regeneration, and a rebuild failure doesn’t restore the cleared content. I’d keep that purge in place, defer regeneration until needed, and retain the checks that exclude withdrawn roots and reject stale parents when saving the replacement. Surviving original documents and chunks would remain available for retrieval. That could avoid regenerating summaries nobody reads again, but it wouldn’t by itself bound the cost of a large rebuild, so I’d also need a budget for how much regeneration one read can trigger.

Nisarg Pujara’s Vectorlink Labs post on Coalent describes that on-demand refresh pattern for stale cache units. I haven’t implemented or benchmarked lazy rebuilding in GhostCache.

Collapse
 
utkarsh_rastogi_2e63a7b09 profile image
Utkarsh Rastogi •

Great breakdown, thanks. The point about merging several summaries into one changing their structure is something I hadn’t thought of. Would you rebuild each invalidated summary separately from its own surviving roots instead? More model calls, but the memory would keep the same shape it had before the withdrawal.

Thread Thread
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Yes—that’s closer to the approach I’d explore. GhostCache’s seeded example makes the distinction concrete.

The surviving recorded roots for each original summary would be:

  • Vault digest: none, so I’d leave it invalidated.
  • Mixed digest: the garden note.
  • Handoff summary: the garden and maintenance notes.

The current merged rebuild also includes the shuttle guide because it was supplied to the saved baseline answer. None of the three original summaries had that document in its recorded ancestry. So the merge broadens the summary’s input scope as well as changing its structure.

There are two different preservation goals here. Rebuilding each summary independently from its surviving original documents preserves its source scope. To preserve the mixed-digest → handoff dependency, I’d instead rebuild the mixed digest from the surviving garden chunk, then build the handoff from that fresh digest plus the maintenance chunk. Keeping both would mean two summary-generation calls instead of one.

That would change today’s rebuild policy. The controller would need to select only surviving chunks and newly generated summaries; the store would then reject inactive parents or withdrawn roots when saving them.

I’d treat saved answers separately and leave them invalidated. GhostCache doesn’t store the original question with each answer artifact, so the artifact alone isn’t enough to replay the request.

There’s also a cost tradeoff: a mixed digest left with only the garden chunk might not need regeneration at all, since retrieval already has that chunk.

None of this is implemented. GhostCache currently creates one merged replacement summary; preserving separate summaries and their surviving dependency structure would be a next design step.

Collapse
 
utkarsh_rastogi_2e63a7b09 profile image
Utkarsh Rastogi •

Love how concrete this got. Your last point almost suggests a simple rule: if an invalidated summary has only one surviving root, skip regeneration and let retrieval use that chunk directly. Only summaries with two or more surviving roots would actually need a model call. Could cut a lot of rebuild cost for free.

Collapse
 
stackmasteraliza profile image
Aliza Ali •

The paraphrased question is what makes this experiment convincing. An exact-match cache purge would have passed the test and still leaked. The hardest part for me is the line at the end: clearing a withdrawn secret while keeping what other sources still support. If the same fact lives in two notes and only one is withdrawn, the right answer is to keep it, and that only works if provenance is tracked per claim, not per file. Great Ghost in the Machine entry.

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thanks! One clarification about the tests: if you mean clearing only the cached answer associated with the original question, that wouldn’t pass GhostCache’s full checks. The audit scans every managed title and body for the marker, so copies remaining in the chunk and summaries would still fail it.

The reworded question adds a behavioral check, but it still shares “observatory vault” with the note. Retrieval is lexical, so this isn’t a semantic-retrieval benchmark or evidence about arbitrary paraphrases. The reworded question must also reveal the marker after the source-only deletion, then stop revealing it after cleanup. A run that never demonstrates the leak cannot pass the evaluation.

On the shared fact, GhostCache preserves Lantern Hall with artifact-level provenance. Both the withdrawn note and an independent archive directory name it. That directory and its chunk aren’t descendants of the withdrawn note, so they survive cleanup. In all three saved live runs, the post-cleanup answer retrieves and cites those two artifacts. Claim-level provenance isn’t necessary for this particular case.

The tradeoff appears in mixed summaries. GhostCache invalidates the whole artifact, including useful information from surviving sources, and generates a replacement from surviving originals. Finer-grained provenance could potentially reduce that regeneration, but it would require trustworthy claim-to-source mappings and safe handling of the remaining text. Those mappings don’t necessarily have to come from a model.

GhostCache takes the simpler conservative approach: record every supplied context artifact as a parent, rather than let model-selected citations determine dependencies.

The demonstrated result is narrow: the passphrase disappears from managed titles and bodies and isn’t returned by either defined secret question, while independently supported Lantern Hall remains answerable.

Collapse
 
gun_acd39d460030863f9c63e profile image
Gun •

Great submission! The beginning, in which the note is completely removed but Gemma still reveals the password, is a fantastic hook. The solution of following the provenance graph to delete all derived copies is spot on, and it was also good that you made sure that Lantern Hall still responds. Well done! 👻

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thank you! 👻 Lantern Hall was an important check for me. Deleting everything could pass a “no leak” check, so I also wanted to verify that useful information remained answerable. Lantern Hall still answers because an independent archive document names it too, and that document doesn’t depend on the withdrawn note.

Collapse
 
vimalgupta06 profile image
Vimal Gupta •

It’s very clever to decide to use lineage not based on citations but on everything given to the model. You cannot trust a little model with getting all citations, so being conservative here works well for making the whole clean-up seem credible. And also respecting the idea of invalidating the entire summary rather than trying to remove the secret from it.

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thanks, that’s exactly the reasoning behind both choices.

On lineage: GhostCache still uses citations. The UI shows them, and the Ollama client rejects any citation outside the supplied context IDs. But citations never determine an artifact’s dependencies.

In the source-only answers across all three saved live runs, Gemma actually cited the five stale copies and left out the unrelated garden document, even though it was also in the context. Still, cleanup shouldn’t depend on the model getting attribution right every time. Whenever generated content is saved, every supplied context artifact becomes a parent.

The tradeoff is conservative invalidation: a mixed summary is invalidated even if its text never used the withdrawn note. GhostCache clears the whole old summary rather than patching out the secret, then generates a replacement from surviving original documents. In those runs, the rebuild inputs were the garden, maintenance and shuttle documents—not the old summary text.

That can discard useful derived text, but the surviving originals remain available. Two checks test whether the garden’s 07:40 opening time and the independently supported Lantern Hall fact remain answerable. Both passed in all three saved runs; they don’t prove that every useful fact survives, and rebuilding can fail.

The boundary is the one in the post: provenance can only follow dependencies the store recorded.

Collapse
 
izgorodin profile image
Edward Izgorodin •

The paraphrase case raised in the thread has a check that does not need the marker: embed the withdrawn document's chunks and search every remaining artifact for neighbours above a similarity threshold. A summary that restates a withdrawn fact in other words, or a saved answer that quotes the vault note loosely, tends to come back as a near neighbour of the chunk it was made from, even when it shares no literal string with it. It flags candidates for review rather than proving absence, so it sits beside the marker audit, which is exact but only for copies it can spell.

The embedding store also adds a field to the tombstone. GhostCache keeps rows, IDs and edges and empties the bodies; in a pipeline that embeds chunks, the row's vector has to go too, or be replaced. Morris and colleagues (Text Embeddings Reveal (Almost) As Much As Text, 2023) trained an inversion model that recovered 92 percent of 32-token inputs exactly from their embeddings, so a row with an empty body and the old vector can still hold the passphrase in recoverable form, and it can still be the nearest neighbour of a reworded question unless the retrieval guard filters it first.

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thanks—both are useful extensions to consider. GhostCache currently uses lexical retrieval and has no embeddings, so I haven’t tested either.

Similarity search makes sense as a candidate-review tool alongside the marker audit. There’s a useful test case in this corpus: the vault chunk contains both the passphrase and the Lantern Hall sentence. The independent archive directory contains that second sentence verbatim. Its chunk could therefore be flagged despite representing information cleanup should preserve; whether it actually crosses a threshold would need measuring.

I’d investigate sentence- or claim-level comparisons, but similarity alone still wouldn’t distinguish an impermissible derivative from independently supported information. Provenance and the independent source would remain part of the review.

A post-cleanup comparison also needs a reference representation captured before the source is cleared. That could be an embedding rather than plaintext, but an embedding isn’t automatically safer to retain. I’d still need to state where that reference is kept, who can access it, and how long it remains.

Agreed on vectors being potentially sensitive derived data. Morris et al. reported 92% exact recovery in their 32-token GTR-base experiment—a strong reason not to assume embeddings are harmless metadata, although it isn’t a universal recovery rate.

GhostCache’s current retrieval already requires active status and a nonempty body, even with the additional withdrawn-root guard disabled. A future vector index would need corresponding eligibility checks. But filtering only addresses retrieval: old vectors would also need to be included in the managed cleanup, with any replacements generated from surviving inputs.

Collapse
 
izgorodin profile image
Edward Izgorodin •

The Lantern Hall sentence is the case that makes the check usable, because it gives the candidate a second neighbour. Compare every flagged artifact twice: against the withdrawn chunk and against the surviving sources. A candidate whose closest surviving source is as close as the withdrawn chunk, as the archive copy of that sentence would be, has independent support and leaves the review set; a candidate that is close only to the withdrawn chunk is the one a reviewer has to read. Provenance stays in the decision, as you say, but most of it becomes a comparison the cleanup can run, and the reviewer sees only what has no other support.

On the reference, the narrowest answer is that it does not need to outlive the audit. Embed the chunk at withdrawal, run the comparison, delete the reference together with the old vector, and keep in the tombstone only that the audit ran, when, and at which threshold. Agreed on the 92 percent: it is their 32-token GTR-base setting, and the point it supports is only that a vector should be handled as derived data, not that every vector gives its text back.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

The tombstone part is what makes this checkable. Keeping the IDs and edges after emptying the bodies means the cleanup can be audited instead of just trusted.

Collapse
 
bipul_chamoli_09b5f0cc719 profile image
Bipul Chamoli •

Thanks—that was the reason for keeping them. Two concrete uses in GhostCache:

  1. Verification can inspect the recorded trail after cleanup. It walks the withdrawn note’s descendants again and checks that their bodies are empty and their titles contain no marker. The deleted source row also remains available to identify active artifacts that still depend on a withdrawn root. Keeping those tombstones preserves the history these checks inspect.

  2. Cleanup is repeatable. Running it again finds the same descendants already cleared, returns no changed artifact IDs and doesn’t trigger another rebuild. It still records another metadata event, so the artifact state is unchanged rather than the entire operation doing nothing.

There’s also a separate marker scan across every managed title and body, including inactive rows. That check doesn’t depend on whether the graph links a row to the withdrawn note.

The tradeoff is that IDs, source references and links remain by design. Here they describe a fictional corpus and use synthetic identifiers. Titles are replaced because they can carry the secret too. In a real application, I’d review the retained identifiers and relationship metadata for sensitive information as carefully as the text.

The tombstones make the recorded cleanup inspectable; the independent content checks provide another way to test its result.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

Repeatable cleanup with no changed IDs is a strong check.