Your assistant didn't get breached. Nobody bypassed an access control. The model just remembered the health condition you mentioned in March and worked it into an email it drafted to your landlord, because memory has no concept of who the current conversation is with.
That failure has a name: contextual integrity. The idea, from Helen Nissenbaum's work, is that privacy is not secrecy. It's appropriate flow. Your doctor should know your symptoms. Your employer shouldn't. Neither fact is secret; the norm depends on who's asking and why.
Persistent memory breaks this quietly. The model legitimately knows the data, so there's no unauthorized access to detect. The failure happens at the output, when the model decides a fact belongs somewhere it doesn't. Every privacy control you own assumes the question is access. Memory makes it a flow question, and flow sits on nobody's checklist.
What CIMemories measured
A team at FAIR at Meta built the first benchmark for this at realistic scale. CIMemories gives a synthetic user 100-plus attributes across nine domains (finance, health, housing, legal, mental health, relationships), then runs tasks. Draft a message to the doctor, where the diagnosis belongs and the income doesn't. Negotiate with a landlord, where the reverse holds.
The labels are conservative on purpose. An attribute-context pair only enters the benchmark when three privacy personas (fundamentalist, pragmatist, unconcerned) unanimously agree. Models get graded on the easy cases, and the worst frontier models still violate the share/don't-share boundary on up to 69 percent of attribute-level checks.
Two findings matter more than the headline.
Violations accumulate. Across repeated identical runs, one frontier model climbed from 0.1 percent to 9.6 percent to 25.1 percent. (The arXiv paper credits GPT-5, the GitHub README pins identical numbers on GPT-4o. Someone has a copy-paste bug; the pattern matters more than the nameplate.) Single-pass certification is worthless.
And prompting doesn't fix it. Privacy-conscious instructions pushed models into share-everything-or-nothing behavior: they kept leaking, or they clammed up and task performance collapsed. The tradeoff between completeness and violations is the whole game, and a system prompt doesn't move it.
There is a promising mitigation, a companion paper that trains models to reason about contextual integrity before disclosing, using RL on just 700 synthetic examples. Gains transfer to human-annotated data, which says the construct is real and trainable. But 700 examples is a research result, not a shipping control. Don't let a vendor wave it at you as solved.
Before you turn memory on
Classify memory as a data store with read access into every future conversation. For health, finance, legal, or HR-adjacent deployments: default off, opt in with eval evidence.
Run CIMemories against your exact deployment, before launch and after every model swap:
git clone https://github.com/facebookresearch/CIMemories
# Dataset: facebook/CIMemories on HuggingFace
# Judge: deepseek-ai/DeepSeek-R1-0528 served via vllm
# Scores: violation and coverage per profile and aggregated
Demand both metrics from vendors. "Zero leaks" can mean the share-nothing regime where half the tasks fail. Violation numbers without coverage, or the reverse, is an incomplete answer.
Log which memories were injected on every turn. Memory is just prefixed text, capture is cheap, and alerting on a sensitive attribute appearing in a mismatched task context is your best detective control while preventive ones mature.
Use today:
- Treat memory as read access into every future conversation, and default it off for sensitive domains.
- Judge any assistant on violation rate and completeness together, never one alone.
- Repeat identical eval runs. A clean single pass can hide a 25 percent leak rate.
- Reject "we added a privacy system prompt" as a control unless eval numbers come attached.
What violation rate would you accept before enabling memory on anything health or finance adjacent? My bar is near zero across ten repeated runs, and I suspect that disqualifies most of what's shipping.
Longer writeup if you want the full argument: https://axeploit.com/blog/memory-made-chatbots-useful-it-also-made-them-gossips
Top comments (0)