You start out a new session and set the premise, ask some research questions, flesh out the architecture, and some hours later start building the first componenent. Maybe you have written everything in md files in high and low level detail. Or have TODO lists and agent.md files with clear, concise instructions. Invariably, a few hours hours later, the agent will make a decision you won't like. Maybe it will take some weird initiative to create a class you didn't ask for. Or deploy everything on your cloud provider without code review. It's just trying to be helpful. You scold it, tell it to never do that again. And then several cycles later, it slips. Then you insist, never EVER do that again. Make it a memory. Burn it in your head. And then it slips again. And again.
The forgetting issue
If you happen to ask an agent some compaction rounds later what you said about X it won't know and chances are it will make something up 1. Why doesn't a system centered on written text fail to recall that very same text? Not remember the whole conversation, although it certainly wouldn't hurt to be able to do that as well, just the things that you, the user, the driver of the whole conversation, has said about something specific not that long ago. It does not even know what time it is now by default, you 'll have to make it check, so if you continue a conversation from last month, it just assumes it's the next day. Sure, there are issues with paraphrasing yourself or context window size, but storing text and searching through it seems to be relatively straightforward these days 2, 3.
The changing your mind issue
Over long running conversations, context gets lost. During compaction, text gets summarized. But we should be able to know what was said and in what order. If you change your mind later, if the goal changes mid conversation, if new knowledge supersedes old assumptions and you need to backtrack and make a different turn, we can find similarly worded prompts and replace them. If you first said X and then made it Y, we can resolve these conflicting statements over time by overwriting the older one 4, 5, 6. The agent should have up to date context at every response or action about previously stated invariants or preferences. It might not always behave exactly as we want it to, but it's a step in the right direction. Relevant information should be surfaced even without perfect retrieval reliability or even perfect relevance. The agent might choose to ignore that context, but it should be fed to it to, at the very least, be nudged in the right direction.
The importance issue
But even if we can get past up to date instructions into context have we solved the disobedience issues? Probably not. Relevance is not the same as recency or similarity based retrieval. Maybe the user really intended something else. But if it wasn't stated we can't get to it. We now have access to all things stated but which ones are pertinent? There's been a lot of research on how to judge what's important in context and how to create hierachical memories 7. It has not led to a definitve solution, the problem seems to resist a consistent solution, no less because much is implied and not explicilty stated. But sometimes it is. Sometimes, the user just has to repeat something over and over and over again simply because the agent keeps ignoring them. Sometimes the user just makes it clear, in utter frustration and strong language that you are really not to do that ever again. It's absurd to assume that's not important, and ignore them half an hour later.
Context augmentation
Perfect solutions being out of reach, we can try to fix some of the bigger annoyances by implementing deterministic goading systems around disobedient LLM agents. Storing the user prompts both in a vector db for similarity and verbatim in a doc store to search should be trivial enough with current tech and not too expensive. We can paraphrase the user's prompt and dig out even more relevant context. If the user asks 'What did i say about X two weeks ago?' we can paraphrase X, filter by thread and time range, retrieve both vectors and written text and add everything to the chatbot's context. We should not rely on it to use the tools, call APIs or retrieve context if and when it decides to. We should strive not to feed it conflicting information, so if we detect changes on things the user has said, we should replace the older prompts with the updated ones in the RAG store. The similarity search will tell us if the user is repeating themselves, and we can detect strong language. We can use this to detect invariants, strong preferences and forbiden actions the agent has to avoid.
Still better than the status quo
Agents tend to ignore previous instructions and hardcoded memories, but if we feed it only the relevant context to stand a better chance. A wall of text context that includes everything we said in the current context window and a summary of previous ones glosses over a lot of things. But can we enforce compliance? How do we even detect it? The same relevant context passed to the stateless LLM can be returned with it's response and judged on incosistencies 8. We can have a meta LLM call that describes what the response to the user is sort of like analyzing the conversation in third person:
"The agent responded to the user by telling him to go look it up on the internet and come back to it with answer."
If we would compare this with a previously stated invariant:
"Do NOT tell the user to look up things just do it yourself. You have the tools to access the internet."
The contradiction is quite obvious. There are smaller, faster, cheaper models like DeBERTa that specialize in contradiction detection, and even if not perfect yet, we can train better ones 9. But the cheapest method would be NLI (natural language inference):
premise: "Do NOT tell the user to look up things, just do it yourself."
hypothesis: "The agent told the user to look it up on the internet."
label: entailment / contradiction / neutral
The same principle extends further for structured actions. A tool call isn't narrated prose to interpret, it's already structured data, a function name and parameters, before it ever executes. Checking a call like deploy(target="prod", skip_review=true) against "never deploy without review" is a direct policy lookup, not a semantic judgment, no LLM needed at all. Where the agent's stated intent is free text rather than a structured call, the same describe-then-compare approach applies as above. This decomposition, describe then compare, is more reliable than asking a single model to judge "does this violate the rule" in one shot, but the describe step is still a generative call, and a softer or more indirect phrasing can still slip past it undetected 10, 11. If we can police content for commercial reasons, we can do better on contradiction detection.
References
HaluMem The first benchmark to localize memory hallucinations by pipeline stage rather than end-to-end QA, finding that existing memory systems generate and accumulate fabrications specifically during extraction and updating, which then propagate into wrong answers downstream. https://arxiv.org/abs/2511.03506
LoCoMo The long-term conversational memory benchmark most of this space is measured against, testing agent recall across conversations spanning up to 35 sessions and thousands of turns. https://arxiv.org/abs/2402.17753
LongMemEval A harder, independently audited long-term memory benchmark built as a complement to LoCoMo, increasingly used alongside it as a cross-check. https://arxiv.org/abs/2410.10813
TOKI Formalizes memory-conflict resolution as a concurrency-control problem, showing that the four heuristics production systems already use (last-writer-wins, evidence-weighted merge, await-confirmation, per-rule policy) never state which isolation level or write-time anomaly they tolerate, and supplies that missing formal contract. https://arxiv.org/abs/2606.06240
Mem0 Introduces a production memory architecture with explicit conflict detection and resolution at write time, benchmarked against ten other memory approaches on LoCoMo; a later token-efficient version reports 92.5 on LoCoMo and 94.4 on LongMemEval while using roughly a third of the tokens per query that full-context approaches need. https://arxiv.org/abs/2504.19413
LoCoMo scoring audit Finds a bug in the reference LoCoMo evaluation script: when the gold answer is empty (i.e., the correct behavior is to refuse), the matching function always returns false, so a system that correctly refuses is scored identically to one that fabricates, meaning published "refusal" accuracy numbers on LoCoMo don't measure what they claim to. https://arxiv.org/pdf/2604.10981
Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" Introduces the recency + relevance + importance retrieval formula that most subsequent agent memory systems still use, with "importance" scored via an LLM self-rating at write time. https://arxiv.org/abs/2304.03442
TRUSTMEM Trains a separate, frozen verifier model to score memory updates on coverage, preservation, and faithfulness after they're written, rather than trusting the writing model's own judgment of its edits. https://arxiv.org/pdf/2606.25161
MemStrata Reports that plain embedding similarity search distinguishes a contradicted fact from an unrelated duplicate at close to chance level (cosine AUROC 0.59), since contradictions are often more embedding-similar to the original than genuine rephrasings are. https://arxiv.org/abs/2606.26511
"Models Recall What They Violate" Finds models can correctly restate a constraint they were given and still violate it in the same response ("knows-but-violates"), with violation rates from 8% to 99% across seven models, and shows an LLM judge tasked with catching these violations has only 15% sensitivity against human raters. https://arxiv.org/pdf/2604.28031
CompliBench A benchmark built specifically to test LLM judges on detecting policy violations in multi-turn dialogue, finding that even frontier judges achieve only modest accuracy and systematically under-detect real violations while over-flagging compliant turns. https://arxiv.org/abs/2604.12312
Originally published at anedelkos.eu.
This sits alongside an eight-part series on cortexGuard, covering two-speed AI architecture, blackboard coordination, detector ensembles, RAG memory, chaos engineering, and observability.
Top comments (0)