I Gave My AI Agent Long-Term Memory. It Started Confidently Lying to Customers About Facts From March.

I gave my AI agent long-term memory in an afternoon. It took me two weeks to realize memory was the problem, not the solution.

The pitch was irresistible. My support agent forgot everything between conversations — a customer would explain their setup on Monday, and by Wednesday the agent was asking them the same three questions again. So I did what every builder does: I bolted on a memory layer. Embed the conversation summary, store it in a vector DB, retrieve the top 5 relevant memories before every reply. Classic RAG-over-history. It worked beautifully in my tests.

Then it started confidently telling customers things that used to be true.

The incident that made me look

A customer on the annual plan emailed asking to change their billing email. My agent replied — warmly, fluently, completely wrong — that they were on the monthly plan and could simply cancel anytime. The customer, understandably, escalated: "Are you telling me I don't have an annual contract?"

I dug through the logs. Here's what happened:

1. In March, the customer had been on the monthly plan. The agent summarized that conversation and stored: Customer is on monthly plan, considering upgrading.
2. In May, they upgraded to annual. That happened in the billing system, not in a conversation — so no memory was ever written.
3. In September, the memory retrieval surfaced the March fact. It was semantically relevant (billing + plan), highly similar to the question, and ranked #1 of 5.
4. The model did what models do: it trusted its context. The retrieved memory outranked the vague instructions in my system prompt about "checking current plan details via the account tool."

The memory wasn't wrong when it was stored. It was stale. And my architecture had no concept of staleness whatsoever. A fact from March and a fact from five minutes ago had exactly the same authority in the prompt.

This is the failure mode nobody warns you about with agent memory: you're not building a memory, you're building a slowly-rotting cache with no TTL and no invalidation. Everyone who has done backend work knows a cache without invalidation is a bug factory. Somehow we collectively forgot that the moment we started calling it "memory."

The honest part: I'd seen the warning signs and ignored them

For a full week before the incident, small things were off. The agent twice referred to a pricing page that I had redesigned in July — describing buttons and plans that no longer existed. Once it apologized for an outage from May as if it were recent ("sorry about the disruption earlier this week").

I read those transcripts and thought hallucination, need a better model. I even bumped up to a more expensive tier for a few days, which cost me money and fixed nothing, because the model wasn't hallucinating. It was accurately reporting what its context contained. The context was the liar.

I lost roughly a week to the wrong diagnosis because "the memory system is feeding it dead facts" wasn't a hypothesis I considered. When you build a feature, you don't naturally suspect the feature. Lesson: when an agent says something confidently wrong, the first question shouldn't be "why did the model fail" — it should be "what was in the prompt, and who put it there?" Log the full assembled context, not just the user message and the reply. I now store the retrieved memories alongside every conversation, which is what let me root-cause this in ten minutes instead of ten days.

What I changed

Five fixes, in the order I'd recommend them:

1. Every memory gets a timestamp and a type. I split memories into two kinds: observations ("customer mentioned they use a Raspberry Pi at home") and state ("customer is on the monthly plan"). Observations age gracefully. State doesn't — state describes something that can change outside the conversation.

2. State memories expire. Anything classified as state gets a TTL: 7 days for plan/account facts, 30 days for softer things like project status. When retrieval pulls an expired memory, it's either dropped or re-fetched from the source of truth. Yes, this means classifying every memory at write time — one extra LLM call, about half a cent. Cheap compared to one furious annual-plan customer.

3. Tools beat memories. The real fix for the incident: plan, billing status, and subscription details are now never answered from memory. The system prompt says: "If the question involves current account state, call get_account — retrieved memories about account state are for context only and MUST NOT be stated as fact." Retrieval can inform tone ("this customer has been with us a while"); the API informs facts.

4. Memories are written as snapshots, not assertions. Instead of Customer is on monthly plan, the agent now stores As of 2026-03-14, customer said they were on the monthly plan. Sounds pedantic. It's not: when that string lands in a prompt, the model handles it completely differently. It hedges, it verifies, it calls the tool. Phrasing the memory with its date attached was the single highest-leverage change I made, and it cost me an afternoon of rewriting the summarization prompt.

5. A nightly reaper. A cron job on the Pi reviews memories older than 90 days and deletes any that were never retrieved. Turns out most memories are write-only. If retrieval never surfaced a memory in three months, it's not a long-tail gem — it's noise with an embedding attached.

The bigger lesson

Agent memory has the exact same failure modes as every caching system that ever existed: stale reads, no invalidation, write amplification, and trust boundary confusion (cache treated as source of truth). The industry gave it a friendlier name so we'd stop applying thirty years of hard-won distributed-systems intuition to it.

If you're adding memory to an agent, ask yourself the cache questions before the AI questions:

• What's the invalidation policy?
• What happens on a stale read?
• Which fields must never be served from cache?
• Can I tell, after the fact, which cached values influenced a given response?

If you can't answer those, you don't have a memory system yet. You have a time-delayed hallucination engine.

After the five fixes, the agent hasn't stated a stale fact as truth in six weeks — and the two times retrieval surfaced something expired, the logs show it called get_account instead. That's the behavior you want: memory for rapport, tools for truth.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.