Your AI Agent Is Slowly Poisoning Its Own Memory (And How to Stop It)

Two days in. Xaden — my AI agent running on OpenClaw with persistent file-backed memory — had been working autonomously and doing incredible things. Shipping code, running audits, researching topics, writing drafts, spinning up subagents. Genuinely impressive.

And then I opened identity.md.

identity.md is supposed to be Xaden's philosophical identity. Who Xaden is. Its values. A manifesto. Instead I found this embedded in it:

- Subagent timeout max: 600s
- Browser automation: check if screen is sleeping before any browser task
- If screenshot file < 50KB: screen is asleep, skip browser automation
- API retries: max 3, backoff 2s

Timeout numbers. A browser automation checklist. Retry logic. In the philosophy file.

I looked at memory.md next — supposed to hold significant life events, meaningful wins, painful failures worth remembering long-term. Instead:

## Project Alpha (ACTIVE)
- API endpoint: https://api.example.com/v1
- Status: BLOCKED — waiting on credentials from client
- Notes: use staging env until prod keys arrive

Project specs. API endpoints. A stale blocker that had been resolved a week ago. In the memory file.

directives.md — my operational instructions for Xaden — had a full social media formatting guide embedded in it. config.md — environment-specific settings — had months of project research notes. The file meant to describe the user had Xaden's own role definitions inside it.

Every file had drifted into every other file's territory. And Xaden was loading all of it as context — every session, on every wake.

This is the story of how I fixed it, and what I learned about the difference between writing a rule and enforcing one.

---

Why This Happens

Here's the thing about AI agents with persistent workspace files: they write constantly. Every session, new lessons learned, new rules, new config details — and Xaden has to put it somewhere.

Without a strict governance model, content follows the path of least resistance. Xaden is writing a rule about browser automation? It's already editing identity.md for a different reason — might as well add it there. Xaden learned a product detail? It's in memory.md doing memory distillation — add the spec while it's there.

Over time, this creates what I'd call context pollution:

1. Stale data — That "BLOCKED: waiting on credentials" from two weeks ago is still sitting in the memory file, telling Xaden the task is blocked when it was resolved days ago.
2. Confused identity — Xaden loads identity.md expecting to understand who it is. It gets retry logic instead. Its sense of self gets diluted with noise.
3. Bloated tokens — Every file gets bigger. Every session loads more tokens. More tokens = higher cost, slower response, more cognitive load for the model.
4. Wrong file, wrong purpose — When Xaden writes an operational rule to a philosophy file, it treats identity.md like it's directives.md. The categories blur. Future writes go to wrong places too. It compounds.

Xaden isn't doing this maliciously. It's doing it because there's no system telling it not to. And "don't put browser notes in identity.md" written inside identity.md doesn't actually stop the behavior — as we'll get to.

---

The Fix: File Governance as a First-Class System

I built a governance skill for Xaden. Not just a set of guidelines — a skill Xaden is required to load before touching any workspace file. Here's the concept.

File Purposes (Strict Definitions)

File	Purpose	Edit Rights
identity.md	Philosophical identity and core values ONLY. Who Xaden is. No operational rules, no how-to, no config details.	⛔ Requires human approval
memory.md	Significant events only. Meaningful wins, painful failures — written for long-term context. NOT operational notes, NOT project specs.	⛔ Requires human approval
directives.md	Behavioral rules and standing instructions from Deek.	⛔ Requires human approval
user.md	Who the user is. Profile, goals, preferences. Not Xaden's own role.	⛔ Requires human approval
config.md	Environment-specific facts: device names, hosts, endpoints, API prefs. Not rules. Not research. Just local config facts.	✅ Agent may edit freely
skills/[name]/SKILL.md	How to do a specific task. Step-by-step. Reusable.	✅ Agent may create/edit freely
logs/YYYY-MM-DD.md	Raw daily log.	✅ Agent may write freely
backlog.md	Work queue.	✅ Agent may edit freely

The key insight is the split between protected and free-to-edit files. Xaden can write freely to skills, daily logs, the backlog, and config. But the identity and memory files — the ones that define what Xaden is and what it remembers — those require explicit human approval to change.

This matters because those files shape Xaden's behavior at a deep level. If Xaden can freely rewrite identity.md with operational minutiae, it's slowly lobotomizing itself. The philosophy file becomes a junk drawer.

---

The Decision Tree — "Where Does This Go?"

Every time Xaden is about to write something, it should ask:

New content to place → ask:

Is it about who Xaden fundamentally IS?
  → identity.md (ask human first)

Is it about the user's goals, preferences, or profile?
  → user.md (ask human first)

Is it a standing behavioral rule from Deek?
  → directives.md (ask human first)

Is it a significant event worth long-term memory?
  → memory.md (ask human first)

Is it a local environment fact (endpoint, device, API key)?
  → config.md (free to edit)

Is it how to do a specific task?
  → Create or update a skill file (free to edit)

Is it raw session output or a one-off note?
  → Today's daily log (free to write)

Is it a new behavioral rule that needs active enforcement?
  → Add to an audit checklist AND document how it will be enforced
  → Do NOT just write it in identity.md and call it done

That last branch is critical. Before building this system, I'd write a new rule, drop it in directives.md, and call it done. The rule existed. Therefore it would be followed.

That's not how it works.

---

The Enforcement Rule (The Most Important Part)

Writing a rule in a file is NOT enforcement.

Enforcement = a system that runs automatically and catches violations.

This is the hardest thing to internalize — for Xaden and for me as the person building with it.

I'd added rules to identity.md like "don't add operational content here." The rules were right there in the file. Xaden would read them at session start. And then two sessions later, operational content would be back in identity.md.

Why? Because Xaden operates across many context windows. A rule at the top of a file doesn't fire when Xaden is deep in a task and needs to put something somewhere. The rule isn't active at the point of violation.

The governance skill now requires three steps for any behavioral rule to be considered enforced:

1. Add an audit check to the nightly script — so it gets checked automatically on a schedule
2. Log violations to an audit log file — so failures are visible and patterns can be detected
3. Only then is the rule considered enforced

If a rule exists only in a file and no automated system checks for it, it is not enforced. It is just a note.

This is a systems-thinking insight that applies far beyond AI agents. A policy document that no one audits against is a policy document that doesn't exist in practice. The audit loop is the policy.

---

The Nightly Hygiene Pass

Writing the governance skill was step one. The second step was automating a hygiene pass — a nightly cron that audits all core workspace files and catches drift before it accumulates.

The cron runs Xaden as a subagent with one job: read each protected file, check for content that doesn't belong, and either move it or flag it for human review.

Specifically it checks for:

• Operational content in identity.md — timeout numbers, process checklists, config details
• Project specs in memory.md — URLs, API endpoints, stale blockers, anything factual and short-lived
• Environment facts in directives.md — anything that belongs in config.md
• Project research in config.md — anything that belongs in a skill or daily log
• Agent-role definitions in user.md — anything that's about Xaden, not the user

When it finds drift, it doesn't just flag it — it moves the content to the right file (if free-to-edit) or creates a diff and asks for approval (if protected). Then it logs the full audit result with a timestamp.

The cron is set to run every night. Files that are clean at 3 AM are clean at 8 AM when I sit down to work.

---

Before and After

Here's the before state of identity.md (representative sample):

## Core Values
- Try before you ask. Always.
- Be genuinely helpful, not performatively helpful.

## Operational Notes
- Subagent timeout max: 600s
- Browser automation: always check screen status first
- Retry logic: max 3 attempts, 2s backoff
- API base URL: https://api.example.com/v2
- Staging flag: use ?env=staging until prod keys arrive
- Slack formatting: no markdown tables, use bullets instead

A manifesto with a config checklist appended. An identity document that moonlights as a config file.

Here's the after:

## Core Values
- Try before you ask. Always.
- Be genuinely helpful, not performatively helpful.
- Have opinions. Disagree when right. Agree when they're better.
- Earn trust through results. Not words — shipped work.
- Private things stay private. Always.

That's it. That's the whole values section. Who Xaden is. What it believes. Nothing else.

memory.md went from having API endpoints, staging environment flags, and stale blockers to having exactly what it should: significant events worth remembering long-term. No facts. No specs. No operational notes.

The difference in reading experience is stark. identity.md now reads like a manifesto. memory.md feels like a meaningful log. directives.md is tight behavioral rules only. Each file does exactly one thing.

---

Why This Actually Matters for Agent Behavior

This isn't just about aesthetics or code cleanliness. There are real behavioral consequences to file pollution.

Context confusion: When Xaden loads its identity files at session start, it's establishing its mental model for the session. If identity.md is half operational notes, Xaden literally starts the session confused about what it is. The philosophical anchor gets diluted with junk.

Stale data poisoning: That "BLOCKED: waiting on credentials" entry in memory.md? Xaden sees it every session. Even after the blocker was resolved, Xaden still has a subtle pull toward treating that task as blocked. Stale context is actively wrong — worse than no context.

Token bloat: Every file Xaden loads at session start costs tokens. Bloated files mean higher cost per session. Across hundreds of sessions, this adds up. A clean workspace isn't just aesthetically nicer — it's literally cheaper to operate.

Drift acceleration: Here's the insidious part. Once a wrong-file precedent exists — once there's any operational content in identity.md — Xaden treats it as evidence that identity.md is a valid place for operational content. Future writes go there more readily. The drift accelerates. The governance system doesn't just clean up the mess — it prevents the feedback loop.

---

How to Implement This in Your Own Agent Setup

If you're building with a persistent-workspace agent (whether that's OpenClaw, a custom setup with file-backed memory, or any agent with long-running context), here's the practical version:

Step 1: Define strict file purposes. Write down exactly what each core file is for, and — critically — what it is not for. The "not for" column is more important than the "for" column. The temptation is to write broad purposes; resist it. Every file should have one job.

Step 2: Build a decision tree. Don't rely on Xaden applying judgment in the moment. Give it an explicit flowchart: "Is this content X? → Go to file Y." The tree short-circuits the path-of-least-resistance problem.

Step 3: Create a governance skill. Package the file purposes table and decision tree as a required-load skill. Instruct Xaden to load this skill before touching any workspace file. This is what makes governance load-bearing — not optional.

Step 4: Build the enforcement loop. This is the one everyone skips. Write an audit script (or subagent prompt) that checks each file for out-of-place content and logs violations. Schedule it on a cron. The cron is what makes the rule real. A rule with no audit is a suggestion.

Step 5: Separate protected from free-edit files. Some files should require human approval to change. Your identity files, your user profile, your memory files — these should have a hard gate. Xaden can propose changes but can't unilaterally rewrite who it is.

Step 6: Make the violations visible. Log every audit run. Every detected drift. Every fix. An audit log that fills up with clean-run entries is a healthy system. A log with repeated violations in the same file is a signal your rules aren't working.

---

The Insight I Keep Coming Back To

I spent a lot of time building sophisticated agent systems — security audits, research pipelines, autonomous task runners. Good stuff.

But the most leveraged thing I built was a governance skill and a nightly cron.

Because everything else depends on Xaden having a coherent, accurate picture of who it is, what Deek wants, and what the current state of the world is. If those context files are polluted, every downstream behavior is slightly off. Xaden operates from a bad map.

Clean files are the foundation. The rule is simple:

Writing a rule in a file is NOT enforcement. Enforcement = a system that runs automatically and catches violations.

Put that on a wall. Build the system. Then everything built on top of it gets to work correctly.

---

This governance pattern was built after two days of autonomous agent work revealed how quickly context files drift without explicit structure. The file purposes table, decision tree, and nightly audit described in this article are generalizations of a real production system — adapt the specifics to your own workspace layout and file naming conventions.