Every AI agent I have ever run starts each session as a blank slate. No memory of what happened yesterday. No recollection of the decision you made three weeks ago. No idea that you already told it twice not to CC a particular person on supplier emails.
For the first few months of running my agent fleet across five businesses, this was the single biggest source of friction. Not the AI capabilities — those were fine. The problem was continuity. Every session felt like onboarding a new contractor who had never met me.
Then I built a memory system. And it genuinely changed how the whole operation works.
The Core Problem
Large language models are stateless by design. When you start a new conversation, the model has zero context from previous ones. It does not remember your name unless you tell it. It does not remember that you prefer bullet points over paragraphs. It does not remember that the legal contract with Supplier X was rejected because of clause 14.3.
Most people work around this by pasting context into every prompt. That is tedious, error-prone, and does not scale. When you are running 23 agents across multiple businesses, you cannot manually brief each one at the start of every session.
I needed a system where agents could wake up and immediately know who they are, who they work for, and what has been happening.
The Solution: Three Layers of Memory
I settled on a three-layer approach. Each layer serves a different purpose and has a different read frequency.
Layer 1: Daily Session Notes
Every meaningful thing that happens gets written to a daily markdown file:
memory/
2026-02-15.md
2026-02-16.md
2026-02-17.md
...
Each file is a raw log. Decisions made, emails sent, errors hit, context that needs to carry forward. The agent writes to it during a session the same way a human would jot notes.
A typical entry looks like this:
## 2026-02-17
- Donna flagged invoice from Reynard as duplicate — held for review
- Confirmed: budget sign-off threshold raised to R50k (was R25k)
- Staging deploy failed — port conflict. Elon investigating.
- Reminder: Warwick starts 1 March, needs system access by Feb 28
At the start of each session, the agent reads today's file and yesterday's. That is usually enough to restore working context.
Layer 2: Curated MEMORY.md
Daily files get noisy fast. You do not want an agent wading through 90 days of raw notes every morning.
MEMORY.md is the distilled version — the things that actually matter long-term:
# Long-Term Memory
## Business Decisions
- Budget approval threshold: R50k (raised Feb 2026)
- Reynard is a contractor — do not include in staff comms
- Prod deploys require owner sign-off, no exceptions
## Agent Notes
- Donna: do not CC owner's personal address on supplier correspondence
- Elon: prefers direct commands over open-ended requests
- Gene: field standup Mon/Wed/Fri at 07:30
## Ongoing Matters
- PAJA application: response due March 15
- FlowCap agreement: pending signature
Agents read this file in main sessions only — not in crons or sub-agents (for privacy reasons). It gets updated a few times a week.
Layer 3: Vector Knowledge Base
For deeper recall — contracts, meeting transcripts, lengthy documents — I run a local vector database. It sits on top of Qdrant and handles semantic search across embedded chunks.
When an agent needs to recall something specific, it queries the KB rather than trying to hold everything in MEMORY.md.
# Store a memory
curl -X POST http://localhost:3001/api/message/sync \
-H "Content-Type: application/json" \
-d '{"message": "IC agreement clause 7: contractor entitled to 15% on completion", "sessionId": "jarvis"}'
The File Structure
Here is exactly what the workspace looks like:
clawd/
MEMORY.md # Curated long-term memory
SOUL.md # Agent identity and personality
AGENTS.md # Operating instructions
memory/
2026-02-15.md
2026-02-16.md
heartbeat-state.json # Tracks last check timestamps
skills/
email-digest/
meeting-prep/
The agent reads files in a specific order at session start:
- SOUL.md — who am I
- USER.md — who am I helping
- memory/today.md and memory/yesterday.md
- MEMORY.md (main session only)
- TODO.md (main session only)
This sequence takes maybe 15-20 seconds of context loading. After that, the agent is fully oriented.
A Real Example
Three weeks ago, one of my agents handled a payment terms negotiation with a supplier. We moved from 30-day to 45-day terms — a small but meaningful cash flow improvement. The decision went into MEMORY.md as a one-liner.
Last week, a different interaction came up where the agent was drafting a new supplier agreement. Without being prompted, it defaulted to 45-day terms and added a comment: "Using 45-day terms as per updated policy."
That is the system working. No prompt engineering, no manual briefing. A well-maintained memory file carried the decision forward across three weeks and multiple context resets.
What This Unlocks
The memory system changes the relationship between agent and operator. Instead of an agent that needs constant re-briefing, you get one that builds institutional knowledge over time.
After six months of running this setup, MEMORY.md is about 1,200 lines of curated context. Business decisions, relationship notes, infrastructure specifics, lessons from failures. Agents that tap into it feel like they have been on the team for years.
The daily memory files are also invaluable for auditing. When something goes wrong — and it will — you can trace exactly what the agent knew at each decision point.
The Gotcha
One thing I got wrong early: treating MEMORY.md as a dump of everything. It became too long, too cluttered, and agents started effectively ignoring large sections of it.
The fix was ruthless curation. If something is no longer relevant, it gets removed. If a decision gets superseded, the old entry gets updated, not just appended. Signal, not volume.
Now I do a memory maintenance pass every few days during a quiet heartbeat cycle: read the recent daily files, pull out what is worth keeping, update MEMORY.md, prune what is stale. Five minutes of maintenance keeps the whole system sharp.
Memory is not glamorous. It is plumbing. But it is the plumbing that makes everything else work — and once you have it, you will not understand how you ran agents without it.
Top comments (0)