🙋
I’m Luhui Dev, a developer who has been breaking down Agent engineering and exploring how AI can be applied in education.
I focus on Agent Harness, LLM application engineering, AI for Math, and the productization of education SaaS.
An agent spent half an hour finding the right test command yesterday. Today, in a new session, it starts trying approaches that already failed yesterday.
A longer context window can ease the problem. But projects keep changing and sessions keep growing. The cost becomes hard to sustain.
After an agent has done something, what does the system actually retain, and how does that help the next time?
Revisiting Anthropic's approach to agent memory, I found four especially useful responsibilities: keep complete records, extract reusable experience, organize scattered knowledge, and give the next session a short starting point.
I want to unpack this structure through the work of a coding agent.
Three different meanings of “remembering”
Suppose a user tells an agent: “This project uses pnpm.”
If the current conversation still contains that sentence, the information is available in context.
If the system writes it to a persistent file that another session can read, that is external memory.
If training changes the model's parameters, that is a different learning mechanism.
All three can make a model appear to “know” something, but they require different engineering approaches.
For application developers, external memory means we can inspect the content, trace its sources, update conclusions, and undo an incorrect write. It has a lifecycle we can manage.
Conversation compaction also needs to be distinguished. Compaction usually helps the current task continue, whereas long-term memory must support reuse across tasks. A compacted summary can become material for memory, but saving it indefinitely does not automatically solve staleness, conflicts, or when to retrieve it.
First decide how long information needs to live, then decide where it belongs.
Four layers at a glance
Agent memory can be organized into four layers according to purpose: transcripts, memory notes, knowledge representations, and injected context. They retain complete experience, lessons distilled from it, organized relationships between knowledge, and the small amount of information needed at the beginning of each session.
These are layers of information organization, not four kinds of database or a pipeline every item must pass through in sequence.
One file store can support several layers. Conversely, a single layer can rely on several storage systems.
Injected context, at the top, is usually generated from a persistently stored entry file or selected memory items and loaded into the model's context window when a session begins.
Putting each layer's content, common organization, and storage options side by side makes the distinctions clearer:
| Layer | Content | Common organization | Possible storage or medium | How it is read |
|---|---|---|---|---|
| Transcripts | Messages, tool calls, execution results, and other events | Chronological logs, JSONL event streams | Log files, database event tables, object storage | Search by event, time, or keyword |
| Memory notes | Lessons, preferences, plans, evidence-backed conclusions | Markdown notes, structured entries | File directories, database records | Retrieve within the task's scope, then read details |
| Knowledge representations | Topic knowledge, conceptual relationships, indexes | Wiki pages, linked documents, topic indexes | Linked document collections, optionally a graph database | Navigate by topic, link, or index |
| Injected context | Frequent conventions, important reminders, memory entry points | A concise entry file, selected memory fragments | Persistent entry content loaded into the runtime context window | Loaded automatically at session start |
For a small project, an append-only log directory, a set of Markdown topic pages, and a short index are enough to experiment with this structure.
Transcripts: preserve the evidence
Consider three events concerning a test command:
E17: Run pnpm test. It fails because the test database is unavailable.
E18: Start the test database provided by the project.
E19: Run pnpm test again. It passes.
A log can establish what was observed at that moment. It cannot guarantee that the agent's explanation is correct or that the project will always behave that way. Keeping the log lets us revisit the actual actions and feedback.
If the log also contains the agent's claim that “any test failure means the database should be restarted,” its presence in the record does not make it equally strong evidence.
I suggest attaching at least a timestamp, project, version or commit identifier, event type, and content location to each event. User requirements, tool observations, and model inferences should be distinguishable. That gives upper memory layers a way back to concrete evidence, instead of a collection of prose that merely cites itself.
Memory notes: turn an experience into conditional knowledge
The events above support a more careful note:
Under project A's current test configuration, integration tests require an available test database. In session 42, the tests passed after the database was started.
Compared with the raw record, this adds a scope of applicability while retaining the observational basis.
For my own implementation, I try to make each entry answer five questions: What is the conclusion? Where does it apply? Where is the evidence? When was it verified? What change would invalidate it?
The fields can vary by project. The priority is to preserve the connection between a conclusion and its evidence. The model does not need to see every piece of metadata. The system can first filter by project and version, then send only the content the current task needs.
A memory entry is valuable when it helps someone act on evidence later.
Knowledge representations: reduce repeated exploration
Once a project has hundreds of memories, filenames alone may no longer tell you what to read. Topic pages can help: a testing page can link to database configuration, while a release page can connect versioning procedures, rollback conditions, and deployment records.
Search answers “Is there something relevant?” A topic page also explains “How do these things relate?”
A user asking “Why does this pass locally but fail in CI?” may not use words that appear in a particular note. Yet a topic page may already organize the differences between environments.
When investigating a CI failure, a topic page can first connect environment differences, database dependencies, and past decisions. For a specific version or error, search can then locate configuration and logs. This avoids rebuilding the whole picture each time without treating the topic page as the complete truth.
Maintaining only a wiki can lose detail. Relying only on search can mean repeatedly paying to reconstruct relationships. They can work together; the point is to reduce detours during the task.
My tradeoff is this: relationships repeatedly used across documents are worth organizing in advance. For an occasional field lookup, direct search may be cheaper. Knowledge organization has construction and maintenance costs too.
Injected context: show the agent where to look
At the beginning of a session, an agent does not yet know what it does not know. A short entry point can tell it where the current project lives, which conventions matter, and where to read the testing and release procedures.
Claude Code's auto memory is one example: only the beginning of MEMORY.md is brought into a new session, while detailed topic files are read on demand. The entry point tells the agent what it remembers and where to find it; it need not contain all the knowledge.
For my own test system, I made the entry point a navigation aid:
Current project: A
Test environment and commands: see testing.md
Release and rollback conditions: see release.md
Long-term writing preferences: see writing-preferences.md
Whether to inject an item can depend on its frequency of use, the cost of omission, and its length. An infrequent rule with a high cost of omission may deserve space more than an ordinary status update that appears often.
Dreaming: maintain the transformations between layers
The four layers give an agent somewhere to save experience and a path back to knowledge. The next question is: As work on the same project continues, can past effort make the next task easier?
Yesterday's pitfalls should be easier to avoid today. Clues scattered across several sessions should come together into a coherent judgment. Procedures that have changed should be removed from the session's starting point. That is when memory begins to deliver cumulative value.
Dreaming handles this background work. During a task, the agent focuses on the immediate problem. Outside the task, it revisits experience, checks conclusions, and adjusts the knowledge structure so later sessions can access useful information more easily. The analogy with human sleep is quite natural.
But once you start designing it, “organization” turns out to contain many judgments.
If a lesson appears three times, is it a pattern, or the same mistake repeated three times? If a new record conflicts with an old note, did the environment change, or do they apply to different projects? A conclusion may deserve a place in detailed notes, but should it also be loaded before every session?
These judgments affect several layers. Correcting a testing rule may require updating the memory note, the testing topic page, and the starting index together.
Dreaming's value lies in keeping experience useful. The implementation challenge is to make each reorganization evidence-based, verifiable, and reversible when it goes wrong.
For now, the key is its place in the architecture: it maintains content and relationships across the layers, preparing more reliable memory for future tasks. Selecting inputs, resolving conflicts, checking conclusions, and writing back safely deserve a separate discussion, which I will continue in a future essay on Anthropic Dreaming.
Final thoughts
I prefer to maintain agent memory as a software asset. It has sources, scope, and a period of validity. The more it accumulates, the more someone needs to address its technical debt.
Logs make experience traceable, notes make lessons reusable, knowledge organization makes relationships easier to find, and a concise entry point tells the next session where to begin. Background organization continually corrects these materials.
The value of memory must ultimately be realized in future tasks.

Top comments (0)