Part 2 of the Building the AI Memory Stack series
When I published the first article in this series, I thought I was writing about context windows.
The more I wrote, the more I found myself bouncing between documents. I had the glossary open in one browser tab. The Memory as Infrastructure article was open in another. A third tab contained notes about Context Hydration. GitHub was open with the SDK specifications, and a handful of Architecture Decision Records were sitting beside my editor.
None of those documents individually contained "the answer." Together, they formed the temporary collection of information I needed before I could make progress.
Then something unexpected happened.
I was no longer writing about context windows.
I was reconstructing a memory hierarchy, and the context window was only its first layer.
That is why this became a series.
In Part 1, I argued that the context window is best understood as the CPU cache of an AI system. It is an execution surface, not a memory system. Like CPU cache, it is optimized for fast access and exists only for the duration of the work being performed.
But caches do not populate themselves.
Neither do context windows.
The Working Set Before the Work
Before I could write, I assembled a working set.
Browser tabs and open documents
- Sovereign Systems glossary
- Memory as Infrastructure
- Context Hydration notes
- SDK specifications
- Architecture Decision Records
- GitHub repository
- Article draft
|
v
Active Working Memory
|
v
Context Window
|
v
Reasoning
That collection was not my long-term memory. It was a temporary working set assembled for one task. My brain still had to compare ideas, notice contradictions, and produce something new. The documents simply gave the reasoning process the information it needed.
Agentic systems work in much the same way.
Before a model begins reasoning, documents have been retrieved, tools have executed, state has been restored, policies evaluated, and responses normalized. The prompt is usually the final artifact produced by an orchestration layer, not the beginning of one.
By the time the model receives its first token, dozens of retrieval, filtering, ranking, and assembly decisions may already have been made.
That assembled execution state is what I call Active Working Memory.
This article is about the second layer in that stack.
Models Reason. Applications Assemble.
One sentence captures the distinction this entire article is trying to make.
Models reason. Applications assemble.
Those are different responsibilities.
A model does not retrieve documents.
A model does not decide which Git commit matters.
A model does not know whether a tool response is stale.
An application does.
Consider a concrete case. An agent is asked to update a Python SDK.
It retrieves the ADR describing the package boundary.
It loads the glossary definition of Active Working Memory.
It checks the current implementation on GitHub.
Only then does it build the prompt.
People often talk about "putting something into the context window."
That wording quietly suggests the context window is responsible for finding, selecting, and organizing information.
It isn't.
By the time inference begins, the application has already decided what the model will, and will not, be allowed to see.
The context window does not begin the process. It receives the result of the process.
Many failures blamed on the model are actually failures of context assembly. The wrong evidence was retrieved. A stale document won. A constraint never made it into the working set. Those are architecture problems before they are model problems.
RAM for Agentic Systems
If the context window is the CPU cache, Active Working Memory is the system RAM.
| Layer | Primary Responsibility | Typical Owner |
|---|---|---|
| Durable Memory | Preserve knowledge | Storage / Application |
| Active Working Memory | Assemble task state | Orchestrator |
| Context Window | Present selected information | Runtime |
| Model | Perform inference | LLM |
Each layer has a distinct responsibility. No layer substitutes for another. A larger context window does not repair poor selection, and a better model cannot reason over evidence it never receives.
Search Finds. Context Assembly Decides.
Search answers one question.
What information exists?
Context assembly answers another.
Given this task, what information belongs together?
Those sound similar. Architecturally, they are completely different.
Active Working Memory presents that second decision to the model.
Two systems can use the same model, the same context window, and the same knowledge store yet produce very different results, because one assembles a concise, relevant working set while the other floods the model with loosely related information.
The model is identical.
The memory architecture is not.
An Architectural Boundary
Once Active Working Memory is treated as a real layer, context assembly stops looking like prompt engineering and starts looking like systems architecture.
Information crosses this boundary only after decisions have been made about relevance, authority, recency, format, and priority.
Every unnecessary document increases Context Tax. Every verbose tool response competes for attention. Every missing source creates a blind spot the model cannot recognize from inside the window.
The context window can only reason over what Active Working Memory hands it.
Looking Ahead
The working set I assembled while writing this article disappeared as soon as the article was finished.
The article remained.
That distinction turns out to matter.
Agentic systems face the same decision. What belongs only in today's working set? What deserves to become tomorrow's memory?
That is where Part 3 begins.
Models don't assemble context. They inherit it.
Working memory is not where knowledge lives. It is where knowledge collaborates.
Top comments (0)