There's a conflation in how most people talk about AI memory that leads them to build the wrong thing.
The context window and memory are not the same thing. They serve different purposes, operate on different timescales, and fail in different ways. Conflating them produces a system that looks like it handles memory but doesn't actually compound over time.
Here's the distinction and why it matters practically.
The context window
The context window is the model's working memory — the span of tokens it can attend to in a single inference pass. Everything outside the window is invisible. Everything inside the window is equally weighted (with some positional decay depending on architecture).
Key properties:
- Session-scoped — it resets when the conversation ends
- Expensive to fill — every token you put in costs inference compute
- Flat — there's no inherent hierarchy or structure; a crucial constraint and a filler paragraph cost the same
- Bounded — even very long context models have limits, and performance degrades with distance from the query
- Non-persistent — nothing in the context window writes itself to storage automatically
The context window is useful for reasoning within a session. It's not useful as a persistence mechanism because it has no persistence.
Memory
Memory, in the sense that matters for ongoing work, is structured information that persists across sessions and is available to be selectively loaded into context when relevant.
Key properties:
- Durable — survives session boundaries
- Selective — not everything needs to be loaded every time; only what's relevant to the current task
- Structured — organized so the right things can be found and loaded efficiently
- Linked — connected to other relevant pieces so that loading one node gives you traversal hooks to related context
- Accumulated — gets richer over time as more decisions, constraints, and state are recorded
Memory is what you build and maintain. The context window is what you load memory into at session time.
Why the conflation is harmful
When people treat the context window as their memory system, they build workflows that:
Paste everything in every session. Load all potentially relevant documents at the start of each conversation. This works until you hit context limits, runs the risk of missing something, costs tokens proportional to how much you paste, and requires you to manually curate what's relevant every time.
Rely on chat history. Use the conversation history as a de facto memory store. This fails when you start a new thread (history resets), when you need to find something specific (chat history is terrible for retrieval), and when you want to share context with a different model or tool.
Build "memory" features on top of context. Tools that summarize conversations and prepend them to the next session are still just filling the context window — with summaries instead of full history. Better than nothing, but fundamentally still ephemeral. The summary is lossy and unstructured.
None of these produce the compounding effect that genuine persistent memory does, because none of them build a durable, structured, queryable store that gets richer over time.
The architecture that works
Persistent memory for AI-assisted work needs three things:
1. External storage. Outside the model, in files or a database. Not in the context window. Not in chat history. Written to disk in a format you control.
2. Structure. Not a prose dump — typed fields with known semantics. A decision record has different fields than a constraint record has different fields than a session-state record. The structure makes selective loading possible and makes the content reliably interpretable.
3. A loading protocol. A defined process for deciding what gets loaded into context at session start. Not everything, every time — just the hub note for the active project, the session-state record, and any directly relevant linked notes. This keeps context costs low and ensures the most important information is loaded closest to the query.
This is what I call the hub-and-spoke pattern: a hub note per project that's always loaded (current state, active constraints, key decisions), with spokes to more detailed notes that get loaded selectively.
The file-native implementation
The simplest implementation of this is also the most portable: plain Markdown files with structured sections and wiki-style [[links]] between related notes.
- No database required
- No API calls for retrieval
- No build step for embeddings
- Works with any model that can read files
- Fully auditable (it's just text)
- Version-controllable
The tradeoff vs. vector databases: discovery is harder (you need to know what you're looking for, or use a lightweight vector search layer for exploration). For ongoing projects where you know what you're working on, direct file reads are faster and more reliable than probabilistic retrieval.
The vault
The Obsidian vault skeleton I use operationalises this pattern:
- Hub templates (the always-loaded entry point per project)
- Session-state protocol (what gets written at close and read at open)
- Note types with defined structure (decisions, constraints, state, open questions)
- Linking conventions that enable traversal
- Optional lightweight vector search for discovery
→ Obsidian Agent Vault on Gumroad — $49
If your AI workflow today is primarily context-window management — pasting, summarizing, re-loading — you're working against the grain of how these systems are designed. The context window is for reasoning. Memory is for persistence. Build the right thing for each.
Tags: #ai #llm #obsidian #pkm #softwareengineering #productivity #machinelearning
Top comments (0)