Most "agent memory" writeups are about getting things in: which vector store, how to chunk, how to embed. The harder question in practice is the opposite one. Your context window is a fixed budget. Every token you spend on a stale tool output is a token you did not spend on the thing that actually decides the next step. So the real engineering problem is not retrieval, it is eviction: what earns a place in context, and what gets dropped or compressed.
Here is the model that has held up for me across a few agent builds.
Context is a working set, not a memory
Borrow the term from operating systems. The working set is the small slice of memory a process actually touches right now. Everything else lives on disk and gets paged in on demand. An agent's context window is a working set, and treating it like long-term storage is the mistake that makes agents slow, expensive, and confused.
Durable memory (the vector store, the notes, the logs) is your disk. It can be huge. The context window is RAM: small, hot, and contested. The job of "agent memory" is mostly the pager: deciding what to bring into the working set for this step and what to let fall back to disk.
Once you see it this way, most context bloat is a pager bug. You retrieved 40 chunks because they matched, kept all 40 in context for the rest of the session, and now every subsequent step pays attention over 40 things that stopped being relevant 10 steps ago.
Two axes, not one
The thing that made my eviction decisions cleaner was realizing they run on two independent axes, and people collapse them.
Fidelity: must stay exact vs can be approximated. Some content breaks if you compress it. A commitment the agent made ("I will not touch prod"), a tool's exact error string, an ID, the user's literal instruction. Others degrade gracefully: a long document you only need the gist of, an old reasoning trace, the transcript of a resolved subtask. Summarize the second kind aggressively. Never summarize the first.
Recency vs salience. Recency is easy and mostly wrong on its own. The token that matters is not the most recent one, it is the most salient one for the current step: the retrieved fact this decision hinges on, the constraint that must not be violated, the goal. Recent context is one bucket of salient content, not the definition of it.
Cross the two axes and you get a priority you can actually implement. High-salience-must-stay-exact is sacred: keep it verbatim, evict last. Low-salience-can-approximate is the first thing you compress or drop. The dangerous quadrant is high-fidelity-but-low-salience: exact content you are paying full price to keep, that is not helping this step. That is where most of the waste lives.
A concrete eviction policy
You do not need anything fancy to start. A policy that beats "keep everything until the window overflows":
- Pin the invariants. Goal, hard constraints, the user's literal ask. These never get evicted or summarized. Small, always present.
- Keep the last N turns raw. Recency is a cheap proxy for salience for conversational coherence. N is small, like 3 to 6.
- Demote retrieved evidence after it is used. A chunk that informed a step gets summarized to its one-line claim plus a pointer back to durable memory. If a later step needs the detail, page it back in.
- Compress resolved subtasks to their outcome. The full transcript of "figured out the config" becomes "config is X, verified." The reasoning got you there, but it is not load-bearing anymore.
- Evict on salience, not age. When you need room, drop the lowest-salience item, not the oldest. An old invariant outranks a fresh but irrelevant tool dump.
The pointer-back part is what makes this safe. You are not deleting information, you are moving it from RAM to disk with a way to fault it back. Summarization without a pointer is just lossy deletion, and that is how agents "forget" things they clearly should have known.
The failure mode to watch
The subtle bug is trusting your own summaries. When you compress retrieved evidence into a one-line claim, that claim is now a memory the agent will act on, and it was written by an LLM that may have smoothed over the caveat that mattered. Keep the pointer to the source, and when a step is high-stakes, page the raw evidence back in instead of trusting the summary. Fidelity and salience decide what to keep; stakes decide when to distrust your own compression.
None of this needs a new framework. It needs you to stop treating the context window as storage and start treating it as a scheduler: a small, contested resource where the real skill is deciding what to page out.
Written with AI assistance and edited by a human. I think about this because I build a local-first personal memory layer, where the "disk" is your own captured work and the pager runs on your machine.
Top comments (0)