A Practical Agent Memory Architecture: What to Keep, Retrieve, Revise, and Forget

A bigger context window enlarges what you can store. It does not decide what your agent should actually use. This is a build checklist for the part that does.

Here is the mental model to start from. Context is the working set, the few things the model reasons over this turn. Memory is the policy that decides what reaches that working set from a much larger store. A million-token window makes the store bigger. It does not hire a librarian, and it does not tell the forklift which box to fetch before the battery dies. A warehouse does not become a library just because you keep adding shelves.

Before the checklist, two evidence points explain why the window alone is not enough.

Why the window is not the answer

Position matters inside one prompt. The Lost in the Middle study found a U-shaped pattern in controlled question-answering and key-value retrieval on 2023-era models: information near the beginning or end was used more reliably than information in the middle. Read it as a demonstrated failure mode, not a fixed score. Synthetic stress tests go further. RULER evaluated 17 models on 13 synthetic tasks and found only about half of the models claiming at least 32K context held its satisfactory threshold at 32K, where satisfactory is RULER's own 85.6% definition. NoLiMa, a 2025 preprint, stripped literal keyword overlap so a match needed latent association; under its v1 evaluation, GPT-4o fell from 99.3% at short context to 69.7% at 32K. Long context is still improving: on RULER's own tables some later models held strong performance through 128K. The takeaway is narrow: usable context can be smaller than nominal context.

Long histories are a separate problem. LongMemEval uses 500 questions across five long-term memory abilities: extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. At roughly the 115K-token setting, tested long-context models dropped 30% to 60%. The window held the history; holding it was not enough. Crucially, the authors split memory into indexing, retrieval and reading, and report that even with perfect-recall memory, structured reading such as Chain-of-Note improved question-answering by up to 10 absolute points across three tested LLMs. Retrieving the right record is only half the job.

Different tests expose different gaps between advertised context capacity and reliable use. Lost in the Middle varies position, RULER adds harder tracing and aggregation tasks, and LongMemEval evaluates memory across long interaction histories. Results are study-specific and do not share one scale. Sources: Liu et al. (TACL 2024), Hsieh et al. (COLM 2024), and Wu et al. (ICLR 2025).

The build checklist

1. Separate three layers, not one transcript

Do not fuse raw history, durable memory and working context into "just save the transcript." Split them:

• Raw event history: an auditable archive, subject to retention and erasure rules.
• Curated memory: small, typed, principal-scoped and governed, with source-event references so consolidation can be unwound.
• Working context: assembled from the curated layer, not replayed directly from untrusted raw events.

This separation is an engineering synthesis across LongMemEval, MemGPT, Anthropic's context-engineering guidance, LangChain's memory docs and the OpenAI Agents SDK sessions model, not a single-source rule.

Raw events can remain auditable within retention limits while principal-scoped curated memory decides what the model sees now. Admission, retrieval, revision and forgetting are separate control points.

2. Implement five decisions as separate control points

Every event runs a gauntlet you can test and instrument independently:

1. Admit. Is this worth remembering, or transient noise for the raw log only?
2. Retrieve. Given the current task, which stored items are relevant?
3. Consolidate. Can related events collapse into one summary while keeping pointers to source events?
4. Revise. Has a higher-trust fact superseded an older one, with provenance for each version?
5. Forget. Should an item leave active memory because it expired, was disproved or became unsafe, or must an authorized erasure remove it across raw and derived stores?

3. Attach metadata to every curated record

Each record carries timestamps, source-event references, confidence, conflict status, expiry and deletion state. User-visible memories stay inspectable and correctable. This is what makes revision provenance-aware and consolidation reversible.

4. Treat forgetting as quality control, not loss

Hoarding every tool output is not photographic memory. It is a garage with a token bill, and it buries the few records that matter under thousands that do not. Forgetting relies on separation, not silent destruction: staleness, disproof or low relevance should deactivate a curated entry while preserving whatever raw evidence retention allows. Authorized user erasure and legally required deletion are different operations and must propagate through raw records, indexes and consolidated memories rather than leaving a derived copy behind. This draws on the forgetting logic in MemoryBank, whose Ebbinghaus-inspired mechanism is a reasonable design choice rather than a universal decay law, plus compaction guidance from Anthropic.

5. Harden the write path against poisoning

Once memory is read back as trusted context, it is a security boundary. AgentPoison, a preprint, poisoned memory and retrieval stores across three controlled agents so malicious records surface later as retrieved demonstrations. MINJA, a preprint under a controlled threat model, showed an attacker using only ordinary queries could inject malicious records. These are experimental threat models, not deployment prevalence, but the mechanism holds: if any utterance can become durable truth, any user can write to your model's future context. OWASP's data and model poisoning guidance recommends provenance tracking, versioning, anomaly detection, sandboxing untrusted data and validation against trusted sources. Mapped onto memory:

• Put a write gate at the trust boundary; keep the admission classifier inside it, since a malicious input may inflate its own importance.
• Records carry provenance; updates are reversible.
• Content from untrusted turns cannot silently promote itself into durable, high-confidence memory.
• Partition curated memory and retrieval by principal so one user's state cannot surface in another's working context.

6. Test each stage separately

Evaluate indexing, retrieval and reading as distinct stages rather than one answer-accuracy number, because a system can retrieve perfectly and still read wrong. Write explicit tests for the write path and the retrieval path. Track stale-memory, conflict and isolation errors, not just correctness. Retrieval and compression add their own failure modes: a retriever can drop the one detail that mattered, a summary can go stale, and an agent controlling its own paging can cascade one bad decision into several.

Prior art you can read as this checklist

Generative Agents, evaluated in a twenty-five-agent sandbox for believable behavior, scored a memory stream on relevance, recency and importance, then wrote back reflections and plans. MemGPT, a preprint, paged information between in-context memory and external storage using function calls. A-MEM, a preprint, creates structured notes, links memories and lets a new memory update an existing one, which is revision as a first-class operation. Mem0, a vendor-authored preprint benchmarked on LoCoMo with LLM-as-judge scoring, reported a 26% relative quality improvement over OpenAI's memory system, plus 91% lower p95 latency and over 90% token savings versus a full-context baseline. Those are author-reported numbers on selected models, not a production guarantee, but the direction is clear: a small curated memory beat a full-context baseline on cost. LangChain, the OpenAI Agents SDK, OpenAI and Anthropic compaction, and Letta all encode the same archive-versus-working-set split under different vocabularies.

When to skip most of this

Do not build a memory platform on day one. For short, bounded tasks, a larger context window plus compaction, or no durable memory at all, is often simpler and safer than a full persistent-memory stack. This is engineering judgment supported by RULER's evidence that long context increasingly works, Anthropic's smallest-high-signal-context guidance, and documented compaction that shrinks history without a separate system. Persistence creates provenance, deletion and poisoning obligations a scoped session never incurs. Start with a retention-scoped raw log and a recent-turn window. Add typed memories only when a repeated, observed failure justifies each type. Give users a way to inspect, correct and erase what the system believes about them.

The best memory system is not the one that remembers the most. It is the one that puts the right record on the workbench at the right moment, can point to where it came from, and knows when to stop bringing it up. A bigger warehouse buys you room. The librarian is still the job.