Agent Memory & Context Engineering

How agents remember - and why deciding what to forget is the real skill

An agent that starts every step with a blank mind cannot really pursue a goal. It would reintroduce itself to you on every message, forget what it just tried, and repeat the same mistake forever. Memory is what turns a stateless model into something that accumulates - that knows who you are, what it has already done, and what it learned last Tuesday. This post is about how that works and, more importantly, about the discipline of deciding what an agent should remember at all.

The context window is not memory.

The first thing to unlearn: a model’s context window is not its memory. The context window is working memory - RAM, not a hard drive. It is finite, it is reset on every request, and every token in it costs money and dilutes the model’s attention. Stuffing an entire conversation history and knowledge base into the prompt does not scale, and past a point it actively hurts - the model loses the important signal in a sea of stale detail. Real memory lives outside the window and is selectively loaded into it when needed.

Four kinds of memory

Borrowing loosely from cognitive science, agent memory is usually split into four types, and good systems use all of them:

• Short-term/working memory - the current conversation and the agent’s recent thoughts and observations. Lives in the context window.
• Long-term episodic memory - a record of what happened: past conversations, decisions, and the outcomes of previous tasks.
• Long-term semantic memory - facts and knowledge: who the user is, domain information, documents. This is what retrieval-augmented generation pulls from.
• Procedural memory - how to do things: learned skills, tool-use patterns, and reusable strategies.

Short-term memory: the rolling buffer

The simplest memory is just keeping recent turns in the prompt. The problem is that conversations outgrow the window, so the standard move is to keep the last few turns verbatim and summarise the older ones into a compact note:

def build_context(history, window=6):
    recent = history[-window:]
    older = history[:-window]
        if older:
            summary = llm(f"Summarize this conversation so far:\n{older}")
            return [f"Summary so far: {summary}", *recent]
    return recent

This keeps the prompt bounded while preserving the gist of what came before. It is crude, but it is the backbone of almost every chat agent in production.

Long-term memory: embeddings and vector search

To remember across sessions, an agent writes information to an external store and retrieves it later by meaning rather than exact keywords. The mechanism is embeddings: each piece of text is converted into a high-dimensional vector, stored in a vector database, and later retrieved by finding the vectors closest to the current query. This is what lets an agent recall a relevant fact even when you phrase your question completely differently from how the fact was stored.

# Write a memory
vec = embed("User prefers window seats and vegetarian meals.")
store.add(id="pref-1", vector=vec, text="User prefers window seats and vegetarian meals.")

# Later, retrieve by meaning
query_vec = embed("book me a flight to Delhi")
hits = store.search(query_vec, top_k=3)
context = "\n".join(h.text for h in hits)   # injected into the prompt

That retrieval step is what gives an agent its long memory without bloating the context window: you store everything but load only the handful of memories relevant to the moment.

Why vector search alone isn’t enough

Vector similarity is powerful but blunt, and it is worth knowing its limits before you lean on it. Similarity relies on relevance - not recency, not authority, not workflow state. It will happily surface an outdated preference the user changed yesterday or a fact from a draft that was later overruled, simply because the words are close. It does not understand which memory is current, which is authoritative, or where you are in a multi-step task. Production memory systems layer on recency weighting, source ranking, and explicit state to compensate. Memory in 2026 is treated as a real engineering discipline with measurable trade-offs, not a database you bolt on and forget.

Context engineering: the real skill

All of this rolls up into the discipline people increasingly call context engineering: deliberately deciding what goes into the context window on every single step. Prompt engineering asks How do I word the instruction? Context engineering asks What information should the model see right now, and what should I leave out? It is closer to managing a tight working-memory budget than to writing clever prompts.

A recent and influential pattern is Agentic Context Engineering (ACE), which treats context as something the agent actively curates through a three-role loop: a Generator produces an attempt, a Reflector evaluates it and flags what was missing or wrong, and a Curator distils the lesson into a growing “playbook” that improves future context. Reported results show meaningful accuracy gains on agent benchmarks without retraining the underlying model - the improvement comes entirely from feeding it better context. That is the whole thesis of context engineering in one experiment.

Practical patterns to start with

• Summarise old turns, keeping recent ones verbatim - bounded prompt, preserved gist.
• Retrieve, don’t dump - pull only the top few relevant memories per step instead of the whole store.
• Tag memories with time and source so you can prefer recent, authoritative information over stale matches.
• Write back what matters - after a task, save the durable facts and lessons, not the entire transcript.

The takeaway

Memory is what separates a chatbot from an agent that genuinely accumulates competence over time. But the headline isn’t “store everything” - it’s the opposite. The skill is curation: deciding, on every step, the smallest set of information that lets the model act well and leaving the rest in long-term storage until it’s needed. Treat context as critical infrastructure, and your agents get sharper, cheaper, and more reliable all at once.

Next in the series: One agent can only do so much. We’ll look at multi-agent systems - orchestration, the A2A protocol, and how to split a goal across a team of specialised agents without the whole thing collapsing into chaos.