Agent Memory Done Right: Short-Term, Long-Term, and the Scratchpad

• Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust
• Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series)
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

---

Your agent has amnesia. Every call to run() starts from an empty
messages list, learns what it needs for one task, and throws it
away. That is fine for a calculator. It falls apart the moment a
user comes back.

Add a second turn and you see the cracks. "What did I ask you last
time?" gets a polite apology. "Remember that I'm vegetarian" lasts
until the process restarts. Hand the agent a long task and the
buffer swells past the model's comfortable range: latency climbs,
quality drops, and eventually a request dies with a context-length
error your on-call engineer gets to read at 2am.

The fix is memory architecture. Not one thing — three. Short-term
memory is the conversation buffer that rides in the prompt.
Long-term memory survives across sessions, usually in a vector
store. Scratchpad memory is the reasoning trace of the current
task, inside the loop but hidden from the user. Most agent bugs in
production come from confusing two of these, or skipping one.

Short-term: the conversation buffer

Short-term memory is the messages list. That is the whole
definition. System prompt, prior user turns, prior tool calls and
their results — everything the model sees on this turn. It is
bounded by the context window.

Those windows sound generous until you run a ten-step agent whose
tools return JSON blobs. Two things degrade on long buffers.
Latency climbs roughly linearly with input tokens. And quality
drops on the "lost in the middle" curve, where the model attends
worse to material buried in the interior of a long prompt.

The rule of thumb: start compressing once you pass ~60% of the
window. Do not wait for the error.

The cheapest compression that works is a rolling summary. Keep the
last N turns verbatim, hand everything older to a cheap model, and
glue the summary back as a synthetic message. The trigger that
matters is token count, not turn count. A single tool result can
blow past a thousand tokens on its own, so a six-turn buffer today
can be larger than a forty-turn buffer yesterday. Measure tokens
with the API, not with a character-count guess.

# short_term.py  --  pip install anthropic
from anthropic import Anthropic

client = Anthropic()
SUMMARIZER = "claude-haiku-4-5"   # cheap model
MAIN_MODEL = "claude-opus-4-8"
KEEP_LAST = 6
TOKEN_BUDGET = 60_000


def count_tokens(messages: list[dict]) -> int:
    r = client.messages.count_tokens(
        model=MAIN_MODEL,
        messages=messages,
    )
    return r.input_tokens

The summarizer is a separate, cheap call that folds old turns into
a short synthetic message:

    joined = "\n".join(
        f"{t['role']}: {t['content']}" for t in old_turns
    )
    resp = client.messages.create(
        model=SUMMARIZER,
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": (
                "Summarize this agent conversation in "
                "under 200 words. Keep decisions, user "
                "preferences, open questions, and any "
                "tool results needed next turn.\n\n"
                + joined
            ),
        }],
    )
    return resp.content[0].text

Now wire the trigger to token count, keep the recent tail verbatim,
and recurse if one pass is not enough:

def compress(messages: list[dict]) -> list[dict]:
    if count_tokens(messages) <= TOKEN_BUDGET:
        return messages
    if len(messages) <= KEEP_LAST:
        return messages   # nothing old to compress
    old = messages[:-KEEP_LAST]
    recent = messages[-KEEP_LAST:]
    summary = summarize(old)
    return [{
        "role": "user",
        "content": "[prior summary]\n" + summary,
    }] + recent

Call compress() before every messages.create() and the buffer
stops growing without bound. The summarizer runs on a small, cheap
model. Token counts come from Anthropic's count_tokens endpoint,
because tool results and image blocks do not count the way plain
text does.

Two traps. The summary is lossy by design, so the agent will
sometimes forget a fact you told it twenty turns ago. If a fact has
to survive, promote it to long-term memory instead of hoping the
summary catches it. Short-term memory is for the flow of the
conversation. Anything you would write on an index card belongs
elsewhere. Second, the summarizer is itself an LLM call — it costs
money and adds latency. Run it async where you can, and fire it on
a token threshold, not on every turn.

Long-term: what survives across sessions

Long-term memory is whatever the agent remembers after the user
closes the tab. It cannot live in a single process's messages
list. It has to be written somewhere durable and readable from the
next request on whatever machine picks it up.

Two families. Structured facts (user.tz, user.diet) go in a
key-value or SQL store. They are cheap, predictable, and cannot
hallucinate, because there is nothing fuzzy in the loop. Everything
with a blurry edge (past conversations, uploaded documents,
episodic recall) goes in a vector store with top-k retrieval
before each turn.

The retrieval pattern is small. Recall, format, prepend:

def build_messages(agent, user_prompt):
    hits = agent.recall(user_prompt, k=5)
    memory = "\n".join(f"- {h.text}" for h in hits)
    return [
        {
            "role": "user",
            "content": (
                f"Relevant memory:\n{memory}\n\n"
                f"{user_prompt}"
            ),
        },
    ]

The trap is cargo-culting a vector store because a tutorial used
one. Vector stores cost real money, add a round trip on every turn,
drift over time, and need TTLs and a reindexing plan. Every one of
those costs is worth paying when you genuinely need semantic search
over unstructured history. None are worth paying for
SELECT tz FROM users WHERE id = ?.

The test I run before adding a vector store: can I name the query
in SQL? "Give me everything this user told me that relates to their
question" is a vector query. "Give me their timezone" is SQL. "Give
me the last three tickets they filed" is SQL, ORDER BY created_at
DESC LIMIT 3. If you can write the query on a napkin, the vector
store is a trap. Reach for it when the napkin fails, not before.

Hybrid is the honest answer in practice. Route at the planner: KV
store for facts with a shape, vector store for the genuinely fuzzy
things. Two stores, one agent, each used for what it is good at.

Scratchpad: the trajectory of the current task

Scratchpad memory is the thought-action-observation history of the
task the agent is working on right now. It is the tool calls it
made, the results it got, the intermediate reasoning. The user
never sees it. It exists so that on step six the agent remembers
which URLs it already fetched on step three.

The scratchpad is not long-term memory — it dies with the task. It
is not the user-visible conversation either, even though both live
in the same messages array inside the loop. Pull them apart:

# scratchpad.py
from dataclasses import dataclass, field


@dataclass
class Step:
    thought: str
    action: str          # tool name
    args: dict
    observation: str


@dataclass
class Scratchpad:
    task: str
    steps: list[Step] = field(default_factory=list)

    def record(self, thought, action, args, obs):
        self.steps.append(
            Step(thought, action, args, obs)
        )

    def as_prompt(self, max_chars: int = 4000) -> str:
        lines = [f"Task: {self.task}", "Trajectory:"]
        for i, s in enumerate(self.steps, 1):
            lines.append(
                f"{i}. {s.action}({s.args}) "
                f"-> {s.observation[:200]}"
            )
        out = "\n".join(lines)
        if len(out) > max_chars:
            out = out[:max_chars] + "\n...[truncated]"
        return out

Scratchpad is deliberately boring: a list of steps, a way to
format them into a prompt, a truncation escape hatch. Feed
as_prompt() into the system prompt or a dedicated trajectory
block before each step.

Why bother? Because this is where loop pathologies happen. The
agent calls the same tool five times because it forgot it already
tried. It invents a parameter it summarized away. It wanders
because nothing points back at the goal. Keep the trajectory
explicit, prune it by step count, and most of that stops being
mysterious.

The rule you can tape to the monitor

Short-term memory is bounded by the model. Long-term memory is
bounded by your storage. Scratchpad memory is bounded by the task.
If you find yourself putting a six-month user fact into the
scratchpad, or re-embedding tool calls into a vector store, or
summarizing the user dialogue into a long-term note, you have mixed
up the three. Keep them separate and most of the hard parts of
agent memory turn back into ordinary engineering.

Getting these three layers right is most of what separates an agent
that survives a second turn from one that does not. Agents in
Production walks through building and shipping the full loop:
memory, planning, tool calls, retries. Its companion, Observability
for LLM Applications, covers tracing and cost accounting for the
retrieval and compression calls above, so you can see which layer
lied when a lookup goes quiet.