Mayur Agnani

Posted on Oct 31 Edited on Oct 31 Originally published at theaicommit.com

How to Tell What Claude Code and Codex Forget When They Compact a Session

#ai #agents #testing

Engineer's view. This is a process restart with a handoff note. Whatever the process wrote down or saved to disk comes back. Whatever sat only in memory is gone, and nothing errors. A coding agent's memory is its conversation, so values it read but never wrote down vanish, and it must read them again.

TL;DR. A compacted agent keeps your instructions and its own edits, but loses most of what it only read. Whether a fact survives depends on how it entered the session, so you can predict the loss and tell the summary what to keep.

Explain Like I'm 5

You are building a huge model castle, and your table is full. So you ask a friend to write a
short note about the castle, then you clear the table.

The note says what you were told to build and which towers you finished. It does not list every
piece you looked at in the box. Those are still in the box, so you can look again.

But some pieces you borrowed for a minute, and they went back to a friend's house. If the note
does not mention them, you will not remember them, and you cannot go look.

So before you clear the table, tell your friend what to put in the note.

The Problem

You have shipped this bug already, in a system with no model in it.

Your service held a session cache in process memory. A deploy restarted it. The health check went
green and the error rate stayed flat. Then users started logging in again, because the sessions
never reached disk. Nothing crashed. The process simply came back without the state that had only
lived in memory.

A coding agent has the same split, and it restarts in the middle of your work.

Its working memory is the conversation. When that fills, the agent runs compaction: it
replaces the conversation with a summary and carries on. In my test session, 78,642 tokens became
2,632. That is 97% gone in about twenty seconds, and the session looks healthy afterward.

The agent then keeps working from the note. If a detail is missing, you see no error. You only see
it read the same file twice, or ask something you already answered. Worse, the detail may have
come from a test run or a web page that no longer exists.

You could treat this as bad luck. It is not. What survives follows a rule.

The fix: know which lane each fact is in, tell the summary what to keep, and write anything you
cannot regenerate to a file before compaction runs.

The Fix: Sort Facts by How They Entered the Session

Compaction is a rebuild, not a shortening. The new context comes from three lanes.

What reloads from disk?

Claude Code reloads the root CLAUDE.md, auto memory and the plan file. It also re-reads up to
five recent files, each under 5,000 tokens. Rules with paths: and nested CLAUDE.md files are
different: they entered as messages, so they get summarized. They load again only when the agent
touches a matching file.

What does a budget keep?

Codex's open-source compaction path keeps your recent messages word for word, up to 20,000 tokens,
walking newest to oldest. It adds a summary and re-injects its starting context. Older messages
depend on the summary.

Which facts does the summary drop?

I tested this rather than assume it. I planted facts in real Claude Code sessions, compacted them,
then asked from memory with tools off. Each of three sessions read 25 small config files.

Every instruction survived, and so did all ten edited files, even after a second compaction. The
agent said so plainly when a value was gone, even when told never to answer "unknown".

The summary kept what mattered to the work so far. In an earlier run, the last task counted timeouts
over 5000. Of six values I probed, only the two timeouts over 5000 survived.

What This Means for You

When this matters. You run long sessions in Claude Code, Codex or any agent that summarizes its
history. It matters most when the agent reads something it cannot read again: a failed test run, a
production log, a web page, or the output of a command that changes state.

How it affects you. A file on disk costs you one more read. The agent gets it back and you pay
tokens. A one-time output costs more, because the evidence is gone. The agent will then rerun the
command, or reason from a vague line in the note. Long sessions may lose more than my small test
did. Factory ran probe questions after compression on 36,611 production messages. All three methods
scored worst on tracking files, at 2.19 to 2.45 out of 5, even though each kept instructions well.

What to do about it. Start with no setup. Before you compact, name what matters:

/compact Keep the exact failing test names, error strings and ids.

A line like that, naming the values I needed, took my test from 0 to 36 of 36.

Next, watch for the moment that hurts. When the agent reads something you cannot regenerate, ask it
to save the facts to a notes file. A file sits in the disk lane.

Then fix the repeated cases. Move a rule that must always hold into your root rules file. Add a hook
that restores your working state after every compaction. Finally, audit a real session to see your
own numbers.

Implementing It

The change.

Four jobs, and one person may do all of them.

Job 1, whoever drives the session. Save anything you cannot regenerate, then compact at a task
boundary with a focus. Codex takes a compaction prompt in its config, so the same habit can be set
once there. In ~/.codex/config.toml:

compact_prompt = "Summarize this session. Keep exact error strings, ids, file paths and test names."

Job 2, whoever owns the repo's agent setup. Check which lane your rules are in. A rule with
paths: enters as a message, so its survival is up to the summary until the agent touches a
matching file again. My short test kept one, but the docs warn it can go. If it must hold across
compaction, move it to the root file. Then add a section the summarizer reads, in CLAUDE.md:

# Compact Instructions
Keep exact error strings, failing test names, ids and every file edited.

Add a hook for state that changes. A SessionStart hook with the compact matcher runs after
every compaction, and its output joins the new context. In .claude/settings.json:

{ "hooks": { "SessionStart": [ { "matcher": "compact", "hooks": [
  { "type": "command",
    "command": "git status --short; cat notes/session.md 2>/dev/null" }
] } ] } }

Job 3, whoever builds their own agent. Do not trust a summary with the file list. Keep it as data
in your loop, and put it back after you summarize.

def rebuild(history, pinned, ledger, summarize, keep_tokens=20_000):
    recent = newest_user_messages(history, keep_tokens)
    note = summarize(history, focus="exact errors, ids, decisions")
    edited = "Files edited: " + ", ".join(sorted(ledger.edited))
    return [*pinned, *recent, edited, note]

Job 4, whoever wants the numbers. code_example.py needs no model. It reads a session transcript,
finds each compaction, and checks which identifiers reached the summary, grouped by lane. Run it on
one of your own sessions:

python3 code_example.py ~/.claude/projects/<project>/<session>.jsonl

An identifier here means an exact string such as a port, path or error code. On 40 compactions
from my own sessions, 4% of the identifiers the agent only read reached the summary, against 19% of those the agent or I wrote. An exact-string match undercounts paraphrase, so
trust the gap more than the percentages.

How you know it worked.

Probe it. Fork a session right after a compaction, turn tools off, and ask for five facts you need.
Offer "UNKNOWN" as an answer so a miss is visible. In Claude Code:

claude -p --resume "$SESSION" --fork-session --tools "" \
  "From memory, no tools. Which files did you edit? What was the exact error? Reply UNKNOWN if unsure."

Before your change, count the misses. After it, the same probe should return your focus items
exactly. Also watch re-reads: the same file read twice right after a compaction means the note
missed it.

When Managing Compaction Is the Wrong Tool

When the task is short. A session that never fills its context never compacts. Spending effort on a
hook then buys nothing.

When the facts are on disk and cheap to read. Losing a value from a config file costs one more
read. Asking the summary to keep every value makes the note longer, and a long note costs tokens on
every later turn. Keep what is expensive to get back.

When you should start over. If the next task is unrelated, /clear is better than any summary. Old
context crowds the new work, however well it was compressed.

When the work belongs elsewhere. Send large research to a subagent, so the reads never enter your
main conversation at all.

My test has limits too. It used small synthetic sessions on one model, and I did not run the Codex
side. Run the probe on your own work before you trust these numbers.

Three questions before you add a hook:

Which fact, if lost, could the agent not get back?
Does that fact reach the model through a file, or only through a message?
Would a probe after compaction return it exactly?

Glossary

compaction — replacing a conversation with a shorter summary so the agent can keep working.
lane — the route a fact takes through compaction: reloaded, kept by budget, or summarized.
probe — a question asked after compaction to test whether a fact survived.
focus — an instruction telling the summary what to keep, as in /compact with text after it.
identifier — an exact string such as a port, id, file path or error code.

DEV Community