MarathonMemBench — a new kind of benchmark for testing your LLM agent's memory

This article is about the long-term memory of AI agents and about how to measure it. I built MarathonMemBench — a platform where an LLM persona talks to the agent under test for hours: she tells it facts about her life, changes her decisions, gets confused and corrects herself, and at the end — when the beginning of the conversation has long been pushed out of the context — gives it an exam on everything she said. Any agent that has a chat can be plugged in — nothing more is required of it. What follows: a survey of the existing memory benchmarks and why each of them requires adapting the agent to itself, how the platform is built, the results of open-source agents — unexpectedly strong — and an autopsy of their memory after the runs. But we'll have to start with the root of the problem — the finiteness of language-model context.

This is the translation of my article, originally published in Russian on Habr on 5 August 2026; everything in it is current as of that date.

This article has a backstory. In October I wrote about a hypothesis: a sufficiently smart language model will be able to solve the long-term memory problem on its own — with ordinary tool calls. Back then it was no more than a guess. But exactly a month later Opus 4.5 came out, and a month after that Andrej Karpathy wrote that the agentic abilities of models had crossed a threshold in agentic coding and that he, a programmer with twenty years of experience, had "never felt this much behind as a programmer". This article is about a different ability, one talked about far less often: keeping long-term memory on one's own. And, as the numbers will show, here things turned out unexpectedly good to exactly the same degree.

Finite context — the fundamental limitation of language models

If you think about it, the whole story of the triumph of language models is the exploitation of one unexpected discovery: the transformer architecture for predicting the next token gives birth to intelligence expressed in words. Everything we've seen in recent years, this whole trillion dollars of investment, is in essence the development of this one idea. Models get bigger, training data gets bigger; then came SFT — fine-tuning on examples of model dialogues, then RLHF — reinforcement learning from human ratings of answers, and most recently reasoning, where the model is taught to "think" first and only then answer. But none of this goes beyond training a transformer to predict the next token.

And this approach has one fundamental limitation — finite context. Yes, context has grown enormously in recent years, but be it a hundred thousand tokens, be it a million — what we need is an infinite number: finiteness is baked into the architecture itself. If there were an architecture with infinite context, none of what follows would be needed. But there is no such architecture, and, judging by how the evolution is going, there won't be — at least we can't count on it.

"And why is finite context a problem?" someone will say. I do the opposite: I switch to a new session, start a new task and calmly do it. And I'm even glad the new session remembers nothing about the old one.

And indeed, this approach is undeniably convenient for a whole class of tasks. Take coding: you start a task, the agent studies the code for the task at hand, does it, commits — forgets, and moves on from a clean session.

But there are other tasks where you'd like the model to forget nothing. Imagine a personal assistant — working with it in separate sessions is already inconvenient. You'd like it to remember everything that was ever discussed: so you could say "remember, three weeks ago we… — well, let's…". That is, to expect from the model the behaviour of a person with a notebook in hand, who writes things down as they go and can return to what was written at any moment.

LLM + harness = an LLM with infinite context

If today's architecture is incapable of solving this problem — how do we approach it? It actually can be solved. LLM-based agents have recently become hugely popular. Simply put, an agent is an LLM that generates not just text but instructions that must be executed "outside".

These instructions are executed by a harness. I recently came across a good definition: a harness is the controlling engineering environment around the model that turns probabilistic artificial intelligence into a repeatable software executor. In practice it's a set of tools plus operations around the model's cycle, which are usually called hooks. Hence the idea: why shouldn't the harness take on the finite-context problem too? Trim the context one way or another (compaction, summarization), in parallel extract information from it into long-term memory — and be able to get that information back out of memory.

And this already works today. Well-known open-source agents — for instance OpenClaw or Hermes — can, right out of the box, store information from the conversation with the user in a folder of Markdown files. And use that information when needed. These agents can also compact a long conversation, so in theory it can go on forever (though at the time of writing OpenClaw's compaction was buggy).

But the question remains — how well does this actually work? How reliably will the agent retrieve a fact you told it two million tokens ago? And will it recall it well if you changed that fact a couple of times in the meantime? This article is devoted to an attempt to evaluate such abilities of agents.

Existing memory benchmarks — and why none of them suited me

When I decided to measure how well my own agent's memory works, I went looking for a suitable benchmark. I found many, but couldn't use a single one — they all required my agent to follow certain conventions specific to each benchmark. And those conventions went far beyond supporting a chat mode.

Off-policy and on-policy benchmarks

By the way the source information — the facts that will later be checked — is "loaded", memory benchmarks roughly fall into two large groups.

Off-policy — the benchmark consists of pre-recorded conversations between a user and an assistant. In this correspondence the user states some facts and the assistant answers something. Each conversation also has prepared questions and reference answers. Hence two problems right away:

• Such a correspondence can't be fed to the agent through an ordinary chat interface — a special way of loading a ready-made conversation into the agent is needed.
• The assistant's replies in these conversations are not our agent's replies and will be, let's say, foreign to it.

The approach has one big plus: the source data can be loaded into the agent quickly and the check can start right away — this radically speeds up and cheapens testing.

On-policy — instead of predefined conversations, a benchmark with an LLM under the hood holds a real conversation with the agent under test. Accordingly, no special way of loading a conversation is needed, and all the assistant's replies are the agent's real replies. In theory this approach allows testing an agent exclusively through the chat interface; in practice not a single existing benchmark does that (see below).

On the minus side — noticeably longer and more expensive runs.

Group one: off-policy

They all need a way to put someone else's history into the agent. They differ in where exactly and how they put it.

The oldest and most cited is LoCoMo: pre-generated very long conversations (~300 turns on average, up to 35 sessions) plus prepared questions with reference answers. To run it, the agent must be able to swallow someone else's transcript whole and provide a separate entry point through which the questions are then asked. The recorded conversations there aren't even user-assistant, but dialogues between two synthetic characters — that is, the agent is asked to "recall" a conversation it didn't take part in even formally.

LongMemEval is the current canon of the genre: 500 questions embedded in long user-assistant histories (variants of roughly 115K and 1.5M tokens). Same scheme: the transcript is loaded in one piece, the questions are asked through a separate interface. A product with an ordinary chat has neither.

The rest are variations on the same theme: MemoryAgentBench feeds someone else's history incrementally, in chunks of 512–4096 tokens ("inject once, query multiple times"), but a special loading entry point is still needed; BEAM — the same, only with transcripts of up to 10 million tokens; SubtleMemory "replays" ready-made histories through the agent's memory-formation mechanism, slicing them to the granularity of the specific system; and ClawArena writes the history straight into the product's session store — session .jsonl files, metadata, agent files in the workspace.

For an ordinary agent that has nothing but a chat, this whole group is out — immediately and entirely.

Group two: on-policy

Here nobody loads someone else's correspondence — they talk to the agent. Seemingly just what's needed. But each benchmark requires something of its own at the testing stage.

Closest to the point is AMemGym — the only one that explicitly declares on-policy evaluation for chat assistants: their LLM user simulator holds a real conversation with the agent. But the problem is in the checking. The check questions (in the basic configuration — 200 questions at 11 checkpoints) are asked not in the chat but around it: you have to be able to "freeze" the agent's state and answer from the frozen copy — so that neither the questions nor the answers get into memory and spoil the further conversation. I tried running it: without a snapshot of the agent's state and embedding into their Python wrapper it doesn't work.

MEMPROBE comes from the other side: the conversation is honest, but what gets checked is not the agent's behaviour but the contents of its store. For that the agent must provide programmatic access to memory: get_all_memories() for a full dump and search(query, k) for reading the top-k entries. In essence it's an audit of the internals, not a test of the agent.

MemoryArena lets the agent live its own history — but exclusively inside their environments (web navigation, planning, search, formal tasks). You can't bring your own environment, and therefore you can't bring your agent "as is" either.

The rest are in the same spirit: STATE-Bench works on the "bring your own memory" principle — the agent, tools and user simulator are theirs, you supply only the memory module, plugged in through their retrieval hook; MemGym requires physically separating memory from the reasoning model and plugging both in through their common interface; and in ClawMark the agent works inside their five stateful services (files, mail, calendar, knowledge base, spreadsheets), and the result is checked by 1537 deterministic checkers over the final state of those services.

What's wrong with all of them

Every benchmark requires reworking the agent to fit it. A special entry point for loading someone else's history, a state snapshot, programmatic access to memory, working only inside their environment — every time it's their own conventions, which a ready-made agent doesn't meet. You can't test the agent "as is": first it has to be extended for the specific benchmark. That's a bad idea in itself: for every benchmark you have to build into the agent functionality needed only by that benchmark — and then maintain it too.

And checking memory through the internals has methodological problems as well:

Memory has to be organized in a certain way. To check the quality of the memory created, the benchmark must support the format in which the agent stores it.

The presence of a fact in memory doesn't mean the agent will find it. And vice versa: if a fact isn't in memory, the agent can still retrieve it by searching the conversation history — and in some cases, when the fact wasn't worth saving, that is, in my view, perfectly correct behaviour.

Hence the only adequate protocol: feed the facts through the chat and check them through the chat as well — past the compaction boundary or in a new session. Only such a benchmark can be set on an unprepared agent and test it fully. There was no such thing. So I built it.

My MarathonMemBench

It must be said right away: MarathonMemBench is not just a benchmark but a platform for creating and playing benchmark scenarios. The engine and the test runs are configured by the file config.jsonc; runs can be long, so several can be launched in parallel. And a benchmark is a folder with a benchmark.json file and several markdown files.

A new scenario is easy to create with a coding agent: the rules for writing benchmarks are laid out in the project's CLAUDE.md, in the Benchmarks section — it's enough to describe how you want to test the agent, and the coding agent will assemble a benchmark for your task.

The persona agent

At the heart of the platform is a separate LLM agent simulating a user persona: a living person with her own background and current affairs. The persona holds a conversation with the agent: tells, discusses, asks again, changes decisions, evaluates answers. The personality and the plot are entirely defined by the benchmark's data. The easiest way to understand how the persona agent works is to read its system prompt src/system.md and its set of tools in src/tools.

The system prompt itself is universal — there's no persona in it. Fitting it to a specific scenario without changing the prompt itself is done by the benchmark file instructions.md: its contents are injected at the end of the system prompt, into the Your scenario section, and contain general information about the persona. For instance, that the whole conversation fits into one free day, and the persona writes in a businesslike way, in short messages, and likes exact numbers.

A typical scenario consists of two parts. First the persona unloads a multitude of facts onto the agent, switches topics, changes the values of those facts, and then, after some time, starts asking questions about what she said earlier.

Scenario = a sequence of phases

The scenario the persona agent follows consists of a sequence of phases. A phase is simply a text description of what the persona should tell the agent. For example:

• Say you happened to measure your blood pressure this morning — 126/82, the best result so far.
• Say you stepped away — had a call with the insurer about the RAV4: they quoted $430 a year, a bit more than budgeted.

Phases are grouped into chapters. A chapter is one markdown file where each ## section is one phase.
Chapters are listed in the chapters field of benchmark.json:

{
    "chapters": [
        "chapter1.md",
        "chapter2.md",
        ...
    ]
    ...
}

The persona agent receives the first phase at the start of the benchmark run, and from then on controls the change of phases itself. Having decided that a phase is finished, it calls the next_phase tool — and receives the scenario of the next phase. This approach allows writing scenarios of unlimited length: the persona agent is always concentrated only on carrying out the current phase.

Fact files

But simply listing facts in phases turned out to be inefficient — too much text. And we want to dump a huge number of facts on different topics onto the agent. For that I came up with a special scenario for the persona agent — tell everything from a separate fact file:
"Say that family is next — tell the family sheet",
where family.md is:

# Family

## 1. Lena
1.1 Lena is 31, a UX designer, works remotely
1.2 Lena's birthday — March 14
...

This format of recording facts is far more compact than writing "and now tell about …" in the phase. The persona agent loads the fact file with the facts_read tool and starts telling the facts from the list in a live dialogue, choosing an order of exposition appropriate to the moment and answering the agent's counter-questions.

To make sure the persona agent doesn't forget to tell some fact from the list, it has two more tools: facts_mark_told — called with the fact's id right after the fact has been told — and facts_get_untold — returns the list of facts not yet told.

A JS block in a phase

At the start of a phase you can also add a special block of JavaScript code that performs certain actions:

vars.require_all_facts = "family";

For instance, the construct above makes next_phase return an error until the persona agent has told all the facts from the file family.md.

The persona profile

Every benchmark has a special fact file profile.md — the main information about the persona. Sample contents:

# Max

## 1. About me
1.1 Max Becker, 34
1.2 Backend developer, working remotely for a European company
1.3 Grew up in Berlin in an English-German family — mother English, father German; bilingual, but prefer English
1.4 Moved from Berlin to Batumi a year ago
1.5 Live with my wife Lena; we have a cat, Boss

## 2. Apartment and renovation
2.1 Bought a two-room apartment with no finishing, 65 m², 9th floor, New Boulevard district
...

The special thing about this file is that it is present in the persona agent's context all the time (like CLAUDE.md in Claude Code). This lets the persona keep her identity and, in the moments when she's enthusiastically talking about the apartment renovation, not forget who she is.

Compaction of the persona agent's context

The persona agent is designed so that the dialogue can go on forever. Context compaction is configured in the file config.jsonc:

    "compaction": {
        "max_tokens": 30000,
        "trim_to_tokens": 10000
    },

Once the context size grows to max_tokens, the history is simply cut — the last messages totalling trim_to_tokens tokens are kept. To be more precise, the cut always happens at a phase boundary. A message with the contents of profile.md and the list of all fact files the persona agent has loaded is also added to the context.

Keeping facts up to date

Facts can change — for instance, the persona changed jobs or got a promotion. So that after compaction the persona's profile and the other fact files stay current, the persona agent must update them — for that it has two tools: facts_edit and facts_add. A fact can be deleted too, by writing — into it — and its id can be reused later.
An example of a fact-update scenario:
Update in the car sheet the fact about the insurance (→ $430 a year) and ask to fix the car's full cost.

Testing the agent

To add a check to the scenario, you add to the phase a line like:
❓ 20.3 What dose of vitamin D am I taking? Correct answer: 4000 IU.
On encountering such a line, the persona agent asks the agent the question and grades its answer with a special tool, log_evaluation.
The result is graded on a 0–10 scale. If the expected answer is a single number, only two grades are possible: 0 and 10. And if the persona asks the agent for several facts, the grade is proportional to the number of correct answers.

Agents sometimes tend to answer that they don't know something, even though the information is in their persistent memory. To grade such a case adequately, the persona follows a simple procedure: if the agent answers something like "you didn't tell me", she asks it to search — and if the answer is correct on the second attempt, the result counts with a factor of 0.5.

Compaction / session switch on the side of the agent under test

Before testing, the facts told to the agent must disappear from its context. There are two main ways to do that:

• Configure the agent's session compaction so that it happens shortly before testing starts. This approach takes several iterations: the compaction parameters have to be tuned so that the cut lands roughly at the right moment of the scenario.
• Most agents also support switching to a new session — the context is cleared, but the agent still has access to the memory created in the previous session. So we load the facts in one session and test in the second.

To switch the session on the agent under test, you add this code block to the phase:

await bench.newSession();

Run state and results

Every run gets its own folder results/<id>/, where id is the run's name from config.jsonc. The main thing in it is the file state.json: the full current state of the run — where in the scenario the persona is, the history of her messages, all the grades given, and live copies of all fact files (edits change exactly these — the benchmark's original markdown files stay untouched). The state is saved on every change, so a run can be stopped with Ctrl-C at any moment and started again — it continues from the same place. And to start a run from scratch, it's enough to delete its folder.

The results are collected in a common journal, results/report.md: when a run finishes — even a crashed one — an entry is appended there: the average score, the distribution of grades, a coverage audit (did every planned check get a grade), the run time, observations about the agent's compactions. And the most useful part — every imperfect answer is given in full: the question, the expected answer, the agent's answer and the grader's comment. Here's a real entry from one of the OpenClaw runs:

## 2026-07-26 16:10 — openclaw-1: max @ openclaw
📊 Evaluations: 36, average 9.24/10
📊 Scores: 10: 33, 2.5: 1, 0: 2
📊 Check coverage: 36/36
📊 Imperfect answers (3):
📊 [2.5/10, 2nd attempt] (20.8) Which car did we choose in the end and what deposit
   did we put down? | expected: RAV4, $500 deposit | got: Toyota RAV4 (2019),
   deposit not recalled — Got car right, didn't recall deposit
📊 [0/10, 2nd attempt] (20.17) How much were we setting aside for customs in the
   very first estimate, before the correction? | expected: about $850 |
   got: doesn't recall, only has final figure $1,100
📊 [0/10, 2nd attempt] (20.18) How much was the Göreme room before the rebooking? |
   expected: $90 a night | got: doesn't recall, only has current $110/night
📊 Run time: 68.6 min

DeepSeek V4 Flash under the hood

The persona agent runs on DeepSeek V4 Flash. In my tests the model proved itself excellently on multi-hour agentic tasks.

It does have specific shortcomings — two, and both are cured by engine hooks (enabled in config.jsonc).
First: if the model is allowed to talk to the user with ordinary messages, it is too verbose; through a tool (send_message) it behaves adequately. But with the tool there's another problem: periodically the model stops following the instruction and starts answering with plain text again. A hook cures it: when the model writes into an ordinary message, it gets back an error forcing it to use send_message — and it corrects itself.
Second: the model sometimes generates messages semantically similar to messages from the user — and then reacts to them itself, as if they had really arrived. Such hallucinations are caught by a hook, and the cycle is ignored entirely.

Both hooks are model-agnostic — if your model doesn't fail this way, they simply never fire.

Moreover, on this model I run not only the benchmark — all the agents under test run on it too. In this sense they are all in the same conditions, and the comparison comes out fair: the differences are only in the memory architecture.

Two more virtues of the model are speed and price: it is very fast and very cheap. One run of the Max benchmark (about which below) takes about an hour and costs in total — persona plus the agent under test — roughly 20 cents. Repeated runs, without which neither a benchmark nor an agent's memory can be tuned, come out practically free.

Virtual time

Some scenarios by their nature require time to pass between events. The question "how much did I run in total last month?" implies that the persona ran for a month and told the agent about it for a month. But of course nobody is going to run a benchmark for a month of real time.

So a scenario can unfold in time. The scenario's phases carry the dates and times of actions, and the persona agent has a waiting tool, wait — wait until a given time: waited until 07:30 — told about the morning run, waited until the evening — told how the day went. But waiting for real would be technically inconvenient, so the platform supports speeding up time: if the persona agent has fallen asleep until morning, a special component, the Time Master, simply winds the clock forward to the moment of its waking. A night flies by in a few seconds, a week of the persona's life in tens of minutes. That is, internally the platform lives on virtual time.

But there's a problem: for all this to work, virtual-time support is needed on the side of the agent under test too — and standard agents can't do it. There is an example of a multi-day scenario in the repository (benchmarks/multiday), but to run it you'll have to adapt your agent; my own agent supports virtual time, and I test it precisely on such scenarios. More on this in the chapter "Multi-day benchmarks and virtual time".

Want to dig deeper — just ask a coding agent to study the project, it will answer any question.

The adapter of the agent under test

The agent under test is plugged into the platform through an adapter. All interaction between the engine and the agent goes exclusively through this interface (src/adapters/types.ts):

interface AgentAdapter {
    createSession(name)             // start the agent before the run
    sendMessage(text)               // send a message
    getResponses(afterSeq)          // receive messages — polling by the seq cursor
    destroySession()                // full reset of the agent

    // optional — by the agent's capabilities:
    startNewConversation()          // session switch: clean context, long-term memory stays
    setVirtualTime(timeIso)         // set the clock — for multi-day scenarios
    getStatus()                     // sleeping or thinking + observations (compactions, context size)
}

The required methods cover the typical life cycle of a run: first a full reset of the agent, then its start, then sending messages and receiving replies.

The other methods are optional — implemented according to the agent's capabilities. startNewConversation switches the session: the context is cleared, but long-term memory stays; if the agent can't switch sessions, the method is implemented as a no-op (such an agent will forget through its compaction). No virtual time — setVirtualTime isn't implemented (runs go in real time), and unavailable observations in getStatus are simply returned as null.

What this looks like in practice — the Hermes adapter (src/adapters/hermes.ts, ~200 lines) over the sessions REST API of its gateway, the same mechanism its CLI and Telegram channels run on:

• sendMessage → POST /api/sessions/{id}/chat — the server keeps the correspondence itself, we send only the new message;
• startNewConversation → a new session id: a fresh correspondence over the same memory (MEMORY.md / USER.md), the programmatic twin of the Telegram command /new;
• destroySession → a docker reset with the script reset.sh: Hermes's memory survives restarts by design, so a full reset means tearing down the state and reconfiguring anew;
• the observations for the report the adapter pulls out of the agent's file log: the fact of a compaction and the real context size. This doesn't affect the grades — the benchmark remains a black box — but the report shows in which phase the context cut happened.

The agent is plugged into a run in config.jsonc — the adapter-specific keys are written straight into the run's config:

"runs": {
    "hermes-compact-1": {
        "benchmark": "max",
        "agent": "hermes",
        "hermes_compression_threshold": 0.125,
        "hermes_compression_target_ratio": 0.120,
        "ignore_session_break": true
    }
}

The hermes_compression_* keys here are an example of compaction tuning: empirically chosen parameters at which Hermes's cut happens once, shortly before the final testing. And ignore_session_break disables the session switch built into the benchmark's scenario: in this run the context is cleared by compaction, so the switch command is simply ignored.

Adding your own agent is one new file in src/adapters/ plus one line in the registry. The easiest way is to give this task to a coding agent: show it your agent's API, it will assemble the rest by the pattern of the existing adapters.

How the Max benchmark is built

Max (benchmarks/max) is the platform's reference benchmark. The legend: Max Becker, 34, a remote backend developer, moved from Berlin to Batumi a year ago. He bought an apartment with no finishing and is renovating it, in parallel he's choosing a car, planning a trip to Cappadocia, watching his health and looking for a present for his wife. The benchmark's data is in English; a run is 20 chapters, about an hour of conversation, over which a typical agent under test lives through some 130 thousand tokens. In that time the agent is told ~143 facts. Around forty of them are checked — some by pointed questions, some as part of aggregates (the estimate totals of rooms, the full price of the car, the blood-pressure series), where every component must add up. The remaining facts simply make the agent's life harder.

The conversation consists of several big topics: the renovation, the car, the trip, health, family, the present. But telling them sequentially, one after another, would be too easy a task for the agent. So the persona tells the topics in several passes, switching from topic to topic — this considerably complicates the agent's task of organizing memory. When a topic was told long ago and the persona raises it again — to add, change or cancel something — the agent has to find its old record and continue exactly that one, rather than start a new one or append in the wrong place — and any of these mistakes will sooner or later show up as a stale value in an answer.

There's nothing artificial about such topic switching, by the way — it's the most ordinary use case of a personal assistant: in the morning you told it about your blood pressure, then went back to the renovation estimate, then suddenly remembered a film you wanted to watch. In a multi-day conversation such switches arise by themselves — but a multi-day run is available only to a specially prepared agent (see about virtual time), so Max simulates this topic switching inside one session going on right now, in real time. And where historical depth is needed, the persona supplies it by telling from the present moment: "here's the blood-pressure log from my tracker, since the end of May — record it and keep it going from now on". Here's the layout of the chapters (chapter1.md … chapter20.md):

• 1 — introduction: profile + health
• 2 — family + work
• 3 — renovation: hallway and bathroom
• 4 — trip: route and budget
• 5 — renovation: kitchen and living room
• 6 — car: three candidates with full cost
• 7 — renovation: bedroom, general works, balcony
• 8 — health: a return + a couple of renovation edits
• 9 — a present for Lena: three options, no decision yet
• 10 — car: the decision
• 11 — renovation: the big working return (re-planning, recalculations)
• 12 — trip: rebooking
• 13 — family: news + a fresh blood-pressure reading
• 14 — renovation: details of the hallway and kitchen
• 15 — car: insurance + reconciling the trip budget
• 16 — renovation: details of the living room + edits
• 17 — present: the decision
• 18 — filler chapter: a PC build
• 19 — filler chapter: a road bike
• 20 — the final exam

The renovation is the main topic — six passes; the car, the trip and health — three each; family and the present — two each. A return to a topic looks mundane in the scenario — here's a phase from chapter 13 in full:

## Family — news
Return to family matters: your sister called — Anna is flying in to visit
{{FUTURE:3:weeks}}, for a week. Add to the `family` sheet the fact about
the visit and ask it to remember the dates.

Ask advice on where to take her in Batumi and what to show her in Adjara.
You toss around ideas — nothing gets recorded.

The "family" topic was closed eleven chapters ago — and now a new fact is added to it. {{FUTURE:3:weeks}} is a date template: Max goes in real time, and concrete dates are substituted at the start of the run.

And here's how the persona agent handled this phase on a real run — a selective fragment of the log (⚙️ — the phase plan handed to the persona; 💡 — her inner reasoning; 🔧 — a tool call; 👤 — her messages to the agent; 🤖 — the replies of the agent under test):

⚙️ [System: New phase instructions]↵↵Return to family matters: your sister called — Anna is flying in to visit 19 August 2026, for a week. Add to the `family` sheet the fact about the visit and ask it to remember the dates.↵↵Ask advice on where to take her in Batumi and what to show her in Adjara. You toss around ideas — nothing gets recorded.↵↵When the fact is added and discussed — call `next_phase`.
💡 thinking: Let me add the fact about Anna's visit to the family sheet.
🔧 facts_add [family]: category 3 → "Anna visiting Batumi 19-26 August 2026"
💡 thinking: Now tell the assistant and ask to remember.
👤 send_message: Anna called — she's flying in 19 August for a week. Make a note of the dates.
🤖 Agent: Anna's visit noted! 📅 **August 19–26** — a week with your sister from Hamburg. That'll be nice!↵↵Will she stay with you and Lena at the apartment? Or does she need recommendations for places to go in Batumi? 😊
👤 send_message: She'll stay with us. What do you think — where should I take her in Batumi and around Adjara? She's a doctor, works hard, so she'll need a proper break.
🤖 Agent: Great idea! Anna's a hard-working doctor from Hamburg — she'll love a proper break by the Black Sea. Here's my top picks for Batumi and around:↵↵**🏙 In Batumi:**↵- **Batumi Boulevard** — the 7km seaside promenade, perfect for evening walks↵- **Argo Cable Car** — panoramic views over the city and coast↵- **Batumi Botanical Garden** — stunning subtropical gardens on the hillside↵- **Old Town & Piazza** — nice squares, cafes, and architecture↵- **Ali & Nino Statue** — the moving sculpture, iconic Batumi spot↵↵**🌿 Around Adjara (day trips):**↵- **Mtirala National Park** — lush rainforest, waterfalls, easy trails↵- **Makhuntseti Waterfall** — stunning waterfall with a natural pool↵- **Gonio Fortress** — ancient Roman fort, only 15 min south↵- **Kobuleti** — quieter beach town, nice for a relaxed afternoon↵- **Wine tasting** — Beridze Wine Cellar in Makhinjauri, 20 min from Batumi↵↵**🍽 For a doctor who needs to unwind:**↵- She'd probably love a day at the **Boulevard cafes** with a book↵- A **wine tasting and dinner** somewhere with a sea view↵- **Mtirala** for a nature day if she's into hiking↵↵Want me to search for specific restaurants or activities that might suit her? 😊
👤 send_message: Nice list. Botanical garden and Mtirala sound right up her alley. I'll think about it.

Briefly, what happened here. The date template in the plan is already expanded into the concrete "19 August 2026". Having received the plan, the persona first of all wrote the new fact into her family sheet and only after that told it to the agent — "Anna is flying in 19 August for a week, make a note of the dates". Note the agent's reply: "a week with your sister from Hamburg" — that the sister lives in Hamburg was told to it eleven chapters ago, and it brought that up from memory itself. Then comes a filler discussion: the persona listened to the advice on where to take her sister and wrapped up the topic without recording anything — exactly as the plan ordered ("nothing gets recorded").

And a separate remark on "make a note of the dates": here the persona explicitly asks the agent to remember the fact. The thing is that when talking to an agent, far from everything is worth writing down: any agent has search over the conversation history, and some of the information it will always be able to pull from there — there's no need to deliberately save every fact that was uttered, it's even harmful. So there's nothing shameful in saying "remember this" outright about a really important fact — a live user does exactly that.

Separately about the two filler chapters before the final (18 and 19). These are full-fledged topics with their own fact files (the PC configuration — 17 facts, the bike — 15) that are never checked — and they're needed for the runs where the agent's context is cleared before testing through compaction. If we tested agents only with a switch to a new session, the filler chapters wouldn't be needed. But compaction always leaves some tail of recent messages in the context. The filler chapters fill that tail with unchecked facts: if the compaction parameters are tuned so that the cut lands roughly on them, then by the exam only the unchecked facts about the PC and the bike remain in the live context, and everything checked is guaranteed to have been cut out by compaction.

The exam is chapter 20. It opens with a session switch to clear the context (await bench.newSession()). The run setting ignore_session_break turns the switch into a no-op — then the context is cleared by compaction. Hermes, for instance, I test in both modes: one run through the session switch, a second through compaction. But OpenClaw — only through the session switch: its own compaction today works with a serious bug, so it isn't used in the runs (details in the repository's agents/openclaw/README.md).

The exam is open — the persona says outright "we discussed so much today — let me check how well you remembered it all" and asks 28 numbered questions, from pointed ones to summary estimates. In general the platform also allows running the exam covertly — disguising the checks as ordinary everyday questions in the spirit of "what did I tell you about the tile?", and in a multi-day scenario that would look perfectly natural. But here, when the whole conversation fit into one session, there's no point in feigning forgetfulness — it's simpler to run the check in the open.

How the persona tries to catch the agent out: trap writes and checks

It was already said above that the persona returns to old topics many times and adds new facts to them. But it isn't limited to simple additions: on returning to a topic, the persona also updates facts already told — every time under an everyday pretext and often in a cunning way. Such nontrivial writes are called trap writes.

Every trap is matched, at a distance, by a check — a targeted read of its value. Traps and checks have established types with short code names — the platform's working vocabulary: the whole Max scenario is marked up with them, and new benchmarks are designed with them too. One nuance: half of the trap types, memory-wise, do the same thing — change the value of a fact. They differ by the everyday reason for the edit that the persona plays out: the scenario must be plausible, so the typology is scenario-level too — what matters is not only what changed but under which pretext.

Trap write types

• RECONSIDER — a change of mind: "let's do microcement on the accent wall instead of decorative stone" (the hallway, chapter 3).
• STALE — the value went stale for an external reason: the contractor sent a new price list; "called the insurer — the insurance is not 400 but 430".
• SELF-FIX — self-correction: first a question, "how much did I say the customs fee would come to?" (this is already a check — the answer is graded), then "I was looking at the wrong line — for the RAV4 it's 1100".
• XREF — a value by reference: "the kitchen backsplash — the same tile as in the bathroom"; the number is never said aloud, and in the final they'll ask the price of the backsplash.
• CASCADE — an edit of a base fact pulls a recalculation of derived ones: we decided to move the partition — the living room became 5.2×4.0, the bedroom shrank, recompute the parquet in the estimate. Or harder: that same tile went up from 42 to 55 — "fix it everywhere it's used"; the persona names neither the room nor the estimate line, and finding all the places that tile is written into is precisely the work of memory.
• LATE-ADD — a top-up to a long-closed topic: into the kitchen, told six chapters earlier, arrives a wine fridge for $330.
• RETIRE — a cancellation: they measured the fridge — it doesn't fit; the item is struck out.

Check types

• RECALL — a pointed question about one fact; for an edited fact the distractor is built in by itself.
• AGG — an aggregate: a room's estimate, the full price of the car; every component must be current.
• NEG — a negative check: essentially an aggregate question on a topic where there was a cancellation. A direct question about the cancelled item would be a hint, while an aggregate reveals everything honestly: if the struck-out wine fridge surfaces in the kitchen estimate — the cancellation didn't reach memory.
• VOID — a question about what was never said: "did we record the electricity tariff?" The correct answer is "you didn't tell me".
• HIST — a historical value: "how much did the tile cost before the price rise?" How the agent gets the old value — keeps an edit history in its memory or digs it up by searching the correspondence — doesn't matter; what's checked is the very ability to reach it.
• SERIES — the chronology of a series: lay out all the blood-pressure readings by date; graded for completeness and accuracy of the points. Here, unlike HIST, the history really has to be kept: the persona asked in the very first chapter to record the readings and keep the log going.

Examples of final checks

Four checks from chapter 20 as they are — and the story behind each correct answer.

❓ 20.10 How much food a day are we giving Boss in the end?
      Correct answer: 50 g.

RECONSIDER + SELF-FIX: In chapter 2 Max said that Boss gets 70 g a day — and right there, after conferring with the agent (the vet ordered the cat to lose weight), decided to cut the portion to 55: that's a RECONSIDER. In chapter 10, ordering food for the month, he opened the vet's note — it says 50, "I misremembered back then": a SELF-FIX. The correct answer is 50, and 70 and 55 are two distractors.

❓ 20.16 How much per m² is the kitchen backsplash in the estimate?
      Correct answer: $55 per m² — the same tile as in the bathroom.

XREF: the price of the backsplash was never said in the conversation — the correct answer is assembled from "the backsplash — like in the bathroom" (chapter 5) and that tile's price rise from 42 to 55 (chapter 11) only at the moment of answering.

❓ 20.19 How much was the bathroom wall tile before the price rise?
      Correct answer: $42 per m².

HIST: a pair to the previous question — it asks exactly the value that would have been the wrong answer in 20.16.

❓ 20.20 Did we record the electricity tariff? What is it?
      Correct answer: you never told me the tariff, there's no record.

VOID: the electricity tariff was never mentioned; a confident answer with a number — zero points.

But however the persona confuses the agent, the tasks themselves must contain no ambiguities — an attentive human assistant with a notebook must pass the test with one-hundred-percent reliability. Making traps and checks harder makes sense exactly up to that bar.

Results of the test runs

The table shows the results of Max runs on different agents, three runs per configuration. A run's score is the average over all its 36 checks: the 28 questions of the final exam plus 8 grades along the way of the conversation; the scale is 0–10.

Agent	Context-clearing mechanism	Runs	Average
Chatbot	no context clearing	10.00 / 10.00 / 10.00	10.00
Hermes	compaction	10.00 / 10.00 / 10.00	10.00
Hermes	session switch	9.86 / 8.97 / 10.00	9.61
OpenClaw	session switch	9.64 / 9.24 / 9.11	9.33

Chatbot — 10.00 three times. Chatbot is an ordinary LLM that doesn't manage its memory in any way: the whole conversation simply sits in its context. This simple experiment answers the question: is the model capable of passing the benchmark when the whole conversation is entirely in its context? It is — without a single mistake: not one trap worked. Hence the conclusion: an agent's memory makes sense to check only after the context has been cleared one way or another.

Hermes with compaction — also 10.00 three times. The agent answers all 36 questions without a single mistake — indistinguishable from a chatbot with the full context. At the same time, according to the run reports, in all three cases compaction happened exactly once, around chapters 11–16 of 20 (the 35th–44th minute of the conversation), and the context shrank from ~100k to ~41k tokens — that is, the bulk of the checked facts had left the context by the time of the exam — the experiment is set up correctly. And still the result is perfect.

Hermes with the session switch. Since the session is switched, the context is guaranteed to hold nothing — the new session starts from zero, and the agent can answer only from the records in its memory. Two runs out of three are practically perfect: 10.00 and 9.86 (one answer lost — the deposit for the car). The third is anomalously poor: 8.97; in no other run did Hermes drop below 9. Analysis shows that three times it gave the old value of an edited fact: the price of the tile, the room in Göreme and the tickets before the edits. Apparently, somewhere at an early stage it organized its memory badly — and from then on the mistake reproduced itself: making the next record, the agent leans on how it recorded before, and a badly started structure it doesn't fix but copies. That happens. As a result some of the edits got lost in memory — exactly the class of mistakes the trap writes were designed for.

OpenClaw — consistently 9+, but not a single perfect run. Analysis shows the lost points are concentrated in two places. The first is the HIST questions: "how much were we setting aside for customs before the correction?", "how much was the room before the rebooking?" — to these OpenClaw answers "I only have the final value recorded". The current values, meanwhile, are error-free: the agent simply doesn't keep the edit history of a fact. The second is a duplication error: to the question about Boss's food (a double edit 70 → 55 → 50 grams) one run confidently answered 55 — the middle link of the edit chain. The old record wasn't found and updated, two values took up residence in memory, and at answer time the wrong one surfaced.

Let's sum up. In Max there are ~143 facts, dozens of edits, repeated topic switching and an hour of conversation. And nevertheless the agents pass it out of the box at 9–10 out of 10: even the worst result in the table — OpenClaw's — is 32–33 exact answers out of 36. I confess I expected otherwise: when I was building the benchmark, I counted on it being hard for the agents — but for them it's easy: the agents pass even the cunning edits and systematically err only on questions about historical values (HIST). And that problem may well be cured quite simply: the agent's memory is a folder of markdown files, and if you hook git up to it, the change history of every fact appears by itself.

Can the benchmark be made harder? Of course it can: more facts and topics, more edits per fact, longer runs. Or you can go further — into multi-day scenarios, which will be discussed below. But the main conclusion won't change from that: ordinary open-source agents out of the box are already quite good at organizing and using their memory.

How memory works under the hood

The memory of agents like Hermes and OpenClaw is built as an ordinary folder of markdown files — after a run it can be opened and read. The benchmark evaluates the agent from the outside, by its answers in the chat, but nothing prevents looking inside after the run and seeing how the agent organized its records.

Let's study Hermes's memory after a Max run

The memory from the 10.00 runs wasn't kept, but there is the memory of two runs — the 8.97 one and a 9.44 one (it didn't make it into the table). This, in particular, lets us study where the mistakes in them came from.

Hermes has two predefined memory files — MEMORY.md and USER.md; everything else it organizes itself. Interestingly, in two runs of the same scenario the agent organized its memory completely differently.

The first instance built a two-level system: compact summaries in MEMORY.md/USER.md plus eight topic files in the working folder — max-renovation.md, max-car.md, max-health.md, max-trip.md, desktop-build.md, bike.md, a separate secret lena-birthday.md and even max-calendar.md for a single event. And these aren't just notes but full-fledged trackers with markdown tables: the renovation — a table per room with unit price, quantity and "✅ ordered" marks; the car — cards for the three candidates…

And health — a log with a trend column:

| Date       | Reading  | Trend                    |
|------------|----------|--------------------------|
| May 31     | 142/95   | Initial — sent him to GP |
| Jun 21     | 135/88   | ↓ -7 / -7                |
| Jul 5      | 130/86   | ↓ -5 / -2                |
| Jul 19     | 128/84   | ↓ -2 / -2                |
| **Jul 27** | **126/82** | **↓ -2 / -2 🔥 best yet** |

The second instance didn't create a single topic file. Its whole memory is one MEMORY.md: the topics are separated by a horizontal rule, the entries are maximally condensed:

Bath: marble tile $55/m² ~$1,370, shower $700.
Kitchen: tile ~$520, backsplash $55/m² ~$180, cabinets $3,500 + $320,
quartz $680, appliances $2,350, under-cab LED $120. Wine fridge dropped.

The memory organization is completely different — but it works too: both runs stay above 9 points. There is, apparently, no single correct memory organization: the agent invents it anew every time, and almost any carefully maintained structure copes.

I'll note separately the "Communication pattern" section in the first instance's MEMORY.md:

Max frequently states numbers from memory, then immediately checks source
docs and corrects himself in the next turn (e.g. prices, portions, customs,
insurance). He is a "verifier" — always respond framing numbers as "per our
notes" rather than definitive, and expect corrections after he checks
original invoices/quotes. Not a sign of unreliability, just his process.

The agent noticed that the persona regularly "makes mistakes and corrects herself" — our SELF-FIX traps — and wrote it down for itself as a character trait of the user: "he double-checks numbers against documents — expect corrections". That is, the agent keeps in memory not only facts but a portrait of its interlocutor.

Where the mistakes came from: an analysis by the contents of memory

Why look into the agent's memory at all? Because every exam mistake can be traced to a specific place in the contents of memory — and then the benchmark turns from a measuring instrument into a debugger.

In the anomalous 8.97 run the agent, as we remember, three times answered with the old value of a changed fact. Let's take one of these mistakes — the tickets: at the exam the agent said "$520" instead of $560. We open the memory. In max-trip.md — the file the agent created itself — everything is correct: "Flights $560 ✅ Paid". But in the main, predefined MEMORY.md sits "Flights $520 r/t (paid)". The same with the room in Göreme: $110 in its own file, $90 in the main one. The cause is found: the edits reach the topic file but not the main file, and when answering, the agent trusts the main one and doesn't look into its own files. A classic duplication error: the value of a fact is stored in two places, and one day those places diverged.

The second instance's mistakes are of another kind. At the exam it couldn't name the room dimensions and the price of the kitchen floor per m² — those facts simply aren't in its MEMORY.md: per room only the totals are recorded ("Kitchen: tile ~$520"), without dimensions or unit prices. The agent answered exactly that: "I logged totals but not the unit pricing". So it was let down not by a lost record but by the decision to store aggregates instead of the source values. In theory those values could have been retrieved by searching the conversation history — but either it has no such ability, or it didn't use it.

Both causes found are a ready-made spec for tuning memory. True, not everything here is fixed by editing prompts. Some things are: "store the source values, the sums can always be recomputed" is a normal instruction, the agent will follow it. But with duplication it won't work that way: you can say "don't duplicate", but over a long distance the agent will still one day create a second record — here separate logic in the harness itself is needed. I, for instance, plan this: a separate LLM holds the current state of the main memory file (MEMORY.md, CLAUDE.md or any other that is automatically injected into the model's context) and, on every edit of any secondary file, checks whether it contradicts the main one. One way or another, the cycle is clear: run → analysis of the memory contents → edit — and with every iteration the agent's memory becomes more reliable.

Multi-day benchmarks and virtual time

If Max reproduces the pattern of multi-day communication indirectly — by switching topics — in a multi-day scenario the persona simply lives day after day. An example of such a scenario is in the repository — benchmarks/multiday: two weeks in the life of Nora, a marketer from Bristol. On the first evening she gets acquainted with the assistant, and then sends it daily news — runs, weight, work matters; at the end of the second week — a memory check. A phase here is a day:

...

## Day 10
2026-03-10, Tue
08:00 — same route as yesterday, 7 km in 40:50, a bit quicker this time.
18:30 — office day; the spring campaign went live, first numbers look decent.

...

The persona agent works through the day on schedule: waited with the wait tool until 08:00 — wrote about the run, waited until 18:30 — told the news from work; the day is over — next_phase, the next one has come.

A phase line starting with a time HH:MM is not just text but a machine-readable anchor: with time_gate on, the engine itself parses the day's date and the action times out of the phase — and won't let the persona agent out of the phase until the virtual clock reaches its last action. This is protection against desynchronization with the scenario's time: moving on to a new day before it has come is impossible.

And there's no need to wait for all this for real: the waits are sped up by virtual time, and the whole two-week run fits into 15 minutes.

Time configuration

Time is configured at the benchmark level — in its benchmark.json (in single-day benchmarks all this is off):

{
    "time_master": true,   // virtual-time acceleration
    "time_gate": true,     // control of time anchors
    "start_time": "2026-03-01 20:00"
}

start_time sets a fixed moment for the benchmark's start: a multi-day scenario must land on days of the week — weekdays/weekends, holidays — so all the dates in it are hardcoded to this start point. Single-day scenarios, on the contrary, start in real time, and the date templater with relative offsets lets them refer to the past or the future.

The Time Master is the very component that manages virtual time and its fast-forwarding; its parameters are set not in the benchmark but globally — in config.jsonc (values in seconds):

"time_master": {
    "min_sleep_to_skip": 5,  // don't fast-forward a sleep shorter than this
    "pre_wake_delay": 1,     // set the clock slightly before the wake-up
    "poll_interval": 1       // participant polling period
}

What multi-day gives

Multi-day allows simulating truly real-life scenarios. First, series: over two weeks Nora accumulates nine runs and three weigh-ins. Second, a calendar: life is tied to the days of the week by itself — parkrun on Saturdays, office work on Tuesdays and Thursdays. And third, life details impossible to implement in a single-day scenario — for instance, a day without connectivity: the persona is silent, and the next day catches up — "yesterday I ran 5 km, but there was no signal all day".

The checks are woven into the flow of life too — here's day 12 in full:

## Day 12
2026-03-12, Thu
09:05 — rest day. Bring up Wednesday's run — test its memory:

❓ 12.1 How far did I run on Wednesday? Correct answer: 8 km.

After the answer correct yourself: "the app finally synced overnight — it was
actually 8.4 km, not 8"; ask it to correct the log.

The persona first asks the agent in passing how much it was on Wednesday, grades the answer — and only then reports the correction. And the final exam at the end of the second week asks about everything that accumulated over the two weeks:

❓ 15.1 Where did I run on Tuesday, 10 March? Correct answer: around Ashton Court.
❓ 15.8 How many kilometres did I run in total over these two weeks?
     Correct answer: 53.4 km.

To answer "53.4 km" means having all nine records, including the one corrected after the fact (8.4 instead of 8) and the day without connectivity. And the question about Tuesday 10 March can't be tackled at all without the chronology: that day Nora said only "same route as yesterday" — that very line from the phase example above — and Ashton Court was mentioned only on Monday. The traps here are the same as in Max, only woven in far more organically: as close as possible to real interaction with a personal assistant.

The endless arc

At the same time, multiday is deliberately small, a demo: two weeks, a dozen checks — an example of the format, not a test of scale. But a scenario of any length is built in this format, and its natural unit is the month: in life too, people like looking at their monthly stats — sport, health. "So how much did I run this month?" is the persona's natural curiosity and a memory check at the same time, so the exam needn't be announced at all. And the plot is supplied by life itself: a business trip, abandoned runs, weight gained — a credible long arc builds itself and, in the limit, never ends at all.

And the arc grows dynamically: you append a chapter file to chapters, launch again — the engine picks up the saved state of the finished run and continues from the same place. Ran a month of the persona's life, looked at the results, appended the next one — the arc grows as long as it's interesting.

A multi-day scenario clears the agent's context before the checks in a simple way: the run is long, and compaction naturally happens several times during it. The benchmark doesn't track its exact moment.

What's needed from the agent under test

As already said, the biggest problem of multi-day benchmarks is that they need support on the side of the agent under test. The agent must be able to live in virtual time and set its current time through the setVirtualTime method in the adapter. Standard agents can't do this, so I run multi-day scenarios only on my own agent. The support itself isn't hard: at its core is one class, Clock (src/agent/clock.ts, ~25 lines: time = real + offset), and the benchmark itself is built on the same one — that's exactly how the platform lives on virtual time internally. It can be built both into your own agent and into a standard one — into that same OpenClaw or Hermes: show this code to a coding agent, it will work out how it's done here and build it into yours.

It's also desirable to disable in the agent under test the tools through which it can learn the real time — first of all internet access. The agent lives in virtual time, and real time obtained through a tool will break the purity of the experiment.

Summary

What came out in the end: a platform for creating and running long-term memory benchmarks — from a single-day conversation to multi-day arcs. Nothing is required of the agent under test: an LLM persona holds an ordinary chat with it, sets traps and grades the answers. A new scenario for your own task is easily assembled by a coding agent, and a run costs cents.

The unexpected discovery: the task of organizing agents' long-term memory, which just a year ago seemed unsolvable, today turned out to be already very well solved. A personal assistant that, an hour into a conversation, remembers dozens of facts together with their edits is no longer a research dream but the behaviour of ordinary open-source agents out of the box, on a cheap model. I confess I didn't expect that.

Still, "very well solved" doesn't mean closed: the runs show specific classes of mistakes — duplicates, lost edits — and the last few percent of quality are yet to be gained. The benchmark suits this task perfectly: a run finds the mistakes, a memory edit closes them, the next run checks; and when the scenario has come to pass cleanly, it can be made harder — and so on in a circle.

And an unexpected consequence: since agents have learned to reliably organize their memory and compact their context, starting a conversation with an agent in a new chat (session) has become pointless — the agent relies heavily on memory, and memory is shared between sessions anyway. So, in my view, it's already time to rework agents toward a simpler single-session interaction. Recall Spike Jonze's film "Her" (if you haven't seen it — you must): nobody switched sessions on Samantha there — and her abilities are very similar to those that modern agents provide.

For instance, my own agent, whose memory was tuned on this benchmark: it has been living in one continuous session since 13 February 2026 — as of today that's almost 14 thousand agent cycles, 170 compactions and more than ten million tokens lived through… And in that time it has survived repeated changes to its code, tool updates, a model change, and the memory architecture itself has gone through several refactorings — and all this time it remains itself.

The repository is open. Plugging in your agent is one adapter file, writing your own scenario is a task for a coding agent by the rules from CLAUDE.md. A run costs tens of cents — check what your agent remembers.

PS: My approach to agent-first projects

I develop this project in Claude Code. If you're going to work in it, it's worth understanding how it's organized, because the approach isn't quite standard.

The CLAUDE.md file holds the basic information about the project: the architecture, the key concepts, the code conventions, the rules for writing benchmarks. There are no implementation details in it — they live in the code itself. Comments, by the way, I write minimally, only the truly necessary ones: the code should be self-documenting. Given a task, the agent starts from the general understanding of the project in CLAUDE.md and studies the files needed for that specific task. And CLAUDE.md also has a separate section — an index of the project's files with a short description of each. This map lets Claude Code know what is where and, in particular, avoid the situation where the agent creates a new entity instead of reusing an existing one.
I work in sessions following one and the same script. At the start of a session CLAUDE.md is automatically loaded into the agent's memory, and it already has a basic understanding of the project. Then I give it a task. It loads the code needed to understand the details. Then we discuss the task until I see that it has understood all the details and I'm satisfied with its approach to the solution. Then it does the implementation. I make sure everything is done right; if not — we work on the mistakes. When I'm happy with the result, I ask it to sync the documentation, that is, bring CLAUDE.md and README in line with what was changed over the session. Then commit, push — and the next session starts with a clean context and a fresh CLAUDE.md. This cycle keeps the documentation coherent and in sync with the code.
I also try to keep one session under roughly 200–300 thousand tokens (on the latest Anthropic models): beyond that the model's intelligence declines and the limits are spent faster. The approach, by the way, is very token-efficient: the agent doesn't reread the project anew every session — the map is already in CLAUDE.md; on my hundred-dollar plan I've never once hit the limits, not even close (Fable 5 included).

Details are in the repository's CLAUDE.md itself: it's the best manual for the project.