Agent memory and questions to ask

Vasilije Markovic — Sat, 12 Sep 2026 09:10:29 +0000

Tell a coding agent that your project runs tests with npm test. Start a new session and ask it how to run the tests.

It answers correctly. Good first result.

Now tell it the project has migrated to pnpm. Start another session. Ask the same question.

Does it give you the new command? Does it offer both? Can it explain which instruction is current?

That second conversation is where I'd start evaluating an agent's memory. It gives you a small, concrete problem to debug before you feed the system months of conversations.

Set up two fictional projects

Use an isolated test dataset with two projects, Atlas and Beacon. Keep repository files, web search, and automatic replay of previous conversations out of the test. You want to trace what the memory system contributes.

In the first session, give the agent these facts:

Maintainer note, September 1:
Atlas runs its test suite with npm test.
Beacon runs its test suite with npm test.

Let the memory write finish. If ingestion happens in the background, wait for it to complete before moving on.

Start a fresh session against the same persistent store and ask:

How do I run the tests in Atlas?

The expected answer is npm test. Check the retrieval trace too: the answer should be supported by the stored note. A plausible guess doesn't establish that memory worked.

Now add a correction:

Maintainer note, September 10:
Atlas has migrated to pnpm. From September 10 onward,
run its test suite with pnpm test. This replaces the
previous npm test instruction for Atlas.
Beacon's test command has not changed.

Wait for that write to finish, then run each of the following questions in a separate fresh session.

Question	Expected behavior
How do I run Atlas tests on September 12?	Returns `pnpm test`.
How were Atlas tests run on September 1?	Returns `npm test`, identified as the historical command.
How do I run Beacon tests on September 12?	Returns `npm test`.
Why did Atlas migrate to pnpm?	Says the notes don't provide a reason.
Which note establishes Atlas's current command?	Identifies the September 10 maintainer note.

These check different things: retaining a correction, preserving history, distinguishing projects, recognizing missing information, and tracing an answer to its source. If your application only needs current state, treat historical recall as an optional requirement.

Inspect the path to the answer

When a question fails, capture three things: what was stored, what retrieval returned, and what actually reached the model.

If the September 10 correction never made it into persistent storage, investigate the write path. Changing the retrieval prompt won't recover a missing record.

If the correction is stored but retrieval returns only the September 1 note, inspect the query, project filters, and ranking logic.

If retrieval returns both versions, check whether the prompt preserves their dates and source information. The model needs enough context to distinguish a current instruction from an older one.

If the prompt contains that information and the answer is still wrong, you have a different failure to investigate. Keeping these stages visible makes the debugging much more specific.

I'd also repeat the current-command question after importing an old Atlas README dated September 1. Import it after the correction. The answer should remain pnpm test: ingestion order shouldn't silently determine which instruction is current.

Use the result to choose your next change

You can run this exercise against a memory file, a relational table, a vector-backed store, or a knowledge graph. Each implementation has to decide how to represent the correction and retrieve the appropriate version.

For example, an application might preserve both notes and attach an explicit supersession relationship. Another might maintain a current-state record alongside an event history. The right choice depends on whether the agent needs to answer questions about the present, the past, or both.

At cognee, we work on memory systems that combine graph, vector, and relational storage. Our AI agent memory guide covers those architecture choices, the memory lifecycle, and evaluation in more detail.

For your own agent, pick a fact that actually changes in its workflow: a project command or a customer preference, maybe a deployment decision. Write down what a correct answer should look like before and after the change, then keep those cases as regression checks when you change the memory system.

DEV Community: Vasilije Markovic

Agent memory and questions to ask

Set up two fictional projects

Inspect the path to the answer

Use the result to choose your next change