Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The memory example is the easy half of the term, and it is worth naming the hard half, because the hard half is the reason memory exists at all. Elon asks what he wanted to buy, and the agent finds it, because the question names the thing to look for. Now change the request. Three weeks later Elon says: draft an offer for that coding company, the budget is whatever is left after the rocket. Nothing in that sentence points back at Monday. A memory that only works when the question names the fact will write a perfectly good offer and never bring up the one note that would have earned its keep, the part about how much.
Those are two different abilities, and they are measured separately. A study this summer ran one memory system on both. Asked directly, it found the stored fact every single time. When the later request merely depended on the fact without naming it, the fact made it into the answer roughly one time in ten. Same store, same facts, different shape of question.
So the test for whether memory works is not the one in the article, although that one has to pass first. It is this: ask the agent for something that depends on what it knows, without telling it what it knows, and check whether the answer changed. If the answer is the same one a stranger would have written, the store is full and the memory is not working.
Thanks for this comment, and yes, exactly! I presented the basic version of memory here, but as you point out, that's actually the easiest case: the request itself tells the agent what information to look for. The implicit case, where the agent has to recognize on its own that some previously stored information is relevant, is much harder. Thanks for adding this nuance! 😊
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The implicit case is where the interesting measurement problem sits too, not only the interesting engineering problem. If you score the final answer alone, a model that gets it right from general knowledge scores the same as a memory that actually delivered, and both read as a win. The cheapest correction I know is a second copy of the same starting state with the decisive fact removed, or replaced by a decoy of the same shape and length. What you report is then the difference between the two runs rather than the score of one, and nothing has to be inspected inside the retrieval step.
It catches the opposite failure as well. A system that fires on every request, including the ones the stored fact should not touch, looks strong on the cases where the fact was relevant and quietly costs you everywhere else. The difference between the two copies is the only number I have found that moves for one of those and not the other.
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The memory example is the easy half of the term, and it is worth naming the hard half, because the hard half is the reason memory exists at all. Elon asks what he wanted to buy, and the agent finds it, because the question names the thing to look for. Now change the request. Three weeks later Elon says: draft an offer for that coding company, the budget is whatever is left after the rocket. Nothing in that sentence points back at Monday. A memory that only works when the question names the fact will write a perfectly good offer and never bring up the one note that would have earned its keep, the part about how much.
Those are two different abilities, and they are measured separately. A study this summer ran one memory system on both. Asked directly, it found the stored fact every single time. When the later request merely depended on the fact without naming it, the fact made it into the answer roughly one time in ten. Same store, same facts, different shape of question.
So the test for whether memory works is not the one in the article, although that one has to pass first. It is this: ask the agent for something that depends on what it knows, without telling it what it knows, and check whether the answer changed. If the answer is the same one a stranger would have written, the store is full and the memory is not working.
Thanks for this comment, and yes, exactly! I presented the basic version of memory here, but as you point out, that's actually the easiest case: the request itself tells the agent what information to look for. The implicit case, where the agent has to recognize on its own that some previously stored information is relevant, is much harder. Thanks for adding this nuance! 😊
The implicit case is where the interesting measurement problem sits too, not only the interesting engineering problem. If you score the final answer alone, a model that gets it right from general knowledge scores the same as a memory that actually delivered, and both read as a win. The cheapest correction I know is a second copy of the same starting state with the decisive fact removed, or replaced by a decoy of the same shape and length. What you report is then the difference between the two runs rather than the score of one, and nothing has to be inspected inside the retrieval step.
It catches the opposite failure as well. A system that fires on every request, including the ones the stored fact should not touch, looks strong on the cases where the fact was relevant and quietly costs you everywhere else. The difference between the two copies is the only number I have found that moves for one of those and not the other.