DEV Community

Discussion on: Agent memory needs more than vector search

Collapse
 
max_quimby profile image
Max Quimby •

The "most similar memory isn't always the right memory" line is the whole game. The example that stuck with me is the one you gave: an old post arguing X and a newer one where you'd changed your mind. A cross-encoder scores relevance to the query, but both of those are highly relevant — what it can't tell you is which one is current. Relevance and currency are different axes, and rerankers only fix the first one.

Two things that helped us before reaching for the cross-encoder tax on every request: (1) a cheap recency/decay prior baked into the candidate scoring, so newer memories win ties for free, and (2) explicit supersession edges — when a memory contradicts an earlier one, link them, so retrieval can prune the stale one instead of feeding the model both and hoping. Overfetch → rerank only the top ~20-30 also keeps the latency you flagged bounded. Have you experimented with letting the memories themselves carry a "supersedes" pointer rather than resolving conflicts purely at query time?

Collapse
 
allenheltondev profile image
Allen Helton AWS Heroes •

I agree that relevance and currency are different problems, which is why I didn't ask the reranker to handle currency. That's what I use the expiration filter for. When I change my mind, the old memory expires and it's excluded from the ranking. So it's basically what you suggested with a supersedes pointer, but used when the memory is written instead of at query time.

I haven't tried a recency prior yet, that's a good idea for ties. On overfetching, even 20 candidates was ~1.2s on my test setup, so "bounded" depends a lot on your hardware.

Collapse
 
izgorodin profile image
Edward Izgorodin •

Expiring at write time moves the currency problem from the read to the write, and the benchmark only exercises the read. The expiry dates come with the generated corpus: the archived billing record expires on 2025-06-14, the day before its replacement is created, so q05 and q06 test a filter whose input was already correct. In a live store something has to set that date when the new post lands, which means finding the old post first. If the write finds it by similarity, it meets the same lookup that, in your own store, scores event sourcing and event streaming as basically the same thing.

The two ways that write fails are not symmetric. A missed expiry turns q06 back into its hard version, a stale record that stays eligible and is more strongly worded than the current one. A wrong expiry removes a memory that was still true, and because eligibility runs before ranking, the reranker never gets the chance to disagree. One more scenario would cover it, with a write step added before the queries: write a changed-mind memory into a group of lookalikes and check which record the write expired, the right one, a neighbour, or none.