DEV Community

Cover image for From “Retrieving” to “Verifying”: How ChronoHybridMem Turns Agent Memory into Evidence
Agent Memory Leaderboard
Agent Memory Leaderboard

Posted on

From “Retrieving” to “Verifying”: How ChronoHybridMem Turns Agent Memory into Evidence

ChronoHybridMem ranked #5 on the first AML Open Leaderboard with an Overall Score of 44.33.

But what makes its approach interesting?

Long-term memory retrieval for an AI agent is often framed as a search problem:

Given the current query, find the most relevant memories.

But finding a relevant piece of information is only part of the problem.

An agent may retrieve something that looks relevant but lacks context. A structured fact may be useful but difficult to verify. A generated summary may contain the right information but provide no clear path back to the original conversation.

This raises a deeper question:

When an Agent retrieves a memory, can it also explain where that memory came from?

ChronoHybridMem approaches this problem from a different angle.

Instead of treating memory as a collection of isolated summaries, it treats memory as a collection of verifiable evidence.

Its core idea is simple:

Agent memory should return not just relevant information, but evidence that can be traced back to its source.

In its v0.2.0 submission to the first Agent Memory Challenge, ChronoHybridMem combined raw message preservation, source-bound structured facts, SQLite FTS5 dual-path retrieval, and constrained candidate reranking.

The result was a system designed around a clear boundary:

The model can choose among existing evidence. It should not be allowed to invent the evidence itself.

After the competition, the team continued experimenting with query planning, collection-aware reranking, evidence graphs, recall-failure diagnosis, and selective gating.

Some of these ideas improved the local baseline.

Others did not.

That distinction is important because the team's post-competition work treats both positive and negative results as part of the engineering process.


1. The Core Idea: An Agent Needs Evidence, Not Just Similar Text

Consider a simple example.

Suppose a conversation history contains two statements:

“Yutu gave a bag of cookies to Tutu.”

and later:

“Yutu works at the court.”

Now the user asks:

“Where does the person who gave the cookies work?”

The answer cannot be obtained reliably by retrieving only one of these messages.

The system first needs to identify that the person who gave the cookies was Yutu.

It then needs to retrieve the information about Yutu's workplace.

More importantly, the resulting answer should be supported by evidence from the original conversation.

This illustrates three requirements for long-term Agent Memory:

  1. Original conversations should not be replaced entirely by opaque summaries.
  2. Retrieval should support both raw textual details and structured representations.
  3. The model should select and rank evidence rather than freely generate new memories.

ChronoHybridMem is designed around these principles.

Instead of asking:

“What should the memory system remember?”

it focuses on another question:

“When the Agent needs to remember something, what evidence can the system provide?”


2. From Conversation to Verifiable Memory Evidence

At a high level, the ChronoHybridMem v0.2.0 pipeline looks like:

Conversation

RAW + FACT Storage

Dual-path FTS5 Retrieval

Candidate Merge & Deduplication

Constrained Reranking

Evidence Set

Answer

The system separates the responsibilities of different components.

During the Add stage, the original conversation is preserved.

The system can also extract structured facts from the conversation.

During Search, the query enters two retrieval paths:

  • raw-message retrieval;
  • fact-level retrieval.

The candidates are then merged and deduplicated.

A language model is used only to rank the candidates.

Finally, the system returns the selected evidence together with its source information.

This creates an explicit separation between:

retrieving evidence

and

generating an answer from evidence.

The final answer is produced through AML's standardized answer process rather than directly by ChronoHybridMem.

This separation allows the memory component itself to be evaluated independently.


3. RAW + FACT: Keep the Original Evidence, Add Structure on Top

One of the central design choices in ChronoHybridMem is that structured facts do not replace the original conversation.

When a message is written, the original message is first persisted as a RAW record.

In model mode, the system can additionally extract structured FACT representations.

For example:

RAW

“Yutu works at the court.”

FACT

“Yutu works at the court.”

The important difference is that the FACT is not treated as an independent piece of truth.

Each structured fact contains a:

source_message_id

that points back to the original message.

This creates two complementary representations.

RAW preserves context

The original message retains:

  • exact wording;
  • relationships between people;
  • temporal clues;
  • surrounding context;
  • details that may be lost during compression.

FACT improves retrieval

Structured facts provide a more compact representation that can be easier to search.

They can help the system locate information that may be difficult to match directly against the original wording.

The source link connects the two

Because every FACT can point back to its source message, the structured representation does not become an isolated summary.

The system can move from:

FACT → source_message_id → RAW

when verification is needed.

This is an important distinction.

Structure is used to improve retrieval, while the original record remains the source of evidence.


4. SQLite FTS5 Dual-Path Retrieval

ChronoHybridMem does not rely on a single retrieval representation.

The Search stage runs two parallel FTS5 retrieval paths.

Path 1: Raw Message Search

The system searches the original conversation records.

This path is useful when the answer depends on:

  • exact wording;
  • relationships;
  • contextual clues;
  • or details that may not survive structured extraction.

Path 2: Fact Search

The system separately searches the structured FACT records.

This path provides a more compact representation for direct fact retrieval.

The two candidate sets are then merged and deduplicated.

Conceptually:

Query

Raw Message FTS5 → Raw Candidates

Fact FTS5 → Fact Candidates

Merge + Deduplicate

Candidate Pool

The goal is not to decide that one representation is universally better.

Instead:

The two retrieval paths compensate for each other's weaknesses.

Raw messages preserve context.

Facts provide compact retrieval targets.

Using both reduces the risk that the entire retrieval process becomes dependent on a single representation of memory.

The Search request also carries an exact user_id boundary, ensuring that retrieved records remain within the correct user's memory space.


5. Constrained Reranking: Let the Model Choose, Not Invent

After candidate retrieval, ChronoHybridMem uses a language model to rank the candidates.

But the model operates under strict constraints.

It receives:

  • the current query;
  • the retrieved candidate set.

It can then return the IDs of candidates that should be ranked higher.

It cannot:

  • create a new memory;
  • invent a candidate;
  • return an arbitrary ID;
  • access another user's records;
  • or replace the evidence with a generated memory.

The system subsequently validates:

  • whether the returned candidate ID actually exists;
  • whether it belongs to the current user;
  • whether a FACT has a valid source message;
  • and whether the source relationship is legitimate.

This establishes a clear division of responsibility:

The model decides which evidence is useful.

The storage and retrieval layer decides what evidence actually exists.

This distinction becomes particularly important when using LLMs inside memory systems.

A model may be excellent at reasoning over information.

But reasoning ability should not automatically give it permission to create the information it is supposed to retrieve.

ChronoHybridMem therefore treats the model as a candidate selector, rather than the source of truth.


6. Why Verifiability Matters

The purpose of this architecture is not simply to increase the number of retrieved memories.

It is to shorten the distance between:

the answer an Agent wants to produce

and

the historical evidence supporting that answer.

Return to the earlier example.

The Agent needs to answer:

“Where does the person who gave the cookies work?”

A useful evidence set might contain:

  1. the original message identifying who gave the cookies;
  2. the structured fact describing that person's workplace;
  3. the source message associated with that fact.

The downstream answer model can then reason over these pieces of evidence.

If necessary, it can trace the structured fact back to the original conversation.

This makes memory more inspectable.

Instead of simply returning:

“The person works at the court.”

the system can provide the evidence chain behind that statement.

This leads to a broader design principle:

A reliable memory system should make it possible to inspect why a memory was retrieved, not merely what was retrieved.


7. Engineering Boundaries Matter Too

ChronoHybridMem's emphasis on evidence is not limited to retrieval.

The team also treats several engineering boundaries as part of memory reliability.

Idempotent writes

The Add operation is designed to be idempotent, reducing the risk of duplicate memory pollution.

Persistent storage

SQLite WAL is used to support stable persistent reads and writes.

User isolation

Both API and SQL-level operations use user_id as an isolation boundary.

Explicit failure

If required configuration for model mode is missing, the system fails explicitly rather than silently switching to unknown behavior.

These details may appear less interesting than a new retrieval algorithm.

But they matter when memory systems move from benchmark demonstrations toward actual Agent infrastructure.

A memory system needs to answer not only:

“Did I retrieve the right information?”

but also:

“Did I retrieve it from the right user?”

“Can I trace it back to its source?”

“Can I reproduce the same behavior?”

“What happens when part of the system fails?”

For ChronoHybridMem, these are part of the same reliability story.


8. Official Result vs. Post-Competition Research

An important distinction in the ChronoHybridMem team's work is between its official AML result and its later local experiments.

The result reported on the AML leaderboard corresponds to:

ChronoHybridMem v0.2.0

Rank #5

Overall Score: 44.33

The experiments described below were conducted after the official evaluation.

They primarily use the public LoCoMo dataset and a local Qwen3-4B proxy model to investigate retrieval mechanisms and engineering hypotheses.

They are not new AML leaderboard scores.

This distinction matters.

The team uses a layered evidence protocol:

  • official leaderboard results demonstrate performance validated by AML;
  • full 1,977-question local runs compare methods under the same local evaluation setup;
  • smaller fixed sets are used for diagnostics, mechanical checks, and upgrade gates.

Only paired comparisons using the same data, model, query plan, and call budget are used to discuss method-level improvements.

This allows the post-competition work to function as research rather than as an attempt to reinterpret the official leaderboard result.


9. After v0.2.0: Turning Memory Research into Falsifiable Experiments

The post-competition development did not simply add more models or retrieval modules.

Instead, the team framed each stage around a falsifiable question:

What failure mode are we trying to solve?

Does the proposed mechanism actually improve it?

Does it introduce additional model calls?

Can its evidence still be traced?

Can the module be safely disabled if it causes regression?

This produced a sequence of experiments from P1 to P5, preceded by two additional local milestones.


10. research-v0.3.0: Lexical + Dense Retrieval

The first post-competition milestone, research-v0.3.0, explored hybrid retrieval.

It retained the existing Add/Search API and original evidence-return contract.

During Search, the system combined:

  • Porter-normalized BM25 retrieval;
  • BAAI/bge-large-en-v1.5 dense retrieval.

The scores from the two candidate sets were normalized and fused.

The top five fused candidates were then reranked using:

answerai-colbert-small-v1

The important constraint remained unchanged:

The reranker could only reorder existing candidates.

It did not generate new evidence or replace the original messages.

On the full 1,977-question LoCoMo local evaluation, v0.3.0 achieved:

  • Hit@1: 0.4355
  • Hit@3: 0.6186
  • Hit@10: 0.7577
  • MRR: 0.5183

Compared with the corresponding lexical baseline, Hit@1 increased from 0.3359 to 0.4355.

The team also tested different reranking pool sizes.

Interestingly, increasing the ColBERT reranking pool did not automatically improve the top-ranked result.

On the fixed 200-question experiment, Top-5 performed better than Top-10 and Top-20 for the final ranking objective.

The team therefore froze the reranking pool at Top-5.

This is an example of an important engineering principle:

A larger candidate pool is not automatically a better candidate pool.


11. research-v0.4.0: Time-Aware Retrieval + Dedicated Reranking

The next milestone shifted the focus from retrieval coverage toward ranking quality.

research-v0.4.0 added a time-aware dense representation and a dedicated local Qwen reranker.

The reranker used:

Qwen3-Reranker-4B

with a yes/no relevance formulation.

The model returned a relevance probability over existing evidence rather than generating new memory.

The system also introduced a controlled temporal key, adding message date information to the retrieval representation.

An interesting result emerged:

The temporal signal alone did not improve Hit@1 on the full dataset.

However, it increased Top-10 candidate coverage.

When combined with the dedicated Qwen reranker, those additional candidates could sometimes be converted into better top-ranked evidence.

The final local result for v0.4.0 was:

  • Hit@1: 0.5225
  • Hit@3: 0.6808
  • Hit@10: 0.7653
  • MRR: 0.5856

Compared with v0.3.0:

  • Hit@1 increased by 0.0870
  • MRR increased by 0.0673

Again, these are post-competition local research results, not AML leaderboard scores.


12. P1: Structured Query Planning

After the two model-side milestones, the research shifted toward a more fundamental question:

Is retrieval failing because the system does not understand what the query actually needs?

P1 introduced structured query planning.

Instead of treating the query as a flat string, the planner decomposes it into:

  • intent;
  • core terms;
  • expansions;
  • entities;
  • temporal cues;
  • up to four evidence needs.

Different fields can then be used by different retrieval paths.

The evidence needs also become reusable signals for later experiments.

Importantly, P1 does not add another model call during Search and does not change the Add/Search API.

If the planner output is incomplete, the system falls back to a safer first-stage retrieval path.

On a fixed 200-question local screening set:

Hit@1: 0.545 → 0.565

MRR: 0.6145 → 0.6292

Hit@10 remained at 0.740.

On the full 1,977-question local evaluation:

Hit@1: 0.5761

Hit@3: 0.7157

Hit@10: 0.7618

MRR: 0.6479

The result suggested that better understanding of the information need could provide more stable gains than simply adding more retrieval models.


13. P2: More Coverage Does Not Necessarily Mean Better Ranking

P2 explored collection-aware reranking.

Multi-hop questions often require multiple complementary pieces of evidence.

A conventional ranking system, however, may place several highly similar records at the top while pushing complementary evidence lower.

P2 therefore attempted to select candidates based partly on how much additional evidence they covered.

The idea sounds intuitive:

If a candidate covers a new evidence need, shouldn't it become more valuable?

Not necessarily.

On a fixed 20-question experiment, P2 kept Hit@1 unchanged while improving Hit@3 and MRR.

But on a frozen 35-case synthetic stratification:

Hit@1 fell from 1.00 to 0.8571

and

MRR fell from 1.00 to 0.9286.

Why?

Because:

Covering more query requirements does not necessarily mean that a candidate is the best first piece of evidence.

P2 was therefore rejected from the default path.

The code remains available for reproduction and failure analysis.

This negative result changed the team's research direction.

Instead of continuing to optimize diversity in the top-ranked candidates, the team began asking:

Where exactly is the correct evidence being lost?


14. P3: Evidence Graphs Need Evidence Too

P3 explored another attractive idea for memory systems:

Can an evidence graph help connect people, places, organizations, relationships, and temporal updates?

The team constructed graph representations for entities and relations.

But it imposed a strict rule:

Every entity mention and every relationship edge must be independently supported by an original message.

The reason is straightforward.

A graph generated by an LLM can look highly structured while still containing unsupported relationships.

For a memory system centered on verifiability, that would simply move the trust problem somewhere else.

Under the strict evidence constraint, the graph became surprisingly sparse.

On a fixed set of 419 original messages, the strict relation graph produced only 3 independently witnessed relationship edges.

A second experiment, P3-B1, used source-local entity mention anchors.

This achieved coverage across:

363 / 419 messages (86.63%)

But on the fixed 20-question test:

Hit@1 fell from 0.40 to 0.35

and

Hit@10 fell from 0.55 to 0.50.

P3 therefore did not become a global default retrieval channel.

The experiment left an important lesson:

A memory structure should not be considered useful simply because it is structured, interpretable, or highly covered.

Its value still has to be demonstrated through paired retrieval gains and source-level auditing.


15. P4: Diagnose the Recall Failure Before Fixing It

P4 was arguably the most important shift in the post-competition research.

Instead of asking:

“What new retrieval module should we add?”

the team first asked:

“Why are we failing to retrieve the correct evidence?”

A fixed 100-question audit identified 30 Top-10 retrieval failures.

They were divided into three categories:

Fusion miss

20 cases

The relevant evidence had been found by at least one retrieval channel but was lost during candidate fusion.

Channel miss

8 cases

None of the lexical retrieval channels found the relevant evidence.

Reranker drop

2 cases

The correct evidence was retrieved but subsequently ranked too low.

This distribution was revealing.

The dominant problem was not:

“The reranker cannot recognize the correct candidate.”

It was:

“The correct candidate often never survives into the reranking pool.”

This changed the optimization target.


16. P4-A: Turn Evidence Needs into Retrieval Channels

To address the largest failure bucket — fusion misses — P4-A reused the evidence needs already generated by P1.

Each evidence need became an independent, bounded retrieval channel.

Each channel was given a fixed number of candidate positions.

These candidates were then merged with the existing retrieval results and passed into the original reranking pipeline.

Crucially:

  • no new evidence was generated;
  • the existing query planner was reused;
  • Search did not require an additional model call.

The goal was simple:

Give important evidence needs a guaranteed opportunity to enter the candidate pool.

On the full 1,977-question local evaluation, P4-A q2 became the strongest post-competition local proxy baseline.

Compared with P1, it recovered 8 questions whose relevant evidence had previously fallen outside Top-10.

Four of those recovered cases moved directly into Top-1.

Hit@1, Hit@3, Hit@10, and MRR all improved modestly.

Again, this is a local proxy result, not a new official AML score.

But the experiment provided a useful diagnosis:

Improving recall before reranking can matter more than making the reranker itself more sophisticated.


17. P5: When Should the System Override Its Top Result?

P5 explored a different problem.

P4-A showed that candidate replacement could rescue some queries.

But the same mechanism could also replace an already-correct Top-1 result.

The team therefore asked:

Can we identify the cases where a candidate swap is actually beneficial?

Several signals were tested:

  • channel count;
  • query-token overlap;
  • temporal/correction strata;
  • model-reported confidence.

None passed the predefined fixed-200-question threshold.

For example:

  • channel count: Hit@1 change -0.005
  • query overlap: -0.035
  • strata narrowing: -0.010
  • confidence gating: did not trigger reliably

More importantly, asking the local model to provide an explicit confidence signal itself reduced Hit@1 by 0.045 in the tested setup.

The conclusion was therefore not that selective gating is impossible.

Rather:

These simple signals are not reliable enough to determine when an evidence swap should occur.

P5 was kept as default-off ablation code, and the team stopped tuning this direction.

Again, a negative result became useful evidence.


18. What These Experiments Suggest About Agent Memory

Taken together, the ChronoHybridMem experiments suggest several broader observations.

1. Query understanding can matter more than retrieval complexity

P1 produced stable gains by making the information need more explicit.

Simply adding another retrieval mechanism is not guaranteed to produce the same effect.

2. More structure does not automatically mean better memory

P2 and P3 both explored more structured candidate selection.

Both demonstrated that additional structure can introduce new failure modes.

3. Recall and ranking are different problems

P4's failure audit showed that many apparent “ranking failures” were actually retrieval failures.

If the correct evidence never enters the candidate pool, a better reranker cannot recover it.

4. Negative results can improve system design

P2, P3, and P5 did not enter the default path.

That is not necessarily wasted work.

By explicitly measuring their failures, the team narrowed the space of plausible design choices.

This is perhaps one of the most interesting aspects of the project:

The system is being developed not by accumulating modules, but by eliminating unsupported assumptions.


19. The Next Challenge: When Lexical Retrieval Cannot Find the Connection

The current architecture works well when useful evidence can be brought into the candidate pool.

But an important class of problems remains.

Consider a user saying:

“I'm thinking about adopting a cat.”

Months earlier, they had said:

“I have a lot of lilies at home.”

The historical statement may be highly relevant to the current decision.

But the connection is not obvious from the query itself.

A retrieval system searching for:

  • cat;
  • adoption;
  • pet;
  • breed;
  • food;

may never search for:

lilies

The problem is no longer simply:

“Can the system rank the correct evidence?”

It becomes:

“Can the system discover that this seemingly unrelated memory matters?”

This is a harder problem for query-conditioned retrieval.

If the relevant memory never enters the candidate pool, even a powerful reranker cannot recover it.

This is one reason the ChronoHybridMem team is now investigating channel-miss cases more closely.


20. Toward Source-Constrained Multi-Hop Retrieval

The next stage of the team's research focuses on cases involving:

  • abstract relationships;
  • identity;
  • personality;
  • decisions;
  • multi-hop reasoning;
  • and implicit connections.

The goal is to explore source-constrained bridging retrieval.

Instead of allowing an LLM to freely invent relationships, the system could:

  1. locate relevant entities or messages;
  2. identify source-supported relationships;
  3. perform a limited expansion;
  4. retrieve additional evidence from the same user, session, or explicitly witnessed relationship.

The expansion would remain tightly bounded.

For example:

  • fixed candidate limits;
  • limited traversal depth;
  • deterministic tie-breaking;
  • strict user isolation;
  • no model-generated relationship treated as ground truth.

The principle remains unchanged:

Expand the search space without expanding the set of unsupported facts.


21. Temporal State and Corrections

Another important direction is temporal memory.

Long conversations are not static.

Users change their plans.

They correct names.

They update schedules.

They revise preferences.

They add exceptions.

Consider:

“I'm moving to Shanghai next month.”

followed later by:

“Actually, the move has been postponed.”

A memory system should not simply retrieve both statements and leave the downstream model to guess.

The next research direction is therefore to represent state changes as an auditable chain:

Original Statement

Update / Correction

Currently Valid State

But the original evidence should remain available.

The goal is not to delete old memories.

It is to make the current state explainable:

Which earlier information changed?

What evidence caused the update?

Why is this the currently valid state?

This is another extension of the same principle behind RAW + FACT:

Interpretation can evolve, but evidence should remain traceable.


22. Toward a More Complete Memory Evaluation Framework

ChronoHybridMem's research also points to a broader question for Agent Memory evaluation.

Memory quality should not be measured only through:

  • Hit@K;
  • MRR;
  • answer accuracy.

Future evaluations may also need to track:

  • evidence recall;
  • candidate-source distribution;
  • model call count;
  • fallback behavior;
  • user isolation;
  • retrieval stability;
  • database state;
  • runtime cost;
  • and source completeness.

A memory system that achieves higher recall by dramatically increasing inference cost may represent a different engineering trade-off from one that achieves similar performance with a smaller budget.

Likewise, a system that retrieves an answer but cannot identify its source presents a different reliability profile from one that returns a fully traceable evidence chain.

This suggests a broader direction for memory evaluation:

Measure not only whether the Agent remembers, but how it remembers, what it costs, and whether the memory can be verified.


23. From Memory Retrieval to Evidence Systems

ChronoHybridMem began with a relatively simple architectural choice:

Keep the original conversation.

Add structure where useful.

Retrieve through multiple paths.

Let the model rank existing candidates.

Return evidence with its source.

The subsequent P1–P5 experiments made the picture more nuanced.

They showed that:

  • structured query planning can provide meaningful gains;
  • more candidate diversity does not automatically improve ranking;
  • graph structure requires strict evidence constraints;
  • recall failures can dominate reranking failures;
  • and simple confidence-based gating is not necessarily reliable.

Together, these results suggest that Agent Memory may be moving toward something broader than conventional retrieval.

A memory system is not simply a database.

It is not simply a vector store.

It is not simply a summarization layer.

It is increasingly becoming an evidence system between an Agent and its history.

The central question becomes:

When an Agent remembers something, can we understand where that memory came from and why it should be trusted?


24. What ChronoHybridMem Adds to the AML Landscape

The first AML leaderboard contains systems that make different architectural choices.

Some emphasize structured memory.

Some emphasize retrieval.

Some rely heavily on learned representations.

ChronoHybridMem represents another point in this design space:

Preserve evidence first, then add controlled structure and retrieval mechanisms around it.

Its Rank #5 result — 44.33 Overall — demonstrates that this evidence-oriented architecture is competitive under the first AML evaluation.

But the more interesting contribution may be the research process that followed.

Rather than assuming that every additional module improves memory, the team explicitly tested hypotheses and removed those that failed.

This makes the project useful not only as a leaderboard entry, but also as a case study in how Agent Memory systems can be iterated.


25. The Broader Question for Agent Memory

ChronoHybridMem ultimately asks a simple question:

When an Agent remembers something, should it be able to show its work?

A useful memory system may need to do more than return:

“I remember this.”

It may need to provide:

“Here is what I found.”

“Here is where it came from.”

“Here is why this evidence was selected.”

“And here is the original record if you want to verify it.”

This changes the role of memory.

Instead of treating memory as a hidden layer that produces an opaque answer, we can treat it as an evidence layer that connects an Agent's current reasoning to its historical context.

That may become increasingly important as Agents move from short-lived interactions toward long-term relationships, persistent tasks, and autonomous decision-making.


Thanks to the ChronoHybridMem Team

We'd like to thank the ChronoHybridMem team for sharing their system design and post-competition research with the AML technical deep dive series.

Their work illustrates an important aspect of Agent Memory research:

Progress does not always mean adding another module. Sometimes it means finding out which modules should not be there.

The goal of the AML solution spotlight series is to make top-performing memory systems easier to understand — not only through leaderboard scores, but through the technical ideas, engineering choices, trade-offs, and failures behind those scores.

More technical deep dives into the first AML leaderboard are coming soon.


ChronoHybridMem

Team: ChronoHybridMem
Team Lead: Haoxuan Meng
GitHub: https://github.com/Tin11Mn/chrono-hybrid-mem

AML

Agent Memory Leaderboard
https://agentmemoryleaderboard.ai/

Leaderboard:
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

Top comments (1)

Collapse
 
ahmetozel profile image
Ahmet Özel

Treating memory as evidence with a path back to the source fixes something summary-based memory quietly breaks: once a fact has been distilled, there is no way to tell a correct summary from a confident misreading of the original turn. Keeping the pointer means a wrong memory is diagnosable rather than just wrong. The temporal half is where I would expect most of the remaining wins. Agent memory is full of claims that were true when written - a preference, a project status, an address - and a retriever with no notion of supersession happily surfaces the older one because it matches the query just as well. Distinguishing an episode, which stays true about its moment, from a claim, which asserts something is true now and needs invalidating by a later one, removes a whole class of confidently stale answers. Traceability also makes evaluation tractable, since you can score whether the cited turn actually supports the memory rather than asking a judge whether the memory sounds right - the second one drifts with the generator, the first does not.