FlowGrid AML Retriever ranked #8 on the first AML Open Leaderboard with an Overall Score of 43.98.
But what makes its approach interesting?
Many Agent Memory systems begin with a familiar question:
Given the current query, how can an Agent retrieve the most relevant past information?
FlowGrid begins one step earlier.
Before a system can retrieve useful memory, someone needs to decide what counts as a decision, what changed afterward, what remains unresolved, and where the supporting evidence came from.
Its original form was not a vector database or a memory API.
It was a Markdown decision log written for people.
That starting point shaped the system’s core idea:
Agent Memory should not only preserve what was said. It should preserve how project judgments were made, what evidence supported them, and how those judgments changed over time.
In its v1.0 submission to the first Agent Memory Challenge, FlowGrid AML Retriever turned this philosophy into a lightweight Add/Search memory service.
It preserved original messages, created multiple retrieval views, combined deterministic retrieval signals, and returned traceable evidence rather than final answers.
The result was a deliberately constrained system:
The retriever should find evidence.
The answer model should reason over evidence.
Neither should silently replace the historical record.
After the competition, the project continued as FlowGrid Agent Memory, where the original retrieval foundation became part of a broader system for memory states, authorization, replacement, deletion, and minimal-context compilation.
But the competition entry remains useful as a focused case study:
What happens when a human-readable project decision log becomes an Agent Memory retriever?
1. The Core Idea: Memory Is More Than Stored Facts
Consider a familiar project scenario.
A team first records:
The release date is August 10.
A few days later, a new note says:
The release date has moved to August 14 because testing needs four more days.
Later, an Agent is asked:
What is the current release date?
A basic memory system may retrieve one date.
A stronger system may retrieve both dates.
But a useful project-memory system also needs to preserve:
- which statement came later;
- whether the later statement was actually an update;
- why the change happened;
- where both statements came from;
- and whether the older record should remain available for audit.
This is the distinction FlowGrid starts from.
Its memory system is not designed only to retrieve a sentence that looks relevant.
It is designed to return historical evidence that can still be inspected when a project decision needs to be understood, challenged, or revised.
Instead of asking:
“What should the system remember?”
FlowGrid asks:
“What record would let a person or an Agent understand why the current project state exists?”
2. The Starting Point: A Markdown Decision Log
Before FlowGrid became an AML submission, it was a Markdown decision log.
The log recorded project judgments in a human-readable format:
- decision status;
- project stage;
- background;
- core question;
- candidate options;
- final choice;
- reasons for rejecting alternatives;
- risk assessment;
- follow-up validation;
- and review entry points.
This mattered for a simple reason.
A project is not only a collection of facts.
It is a sequence of choices made under uncertainty.
A Markdown file can be opened by a human, reviewed in Git, compared across commits, and read even when no Agent is available.
Someone can see what the team decided, which alternatives existed, why a direction was chosen, and what conditions would trigger reconsideration.
As projects and Agent collaboration expanded, this single-file approach encountered predictable limits.
Decisions became distributed across messages, sessions, and files.
Older states could remain in context after new information appeared.
New evidence could fail to reach the task that needed it.
FlowGrid gradually added source tracing, temporal states, conflict preservation, and retrieval.
The AML branch then reduced this broader system into the competition’s required Add and Search interfaces.
3. From Human Decision Records to an AML Retriever
AML separates memory retrieval from answer generation.
The memory system receives conversation history through Add.
Later, it receives a query through Search.
The system returns retrieved memory evidence.
A standardized answer model then uses that evidence to generate the final answer.
FlowGrid AML Retriever operates within exactly that boundary.
It does not generate the final answer.
It does not claim that a derived summary is more authoritative than the original conversation.
It stores historical messages, retrieves relevant evidence, and returns enough metadata for the next stage to understand where the evidence came from.
This separation allows three different questions to be evaluated independently:
- Was the memory written correctly?
- Was the right historical evidence retrieved?
- Did the answer model reason correctly over that evidence?
For FlowGrid, this boundary is not just an evaluation requirement.
It is part of the system’s reliability model.
4. Add and Search: A Small API with Strict Guarantees
FlowGrid AML Retriever exposes two core operations.
Add receives user, session, and message data.
The system returns success only after synchronous persistence is complete. Once the request succeeds, the newly written message is already searchable.
Repeated requests with the same request_id and user_id are handled idempotently. This prevents a network retry from writing the same memory twice.
Search is always restricted to one exact user_id.
It returns memory evidence ranked by relevance.
The official platform then performs Answer and Eval.
These details may sound operational rather than algorithmic. But memory reliability depends on them.
A system needs to answer more than:
“Did I retrieve useful information?”
It also needs to answer:
“Did I retrieve it from the correct user?”
“Was the message already committed when I said it was stored?”
“Did a retry create duplicate memory?”
“Can the returned result be traced back to a source?”
For FlowGrid, these are part of the same evidence story.
5. Preserve the Original Message
Long conversations rarely arrive in a retrieval-friendly form.
Some questions need one direct fact.
Some require two or three adjacent turns.
Others depend on seeing conditions, actions, and outcomes together.
If a system only retrieves isolated messages, pronouns, causal explanations, and time conditions may become detached from the statement they qualify.
If it stores only large summaries, names, numbers, dates, and exact phrasing can be diluted.
FlowGrid therefore preserves every original message.
The original record remains the source of evidence.
Derived retrieval views may make the record easier to find. They do not replace it.
This produces a basic rule:
Retrieval representations can change. Historical evidence should remain recoverable.
6. Three Retrieval Views, One Evidence Source
FlowGrid creates three retrieval views from the same conversation history.
Single-message view
This view helps locate direct statements, names, numbers, and dates.
It is useful when the answer is explicitly stated in one message.
Sliding-window view
This view includes adjacent dialogue turns.
It helps restore references, conditions, and supplementary information that a single message may not contain on its own.
Session-segment view
This view preserves a larger local event structure.
It is useful when a question depends on seeing a sequence of related statements within the same conversation segment.
The important point is that these views change retrieval granularity, not evidence ownership.
Each derived view retains source_message_ids.
The system can always return from a window or session segment to the original messages that created it.
Conceptually:
Original Message
↓
Single Message / Sliding Window / Session Segment
↓
Retrieved Candidate
↓
Source Message IDs
↓
Original Evidence
This avoids a common memory-system failure mode:
Derived representations become easy to retrieve but difficult to verify.
7. Why Multiple Views Matter
Suppose a user says:
Maya is planning the launch.
Later:
She wants the release moved because the final test is incomplete.
And later still:
What changed about the launch schedule?
A single-message search may find the release change.
But it may not clearly resolve who “she” refers to.
A larger session segment may preserve the identity, the proposed change, and the reason together.
At the same time, a broad segment can be less precise than a single message when the question asks for a date, a number, or a direct statement.
No one retrieval granularity works for every question.
FlowGrid’s answer is not to choose one universal memory unit.
It is to keep several views available while binding them all to the same underlying historical record.
8. Deterministic Hybrid Retrieval
FlowGrid v1.0’s default retrieval path does not use embeddings or external LLM calls.
It relies on Python’s standard library, SQLite, and FTS5.
The system combines several interpretable signals to address common failures in long-conversation retrieval.
SQLite FTS5 provides the lexical retrieval foundation.
English, numbers, and dates are handled as lexical units.
For Chinese text, character unigrams and bigrams help compensate for the limitations of word segmentation.
A lexical approach has clear limits.
It struggles with paraphrases that are semantically similar but use very different wording.
But it remains strong for proper nouns, dates, numbers, direct quotations, and exact terms from the original conversation.
For many project-memory tasks, those details matter.
9. Retrieval Signals Beyond Keywords
After initial candidate retrieval, FlowGrid calculates additional deterministic signals.
These include:
- original-text substring matches;
- term coverage;
- named entities;
- numbers;
- dates;
- answer options;
- temporal signals;
- and adjacent-context signals.
For multiple-choice queries, the answer options can also participate in retrieval.
This gives the system extra clues for people, locations, and candidate facts.
The point is not to claim that every signal is universally useful.
The point is to reduce dependence on one signal alone.
A keyword match may locate a date.
An entity signal may recover a person.
A window view may restore the condition surrounding that person.
A temporal signal may help distinguish an older statement from a later one.
10. Fusion at the Rank Level
Different retrieval methods produce scores on different scales.
A BM25 score is not directly comparable to a custom entity-match score or a temporal feature score.
FlowGrid avoids treating these values as if they meant the same thing.
Instead, BM25 ranking and deterministic feature ranking enter weighted Reciprocal Rank Fusion.
The fusion happens at the ranking level.
This makes it possible to combine multiple retrieval paths without forcing their raw scores into a shared meaning they do not naturally have.
After fusion, duplicate views and evidence completely covered by stronger results can be removed.
This creates space for more independent sources in the final evidence set.
The downstream model receives not a single opaque score, but a ranked collection of traceable historical records.
11. The Competition Result
In the first Agent Memory Leaderboard academic textual-memory ranking, FlowGrid_AML_Retriever v1.0 achieved:
Rank #8
Overall Score: 43.98
The first-place score was 45.06.
The gap was 1.08 points.
FlowGrid did not rank first in any individual category.
Its scores included:
- Explicit fact recall: 55.59
- Relational and multi-hop compositional reasoning: 45.19
- Personalization and care: 51.29
- Temporal and event-sequence reasoning: 21.13
- Memory governance: 27.86
The latter two were the system’s relative weaknesses.
This result is interesting because the default submission did not rely on an external model call or a heavyweight retrieval stack.
Its competitiveness came from preserving evidence, using multiple views, combining complementary signals, and keeping every derivation connected to source messages.
12. A State Update Is Not Just a Later Timestamp
Temporal memory is one of the hardest parts of long-term Agent Memory.
Return to the release-date example.
The system can find both August 10 and August 14.
But a later message is not automatically a replacement for an earlier one.
A newer message about the same project may be unrelated.
It may mention a date in a different context.
It may discuss a possible plan rather than confirm an update.
A naive strategy such as “always down-rank old values” can improve some temporal questions while damaging others.
FlowGrid’s post-competition experiments showed exactly this.
A more aggressive old-value penalty improved part of the temporal behavior but reduced overall MRR.
The reason was concrete:
Topic similarity plus recency was not enough to prove a state update.
13. Protected State-Update Reranking
FlowGrid v1.1 introduced a narrower post-competition mechanism: protected state-update reranking.
The system adjusts ranking only when three conditions hold:
- the query has temporal intent;
- the old and new evidence are highly related;
- the newer message contains explicit update, correction, delay, or invalidation semantics.
Only then does the system slightly promote the newer evidence and lower the older one.
The older record is not deleted.
It remains preserved and retrievable.
This design reflects a broader principle:
A system should not infer a new state merely because a similar statement appeared later.
The mechanism is intentionally limited.
When the evidence is insufficient, the system keeps the original ranking.
14. Governance for Humans and Governance for Models
FlowGrid’s original governance problem was larger than an AML retrieval benchmark.
A project owner may need to know:
- Which state is currently adopted?
- Who changed it?
- When did it change?
- Why was the previous state preserved?
- Is a conflict resolved?
- Can the team recover context if the current judgment turns out to be wrong?
A standardized answer model has a narrower need.
If it sees both old and new timestamped evidence, it may be able to infer the current state.
But a leaderboard result raises an important question:
Does storing more governance metadata actually improve the evidence returned to the answer model?
Not always.
A system may have many internal state labels while still returning weak evidence.
An outdated value may remain influential as long as it appears in Top K.
FlowGrid’s post-competition response is to treat governance mechanisms as testable interventions rather than automatic improvements.
A mechanism should enter the default path only if it can show:
- what error class it repairs;
- what other queries it might harm;
- and whether it meaningfully changes the returned evidence set.
15. Official Results and Local Experiments Are Different Evidence
The official AML score of 43.98 and the #8 rank apply to FlowGrid_AML_Retriever v1.0.
They do not apply automatically to subsequent versions.
The public FlowGrid Agent Memory repository includes an AML Add/Search adaptation layer at v1.1.0, including protected state-update reranking.
That newer mechanism does not yet have a new official AML leaderboard result.
Its metrics come from deterministic synthetic experiments designed for local version comparison.
They should not be presented as a new AML overall score.
In local paired experiments using classic, medium, and mixed settings, three fixed seeds, and top_k 100, the v1.0 baseline achieved:
Recall@20: 0.9948
Recall@100: 1.0000
MRR: 0.6728
With protected state updates, v1.1 maintained the same Recall@20 and Recall@100 while increasing MRR to 0.6948.
A more aggressive temporal strategy could raise MRR to around 0.7040.
However, it reduced Recall@20 in one seed from 0.9870 to 0.9844.
FlowGrid did not choose that setting.
The system prioritizes stable high recall rather than exchanging recall robustness for a small increase near the top of the ranking.
16. What the Local Experiments Actually Show
The local experiments support a limited conclusion.
Under the current synthetic evaluation protocol, explicit update semantics can reduce errors introduced by blunt temporal reranking.
They do not prove that the same gain will appear on AML’s hidden data.
This distinction is important.
A good Agent Memory research process should separate:
- official leaderboard validation;
- local paired experiments;
- diagnostic experiments;
- and product-engineering acceptance checks.
These forms of evidence answer different questions.
The leaderboard tests a submitted version under a shared evaluation contract.
Local experiments compare alternative mechanisms under controlled conditions.
Diagnostic experiments identify why a specific behavior occurs.
Engineering checks determine whether a feature is stable enough for a product path.
Conflating them makes memory research less trustworthy, not more.
17. Where Rank #8 Came From
FlowGrid’s result did not come from a single decisive model component.
It reduced several common failure modes at once.
Original messages were preserved, so writing did not lose detail through summarization or fact extraction.
Single messages, windows, and session segments provided multiple retrieval scales.
Chinese character fragments, entities, dates, numbers, and answer options supplemented baseline lexical retrieval.
Derived views remained bound to original messages.
Results could be verified and deleted at the user level.
The default path used no third-party Python dependency and no external model call.
Add, Search, idempotency, user isolation, and write-then-search behavior could all be reproduced deterministically.
This is not a claim that lightweight lexical retrieval solves Agent Memory.
It is a demonstration that evidence-first retrieval can remain competitive without making every stage dependent on external models.
18. The Trade-Offs Are Real
FlowGrid’s limitations are as clear as its strengths.
A mostly lexical system has limited ability to recover paraphrases with large semantic distance from the original statement.
Temporal paraphrases can be especially difficult.
SQLite is well suited to a lightweight single-node implementation, but it is not itself a distributed memory-service architecture.
Soft reranking can improve evidence ordering, but it cannot replace complete state resolution.
The system can preserve conflicting history.
It cannot automatically decide every real-world conflict without a stronger policy, additional evidence, or human authorization.
Rank #8 shows that the approach was competitive in the first AML leaderboard.
It does not show that long-term memory governance is finished.
19. After AML: FlowGrid Agent Memory
After the competition, the project’s main direction shifted toward FlowGrid Agent Memory.
The AML Add/Search interface remains as a compatibility and evaluation layer.
The broader product core separates memory into three layers.
Raw events
Immutable records of what happened and where the information came from.
Candidate memories
Memories proposed by rules or models.
Candidate, inferred, and unknown states do not automatically become user-confirmed facts.
Confirmed current state
Information accepted by a user, project owner, or authorized policy.
Superseded, rejected, and deleted information does not return to ordinary Agent continuation context.
However, it can remain available in an authorized audit mode.
20. Current State Resolver and Context Compiler
The general-purpose system introduces two core components.
The Current State Resolver determines what is currently valid.
The Context Compiler creates the smallest appropriate memory package for a task, based on scope, permissions, and token budget.
This is a significant shift from a competition retriever.
A benchmark often asks:
Can the system return relevant evidence?
A real long-term Agent also needs to ask:
Is this evidence currently active?
Is it authorized for this task?
Does the Agent need raw history, a confirmed state, or both?
How much context should be included?
Models can help extract potential memory.
But FlowGrid keeps a source-anchor requirement and limits model output to the candidate stage.
A model can propose a memory.
It cannot silently promote its own proposal into confirmed project state.
21. From Benchmark Adapter to General Memory Infrastructure
This distinction protects the general-purpose product from becoming a collection of benchmark-specific rules.
The competition interface belongs in adapters and evaluation configuration.
The governance core should remain applicable to real projects where memory is not only a retrieval task.
The next evidence target is therefore not merely another leaderboard score.
It is whether memory improves continuation across long-running real tasks.
Can an Agent resume work with less repeated explanation?
Can it distinguish active project decisions from rejected alternatives?
Can it retrieve relevant history without reintroducing invalidated context?
Can a human understand why a memory appeared?
These are harder questions than retrieving one correct sentence.
They are also closer to the problems persistent Agents will face in real work.
22. From Decision Logs to Evidence Systems
FlowGrid began with a simple human need:
Keep a readable record of why a project chose one direction over another.
That led to a Markdown decision log.
The log then developed source tracing, temporal state, conflict preservation, and retrieval.
AML Retriever adapted the approach to Add and Search.
FlowGrid Agent Memory extends the same foundation into raw events, candidate memories, confirmed states, authorization, and minimal context.
Across these stages, the core principle remains consistent:
Interpretation can evolve. Evidence should remain traceable.
23. What FlowGrid Adds to the AML Landscape
The first AML leaderboard includes systems that make different architectural choices.
Some emphasize learned representations.
Some prioritize structured memory.
Some focus on retrieval pipelines.
Some use models heavily during write or search.
FlowGrid represents a different point in the design space:
Preserve the original history.
Create multiple retrieval views.
Use deterministic signals where possible.
Return evidence that can be checked.
Keep state updates narrow until they are supported by evaluation.
Its #8 result does not claim that this is the only correct design.
It shows that a lightweight, evidence-first retrieval system can compete under a shared benchmark.
The more interesting contribution may be its path from human-readable decision records to model-facing evidence retrieval.
24. The Broader Question for Agent Memory
When an Agent says:
“I remember this.”
What should it mean?
Should it mean:
“I generated a plausible summary of the past”?
Or should it mean:
“Here is the historical evidence I found, where it came from, what changed afterward, and why this record is relevant now”?
FlowGrid argues for the second interpretation.
Memory should not become a hidden layer that quietly influences an Agent’s decisions.
It should become an evidence layer between an Agent’s current reasoning and its historical context.
That becomes increasingly important as Agents move from short conversations toward long-running projects, persistent collaboration, and decisions with real consequences.
25. From “I Remember” to “Here Is the Record”
FlowGrid’s central contribution is not a claim that every project decision can be automatically resolved.
It is a commitment to keep the record available.
What was said.
Where it came from.
What changed later.
Which state is currently adopted.
Why a current judgment should be trusted.
The first AML result showed that this evidence-retrieval foundation could be competitive.
The post-competition product direction recognizes that retrieval alone is not enough.
Long-term Agent Memory also requires state, authority, context control, and the ability for people to inspect the path from history to action.
A useful Agent Memory system may need to do more than say:
“I remember this.”
It may need to say:
“Here is the evidence.”
“Here is where it came from.”
“Here is what changed afterward.”
“And here is why this is the current record.”
Thanks to the FlowGrid Team
We would like to thank the FlowGrid team for sharing its system design, post-competition experiments, and broader product direction with the AML technical deep dive series.
Their work highlights an important dimension of Agent Memory:
Long-term memory is not only about retrieval quality. It is also about whether the historical path behind a decision remains readable, inspectable, and recoverable.
The goal of the AML Solution Spotlight series is to make top-performing memory systems easier to understand—not only through leaderboard scores, but through their technical ideas, engineering choices, trade-offs, and unresolved questions.
More technical deep dives into the first AML leaderboard are coming soon.
FlowGrid AML Retriever
Team: Artificial Idiot Research Institute
Project Lead: Lingxiao Du
GitHub: https://github.com/dlxeva/flowgrid-aml-retriever
Post-competition general product: https://github.com/dlxeva/flowgrid-agent-memory
Website: https://aizhiz.com/
X: https://x.com/dlxeva
The Second Agent Memory Challenge opens on September 20, 2026.
This cycle includes three tracks:
Textual Memory — long-horizon conversation retrieval, temporal reasoning, and user-profile alignment
Coding Memory — development-context retrieval for real software-engineering tasks
Multimodal Memory — cross-modal memory association across text, images, audio, and video
Each track includes both an Open-Source Methods Ranking and an Industry Systems Ranking.
The challenge is globally open to universities, research institutions, companies, open-source teams, and individual developers. There is no registration fee.
The open-source ranking includes a total prize pool of RMB 150,000, with awards including:
First Prize: RMB 20,000
Second Prize: RMB 8,000
Third Prize: RMB 3,000
Best Technical Innovation Award: RMB 5,000
Important dates:
Registration and submission entry opens: September 20, 2026, 00:00
Rolling evaluation period: September 20 – October 31
Submission deadline: October 31, 2026, 23:59
Evaluation queue closes: November 4, 2026, 23:59
Results planned for release: mid-November 2026
Each participant may initiate up to two full official evaluations.
Once evaluation begins, the submitted version is frozen. The platform then evaluates the system automatically through the standardized Add/Search interface.
Whether you are building a memory layer for conversational Agents, Coding Agents, or multimodal systems, the second AML cycle is an opportunity to test what your architecture can contribute under a shared and reproducible evaluation setting.
Registration opens on September 20. We look forward to seeing what the next generation of Agent Memory systems can do.
AML
Agent Memory Leaderboard
https://agentmemoryleaderboard.ai/
Leaderboard
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard
Content Notice
This article is based on publicly available system materials and the first-cycle AML Coding Memory leaderboard results. It reflects an analytical interpretation of the systems discussed and does not represent an official technical recommendation from AML.
Scores apply only to the referenced evaluation cycle, track, submitted system version, and official leaderboard release. They should not be interpreted as guarantees of performance in every software-engineering scenario.
Top comments (3)
how you measure the real effect of memory on Agent work, rather than retrieval quality alone. In my benchmark work, memory has a cost in tokens, latency, and attention. It can also be damaging when the relevant fact is absent, superseded, contradictory, or merely adjacent.
How do you evaluate whether retrieved memory improved the final task outcome, instead of simply adding more context for the Agent to process? Do you measure completed work, negative transfer, and the cost of the memory path together?
This is exactly why we started with task-level metrics rather than retrieval metrics. We currently measure overall solve rate, new-feature solve rate, and bug-fix solve rate, together with input/output tokens, retrieved context size, write time, and search latency.
The core question is: does memory help the Agent complete more work, and at what cost?
We also agree that harmful memory needs to be evaluated explicitly. In follow-up work, we plan to build a dedicated benchmark around outdated, superseded, contradictory, or misleading memories, and introduce separate metrics for their impact — including negative transfer — rather than letting those cases disappear inside the average solve rate.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.