You need your agent to remember things between sessions, you search for the best persistent memory API, and every list you find is a feature grid where every row has a tick in every column. Everyone has memory. Everyone has a graph. Everyone learns. Then you pick one, and three weeks later the question that actually mattered for your project turns out to be the one the grid never asked.
So I stopped asking which system is best and asked the same six questions of six systems instead: Mem0, Letta, Zep with its open-source engine Graphiti, Cognee, Supermemory and Mnemoverse. The answers come from each vendor's own pages, read on 18 September 2026. I work on the last one, which is reason to weigh my reading of it harder.
One line to carry: every one of these six says somewhere that it learns or improves, two of them name the step that turns a rating into ranking, five of them can be run with no vendor account, and the right choice depends on which of those you actually need.
1. How is memory stored and found?
Six different mechanisms, and the differences are real.
Mem0 fuses several signals at read time, semantic similarity, keyword matching and an entity layer, with dates reordering results on its hosted platform. Letta's agent memory is Markdown files in a git repository, and its FAQ says plainly that "MemFS does not include a semantic or vector index by default". Zep and Graphiti build a temporal knowledge graph and retrieve through a hybrid of semantic search, keyword matching and graph traversal. Cognee runs a relational store, a vector store and an LLM-built graph together, and defines itself against plain retrieval: "Classic RAG embeds text chunks and retrieves by similarity." Supermemory describes a learning model on top of a graph of typed connections between facts.
Ours stores memories written with the concepts you give them and retrieves by meaning. What makes it different is in the ranking: the reference documents the relevance of a read as "Final score (similarity * valence modulation)", and valence is the thing outcome feedback moves. More on that in question 3.
2. Does the same memory follow you between tools?
All six document a route. Mem0 says its memory "persists across sessions, tools, and runs". Supermemory's MCP documentation says "Supermemory MCP gives every MCP-compatible assistant a shared memory layer". Cognee's MCP overview describes an API mode that can "Connect multiple clients to a shared knowledge graph". Zep's hosted product and Graphiti's MCP server both document editor setups. Ours is one account behind one key or OAuth, with Claude Code, Cursor, VS Code, Windsurf and Claude Desktop each documented in its own configuration shape.
Letta answers differently. Its route into editors is its own Agent Client Protocol adapter, set up for Zed, JetBrains IDEs and Obsidian, plus an OpenAI-compatible API on its App Server. If your editor is Cursor or VS Code, check that route before assuming it.
3. Does anything change when a memory turns out to be wrong?
This is the question the grids never ask, and it is where the six separate most.
Mem0 has a feedback endpoint, and its page describes the effect this way: "Over time, Mem0 continuously learns from this feedback, refining its memory generation and search capabilities for better performance." That page names no step from the rating to the ranking. The same product documents a separate feature, memory decay, step by step and precisely: "Decay never zeroes a candidate out: at worst it scales its score by 0.3×". Both are on the hosted platform.
Zep had fact ratings, a rule written up front and scored against every fact. Its February 2026 deprecation notice says: "Fact ratings are being deprecated entirely." Letta records a positive or negative signal per agent step, and the pages I read document no effect of it on memory. Supermemory keeps inferred memories low until they are approved, expires temporary facts and strengthens preferences with repetition, and on the pages I read documents nothing tied to whether a recalled memory actually helped.
Two name the step. Cognee stores ratings and can apply them to ranking, and its release notes say the default influence "remains 0.0", so it is off until you turn it on. Ours takes a rating through a memory_feedback call, and the valence it moves is part of the documented ranking formula above. The call is one of the tools in the same MCP connection the agent reads through, so the agent that saw the outcome can report it in the same session. In both, what moves the ranking is a rating actually being sent.
4. Can several agents or people share one pool?
Six shapes. Mem0 has group chat with every write scoped by the caller. Letta has shared memory blocks. Zep has standalone graphs, and Graphiti gives each team its own graph namespace. Cognee treats an agent as its own permission-holding principal, and warns that its default embedded store is not for agents writing at once. Supermemory shares through spaces, where "Teammates collaborate within the spaces they are allowed to read or write." Ours calls them Rooms: a shared space separate from anyone's personal memory, with membership checked on every request.
5. What does it cost to start, and can you run it yourself?
Five of the six can run with no vendor account on your own machine, on a model you bring: Mem0's open-source library, Letta's local runtime, where "no Letta account is required", Cognee's core, Graphiti, and Supermemory's local binary. Zep's own local edition is gone, in its FAQ's words: "Zep Community Edition, which allows you to host Zep locally, is deprecated and no longer supported." Graphiti is the open path there.
Letta puts a number on its hosted free tier: "Free plans are limited to 3 stateful agents."
Ours has a free tier and is managed by default: the service you connect to is the product, and we run the engine for you. Enterprise adds dedicated instances that we provide and operate, a choice of data residency, and self-hosting by agreement when security or compliance requirements call for it.
6. What does each one tell you about its own limits?
The useful ones say it themselves. Mem0's graph documentation says its entity layer does not assign typed relationships between entities. Letta's FAQ says there is no vector index by default. Cognee says its embedded store is not for concurrent agents. Zep says fact ratings are going away. Ours puts its terms in the same places: the pricing page says who runs the engine and when you can run it yourself, and the reference gives the ranking formula that a rating moves.
The six answers in one table
| memory | across tools | outcome feedback | sharing | run it yourself | |
|---|---|---|---|---|---|
| Mem0 | semantic, keyword and entity signals fused | yes | endpoint, no step named; decay specified | group chat, caller-scoped | yes, open-source library |
| Letta | Markdown files in git, no vector index by default | via its ACP adapter and an OpenAI-compatible API | per-step signal, no memory effect documented | shared memory blocks | yes, local runtime |
| Zep / Graphiti | temporal knowledge graph | yes | fact ratings, being deprecated | standalone graphs; team namespaces | Graphiti only |
| Cognee | relational, vector and LLM-built graph | yes, API mode | step named, off by default | principals and permissions | yes |
| Supermemory | learning model over a typed graph | yes | review, expiry, repetition | spaces | yes, local binary |
| Mnemoverse | concepts plus a ranking that feedback moves | yes, one account | step named, an MCP tool the agent calls | Rooms | Enterprise, by agreement |
So which is best?
The one whose answer to your hardest question is the one you need. If you must run it yourself, five document a local path with no vendor account, and ours offers it on Enterprise by agreement. If you need wrong memories to lose ground over time, two name how that happens, and in both it depends on a rating actually being sent. If your agents live in Letta or your editor speaks ACP, Letta's route is built for you and the others are not. Picking by the number of ticks in a grid is how you end up re-asking this question in three weeks.
Which of the six questions turned out to matter most after you had already chosen?
Disclosure: I work on Mnemoverse, one of the six systems above, so weigh the argument accordingly. The full comparison with every source page is on our library, and the MCP server is open source (MIT): github.com/mnemoverse/mcp-memory-server.
Top comments (13)
The repo is up on my profile: rag-golden-set-toolkit. The article 3 skeleton
is in article3-draft.md, and the first full draft lands as the next commit —
diff welcome, nothing publishes before your read.
Before the full draft lands, one gap between section 3 and the repo is worth closing. Section 3 reports eleven re-runs out of thirty-four with both flips at rank four, a result that needs ranks recorded before the swap, and rule 4 in the README tells a reader to capture them. Yet the rows in golden_set.example.json carry seven fields and none of them is a rank, and ci_eval.example.py checks whether trap_chunk_id is among the retrieved ids without ever writing down where in that list it sat.
The runner already has the retrieved list in hand, in the order the retriever returned it, so the trap's position costs one field next to the verdict and one in the row, written back the way last_validated has to be. With it, every nightly run becomes the pre-swap pass that rule 4 asks a reader without your chunk-diff sidecar to schedule separately, and after the next swap the re-run queue sorts itself from the repo alone.
If the field lands no later than the full draft, section 3 and the code will agree in the version I diff.
Edward, you are right, and the gap is exactly the kind this review contract
exists to catch: section 3 reports both flips at rank four and rule 4 tells a
reader to capture ranks, but the example rows carried seven fields with no
rank among them, and the runner tested membership without ever writing down
where the trap sat. The prose claimed a baseline the code never recorded.
That is the same family of error as the quotation and the "three of four"
line, one direction reversed: this time the record said less than the article
did.
The field lands as its own commit, before the full draft. The row gains
last_rank as its eighth field; the runner gains rank_of(), which returns the
trap's 1-indexed position in the retrieved order, or None when it never
arrived, so the absence stays visible as the "test did not run" case rather
than collapsing into a zero. run() prints the rank beside the verdict and,
with write_back on, stamps last_rank and last_validated back into the row the
way last_validated always had to be. A small revalidation_queue() then sorts
stale rows boundary-neighbors-first from the stamped ranks alone, rows with no
rank to the front, which is your sentence made executable: every nightly run
is now the pre-swap pass, and after the next swap the queue sorts itself from
the repo with no chunk-diff sidecar and no separately scheduled run.
The commit message will be "Stamp trap rank so every run is the pre-swap
baseline", and it lands before the draft commit, so section 3 and the code
agree in the version you diff. Rule 4 in the README changes with it, from an
instruction to schedule a pre-swap pass into a statement that the runner
already stamps one. Thank you for diffing the repo and not just the prose —
this is the fourth time the contract has caught something real, and the first
time it caught it in code rather than in sentences.
The first full draft of Article 3 has landed as its own commit in
article3-draft.md: six sections, the verdict triple and the protocol-state
test credited to you in Methods, the rank stamp from your repo diff already
wired into section 3, and the open questions parked at the end. Diff welcome;
nothing publishes before your read.
Read against the threads it cites, the draft keeps the method intact, and what needs changing is mostly the retelling. Four things matter most. Under two names, every "we" and "our" reads as mine as well, while the suite, its incidents and its numbers are yours, so one sentence under the byline saying so lets those sentences stand. Section 1 gives me the filter clause, which is your improvement: I proposed a second run with the trap removed, and said in the thread that your version is better. Section 3 now says eleven of thirty-four re-runs changed verdict or rank, where the thread says eleven of the thirty-four were re-run and the two that flipped were both at rank four. One sentence on the other twenty-three would also help: I called those numbers the ordering being tested, and they are that only if the twenty-three were re-run as well.
And section 5 credits my survey with the three claims, but they came from my reply in this thread, and the survey is a map of what vendors document, with nothing run, restarted or swapped. The section says it cites by link and has no links yet, so that sentence is the natural place for one: "survey of persistent memory APIs" linked to mnemoverse.com/docs/library/best-p..., the full version with every source page.
The rest is smaller and sits in an issue on the repo, each with the place, the current text, a proposal and its source: the A2A wording against the 1.0 spec, where my own comment was loose as well, the guard the calibration row needs from the feedback thread, the counterfactual pair and the prompt hash that the runner does not have yet, two write-back details, and a few credits, mine included, that say something other than what those people wrote.
Question 3, but for auditability rather than ranking. Of the six, two name the step from rating to ranking — and neither exposes it as a verifiable record. In a governance context, "why did the agent believe this, and when did it stop" is the question you can't answer after the fact unless it was signed and logged when it happened.
Auditability is the right name for the other half of question 3, and it needs a different record from the one ranking uses. A rating record says who marked which memory and when. What a governance review asks for is the read record: what the agent was actually shown when it acted, in what order, and which version of each memory that was. The two diverge the moment a rating lands, because the ranking is recomputed and the earlier order is gone unless it was written down at the time.
So the read record has to be written at read time. It cannot be rebuilt from ratings afterwards, because each rating changes what a replay would return. And "when did it stop" adds one more field on top: the moment a belief was superseded or pushed out of the results the agent sees, which is the event a review most often needs and the one most easily lost.
The first full draft of Article 3 has landed as its own commit in
article3-draft.md: six sections, the verdict triple and the protocol-state
test credited to you in Methods, the rank stamp from your repo diff already
wired into section 3, and the open questions parked at the end. Diff welcome;
nothing publishes before your read.
The draft I can diff is still the one from 1 October: article3-draft.md on main is unchanged since ba2103e, the commit the notes in github.com/kaziava/rag-golden-set-... were written against, and none of them is in it yet. When the next commit lands I will diff it against those notes and keep in the issue the ones that still apply.
The section 3 point from my reply to the same notice that day now has a second reason. If the far jump Ahmet raised under your post happened in that swap, it sat among the twenty-three rows the queue left for later, so the sentence on them belongs in the commit that brings the bridge run.
This reads like the next chapter of your npx post, pointed one layer up.
There you mapped five jobs onto six vendors and showed that the word alone
carries no information; here the question shifts from "does the process start"
to "does the memory survive," which is the precondition the npx post left
open. A memory server that starts cleanly but loses its state across a restart
is a green result whose precondition was never asserted, and it is exactly
the failure mode my golden set catches in retrieval.
The mechanic we've been building in the Article 3 thread lands directly here.
Vendor tag plus replay becomes the persistent-memory equivalent of our
embedder tag plus route stamp: every memory carries the vendor and model it
was earned under, and after a swap the old-tagged memories are stale by
default rather than silently trusted. Your own line about "a question whose
subject is empty by construction" becomes the negative memory test: a
question about something never stored, so any confident answer fails without
anyone judging content. Two independent stacks, one principle, and your
persistent-API survey is the memory-side replay I was waiting on before I
could write the cross-domain section of Article 3.
So this is the ping I owe you. The rag-golden-set-toolkit repo is ready on my
profile — three toolkit files plus article3-draft.md as a fourth, the way we
agreed. The draft will be complete on what it has: eleven of thirty-four with
both flips at rank four, your rank ordering and replay form credited in
methods, the cross-domain frame as a hypothesis the suite can test. Your name
in the byline, no title, no company, no asterisk. First full draft lands as
its own commit and I'll notify you here so you can diff rather than re-read.
Nothing publishes before your read.
And the question I genuinely want your answer to, continuing your own method
rather than proposing a new one: persistent memory APIs differ in what they
promise across a restart — some guarantee durability, some guarantee order,
some guarantee neither. In eval terms, durability is "the memory exists,"
order is "the memory sits where the agent expects it," and the honest verdict
is sometimes "the memory exists but is not where the precondition required."
Do you read those three as separate claims a vendor should stamp on the API,
the way our embedder tag stamps the model that earned a retrieval verdict? Or
is the honest answer that no vendor does that yet, which is exactly why agents
keep leaning on memories across swaps without anyone noticing the
precondition has moved? Asking because if the answer is "no vendor stamps
it," then this post becomes the raw material for a persistent-memory negative
test, the same way your npx post became the raw material for the replay
mechanic.
The survey is a map, not the replay, and that matters for how section 5 cites it. It reads what each vendor's own pages said on 18 September; nothing in it was run, restarted or swapped. So it can stand as the list of what vendors document, and the replay stays an open measurement, which is how your skeleton already frames it: a hypothesis, cited by link once a result exists.
On the question, yes, those are three separate claims, and a negative test can check each one without waiting for anyone to stamp it. Durability: write, restart, read the record back by id; the claim is only that it exists. Order: the same query before and after the restart, compared against the spread of repeated runs, because one identical ordering can also be noise. Eligibility: the question whose subject is empty by construction, next to the same question after the fact is written, which is where your "test did not run" verdict applies. A store can pass the first and fail the other two, and a single green covering all three is the twin green lie one layer up.
I read the skeleton: byline and credits are as agreed, and section 5 framed as a hypothesis is right. I will diff the full draft when it lands.
Edward, thank you for reading the skeleton before the draft — and for the
three-test menu, which is exactly the shape Section 5 was missing. Durability
as write-restart-read-by-id, order as the same query before and after the
restart, eligibility as the empty-subject question next to the same question
after the fact: three claims, three negative tests, no vendor stamp required.
That menu goes into the draft verbatim in structure, credited to you.
Two of your sentences are doing more work than they look. First, "compared
against the spread of repeated runs, because one identical ordering can also
be noise." That is the statistical guard my rank ordering lacked until you
gave it pre-swap ranks: an order claim is only meaningful against a variance
baseline recorded before the event. In the draft I will pair it with the
reproducibility note — both are cases of "free only if you recorded the
baseline first." Second, "a single green covering all three is the twin green
lie one layer up." That is the framing sentence for Section 5, quoted with
your name. And it changes the harness output, not just the prose: the runner
will print a verdict triple per store — durability, order, eligibility — so no
aggregate boolean can ever cover all three again. The CI summary may count
triples; it may not collapse them.
On the survey: agreed, and the draft keeps your framing. Section 5 cites it
as a map — what each vendor's own pages documented on 18 September — while the
replay stays an open measurement, cited by link once a result exists. A map
and a measurement are different sources, and the draft will not let one
impersonate the other.
So: the full draft lands as the next commit in article3-draft.md, with the
verdict triple in the methods, your two sentences credited where they stand,
and the byline and credits exactly as you confirmed them. I will ping you in
this thread the moment it lands. Diff welcome; nothing publishes before your
read.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.