Open source has a discovery problem, and it isn't search. It's that attention gets mistaken for quality.
A GitHub star costs one click. A benchmark run costs days or weeks of careful work. Yet when a developer goes looking for a tool, the star count decides what they try first. That decides which projects get issues filed, contributors, tutorials, and trust. Careful new tools get buried under whatever launched loudest.
I've been on the receiving end of this for a while, so let me make it concrete with numbers anyone can check.
The claim
MemPalace is one of the most-starred open-source AI memory projects, with well over 60,000 stars. Its README describes it as the best-benchmarked open-source AI memory system.
"Best-benchmarked" is a strong phrase. It isn't a statement about popularity or vibes. It's a statement about measurements, and measurements can be compared.
The numbers
On LongMemEval (500 questions, the s_cleaned split), here is what each project publishes.
MemPalace (from its README):
| Mode | R@5 |
|---|---|
| Raw | 96.6% |
| Hybrid (v4), held-out 450 questions | 98.4% |
| With an LLM rerank step | 99% or higher |
palace-rs (from its repo):
| Metric | Result (all 500 questions) |
|---|---|
| R@1 | 0.912 |
| R@5 | 0.986 (493 of 500) |
| R@10 | 0.994 (497 of 500) |
The benchmark itself is designed around 470 questions: the other 30 are abstention questions, which are meant to be excluded from retrieval scoring. On that standard 470-question set, palace-rs scores R@5 0.987 (98.7%). That is above MemPalace's 98.4% hybrid number, the one that depends on boosts built from looking at three specific questions, and it needs no such boosts. I report the all-500 figure above as well so nobody can say the exclusion flatters the result.
The palace-rs run uses its normal exchange-pair chunking, a MiniLM-family local embedder, and hybrid search (BM25 weighted 0.35, cosine 0.65). It uses no generative LLM anywhere in the retrieval path. All 500 questions are scored, including the 30 abstention questions, which all have answer sessions and so can be scored for recall.
That puts palace-rs at 98.6% R@5 against 96.6% raw. It's also slightly ahead of the 98.4% hybrid number, which is the one MemPalace itself calls its honest figure.
The gap is small. On 500 questions, 2 points is 10 questions. I'm not claiming a breakthrough. I'm making a narrower and more useful point: a project with about ten stars published the higher number, and nobody would have found out from the star count.
Where the 98.4% comes from
This part matters more than the headline, and MemPalace deserves credit for how it handled it.
Its hybrid mode adds targeted boosts: one for quoted phrases, one for person names, and one for nostalgia-style phrasing. Those fixes were developed by looking at three specific questions out of the 500.
That is a normal thing to do while improving a retriever. But fixing the failures you can see in the test set stops being a clean measurement. So they publish the held-out 450 as the honest number, which is the right call and the transparent one.
Compare that with the approach palace-rs takes. It reports all 500, and no retrieval parameters were fitted on the test set. The weights, chunking, and scoring were set independently of those questions. Tuning retrieval for three particular questions is not the same thing as having a better baseline, and I'd rather publish the baseline.
The stack difference
Benchmark numbers are one axis. What you actually have to run is another.
MemPalace defaults to ChromaDB. That's a vector database dependency: a service, a client library, a storage format, and a set of failure modes that come with it.
palace-rs is one binary and one SQLite file. Embeddings are computed locally. There is no server to start, no extra process, and nothing to keep in sync. You can copy the whole memory by copying a file. You can back it up with cp.
Simpler to run, and (on published numbers) not worse at retrieval. That combination is the thing I'd want to know about if I were choosing a tool, and star counts hide it.
Why this is bigger than two repos
I don't actually care about the 2 points. I care about what the situation shows.
Stars are a poor proxy for quality, and we use them as one anyway. A star is cheap, public, and instantly legible. Quality is expensive, takes effort to verify, and doesn't fit in a badge. So the cheap signal wins by default.
That causes real damage:
- Good tools starve. A new project doing careful work has no way to get in front of people who'd benefit from it. Without attention it doesn't get issues, without issues it doesn't improve, and without improvement it never earns attention.
- Marketing rewards itself. A strong claim attracts stars, and stars make the claim look more credible, which attracts more stars. At no point does a measurement have to enter the loop.
- Users inherit the cost. If the popular option is heavier, slower to set up, or less accurate than a quieter alternative, every adopter pays for that, and most never find out.
- Claims drift. When a superlative like "best" is never checked, it stops needing to be true. It only needs to be said first and loudest.
None of this requires bad faith. A popular project can make an honest claim that simply becomes outdated, and a launch can go viral for reasons unrelated to engineering. But a claim stays rubbish if it's wrong, however many stars sit next to it.
What I'd like to see instead
If we want popularity to track quality even loosely, claims about quality have to be cheaper to check. A few habits would help:
- Publish reproducible commands. If I can't rerun your benchmark in one command, it's an anecdote.
- Publish per-question outputs, not just aggregates. Aggregates hide where a system fails. Per-question results let anyone audit them.
- Separate tuned from held-out numbers. If you fixed failures you saw in the test set, say so and report the clean split too. MemPalace does this, and more projects should.
- State what's in the loop. "No generative LLM" and "with an LLM rerank" are different systems. Put them in different rows.
- Report on the full, standard question set. Same dataset and same question count is what makes two READMEs comparable at all.
- Treat superlatives as claims. "Best", "fastest", and "most accurate" need a measurement and a date next to them.
What this post does not prove
I want to be clear about the limits, because a post about checking claims should hold itself to the same standard.
- Each project ran its own runner. Both numbers are on the same 500 LongMemEval questions, so the comparison is like for like on the data. Anyone who wants to rule out runner differences can rerun both.
- LongMemEval S is one benchmark. It says nothing about every kind of memory workload. I'm comparing on the split both projects report, not claiming a win everywhere.
- Two points is small. It's enough to disprove "nobody is ahead". It's not enough to call anyone dramatically better.
- MemPalace also reports a 99%-or-higher figure with an LLM rerank step. That's a different system with a generative model in the loop. I'm comparing retrieval without one.
Check it yourself
The palace-rs run is on all 500 questions, with the command and per-question outputs in the repo. MemPalace's 96.6% is on the same 500. Rerun either one and tell me if you get different numbers.
If you maintain a memory tool, an agent framework, or anything else with a benchmark number in the README, do the same. Give the command, give the raw outputs, and let people check.
The takeaway
The next time you pick a tool, put the star count aside for a minute and ask three questions:
- What exactly did they measure?
- Can I reproduce it?
- What do I have to run to get it?
If the answer to all three is on the page, that tool has earned your attention, with ten stars or sixty thousand.
The numbers are public. Check them. Don't trust the stars, and don't just trust me either.
palace-rs is a local-first AI memory tool: one Rust binary, one SQLite file, no external services. Repo and benchmark outputs below.
Links
- palace-rs: https://github.com/AncientiCe/palace-rs
- MemPalace: https://github.com/mempalace/mempalace
- LongMemEval benchmark: https://github.com/xiaowu0162/LongMemEval
Top comments (0)