Bench'd (benchd.ai) calls itself the neutral benchmark authority for AI memory, and sells vendors a verification badge from $299 to $3,999.99 a month.
I ran my memory system through their harness. Then I checked their board. Every number below is from their own site and repository, read on 2026-08-29, and every one takes seconds to verify.
The numbers are fake
Their three track leaders, recomputed against the harness's own aggregation rule:
| Track | #1 | Score | What its own dimensions say |
|---|---|---|---|
| Knowledge Brain | gbrain | 100.0 | perfect recall, reliability score 4.0 |
| Agent Memory | Letta | 80.0 | own dimensions average 26.7 |
| Conversational Memory | LangMem | 60.0 | dimensions are all zero |
A 100.0 beside a reliability score of 4.0 is the signature of an adapter echoing expected answers. An 80.0 built from dimensions averaging 26.7 is an aggregation bug. A 60.0 from all-zero dimensions is not a number.
None of the three has a manifest anyone can download. And per Bench'd's own methodology page, rows marked Community-Verified are "independently run" by Bench'd. All three are. So these are not vendor claims. Letta did not submit an 80.0. gbrain did not submit a 100.0. The vendors named in those rows are being misrepresented by the harness, not caught by it.
Filed as [issue #5 (https://github.com/benchdai/harness/issues/5) on 2026 08-23. Still on the board.
The trust is fake
Their methodology page has a section headed Verify a Receipt Yourself: "You don't need to trust us. Every receipt can be independently verified." It gives you this:
```bash curl -sL https://benchd.dev/api/receipt/run_abc123.json -o receipt.json
`benchd.dev` does not exist. NXDOMAIN against 1.1.1.1, against 8.8.8.8, and against my local resolver. `benchd.ai` resolves fine.
```console $ nslookup benchd.dev 1.1.1.1 *** one.one.one.one can't find benchd.dev: Non-existent domain
The entire independent-verification story terminates at a domain that has never existed. Nobody has ever verified a Bench'd receipt using Bench'd's published method, because there has never been a host to fetch one from. That check took four seconds.
Two more, quickly. Their trust page defines authenticity as a signature from "a key published on this page"; the key signing their own shipped manifests matches neither published fingerprint. And their trust page promises they take no payment from vendors on the leaderboard, while their pricing page sells those vendors a leaderboard badge. An organisation that ranks vendors cannot also invoice them.
Nobody is home
- The harness repo, the entire basis of the "independent, reproducible, open" claim: no commit since 2026-06-06. Eighty-four days.
- Five open issues, from two different vendors. Every comment on all five is from the person who filed it. Dustin-a11y has been waiting since 14 July.
- Their submission endpoint caps uploads at ~4.5 MB, so no full trace manifest fits. Not even their own 16 MB manifests fit.
Submit real traces, get rejected by an unmaintained upload cap. Submit nothing, be Community-Verified at 100.0.
Meanwhile the board publishes receipts dated two days ago, and the pricing page keeps selling.
The control finished third
Bench'd runs a control: GPT-4o-mini with no memory attached. Conversational Memory track, 2026-08-29:
| # | System | Score |
|---|---|---|
| 1 | LangMem | 60.0 |
| 2 | LlamaIndex Memory | 59.0 |
| 3 | no memory at all | 57.6 |
| 4 | LangChain Memory | 34.0 |
| 5 | Mem0 | 32.4 |
A benchmark whose control finishes third is not measuring the thing it is named after.
This is the industry, not one website
Bench'd is downstream of a worse problem: the numbers it aggregates come from READMEs, and the READMEs do not survive checking.
Over the past months I have checked the benchmark claims of several memory projects and have not yet found one that held up. A published 96.6% recall that becomes 87.0% when rescored with the benchmark's own scorer, because the field reports recall_any where the benchmark specifies recall_all. A "clean held-out" set built after the tuning it was meant to hold out. LoCoMo ground truth with roughly 99 wrong or misattributed answers, per @dial481's audit, and a judge accepting ~63% of deliberately wrong answers. Session-level LoCoMo recall quoted without mentioning that a random ranker scores 0.371 at the depths everyone reports.
None of that is fraud. It is a field where the number goes in the README, the arithmetic goes nowhere, and nobody is expected to check. Every project that publishes a soft metric under a hard name makes it more rational for the next one to do the same. The honest ones lose ground for being honest, which is the part that actually disrupts the market.
What I measured, briefly
RE-call, through their harness, unmodified scoring path, on 2026-08-23: 69.0 LongMemEval, 71.6 LoCoMo, $6.64 total spend. The no-memory control is 57.6, so the number that means anything is +11.4 over having no memory. I am giving you the control so you can compute that instead of trusting a ranking.
Two disclosures, since I am demanding them from everyone else. I set RE-call's abstention threshold to zero: every question is answerable by construction and the judge scores "insufficient information" as wrong, so honest abstention forfeits 26.7 points, and refusing when nothing clears the threshold is the whole premise of my product. And I compare only within the Conversational Memory track.
pip install benchd-harness
benchd verify longmemeval-v1.manifest.signed.json
Signed manifests, run records and SHA256SUMS: benchd official-2026-08-23.
Pre-registered before measurement, wrong predictions left standing.
Over to you
Every check above took minutes. The DNS one took four seconds. If the checks are that cheap and keep coming back positive, the problem is not difficulty, it is that nobody is expected to look.
Three questions, and I will answer every reply:
Is this just normal now? Have you ever gone to verify a benchmark number in a README and found it did not hold? I want to know whether my hit rate is bad luck or the base rate, because from where I am standing it looks like the base rate.
What do you do when it is your project on the losing end? You ship a real number. Someone ships a bigger one that nobody checks, and takes the users, the stars and the funding with it. Do you match them and publish your softest metric, spend your time auditing theirs, or eat it and keep building? I have picked one of those three and I am not sure it was right.
What actually gets an unreal number corrected? This is the one I most want answers to. A filed issue gets ignored. A polite email gets ignored. A public writeup costs you a week and makes you look like a competitor with an axe. What has worked for you: reproduction repos, pressure from the benchmark's own authors, a shared audit registry, something I have not thought of? Not how to attack anyone, how to make the correction actually land, because everything I have tried so far has mostly not.
If any line here is wrong, name it and I will correct it publicly, same prominence, same day. I put that offer in their issue tracker before publishing rather than after.
Top comments (0)