Would Your AI Benchmark Survive Being Audited Like the Vendors It Judges? Ours Didn't, at First

Part 1 of 2: how we found our own AI-memory benchmark tool calling a fictional API for one of the four vendors it tests, and the two PASS verdicts the honest fix forced us to take back

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/memtrust, an independent, reproducible benchmark harness for agent-memory backends. pip install memtrust-cli.

---

One of the four agent-memory vendors we benchmark, MemPalace, has two public GitHub issues about its own benchmark numbers. One has 233 thumbs-up reactions and 39 comments, and the other has 42 and 8 (both verified live via the GitHub API on 2026-07-17). They document a headline 100% LongMemEval score measured with Haiku reranking that was never reproducible from the repo's own benchmark scripts, and it was eventually pulled from the README as unverifiable. A separate 96.6% figure that gets quoted everywhere turns out to be mostly the default embedding model doing the work in raw mode. A "lossless" compression claim drops that same 96.6% to 84.2% once you actually measure it, a 12.4 percentage point gap. Two pull requests attempting to fix the reporting, MemPalace/mempalace#433 and MemPalace/mempalace#729, were both closed unmerged the same day, April 12, 2026. #729 was closed within seven minutes of being opened (verified against the PR's own timestamps on GitHub).

This isn't an isolated case. Stanford's 2026 AI Index Report has a line in its technical performance chapter that should worry anyone citing a benchmark number in a sales deck. On SWE-bench Verified, model performance went from 60% to near 100% in a single year (Stanford HAI, 2026 AI Index Report). A jump like that signals a benchmark saturating faster than anyone can redesign it, which means the number stopped measuring the thing it was built to measure. This problem has real precedent: a 2019 study built fresh test sets for ImageNet and CIFAR-10 matching the original collection methodology as closely as possible, and found accuracy dropped 11 to 14 percentage points on ImageNet and 3 to 15 points on CIFAR-10 the moment the benchmark stopped being one everybody had already been implicitly tuning against for years (Recht et al., ICML 2019).

We built memtrust to stop guessing which vendor's number to trust. We run LongMemEval, LoCoMo, and a growing set of evals we wrote ourselves against all four backends the same way, and we publish the raw logs so the numbers can be whatever they actually are. While doing that work, we found something uncomfortable: our own tool had been making the same kind of unverified claim about itself, in its own code, since the very first version.

A fresh clone, zero credentials, memtrust run --backends mempalace,mem0,zep,openviking --eval all. Every backend reports a real SKIPPED status, and the run still exits cleanly with a real report. That's the honesty standard the tool is supposed to hold itself to, and it matters for what comes next.

Our MemPalace Adapter Called a Fictional Class

The adapter talking to it is a normal piece of glue code. memtrust calls store(), query(), update(), and delete(), and the adapter translates those into whatever the vendor's real API expects. Every version of that adapter, until this rewrite, called mempalace.Palace(storage_path=...), a class with .remember(), .recall(), and .invalidate() methods.

We ran one line to confirm it before starting the rewrite:

$ python3 -c "import mempalace; hasattr(mempalace, 'Palace')"
False

The exact terminal check that surfaced this. Palace is not an attribute of the installed mempalace package, version 3.5.0. Every store/query/update call this adapter ever made against it ran through a class that doesn't exist.

We grepped every ^class definition across the entire installed package to make sure we weren't missing a re-export somewhere. mempalace/palace.py, the file the class name implies it should live in, defines exactly two things: MineAlreadyRunning and MineValidationError, both exceptions. There is no Palace class anywhere in the package.

That means store(), query(), and update() had never worked against the real vendor package in this project's entire history. Every test that appeared to pass was exercising a hand-written fake standing in for a guess about an API that was never confirmed, and that guess was wrong. The adapter's own pre-rewrite documentation had already flagged this honestly as a low-confidence guess: "Confidence: MEDIUM on product behavior, LOW on exact Python method names." Low confidence turned out to mean zero confidence, because the class was fictional.

Here's the part that made this more than an embarrassing bug: memtrust exists to catch a vendor claiming something its product doesn't actually do, and we'd just found our own tool doing precisely that about its own capability, unnoticed the whole time.

Rewriting the Adapter Against the Real MCP Server

The fix required calling the vendor's real MCP server directly. Its tool calls dispatch to plain, real, module-level functions: tool_status, tool_list_wings, tool_list_rooms, tool_add_drawer, tool_search, tool_update_drawer, tool_delete_drawer, tool_kg_add, tool_kg_invalidate, tool_kg_query. Those functions had already been in limited, correct use in one narrow corner of the same adapter for metadata calls. The rewrite made them the only way the adapter talks to it for everything.

We verified the vendor's docstrings by running the code. Every response shape documented in the rewritten adapter was captured by calling the real functions live against a real local, chromadb-backed palace. We ran pip install mempalace, pointed MEMPALACE_PALACE_PATH at a temp directory, called mempalace.mcp_server.tool_add_drawer(...), and read what actually came back. Live-verification caught behaviors that a docs read alone would have missed or gotten wrong:

• tool_search results carry no per-record ID field at all. There's no id, no drawer_id, nothing, just content and scoring fields. We considered recomputing an ID client-side from the content, then rejected it: the vendor's update_drawer() keeps a drawer's original ID stable across a content edit on purpose. We verified this ourselves by updating a drawer and confirming the pre-update ID still comes back. Recomputing from a query response's current content would silently produce the wrong ID for anything that had ever been edited. The adapter now reports memory_id="", which is an honest admission that the real vendor response has nothing to put there.
• The knowledge-graph subsystem ignores MEMPALACE_PALACE_PATH entirely when called as a library. We traced this to a module-level flag that only ever gets set to True when the process's own command-line arguments carry a --palace flag. A library caller never passes that argument. The practical effect is bad: two MemPalaceAdapter instances pointed at two different storage paths in the same process both read and write the exact same physical knowledge-graph file on disk. That's a real vendor bug we found by exercising the adapter's own test setup, and it was an unreported issue, so we wrote it up.

One finding cut the other way. The vendor's own docstring for tool_add_drawer claims that content chunked across multiple physical rows can't be updated or deleted by its logical ID. It states, "tool_get_drawer(drawer_id) and tool_delete_drawer(drawer_id) report 'not found' on the chunked path." We read the actual function that resolves those calls, and it handles the chunked case correctly. Live-testing confirmed it: storing 2,000 characters of content, well over the 800-character default chunk size, then updating or deleting it by its logical ID worked right every time we tried it. An earlier draft of this rewrite had trusted the vendor's docstring text here without checking it live, which would have shipped a false "confirmed limitation" in the other direction. Both kinds of error, trusting a fictional API and trusting stale documentation, come from the same root cause: trusting written claims over executable code.

Re-Verifying Our Own Prior Verdicts

A rewrite fixing the fictional API is still just a fix made and validated by the person who wrote it. The standard this project applies to every vendor is that a self-report requires independent verification against reality, so we applied that same standard to ourselves.

Thirteen rows in memtrust's issue-validation log, our internal record of every real GitHub issue we've investigated against these four backends, had a verdict on this vendor's repo sitting on PASS or deferred. They were built and tested against the old, fictional adapter. Once the real rewrite existed, every one of those thirteen went back through independent, freshly spawned reviewers who had no memory of building the original fix. They checked each claim against the real, working code from scratch.

Eleven of the thirteen held up. Two of those eleven had real, working mechanisms that the rewrite had accidentally dropped along the way, ordinary collateral damage from restructuring a file: an authored_at ranking fallback and a docstring section distinguishing "no API key required" from "no network access required" both got restored, and we wrote new tests proving them against the real adapter this time. One row was confirmed correctly still failing, matching how it had been marked before. The remaining two rows turned out to be wrong and got downgraded.

Downgrading PR #1005 for Unshipped Capabilities

A prior version of memtrust claimed it could detect when the vendor's search silently degrades, falling back and reporting a partial result. It stated the claim was "confirmed against the real, merged PR diff" for MemPalace/mempalace#1005. Two things turned out to be false in that sentence. First, PR #1005 was never merged. GitHub shows it closed unmerged by its own author as self-superseded. We confirmed this directly against the live PR. The retrieval-fallback mechanism it proposed did land eventually through separate PRs, but the observability fields memtrust's code was reading, warnings and available_in_scope, were never shipped in any of them. Second, we grepped the real, installed package's source for those exact field names and found zero matches anywhere. The code path that would populate them does not exist in the shipping product.

The parsing logic itself is fine and won't crash or misread a response if that shape ever does appear. It simply never fires because the vendor never sends that response. We had correctly implemented a check against a capability the vendor hasn't built yet, and that's a weaker claim than the one we had been making. The verdict moved from PASS to PARTIAL to reflect that gap.

Downgrading Issue #1733 for Measuring the Wrong Code Path

The second downgrade cuts deeper. A contributor named Kartalops filed a real, well-diagnosed bug. The vendor's memory "wake-up" sort was supposed to prioritize high-importance, recent memories. Nothing in the real ingest path ever wrote the importance field it sorted by. Every memory defaulted to the same value, and the sort silently degenerated into plain insertion order. A prior memtrust fix built a ranking-quality signal and marked this PASS on the theory that it could now detect that exact failure shape.

Once we traced the real code paths after the rewrite, the fix and the bug turned out to live in two different parts of the vendor's own codebase. Kartalops's bug is in mempalace/layers.py, in a function called Layer1.generate(), the actual "wake-up" sort. memtrust's adapter has never called that function in any version, including the fictional one. What memtrust actually calls is tool_search, a structurally separate code path that sorts purely by vector similarity. The ranking-quality signal we built is real and does catch a real class of ranking degradation, just a different one. It was checking a field that belongs to a method this adapter has never invoked.

A capability gap sits a category above a parsing bug or a missed edge case. There is currently no way for memtrust to reach, measure, or detect the bug Kartalops reported through any code path this adapter has, so the verdict moved from PASS to FAIL, capability gap. That's the honest label for a simple fact: right now, we can't do this.

The Final Validation Numbers Across All Backends

Across all four backends, 197 real GitHub issues and PRs have been investigated this way. Each one asks the same question: given the code as it actually exists today, can this tool diagnose or resolve the exact failure that was reported? Here are the results after this re-verification and the two downgrades above:

Verdict	Count	Share
PASS	55	28%
PARTIAL	16	8%
FAIL, capability gap	42	21%
FAIL, not applicable	84	43%

197 issues investigated in total.

55 out of 197 is a sobering result compared to a vendor's 96.6%, but it's the number that was still standing after we went looking on purpose for the kind of overclaiming this project exists to catch, and we found it in our own code before anyone else did. A PASS rate that survives someone actively trying to break it carries weight that an unchallenged PASS rate lacks.

Known Gaps in the Remaining Adapters

This adapter is the one we can currently stand behind at this level of confidence, because it's the one we rewrote against real, live-verified functions and then independently re-checked. The other three adapters are still works in progress. memtrust's own methodology documentation grades adapter confidence on a scale from High to Low, and today that grade actually varies quite a bit from one adapter to the next. We assign high confidence where source code or a documented SDK was read directly and exercised against real installed packages. We assign low confidence anywhere a method signature is still a best-effort reconstruction from documentation that lacked full availability during the build. OpenViking's adapter is flagged as the one most likely to need correction against a live instance. Its public documentation covers resource and skill ingestion in detail while leaving out a confirmed endpoint for writing or querying a conversational memory entry.

This fix settles something narrower than every number in this project. One previously false claim is now either true or honestly labeled as pending. The process used to get there involved independent re-verification by reviewers separate from the original fix, and we intend to keep pointing that same process at the rest of the codebase.

Finding a Vendor Bug During Our First Live Run

Everything above covers a bug in our own MemPalace adapter. While we were fixing it, memtrust also produced its first live benchmark result against a different backend, and getting that result turned up a second bug, this one belonging to the vendor.

mem0_direct runs the actual, self-hosted mem0ai open-source library in-process, backed by a local Qdrant instance and OpenAI for embeddings and extraction. This targets the self-hosted library only, not the commercial product built on top of it, and that's an important distinction to keep in mind for everything below.

A clean mem0ai install with nothing set but OPENAI_API_KEY fails every single LLM-based extraction call out of the box, for anyone. mem0's own default model, hardcoded in mem0/llms/openai.py as "gpt-5-mini", is a reasoning-tier model that only accepts the API's default temperature. mem0's own reasoning-model detection in mem0/llms/base.py checks its model-name set for the string "gpt-5o-mini", one character off from the model it actually ships, so the check never matches, and mem0 sends temperature=0.1 on every call regardless. Every extraction call comes back a 400 error for anyone who just sets an API key and runs it. Our Mem0DirectAdapter now works around it by passing is_reasoning_model=True, mem0's own documented override for this exact situation. mem0ai/mem0#6085 documents the same failure independently; a different user filed it on 2026-07-04, 16 days before we hit it ourselves.

With that fixed, we pointed it at contradiction detection, compression/round-trip fidelity, and extraction quality. Contradiction detection ran seven cases: we told it a fact, then contradicted it later, and mem0 silently overwrote the old value every single time, yielding a 100% silent-overwrite rate and 0% flagged as a conflict. Compression/round-trip fidelity ran five cases and came back with 31.6% mean literal fidelity. That low score is expected, since mem0 performs semantic extraction only, so a literal-reconstruction score was always going to be low; it just quantifies the cost of that design choice. Extraction quality ran fifteen cases, twelve deliberately junk and three deliberately valid. All twelve junk inputs were correctly rejected, and all three valid ones were lost on retrieval. That valid-case sample is only three, small enough that it's worth digging into further rather than treating as conclusive.

We wanted to include this here right away. The same standard that caught our own fictional API in the MemPalace adapter caught a real bug in a vendor's default configuration on the very first live run against a second backend, which tells us the methodology holds up against everyone's code, including our own.

Existing Methods for Verifying Benchmark Claims

Set aside memtrust for a moment. If you're evaluating an AI-agent memory backend, or any AI system whose vendor publishes its own accuracy claims, you have roughly three existing paths to figuring out whether a number is real, and each one leaves a gap.

General-purpose LLM and RAG evaluation frameworks give you real, usable metrics. They offer faithfulness, answer relevancy, hallucination detection, and custom LLM-as-judge scoring built for evaluating a single model's output against a single-turn or single-session query. We checked the documentation of RAGAS, DeepEval, and TruLens, the three most widely used open-source frameworks in this category, directly. None of them ship a first-class way to test memory. They lack tests for recall across separate sessions days apart, what happens when a new fact contradicts an old one, or how retrieval quality decays as the knowledge base ages. The closest built-in metric tracks retained facts "throughout a conversation," by its own documentation's wording, which means one continuous session, stopping short of the multi-session boundary an agent-memory product is actually supposed to solve. You can build that evaluation on top of these frameworks' general primitives, but nobody ships it out of the box.

Vendor self-reported benchmarks are the default path because they're the ones already sitting in the product's own documentation. Setting dishonesty aside, the structural problem is that self-grading and public grading face different incentives, and there's a documented asymmetry in how freely that grading gets checked. A 2024 industry analysis found that four of thirteen vector-database vendors it reviewed contractually prohibited customers from publishing independent benchmark results against them (BenchANT, "To Benchmark Vector Databases or to Get Sued for Breaching a DeWitt Clause?", 2024). Vendors remain free to publish comparative numbers about themselves; whether someone else can check that number against reality is a decision the vendor's own contract makes for you, in a documented number of real cases.

Academic benchmark suites are the strongest, most rigorous foundation available, and also the ones with the least reason to overstate a result. LongMemEval, from researchers at UCLA, Tencent AI Lab, and UC San Diego, tests five distinct long-term memory abilities across curated question sets embedded in chat histories that scale up to 500 sessions and roughly 1.5 million tokens. Its own headline finding is highly critical of the systems it tested: ChatGPT, running on GPT-4o, answered 91.8% of questions correctly when given the full conversation directly, and only 57.7% once its own memory retrieval sat between the model and the answer (Wu et al., "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory," ICLR 2025). LoCoMo, from UNC Chapel Hill, USC, and Snap Research, built very long synthetic conversations and found an even starker gap. Human accuracy on its question-answering task measured 87.9 F1, while the best model tested, GPT-4-turbo, reached 32.4 (Maharana et al., "Evaluating Very Long-Term Conversational Memory of LLM Agents," ACL 2024). Both are the kind of rigorous, adversarial ground truth a vendor benchmark should be measured against, but they still leave an operational gap: you still need to know whether a specific vendor's product, wired up the way you'd actually deploy it, behaves the way the vendor's marketing page says it does, and whether it catches the specific failure a real user already hit in production.

A persistent gap runs through all three of these approaches. None of them makes it someone else's ongoing job to point an adversarial standard at real, reported production failures against a specific vendor's actual shipped product, or to republish the raw evidence and take back a result once it's shown to be wrong. That's the job memtrust is built to do, and it treats its own prior verdicts as requiring the same fresh, adversarial check as a vendor's claim.

Process Changes Driven by the Fictional API Bug

The surface lesson is: verify vendor APIs before you ship an adapter. The narrower, less comfortable lesson underneath it is that a tool built to catch overclaiming is just as capable of producing it. The only thing that caught it here was treating our own PASS verdicts with the same suspicion we point at everyone else's, on a fixed schedule rather than only when something goes wrong.

The test suite behind all of this currently runs 590 passing tests, 8 skipped, entirely offline, with zero live vendor credentials required. That number proves the eval logic is internally consistent. It doesn't prove any single adapter's wire format matches a live vendor's real API, and that's why the re-verification step above exists as a separate, independent check.

Part 2 of this series covers what happens when you combine this verification discipline with a distribution layer that lets an agent or CI job install and run the tool with no human in the loop.

Repo: github.com/RudrenduPaul/memtrust. The full 197-row validation log, commit by commit, is in the repo if you want to check any of the numbers above yourself. That transparency is the entire point of publishing them this way.

---

References

• Stanford Institute for Human-Centered Artificial Intelligence. The 2026 AI Index Report. 2026.
• MemPalace/mempalace#27: "Multiple issues between README claims and codebase." Opened April 7, 2026.
• MemPalace/mempalace#29: "Multiple issues with benchmark methodology and scoring." Opened April 7, 2026.
• MemPalace/mempalace#433: "feat: rule-based contradiction detection (issue #27)." Closed unmerged April 12, 2026.
• MemPalace/mempalace#729: "fix: Multiple issues between README claims and codebase (closes #27)." Closed unmerged April 12, 2026, seven minutes after opening.
• BenchANT. "To Benchmark Vector Databases or to Get Sued for Breaching a DeWitt Clause?" 2024.
• Wu, Wang, Yu, Zhang, Chang, Yu. "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory." ICLR 2025, arXiv:2410.10813.
• Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang. "Evaluating Very Long-Term Conversational Memory of LLM Agents." ACL 2024, arXiv:2402.17753.
• Recht, Roelofs, Schmidt, Shankar. "Do ImageNet Classifiers Generalize to ImageNet?" ICML 2019, arXiv:1902.10811.

---

Co-authored by Rudrendu Paul and Sourav Nandy. Rudrendu is an engineer building agent-native open-source tooling, including memtrust and a companion suite of AI-agent infrastructure projects. Find the code at github.com/RudrenduPaul.