Giving an AI Agent a Benchmark Tool It Can Install and Actually Trust

Part 2 of 2: the npm-to-uv-to-PyPI bridge that closes the installation gap, and the adversarial re-verification discipline that makes the output worth acting on autonomously

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/memtrust, an independent, reproducible benchmark harness for agent-memory backends. pip install memtrust-cli.

Quick summary: memtrust-cli is now on npm. npx memtrust-cli runs a Python benchmark tool automatically using uv, with no Python toolchain required. Below is the full bridge design, a concrete temporal-KG boundary bug the eval caught, and the verification discipline that makes the output worth acting on autonomously.

---

In Part 1 of this series, we found our own MemPalace adapter calling a fabricated API method. Every result it had ever reported for that backend was fiction. Closing that gap meant an adversarial, from-scratch re-verification of every adapter in the project. That story stands on its own, and this one picks up right after it to answer a related question: when an autonomous agent runs this tool, what does "trustworthy" actually mean?

Here's the real command surface memtrust ships today. This is the output an agent or a CI script parses.

Every flag shown here is real, pulled straight from memtrust run --help. This GIF shows the live output.

Defining "agent-native" requirements

"Agent-native" gets used loosely enough that it needs a strict definition. A tool that's actually agent-native has to satisfy two conditions at the same time:

1. Installable automatically. An agent or a CI job needs to go from "nothing installed" to "tool running" using a single programmatically constructed command, and it needs to run on whatever runtime the environment already has.
2. Machine-legible. Every response has to be structured data a program can parse and act on directly.

Most developer tools, including memtrust's earlier releases, miss both marks. Running pip install memtrust-cli assumes the system already has a specific Python version provisioned, a virtual environment configured, and a writable path for pip. A CI runner or an agent sandbox shipping only Node hits a dead end there. Early memtrust CLI output also targeted human readers only. We printed colored tables, progress bars, and summary lines.

The npm side of this project closes the installation gap. The rest of this article focuses on the legibility problem, which turned out to be the harder challenge of the two.

Live on both registries

You can install memtrust-cli via npx memtrust-cli right now. It's on the npm registry. Running npm view memtrust-cli version against the registry today returns 0.3.3, which matches the version on PyPI.

We shipped a complete, tested npm package at npm/memtrust-cli/ in the repo alongside six per-platform companion packages (@memtrust-cli/darwin-arm64, @memtrust-cli/darwin-x64, @memtrust-cli/linux-arm64, @memtrust-cli/linux-x64, @memtrust-cli/win32-arm64, @memtrust-cli/win32-x64). The publish process followed a strict dry-run discipline: npm pack --dry-run produced a correct tarball, then npm publish --dry-run walked through the whole publish flow with registry auth included, and only after that did the real npm publish run. Running a dry run before the real thing is the same verification habit we apply to eval claims. We just applied it to a release instead.

The live path on PyPI is pip install memtrust-cli, version 0.3.3, confirmed against PyPI's own API on 2026-07-21, the date of the npm publish. A mirror package named memtrust currently sits at the same version for anyone who reaches for that shorter name, kept in sync by a manual release step rather than an automated one. Every example below works today through either pip or npx.

Here's the full implementation of bin/memtrust.js. The entire design fits in well under a hundred lines:

// Pinned to this npm package's own version, not left floating: "uv tool run
// --from memtrust" with no version qualifier always fetches whatever is
// currently newest on PyPI. Pinning to this package's own version keeps the
// two registries in lockstep: bump this package.json's version when PyPI's
// memtrust ships a new release, and every subsequent npm install/npx run
// resolves to that exact pinned release, not whatever is newest at run time.
const PACKAGE_VERSION = require(path.join(__dirname, "..", "package.json")).version;
...
const result = spawnSync(
  uvPath,
  ["tool", "run", "--from", `memtrust==${PACKAGE_VERSION}`, "memtrust", ...process.argv.slice(2)],
  { stdio: "inherit" }
);

The script resolves a platform-specific uv binary from require.resolve(), hands off every argument unchanged, and delegates all memtrust logic to the binary, pinning only the PyPI release to its own package version. Reading the whole file top to bottom takes about a minute, which was the point: we wanted a small bridge you can audit at a glance, one that gives you both determinism and simplicity without hiding anything.

We spent a good amount of time deciding where to actually source the uv binary from. Each platform package's prepack script fetches the matching uv release archive directly from Astral's GitHub releases, along with the per-archive .sha256 checksum file uv publishes alongside it, and the script requires an exact match before it will proceed. The maintainer running the release does this verification once, at npm publish time. We chose that tradeoff on purpose: a maintainer verifying the checksum with full visibility into the supply chain gives a stronger guarantee than repeating the check on every install, and downstream npm install runs simply rely on that already-verified state. An offline or network-restricted install never has to reach out to Astral's servers at all, since the uv binary already sits in node_modules ready to run.

uv tool run --from memtrust==<pinned version> memtrust <args> does the heavy lifting after that. It provisions a Python interpreter if the system doesn't have one, installs that pinned memtrust release from PyPI into an isolated tool environment on first use, then caches and runs the tool. An agent or a CI job with Node available ends up with a fully working Python benchmark tool, no extra setup required.

Node's footprint is big enough to justify closing this gap on its own. Sonatype's 2026 State of the Software Supply Chain Report puts npm at 7.97 trillion package downloads across 2025, up 65.43% year over year, against 804.97 billion for PyPI in the same period (Sonatype, 2026). Take that as a rough proxy for what's likely sitting in CI images: Node-based tooling reaches a huge share of environments by default, which raises the odds that Node is already present even where a curated Python toolchain isn't. A Node wrapper bridges that gap directly.

The uv tool itself has real adoption behind it. Astral's repository sits at 87,595 GitHub stars as of 2026-07-17 (GitHub API, astral-sh/uv). pypistats.org reports 165.67 million downloads in the trailing month. The Python Developers Survey run by the PSF and JetBrains found uv usage went from 0% to 11% adoption in the single year it was introduced, reported in JetBrains' "The State of Python 2025." Stack Overflow's 2025 Developer Survey puts uv at 9.5% adoption against pip's 40.9% and npm's 56.8%. Building a bootstrap step on top of uv was an easy call given those numbers.

Generating structured JSON output

Every memtrust run invocation writes a full JSON report to disk by default, saving to ./memtrust-report-<date>.json unless you point it elsewhere with --output. The command produces that JSON report on top of its console summary, so you never have to remember a --json switch. The report's results block breaks down per-backend, per-eval outcomes using the same signal taxonomy the underlying evals use internally. The memtrust run --eval command currently exposes seventeen registered evals, from longmemeval through temporal_kg_boundary. A program reading the file gets the same verdict a person reading the console table would get.

Here's the actual payload an agent or CI job parses. We pulled this directly from a real memtrust run --backends mempalace,mem0,zep,openviking --eval all invocation against memtrust 0.3.3. The run used zero configured credentials, and we left the output completely unedited:

{
  "mempalace": {
    "status": "skipped",
    "reason": "mempalace is not configured: environment variable MEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup instructions.",
    "missing_env_var": "MEMPALACE_STORAGE_PATH"
  },
  "mem0": {
    "status": "skipped",
    "reason": "mem0 is not configured: environment variable MEM0_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup instructions.",
    "missing_env_var": "MEM0_API_KEY"
  }
}

The output is strict key-value pairs. The status and missing_env_var fields give programs machine-checkable data to act on right away, which leads to a design choice worth calling out.

We added a memtrust report <path> subcommand for reading a prior run's JSON back and printing a formatted summary. That's useful when the agent running the eval and the person or system reviewing it are two separate processes.

There's an easy way for a parser to get this wrong, though. A backend missing a configured credential environment variable keeps the run alive: it prints SKIPPED in that backend's section and continues on. A CLI running unattended in CI needs to behave this way, since a missing API key for one backend shouldn't stop the rest of the eval suite from finishing. But that means a program parsing the JSON has to check each backend's own status field. If it relies only on the command's exit code, it'll wrongly assume a skipped backend actually passed.

The CLI also includes a --sign flag backed by an Ed25519 keypair generated with memtrust keygen. This writes a signed receipt alongside the JSON report, and the signature proves which key produced the file and that the contents haven't been altered. We built this for agent-to-agent consumption: structured output only means something if the agent reading it can verify where it came from, and an unsigned JSON file is trivial to spoof. Signing stays optional for standard runs, but it's there for reports that need to cross a trust boundary.

Current approaches to agent tool distribution

Several existing practices try to close the agent-tool distribution gap. It's worth being precise about what each one actually solves, and where each one stops short.

Protocol-level standards for agent-tool-calling give a direct answer for tool invocation. The Model Context Protocol, introduced in November 2024 and donated in December 2025 to the newly formed Agentic AI Foundation, a directed fund under the Linux Foundation, reported over 97 million monthly SDK downloads and more than 10,000 active public servers at the time of that announcement. A protocol standard solves the invocation contract: it defines how a tool describes its own capabilities and how an agent calls them in a predictable shape. But the underlying tool still bears full responsibility for returning truthful values. Standardizing the envelope leaves the contents completely unverified.

Traditional package-manager install flows, including pip and npm's default behavior, need a human reading terminal output to make judgment calls. The user has to accept prompts, notice warnings, and interpret stack traces. A default pip install or npm install outputs unstructured text, because the people who designed these tools built them for a person watching the terminal, not a program parsing it.

Benchmark and leaderboard self-reporting in the broader ML ecosystem has a well-documented gaming problem. A 2025 analysis found that one major model provider privately tested 27 model variants against a popular crowdsourced arena benchmark before release. Proprietary models received access to 54.3% to 70.1% of the benchmark's evaluation data, while openly available models got far less. That additional arena data access alone produced score gains of up to 112% relative to a baseline (Singh, Nan, Wang, et al., "The Leaderboard Illusion," NeurIPS 2025). A separate survey documented the broader pattern of train/test contamination undermining leaderboard comparisons across the field. A self-reported number that relies entirely on trust is easy to optimize toward instead of actually earning.

Detection-capable claims in vulnerability and bug-finding tooling frequently get conflated with live-reproduced claims. The industry has started to recognize this distinction directly. Recent work building automated exploit-verification pipelines highlights the difference: identifying a plausible flaw is a categorically easier claim than producing a working reproduction against a real running system, and treating the two as equivalent overstates what the detection tool actually found (a 2025 paper on automated exploit generation and validation makes this argument in those terms).

There's a gap running through all four of these categories. Each one lets a tool output well-formatted or plausible data without proving it's true. A protocol standardizes the call. A package manager gets the bits onto disk. A leaderboard reports a number. A detection tool flags a candidate. What none of them do is the adversarial re-verification this project treats as mandatory. An agent-native benchmark tool has to add that on top of everything those categories already handle well: install cleanly, call predictably, report a number, flag a candidate, and then prove that number or flag actually holds up when an independent engineer checks the work.

Verifying the tool's internal claims

The distribution mechanics only matter if the underlying tool is trustworthy. An agent calling memtrust run autonomously parses the JSON and decides what to do next, with no visibility into any silent failures behind it. Part 1 laid out exactly what's at stake here: a fictional API call inside that adapter forced every downstream consumer to act on claims nobody had actually verified.

The discipline from that re-verification keeps running on repeat. Every time the project's backlog surfaces a missing capability, an engineer looks at the actual gap, builds the smallest piece of it that can genuinely be closed, and re-verifies the result independently before calling it done. We scope partial fixes to their real boundaries on purpose, to keep the backlog honest rather than padded. A recent knowledge-graph boundary bug is a good example of this loop in action, and the memtrust eval suite missed it entirely until a few days ago.

Fixing the temporal-KG boundary bug

The bug lives upstream, inside the vendor's own knowledge-graph store. Issue MemPalace/mempalace#1913 reported a race condition: querying a knowledge-graph fact with an as_of point-in-time filter at the exact instant that fact was invalidated returned both the old value and the new value at once. The code used a closed-interval SQL comparison (valid_to >= as_of) where it needed a half-open interval (valid_to > as_of). A fact ending exactly at the query instant still matched, and if its replacement started at that same instant, a single-valued predicate like uses_model reported two contradictory answers at once with no warning at all. The pattern that triggers this is invalidating a fact and adding its replacement at the same shared timestamp, and the vendor's own pre-fix onboarding guidance told every agent to use exactly that pattern. The maintainers fixed it in merged PR #1914, switching to the half-open interval and adding a new supersede() primitive as the preferred atomic replacement.

The memtrust backlog had flagged the capability to detect this class of bug months earlier. We'd bundled it together with a second, unrelated capability (drawer neighbor-expansion leak scoping) into one item, and that item sat open because it was too big for a single scoped fix. Forcing it through just to close the backlog line would have meant shipping something half-built, or quietly widening what counts as "done," and the project rules rule out both of those. The item stayed open until we could shrink it down to a manageable unit.

The scope-expanding investigation is where the real work happened. Splitting the bundled item meant reading the adapter code directly, and that reading turned up something useful: the wiring for kg_query(entity, as_of=..., direction="both") already existed post-rewrite and passed straight through to the confirmed tool_kg_query. The evals were simply passing a zero value for as_of. So the actual capability gap was building a classifier and a test to exercise the path that already worked, which was a much smaller job than the original bundled item made it look. You only find that kind of distinction by reading the actual code with the goal of shrinking the claim down to what's true. Re-estimating a vague item over and over just hides these details instead of surfacing them.

We built a new TemporalBoundarySignal enum (CLEAN, DOUBLE_COUNT, NOT_APPLICABLE) in adapters/base.py, and added a dedicated eval at evals/temporal_kg_boundary.py. This eval seeds a fact, invalidates it, and adds its replacement at the same boundary instant to match the shape of #1913. It queries the database at that instant and classifies the response two separate ways: first, it checks the adapter's own self-reported signal, and second, it computes an independent classification straight from the raw list of facts the query actually returned. That second check is what forces the eval to verify the raw data on its own rather than trusting the adapter's word for it. A disagreement between the two signals points to a bug in the adapter's own classification logic, which is a different failure from a backend double-count, and the eval catches both separately.

The test suite behind this holds to the same standard. It runs against two from-scratch fake implementations that faithfully reproduce the real pre-#1914 closed-interval comparison and the real post-#1914 half-open comparison, computing a real answer from real logic each time rather than leaning on hardcoded stubs.

Fifteen tests, all passing, against fakes that implement both the buggy closed-interval logic and the fixed half-open logic in full.

We'll lay out the limitations here as plainly as the eval's own docstring does. First, the real mempalace package is still absent from the build environment. PR #1914 is merged to the vendor's develop branch, but it's absent from the 3.5.0 release this adapter was built and live-verified against. That makes this a detection-capable harness: we proved it correct against faithful synthetic reproductions of both the buggy and fixed comparison logic, but it currently only runs against synthetic targets. The eval should flag #1913's failure shape the moment we point it at a pre-fix deployment, but we still need to actually run that live test before we call this done. Second, temporal_kg_boundary is now wired into the CLI's eval registry as the seventeenth entry memtrust run --eval can invoke by name. It reports not_applicable against any backend other than mempalace, since MemPalace is the only adapter implementing the knowledge-graph calls this eval needs.

Before this change merged, a second engineer independently re-ran the full suite and confirmed every test passed. ruff check, ruff format --check, and mypy --strict all ran clean against the actual merge commit. The suite has grown since then: we're at 590 tests passing and 8 skipped, at 93% overall coverage as of this writing. We applied the same discipline from Part 1's re-verification here, on a much smaller change, because the habit has to hold on boring changes just as much as it held on the dramatic one.

The drawer neighbor-expansion half of that original bundled backlog item remains open and untouched on purpose. Splitting the item let us close only the piece we could actually verify, and we left the other half exactly where it was.

Catching regressions in mem0 during a live run

The same habit produced a second example a few days later, from a real live benchmark run. The mem0_direct adapter uses self-hosted mem0ai running in-process against a local Qdrant instance and the OpenAI API. It ran contradiction detection, compression/round-trip fidelity, and extraction quality. Contradiction detection came back with 7 for 7 cases silently overwritten and 0% flagged: if you tell it your meeting moved from 2pm to 3pm, it stores the new fact and reports zero changes. Compression/round-trip fidelity landed at a 31.6% mean literal fidelity across 5 cases, which we expected, since mem0's design goal is semantic extraction, not literal fidelity. Extraction quality ran 15 cases, 12 deliberately junk inputs and 3 deliberately valid ones. The system correctly rejected all 12 junk inputs, but lost all 3 valid ones on retrieval. That's a small sample, and we're treating it as a signal worth investigating further rather than a conclusion.

Getting any of those numbers required fixing a bug first: the gpt-5-mini/gpt-5o-mini single-character mismatch described in Part 1, which mem0ai/mem0#6085 had already documented independently, 16 days earlier. That's the value of an automated npm install path in practice: a CI job that pulls this tool in on a schedule, with almost no setup effort, catches real regressions automatically.

Unifying distribution and verification

An agent that installs your CLI automatically but gets back false data hasn't actually solved anything. It still needs a human to check the result behind it, which means a benchmark tool with airtight verification discipline that only runs for a person with a curated Python environment never reaches the audience it's meant for. Agents and CI pipelines make decisions with no human supervision in the loop, so a tool built for autonomous execution has to hold its own claims to the same evidentiary bar it applies to the systems it's benchmarking.

We still have real limitations here. We've wired temporal_kg_boundary into the CLI, but we've only pointed it at the synthetic fakes described above, and we still need to test it against a live, pre-fix MemPalace deployment before it's more than a detection-capable claim. A Kuzu graph-database driver stays unbuilt on purpose, since the upstream project backing it formally deprecated that path and dropped default test coverage for it, and building against an abandoned dependency is a bad trade regardless of what it might buy us. "Detection-capable" and "field-tested against a live deployment of the exact bug" are two different claims, and security and eval tooling generally blur that line. We're calling it out directly so the softer claim has to stand on its own instead of borrowing credibility from the stronger one.

Installation and verification both have to work, together, for any of this to mean something. An agent that can install your tool in one command but can't trust what it prints back hasn't gained much over the tools that came before it.

Repo: github.com/RudrenduPaul/memtrust.

---

References

1. Sonatype. "2026 State of the Software Supply Chain." 2026.
2. GitHub API, astral-sh/uv repository metadata, fetched 2026-07-17.
3. pypistats.org: uv package download statistics. Fetched 2026-07-17.
4. JetBrains. "The State of Python 2025." August 2025, based on the Python Developers Survey (PSF + JetBrains).
5. Stack Overflow. 2025 Developer Survey, Technology section. 2025.
6. MemPalace. mempalace#1913, "kg query --as-of returns two values for a single-valued fact at a supersession boundary."
7. MemPalace. mempalace#1914, "fix: half-open as-of interval + supersede() to end KG boundary double-count." Merged.
8. Anthropic. "Donating the Model Context Protocol and Establishing the Agentic AI Foundation." December 2025.
9. Linux Foundation. "Linux Foundation Announces the Formation of the Agentic AI Foundation." December 2025.
10. Singh, Nan, Wang, et al. "The Leaderboard Illusion." NeurIPS 2025, arXiv:2504.20879.
11. "Benchmark Data Contamination of LLMs: A Survey." arXiv:2406.04244, June 2024.
12. "CVE-Genie: From CVE Entries to Verifiable Exploits." arXiv:2509.01835. 2025.

---

Co-authored by Rudrendu Paul and Sourav Nandy. Rudrendu is an engineer building agent-native open-source tooling, including memtrust and a companion suite of AI-agent infrastructure projects. Find the code at github.com/RudrenduPaul.