DEV Community: Rudrendu Paul

Threshold testing is blind to agent regression; we built the statistical alternative, validated against 29 PRs

Rudrendu Paul — Tue, 28 Jul 2026 16:53:59 +0000

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/agent-eval, a statistical regression-testing library for LLM agents. pip install agent-regress-cli.

You change one line in a system prompt. The next day's eval run shows accuracy down three points. Is that a real regression, or just normal LLM sampling noise? Most eval suites can't tell you, because they check whether a single response cleared a threshold instead of measuring how the whole distribution of behavior shifted. You might tweak a prompt, swap from GPT-4o to a cheaper tier, or quietly bump a dependency in LangGraph or CrewAI, and the eval suite still shows green regardless.

agent-eval answers that question directly. It runs an agent 50 times on version A and 50 times on version B, then reports a p-value and an effect size so you know if anything actually changed:

from agent_regress import compare

report = compare(
    version_a=agent_v1,
    version_b=agent_v2,
    test_suite=test_suite,
    n_runs=50,
    metric="tool_accuracy",
)

print(report)
report.assert_stable()  # raises AssertionError if behavior regressed

Here's what a caught regression looks like, straight from the repo's own example running live:

This is the core value in one screenshot: agent-eval runs 500 real trials per version and reports a statistically confident REGRESSED verdict instead of just a lower score.

We didn't want to ship that and call it done. We wanted to know if it actually held up against real, merged, contested code changes from the frameworks people run agents on today. So we pulled 239 real merged PRs across six repos: LangGraph, CrewAI, and the OpenAI Agents SDK, plus EleutherAI's lm-evaluation-harness, DeepEval, and Hugging Face's evaluate (three adjacent eval and benchmark tools we sanity-checked the same pitch against before focusing on the framework integrations). For each one, we asked a simple question: if we pointed agent-eval at exactly this kind of change, would it deliver what a cold pitch about it claims? The rest of this piece covers what that campaign found, including the part where our own tool crashed on the first call.

Gartner expects more than 40% of agentic AI projects to be scrapped by the end of 2027, and the reason cited most often is "escalating costs, unclear business value, or inadequate risk controls" (Gartner, June 2025). "Inadequate risk controls" is often just this problem in disguise: nobody can tell whether the agent's behavior actually changed after the last deploy.

Why threshold testing misses this

Most eval tools answer one question: did this response pass? DeepEval, Promptfoo, and Braintrust are all built around thresholds like a rubric score, an exact match, or a judge-model verdict. None of them tell you whether an agent's behavior meaningfully shifted between two versions of the same system. A 3-point accuracy drop between v1 and v2 might be real, or it might just be ordinary LLM sampling variance. Without a distributional test, you can't tell the difference. Teams usually end up ignoring small drops and eating the risk, or escalating everything and drowning their on-call rotation in false alarms.

The eval space isn't short on tools, and it's consolidating. OpenAI acquired Promptfoo in March 2026, keeping it open-source but folding its team in-house. That's a sign the eval layer is becoming infrastructure people actually build on top of. DeepEval and Braintrust remain strong at answering whether a response cleared a rubric, an exact match, or a judge-model score. None of the three report a p-value, an effect size, or a bootstrap confidence interval on whether behavior shifted between two versions. That's a different statistical question than the one they were built to answer.

This distinction matters more as the surrounding ecosystem keeps growing. LangGraph alone is at 37,447 GitHub stars and roughly 66.7 million PyPI downloads a month as of this writing. CrewAI sits at 55,648 stars and about 11.3 million downloads a month. A lot of production agents change versions on a normal release cadence, and threshold evals simply cannot tell you on their own whether any given release changed real behavior or just changed the score.

How the statistics actually work

Three statistical tools do the actual work. Mann-Whitney U compares the two score distributions without assuming they're normal, which matters because LLM output scores usually aren't. A bootstrap confidence interval (1,000 resamples) adds magnitude to that significance call, measuring how large the shift actually was. Cohen's d separates statistically significant from operationally significant: a shift at p = 0.001 with d = 0.04 is real but irrelevant, while a shift at p = 0.06 with d = 0.5 is large and needs more samples to confirm. The default CI gate only fires when both p < 0.05 and d ≥ 0.2, which keeps a 50-run comparison from failing your build over noise.

Here's what those numbers look like on the tool's own demo comparison, swapping GPT-4o for GPT-4o-mini to cut cost:

50 runs on version A: mean tool accuracy 0.840, SD 0.060
50 runs on version B: mean tool accuracy 0.700, SD 0.090
p = 0.0031, Cohen's d = -0.610, 95% CI [-0.221, -0.067]
Verdict: REGRESSED, block the deploy

We measured statistical overhead on an M3 Pro at about 27ms per comparison at n=50. The agent calls themselves remain the real bottleneck; the math barely registers.

None of this is exotic. The core call is one line of scipy, and we say that plainly because it matters for where this tool actually earns its keep. Any eval platform could bolt those statistics on in an afternoon. The hard part to copy is the version-specific regression history and the framework integrations, because those are what make the statistics usable from a real agent in CI without hand-rolling a harness every time. That's what turns a leap of faith, like swapping to a cheaper model, into a testable claim. A REGRESSED verdict with an attached p-value and Cohen's d either backs that decision or kills it before it ships.

The bug that would have crashed every single comparison

This is where the story stops being a features list. Before we trusted the framework integrations enough to write outreach copy about them, we ran a validation pass, the first one, before we'd shown the tool to anyone outside our own repos. We took a real, merged PR from LangGraph, CrewAI, or the OpenAI Agents SDK and checked whether agent-eval could actually produce the comparison a pitch about that PR would claim.

The very first thing we found was a P0. openai_agents_runner() called agent.run(query). That method doesn't exist on the real SDK's Agent class; only Runner.run(agent, input) does. We confirmed it directly: hasattr(agents.Agent(...), 'run') returns False, and hasattr(agents.Runner, 'run') returns True. The integration raised AttributeError on its first call, for any test suite, any agent, any claim. It was a hard crash that took down the whole integration. This one wrong method name accounted for 15 of the 29 code-fixable gaps the campaign found, or 51.7% of all of them.

We reproduced that exact crash live against the real, currently installed OpenAI Agents SDK, running the actual pre-fix code straight from the commit history:

This isn't a mockup. It's the real buggy call site from the commit before the fix, run against the real SDK, producing the actual AttributeError that broke every comparison.

We fixed the call site and kept going. A crash that big rarely travels alone, and this one didn't either. langgraph_runner() built a fresh, stateless dict on every call, with no thread_id and no checkpoint continuity, so it couldn't represent multi-turn conversations at all. Meanwhile crewai_runner() only wrapped a full Crew.kickoff(), with no way to isolate one tool's behavior. run_suite() had no cache-busting nonce, which meant any framework with its own persistent result cache (LangGraph's cache_policy is a real example) would silently collapse the repeated-sampling variance the whole statistical approach depends on. That last issue turned into a second bug once we looked closer: a piece of state-narrowing logic downstream was dropping the cache-bust key we'd added. Re-review caught it; the first pass missed it.

Two commits closed most of it. 8411eb5 fixed the P0 and 16 other gaps, taking the test suite from 121 tests to 264 passing. f752c11 closed 6 of the 7 residual gaps that a second, independent re-validation pass surfaced, including an async cancellation liveness bug in the store layer. The 7th gap tied to a LangGraph interrupt/resume report and needed one more manual revalidation pass before we could confirm it closed.

Both passes independently caught the same mistake on the first attempt: a proposed fix credited code that was technically correct but still sitting uncommitted in the working tree. You only find that kind of thing by actually running an adversarial check instead of trusting your own fix, and we had to learn that lesson twice. This is a specific, repeatable failure mode of self-reported "this is fixed now" claims, which is why the validation methodology below runs every verdict through a propose-then-refute step instead of taking the first answer.

29 real merged PRs, and what it took to actually test them

We anchored every fix above on a real, merged pull request. The full validation set covered 239 rows across six repos. We excluded 209 of those because the underlying PR never carried the kind of behavioral-drift risk the pitch assumed: docs-only changes, trivial fixes, or PRs that plain didn't exist. Of the 29 rows where the code gap was real and fixable, all 29 now pass. One additional row (@sibblegp) has its code gap fully closed too, but it stays below PASS for a reason no amount of engineering can fix: there's no genuine merged PR to anchor the claim on.

A representative slice of those 29 spans all three frameworks and most of the fix categories above:

PR	Repo	What it changed	What `agent-eval` needed to test it
langgraph #5243	LangGraph	New typed `context=` API, replacing untyped `config['configurable']`	`langgraph_runner()` gained `config`, `context`, and `thread_aware` kwargs so both invocation styles compare directly
openai-agents-python #2463	OpenAI Agents SDK	Fixed agent-as-tool silently dropping the parent run's `RunConfig`	Fixed the P0 crash, then added `capture_trace=` telemetry so RunConfig inheritance is observable per nested call
crewAI #6236	CrewAI	Optional Pydantic `output_schema` on tools, structured JSON instead of `str()`	New `crewai_tool_runner()` isolates a single tool, plus a `schema_conformance_scorer()`
langgraph #7746	LangGraph	Reworked checkpoint snapshot cadence to key on supersteps, not just updates	`thread_aware=True` for per-case thread continuity across repeated runs
crewAI #6134	CrewAI	Security fix: file tools were leaking absolute filesystem paths in responses	New `no_path_leak_scorer()` flags raw paths in tool-response text
openai-agents-python #2214	OpenAI Agents SDK	Stopped image/audio/file tool outputs from being silently dropped to text-only	`capture_trace=` intercepts the SDK's real conversion function directly, so regressions of this exact class are now detectable
langgraph #3126	LangGraph	Reworked `ToolNode` dispatch, risking duplicate tool calls on interrupt/resume	`tool_call_trace_scorer()` plus a real `langgraph_interrupt_resume_runner()` that drives an actual interrupt-then-resume cycle
openai-agents-python #2902	OpenAI Agents SDK	Added a persistent MongoDB session backend for multi-turn continuity	`session_aware=True` reuses one session per logical test case instead of building a fresh one every call
langgraph #4486	LangGraph	Added node/task-level result caching, which can mask repeated-sampling variance	`cache_bust_key_fn()` plus `graph_has_cache()`, and the dropped-nonce bug described above
langgraph #6701	LangGraph	Fixed a cancelled-future re-queue bug in the async batching store layer	New `arun_suite()`, `langgraph_async_runner()`, and `concurrent_cancellation_probe()` to reach async liveness bugs at all
openai-agents-python #2328	OpenAI Agents SDK	Fixed a hard crash using DeepSeek's thinking mode through LiteLLM	Fixed the P0 crash; verified end-to-end against a live install
crewAI #6079	CrewAI	Four pluggable storage backends for memory, knowledge, RAG, and flow persistence	`crew_factory=` builds a fresh `Crew` per invocation, avoiding shared-mutable-state corruption under concurrent comparison
crewAI #4446	CrewAI	Substantial refactor of the Brave Search tool integration	`subprocess_runner()` compares `crewai-tools` versions pre- and post-PR
openai-agents-python #1744	OpenAI Agents SDK	Added support for Anthropic's extended/interleaved thinking via LiteLLM	Fixed the P0 crash, closing the runner-level failure that blocked exercising this feature at all

Every link above resolves to a real, merged PR. That covers 14 of the 29. The complete list, with a real PR link (or an honest note where one isn't on record) and the exact fixing commit for every row, is in the repo at docs/validation.md. The pattern across all 29 is the same: the plumbing connecting agent-eval's statistics to a specific framework's real API surface was incomplete, and that only became visible by testing against what maintainers actually shipped.

Supported frameworks

Four framework integrations are shipped as of v0.1, each installable as its own extra:

LangGraph: pip install agent-regress-cli[langgraph]
OpenAI Agents SDK: pip install agent-regress-cli[openai-agents]
CrewAI: pip install agent-regress-cli[crewai]
LangChain LCEL: pip install agent-regress-cli[langchain]

AutoGen support is planned for v0.3. A Vercel AI SDK integration for TypeScript agents is planned for v0.4. Until then, use subprocess_runner() as a workaround, which compares two installed package versions by running them as subprocesses.

If you're on a framework not listed here, the fastest path is opening an issue with the framework name and a link to one representative merged PR. The validation methodology above is documented and repeatable: point it at your integration and we can work through where it breaks.

Standard benchmarks

Beyond version-to-version comparisons, agent-eval ships harnesses for the three standard benchmarks in the field, so you don't have to write a separate scaffold for each. Tau-bench pass^k runs an agent across k independent attempts (k=1, 4, 8), because single-run scores hide reliability degradation that a repeated-attempt curve catches. The GAIA harness splits results by Level 1-3 difficulty instead of collapsing everything into one accuracy number, since a change that helps easy tasks can quietly hurt hard ones. The SWE-bench harness reports a scaffold pass rate, isolating what the agent framework itself contributes from what the underlying model contributes. All three are version-controlled in the repo's leaderboard/ directory, and results are independently reproduced before a submission gets merged.

Using agent-eval in CI

agent-eval shipped as a pure Python import for its first several months. As of this week, it's also a real CLI, which is how most people will actually run this in a pipeline rather than as a Python import.

One naming note before you install anything: the repo is agent-eval, the PyPI package is agent-regress-cli, and the Python import is agent_regress. Three names for one project, because agent-regress on PyPI was already blocked as too similar to an existing, unrelated project by the time we tried to publish, so the distribution got renamed and the import path stayed put.

pip install agent-regress-cli
# or
uv add agent-regress-cli
# or, via npx if you'd rather not touch Python directly
npx agent-regress-npm-cli --help

The Python API's compare() takes two callables and runs them itself. That shape has no clean command-line equivalent, so the CLI starts one step later: it takes two JSON files containing flat arrays of per-run scores your own harness already computed, then runs them through the same Mann-Whitney U, bootstrap CI, and Cohen's d pipeline the Python API uses internally:

agent-regress compare \
  --version-a-results v1_scores.json \
  --version-b-results v2_scores.json \
  --metric tool_accuracy \
  --json \
  --fail-on-regression

--json prints a single, parseable JSON object to stdout and routes warnings to stderr so stdout stays clean. The CLI's own source calls this "the agent-native machine-readable surface," meant to describe a report that an orchestration script, a CI job, or another agent can consume directly instead of scraping human-formatted text. --fail-on-regression exits 1 on a REGRESSED verdict, so the same command drops straight into a CI gate.

You pass in two pre-computed score files and get a full statistical verdict back, without needing any Python imports to run it.

Here's the shape of that gate as an actual GitHub Actions workflow:

# .github/workflows/agent-regression.yml
name: Agent regression check
on: [push, pull_request]

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install agent-regress-cli[langgraph]
      - name: Run agent comparison
        run: |
          python scripts/run_agent_suite.py \
            --version-a prod \
            --version-b staging \
            --output-a v1_scores.json \
            --output-b v2_scores.json
      - name: Statistical regression check
        run: |
          agent-regress compare \
            --version-a-results v1_scores.json \
            --version-b-results v2_scores.json \
            --metric tool_accuracy \
            --json \
            --fail-on-regression

scripts/run_agent_suite.py is your own harness, whatever runs your agent 50 times per version and writes out the score arrays. The gate itself, the last step, exits 1 on REGRESSED and 0 on STABLE. Drop it into any workflow that already runs your agent.

Honest limitations

We'd rather state the limits plainly than let the 29-for-29 number imply more than it does. Of the 239 rows in the full validation set, we excluded 209 (87.4%) simply because the pitch itself didn't apply to those PRs. That's a very different claim from "29 out of 239 attempts succeeded." This campaign says nothing about agent-eval's coverage of frameworks like AutoGen, which is still on the roadmap. It also doesn't touch correctness questions the stats module was never meant to answer, like whether a scorer function itself is well-designed. The framework-specific plumbing is the real work here. This campaign spent two commits building and re-testing it, and the version-specific regression history only accumulates from real use. The statistical core itself is just one scipy call anyone could copy in an afternoon.

Two more limits are worth stating outright. First, sample size: both assert_stable() and RegressionGate warn but don't fail when either version has fewer than 50 runs. That's the sample size needed to reliably detect a moderate effect (Cohen's d of 0.2) at 80% power. Below 10 runs per version, agent-eval returns INSUFFICIENT_DATA instead of a REGRESSED/STABLE/IMPROVED verdict, because the sample is simply too small to trust any statistical conclusion. Second, agent-eval never makes a model call itself. compare() runs whatever callables you hand it, version_a and version_b, against your own test suite. There are no API keys to manage and no hidden model cost inside the tool. AutoGen and the TypeScript-native Vercel AI SDK integration remain unshipped; we're tracking them above rather than quietly dropping them.

Try it against your own agent

from agent_regress import compare, RegressionGate

gate = RegressionGate(p_threshold=0.05, min_effect=0.2)

def test_tool_accuracy():
    report = compare(version_a=prod_agent, version_b=staging_agent, test_suite=suite, n_runs=50)
    gate.check(report)  # raises AssertionError on regression, warns if n < 30

Apache 2.0, self-hostable, no SaaS account required. docker compose up gets you a working example and the leaderboard UI in one command if you want to see it running before wiring it into anything. The stats module (src/agent_regress/stats/) carries a hard constraint in the codebase itself: it has to stay pure Python and scipy, with no LLM calls, ever. That way the statistical core never becomes a moving target with its own model dependency.

If you maintain or contribute to LangGraph, CrewAI, or the OpenAI Agents SDK, there's a decent chance one of your own merged PRs is hiding in that 239-row dataset. The good-first-issues list is open if you want to help close the remaining gaps. The leaderboard takes PRs too: you can run one of the benchmark harnesses above against a framework or model we haven't covered, then drop the resulting JSON into leaderboard/ matching leaderboard/schema.json. It gets independently reproduced before merging.

Two things are on the roadmap for the next release. One is AutoGen support. The other is deeper CI ergonomics for the three frameworks already shipped: a native GitHub Actions annotation format and a richer --json schema for dashboards. We'd rather ship the one people actually hit first, so if you have a strong preference, or if you maintain a framework we haven't covered yet, open an issue. The validation methodology is documented and the pattern is repeatable.

Repo: github.com/RudrenduPaul/agent-eval. If this caught a regression you would have shipped, or if the validation methodology gave you a pattern worth repeating on your own framework, a star helps other teams find it before they need it.

References

Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027", June 25, 2025
OpenAI, "OpenAI to acquire Promptfoo", March 9, 2026
GitHub and PyPI star/download counts self-reported via public API as of July 16, 2026

Co-authored by Rudrendu Paul and Sourav Nandy.

Rudrendu Paul and Sourav Nandy build open-source developer tools for the AI agent ecosystem. They are the co-authors of agent-eval (agent-regress), a statistical regression-testing library for LLM agents, and spent the past several months validating it against real merged PRs in LangGraph, CrewAI, and the OpenAI Agents SDK.

How We Built a Way to Replay AI Agent Failures Without Paying to Reproduce Them

Rudrendu Paul — Tue, 28 Jul 2026 16:53:55 +0000

What testing our tool against 104 real GitHub issues across eight agent frameworks taught us

By Rudrendu Paul and Sourav Nandy

Repo: github.com/RudrenduPaul/agent-observability. It records and replays AI agent runs so you can debug a failure without paying to trigger it again. Install it with pip install agent-observability-trace-cli.

Quick summary: We tested our approach against 104 real GitHub issues from eight different agent frameworks. It worked for almost all of them. Here is what we learned, including the cases where it did not work at first and what we had to change.

Imagine your LangGraph agent fails on step 8 of a 10-step process. LangSmith shows you the trace, so you can see what happened.

But to actually reproduce that failure so you can study it, you have to run the whole agent again. That means eight more calls to the language model, about 30 more seconds of waiting, and roughly 15 cents in API costs. And because language models do not always answer the same way twice, the failure might not even happen again.

Nobody has really solved this problem yet.

Here is what our approach looks like instead: record a real network call once, then replay it as many times as you want, with no network traffic and no cost.

Record it once, while it's live. After that, every replay is the same run, with the same result, no API calls and no money spent triggering it again.

Even advanced AI models struggle with a task that sounds simple: look at a multi-agent system that just failed, and say which agent caused it. In 2025, researchers from Penn State and Duke built a dataset of 127 real multi-agent failures and tested whether AI models could correctly identify the cause. The best model only picked the right agent 53.5% of the time, and only found the exact failing step 14.2% of the time. The researchers concluded that even the most advanced reasoning models are not reliable enough for this task yet.

You cannot fix this with a better dashboard. Most "agent observability" tools are answering the wrong question. Knowing what happened is fairly easy. Making that exact failure happen again, cheaply and reliably, so you can actually study it, is a completely different problem, and almost nobody has built for it.

This matters right now because agent deployments are growing faster than anyone's ability to debug them by hand. Gartner predicts that 40% of enterprise applications will use task-specific AI agents by the end of 2026, up from under 5% in 2025, one of the fastest adoption curves it has ever tracked. McKinsey's 2026 State of AI report is more cautious: 23% of organizations say they are actually scaling an agent system somewhere in the business, and no single business function has passed 10% adoption yet. Put those two numbers side by side and the picture is clear. Agents are moving into production much faster than the tools to debug them are maturing.

The real gap in agent observability

When people talk about "agent observability," they usually mean tracking an agent's calls, sending them to a dashboard, and looking at a trace when something breaks. That gives you visibility into what happened. It does not give you an easy or cheap way to make the same problem happen again so you can actually study it.

That is the important difference. Trace tools tell you what happened. They do not tell you how to make the same thing happen again, cheaply and reliably. Those are two separate problems. The first one is mostly solved already. Almost nobody has solved the second one.

The cost of the API calls to trigger a failure again is actually the smaller cost. The bigger cost is the engineering time spent re-running a production failure over and over, hoping to catch it in the act. That is what really slows teams down once their agents move past the demo stage.

Other parts of software have solved this kind of problem before. Mozilla's rr debugger has been able to record and replay the exact behavior of an ordinary Linux program since 2017, and it is solid enough to work on something as large as Firefox's codebase. ACM Queue wrote about the same general idea in a widely cited 2020 piece called "To Catch a Failure: The Record-and-Replay Approach to Debugging." In the Ruby and Python testing world, a library called VCR has let developers record one real HTTP call and replay it in their tests for well over a decade. The idea itself is not new. What is new is applying it carefully to the kind of network traffic that AI agents create. Multi-agent systems make this more urgent, since the research numbers above are exactly what you would expect when nobody can cheaply reproduce a failure to study it in the first place.

Finding out where our approach breaks down

Before writing anything about this idea, we tested it. We went through every GitHub issue where a developer had reported a real, specific problem, not a documentation typo or a feature request, across eight of the most widely used agent frameworks. For each one, we asked a simple question: if we pointed our tool's actual recording system at this exact failure, would it capture enough information to explain what went wrong?

We found a working fix for 104 of those issues, spanning langchain-ai/langgraph, langchain-ai/langchain, openai/openai-agents-python, run-llama/llama_index, microsoft/autogen, crewAIInc/crewAI, pydantic/pydantic-ai, deepset-ai/haystack, and agno-agi/agno. Every one of those fixes is already in the project's main branch, and the whole test suite, 1,367 tests, passes cleanly.

The process mattered more than the final count. We started out assuming that watching the HTTP traffic between a Python program and the model's API would be enough, since that is how most Python AI tools actually talk to a model. That assumption turned out to be wrong fairly quickly, and it was wrong in a useful way. It kept failing on the same kinds of traffic, not on random edge cases.

Google's Gemini and Vertex AI tools normally send requests over gRPC instead of ordinary HTTP. A tool that only watches HTTP traffic sees nothing at all in that case. That gap is exactly why langchain-ai/langchain#8304 happened: VertexAI's text-bison model quietly returned no output, and there was no record of why.
AWS's SDK for Bedrock and SageMaker manages its own network connections in a way that sits below the layer we were watching. That gap left crewAIInc/crewAI#4749 with no captured evidence to work from, after a Bedrock error on a multi-tool-call request.
OpenAI's Realtime API and any MCP tool server use long-lived connections instead of a single request and response. When tool calling silently broke in a Realtime Agent, in openai/openai-agents-python#2308, we could not just extend our existing tool to handle it. We had to build a completely separate system to capture that kind of connection.
We also found a subtle bug involving HTTP clients that get created once, when a program first starts, before any recording session has begun. This is exactly how langgraph dev starts up. Our early fix only patched clients created after recording started, so it silently missed every one of these. Fixing it meant changing how our tool decides which client to intercept, not just flipping a setting.

We are not blaming any of these frameworks. This is simply how complicated the real problem is. Capturing everything an agent does takes four or five separate pieces of work, not just one. The only way to find the missing pieces is by testing against failures real users have actually hit. Reading the documentation and guessing will not get you there.

Here is what the actual process of checking a run looks like: list the runs you have recorded, then point the inspect command at one to automatically flag anything that looks broken.

This is the same process we repeated 104 times against real GitHub issues: find the run, point inspect at it, see whether the evidence is actually there.

Twelve examples out of the 104

Every row below links to the real issue. This is a representative sample of what we found. The complete list of all 104 issues, with the exact fix for each one, is in the repository.

Issue	Framework	What broke	What fixed it
agno-agi/agno#5298	Agno	An `UnboundLocalError` during streaming under Uvicorn	We added support for Agno's own streaming setup, along with timestamps on each streamed chunk, so the timing and state around the failure became visible.
crewAIInc/crewAI#4749	crewAI	An AWS Bedrock error on a request with multiple tool calls	We built a way to watch `botocore`/`boto3` traffic directly, since AWS's SDK bypasses the normal HTTP layer entirely and nothing had been recorded at all before.
langchain-ai/langchain#8304	LangChain	VertexAI's `text-bison` silently returning no output	We built support for capturing gRPC traffic, since some SDKs default to gRPC instead of HTTP.
langchain-ai/langgraph#2920	LangGraph	A report that the model felt slow inside a LangGraph agent, with no way to tell where the time was going	We added timing for each exchange and a metric for unaccounted time, so it became clear how much delay came from the model versus the framework itself.
langchain-ai/langgraph#6202	LangGraph	The frontend stopped receiving stream updates mid-run, even though the backend was still running	We added a pass-through streaming mode plus new tracking on the outbound delivery path, so this kind of silent stall becomes visible.
langchain-ai/langgraph#7714	LangGraph	A report of checkpoints using 85% more storage than expected, with no way to confirm it	We built a way to measure checkpoint size and content independently, so this kind of claim can actually be checked.
openai/openai-agents-python#2308	OpenAI Agents SDK	Tool calling silently failing to work with a Realtime Agent	We built a dedicated way to capture the Realtime API's long-lived connection traffic.
openai/openai-agents-python#1671	OpenAI Agents SDK	Occasional "tool not found" errors after an agent handoff	We added a check that compares the failing tool name against the registered tools by similarity, which catches near-misses and mismatched capitalization.
pydantic/pydantic-ai#2837	pydantic-ai	A `FallbackModel` quietly swallowing an error raised inside the agent	We added a way to watch `boto3` traffic directly, along with a pydantic-ai integration that hooks into the agent's own run and state transitions.
run-llama/llama_index#7170	LlamaIndex	The agent occasionally calling tools that do not exist, then crashing	We added an automatic check that compares every tool call against the run's actual list of registered tools, so made-up tool names get flagged right away.
run-llama/llama_index#18937	LlamaIndex	Google's GenAI SDK silently disabling tool calling, with no visible cause	We fixed the ordering bug in how we intercept clients, and added dedicated support for the google-genai SDK.
langchain-ai/langgraph#6449	LangGraph + MCP	An error from an MCP server that was not being handled correctly, with no evidence captured	We built a way to record MCP's raw message exchanges directly, since MCP tool calls do not travel over HTTP at all.

Where this approach genuinely does not work

A tool like this earns trust by being honest about its own limits, so here is where it stops working.

The tool only sees traffic sent by your own program. It cannot see the internal calls made by a separate hosted service that you do not run yourself. Our support for gRPC, the protocol some AI providers use instead of HTTP, is real but only partial. We fully capture the common request-and-response pattern. We do not yet capture gRPC's streaming modes, including the newer async streaming style some tools use. Recording only starts once an actual request exists. If something fails earlier, for example while an SDK is still building that request, there is nothing for us to capture. If you have set up one of our framework-specific integrations, its own error handling can sometimes still catch that earlier kind of failure. If you have not set one up, you will need to catch that error somewhere else in your code.

We also want to be honest about what this tool replaces and what it does not. Sometimes you need to understand behavior across live traffic where you have not seen a specific failure yet. For that kind of open-ended debugging, you still want a normal dashboard tool watching everything as it happens. Our tool is built for the moment right after you already know something broke, and you need to study that exact failure closely, as many times as you need, at no extra cost. It is not meant to help you discover unknown problems in the first place. These are two different jobs, and treating them as the same thing oversells both.

If your team is about to run a lot more agents in production, here is the simple takeaway. Do not treat the ability to reproduce a failure offline as a nice-to-have you will get to later. Treat it as something you need in place before you scale, the same way you would want proper monitoring in place before scaling a large system. The numbers from McKinsey and Gartner above show that scaling is already happening faster than most teams' debugging tools can keep up with. Once a failure is something you can reproduce locally instead of re-running against a live API, the time it takes to fix a bug can drop from a frustrating afternoon to just a few minutes, because the run is already sitting on your disk.

Why this gets more urgent once agents call other agents

Everything above describes one team debugging its own agent. The next problem is much bigger. Agents are starting to call other organizations' agents directly, as a real piece of infrastructure, not just an experiment.

The Agent2Agent protocol, now managed by the Linux Foundation, passed 150 supporting organizations and 22,000 GitHub stars in its first year, with SDKs in five languages already in production use across supply chain, financial services, insurance, and IT operations. Separately, Gartner expects AI agents to be involved in 90% of business-to-business buying activity by 2028, representing more than $15 trillion in spending.

Put that next to what we found earlier. Figuring out which agent caused a failure, inside your own system, is already unreliable, with even the best AI models only getting it right about half the time. That problem does not get easier once the failure crosses into someone else's agent. It gets harder, because you no longer control both sides of the interaction, and there is no shared standard yet for tracing a failure across company boundaries the way there is for tracing normal software within a single company.

This is exactly where debugging tools need to change. They cannot stay something a person clicks through on a dashboard. They need to become something another program, or another agent, can call directly. That is the reason our command-line tool always offers a plain, structured data option instead of only a page meant for a person to read. The command agent-trace show <run_id> returns the full recorded run as JSON by default. The commands agent-trace list, agent-trace inspect <run_id>, and agent-trace diff <run_a> <run_b> all support a --json option too, so another program, or a CI job, can read the result with no person involved. That is a different goal than a dashboard built for human eyes, and it is the goal that actually matters once debugging becomes a normal step inside an automated pipeline instead of a stressful afternoon for one engineer.

What other tools solve, and what they miss

Most existing tools in this space fall into four groups. Each one solves a real part of the problem, but leaves the same gap open.

LangSmith, Langfuse, and Helicone are the most common tools people already use. LangSmith is a strong choice if your team already uses LangChain, and it is genuinely good at showing traces, managing datasets, and versioning prompts. Langfuse offers similar strengths as a fully open-source tool you can host yourself, backed by solid database storage. Helicone works as a simple proxy that sits in front of your API calls and records the raw traffic, with very little setup required. All three are good at showing you what your system is doing across many runs. But with all three, reproducing one specific failing run still generally means running it again against the live model, with whatever cost and unpredictability that carries.

There are also tools built on the OpenTelemetry community's GenAI standards, which have been under development since 2024. These make traces look the same across different vendors, so you are not locked into one tool. That is a genuinely useful improvement. It does nothing, though, to help you cheaply reproduce the run that created the trace in the first place.

Older application monitoring tools have added AI-specific features on top of their existing systems. This is a real strength if you need to see agent problems alongside the rest of your infrastructure. But the gap here is structural. A response can come back with a normal success code and still contain a wrong answer or leaked data, and these tools were built to watch status codes and response times, not to check whether the content of a response is actually correct.

Finally, there are proxy and gateway tools that sit at the edge of your network and log traffic, mainly to track cost and usage. They are easy to add, but that ease is also their limit. They see raw traffic, not what is actually happening inside your agent's reasoning, and that internal state is usually exactly what you need to explain a failure.

A gap runs through all four of these approaches. None of them make it cheap, reliable, and free to reproduce one specific known failure. None of them go beyond your normal HTTP client to also cover the gRPC, WebSocket, and other connection types that agent frameworks increasingly use underneath their SDKs. We built a tool that works at that lower, transport level specifically to close that gap, and then tested it broadly enough to cover the connection types agents actually use, validating it against real reported failures from real users.

Try it

The repository is called agent-observability. The Python package is agent-observability-trace-cli. Once installed, the command-line tool is agent-trace, and the Python package you import is agent_trace.

pip install agent-observability-trace-cli
# framework extras, e.g.:
pip install agent-observability-trace-cli[langgraph]
pip install agent-observability-trace-cli[openai-agents]

# Record a run (your script just needs `import agent_trace` somewhere)
agent-trace run --name my_agent -- python my_agent.py

# List recorded runs
agent-trace list

# Replay offline, zero network, zero cost
agent-trace replay <run_id>

# Inspect a run for known failure patterns
agent-trace inspect <run_id>

# Diff two runs
agent-trace diff <run_a> <run_b>

If you would rather control things directly in code instead of using the command line, you can use the Python API:

from agent_trace import tracer
import httpx

@tracer.instrument(record=True)
def fetch_data(query: str) -> dict:
    with tracer.span("http-call") as span:
        resp = httpx.get("https://httpbin.org/get", params={"q": query})
        span.set_attribute("http.status_code", resp.status_code)
        return resp.json()

In CI, the same commands work the same way. You just give the run a specific ID so the next step knows which run to check:

# .github/workflows/agent-debug.yml
name: Capture agent runs for debugging
on: [push, pull_request]

jobs:
  capture:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install agent-observability-trace-cli[langgraph]
      - name: Run and record agent
        run: agent-trace run --run-id ci-run --name ci-run -- python my_agent.py
      - name: Check for known failure patterns
        run: agent-trace inspect ci-run --json
      - uses: actions/upload-artifact@v4
        with:
          name: agent-trace-runs
          path: ~/.agent-trace/runs/

This records every run, saves it as a downloadable file, and lets you replay it locally in under a millisecond, with no API calls and no cost.

In our own testing, recording added about 0.011% overhead to a GPT-4o call, a tiny fraction of the call's normal response time. Replay runs in under a millisecond, and the result is identical, byte for byte, to the original run.

Every command supports a JSON output option, so another program can read the result directly. The run command writes its normal progress messages to one output stream, and a single final summary as JSON to another, since we cannot control the output format of whatever program is running underneath it. This connects back to what we described earlier about agents calling other agents: a debugging step becomes something a program can run on its own, the same way a person would run it by hand.

This is what an automated pipeline, or another agent, actually reads: structured data, not a page built for a person to look at.

One question we have not settled yet: should replay keep working the way it does today, as a small piece of Python code you wrap around your function, or should it become a plugin that works with any OpenTelemetry-based pipeline with no code changes at all? We would genuinely like your opinion. The discussion is open in our issue tracker, which is where that conversation belongs.

Repo: github.com/RudrenduPaul/agent-observability. If this would have saved you from re-running a live API just to catch a bug, starring the repo helps other teams find it.

You record a run once with agent-trace run, and you can replay it as many times as you want with agent-trace replay. The failure is already sitting on your disk, waiting for you to look at it again.

References

Zhang, Yin, Zhang, Liu, Han, Zhang, Li, Wang, Wang, Chen, Wu. "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems." ICML 2025 Spotlight, arXiv:2505.00212.
O'Callahan, Jones, Froyd, Huey, Noll, Partush. "Engineering Record and Replay for Deployability." USENIX ATC 2017, arXiv:1705.05937. See also the rr project.
"To Catch a Failure: The Record-and-Replay Approach to Debugging." ACM Queue, 2020.
Gartner. "Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025." August 2025.
McKinsey State of AI 2026, as covered by Forbes, March 2026.
Linux Foundation. "A2A Protocol Surpasses 150 Organizations, Lands in Major Cloud Platforms, and Sees Enterprise Production Use in First Year." April 2026.
Gartner B2B commerce forecast, as covered by Digital Commerce 360, November 2025.
OpenTelemetry. GenAI Semantic Conventions. 2026.

Co-authored by Rudrendu Paul and Sourav Nandy. Rudrendu is a founder and engineer building open-source tools for AI agents, including agent-trace (Agent Observability) and a related set of AI-agent infrastructure projects. Find the code at github.com/RudrenduPaul.

Giving an AI Agent a Benchmark Tool It Can Install and Actually Trust

Rudrendu Paul — Tue, 28 Jul 2026 16:52:00 +0000

Part 2 of 2: the npm-to-uv-to-PyPI bridge that closes the installation gap, and the adversarial re-verification discipline that makes the output worth acting on autonomously

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/memtrust, an independent, reproducible benchmark harness for agent-memory backends. pip install memtrust-cli.

Quick summary: memtrust-cli is now on npm. npx memtrust-cli runs a Python benchmark tool automatically using uv, with no Python toolchain required. Below is the full bridge design, a concrete temporal-KG boundary bug the eval caught, and the verification discipline that makes the output worth acting on autonomously.

In Part 1 of this series, we found our own MemPalace adapter calling a fabricated API method. Every result it had ever reported for that backend was fiction. Closing that gap meant an adversarial, from-scratch re-verification of every adapter in the project. That story stands on its own, and this one picks up right after it to answer a related question: when an autonomous agent runs this tool, what does "trustworthy" actually mean?

Here's the real command surface memtrust ships today. This is the output an agent or a CI script parses.

Every flag shown here is real, pulled straight from memtrust run --help. This GIF shows the live output.

Defining "agent-native" requirements

"Agent-native" gets used loosely enough that it needs a strict definition. A tool that's actually agent-native has to satisfy two conditions at the same time:

Installable automatically. An agent or a CI job needs to go from "nothing installed" to "tool running" using a single programmatically constructed command, and it needs to run on whatever runtime the environment already has.
Machine-legible. Every response has to be structured data a program can parse and act on directly.

Most developer tools, including memtrust's earlier releases, miss both marks. Running pip install memtrust-cli assumes the system already has a specific Python version provisioned, a virtual environment configured, and a writable path for pip. A CI runner or an agent sandbox shipping only Node hits a dead end there. Early memtrust CLI output also targeted human readers only. We printed colored tables, progress bars, and summary lines.

The npm side of this project closes the installation gap. The rest of this article focuses on the legibility problem, which turned out to be the harder challenge of the two.

Live on both registries

You can install memtrust-cli via npx memtrust-cli right now. It's on the npm registry. Running npm view memtrust-cli version against the registry today returns 0.3.3, which matches the version on PyPI.

We shipped a complete, tested npm package at npm/memtrust-cli/ in the repo alongside six per-platform companion packages (@memtrust-cli/darwin-arm64, @memtrust-cli/darwin-x64, @memtrust-cli/linux-arm64, @memtrust-cli/linux-x64, @memtrust-cli/win32-arm64, @memtrust-cli/win32-x64). The publish process followed a strict dry-run discipline: npm pack --dry-run produced a correct tarball, then npm publish --dry-run walked through the whole publish flow with registry auth included, and only after that did the real npm publish run. Running a dry run before the real thing is the same verification habit we apply to eval claims. We just applied it to a release instead.

The live path on PyPI is pip install memtrust-cli, version 0.3.3, confirmed against PyPI's own API on 2026-07-21, the date of the npm publish. A mirror package named memtrust currently sits at the same version for anyone who reaches for that shorter name, kept in sync by a manual release step rather than an automated one. Every example below works today through either pip or npx.

Here's the full implementation of bin/memtrust.js. The entire design fits in well under a hundred lines:

// Pinned to this npm package's own version, not left floating: "uv tool run
// --from memtrust" with no version qualifier always fetches whatever is
// currently newest on PyPI. Pinning to this package's own version keeps the
// two registries in lockstep: bump this package.json's version when PyPI's
// memtrust ships a new release, and every subsequent npm install/npx run
// resolves to that exact pinned release, not whatever is newest at run time.
const PACKAGE_VERSION = require(path.join(__dirname, "..", "package.json")).version;
...
const result = spawnSync(
  uvPath,
  ["tool", "run", "--from", `memtrust==${PACKAGE_VERSION}`, "memtrust", ...process.argv.slice(2)],
  { stdio: "inherit" }
);

The script resolves a platform-specific uv binary from require.resolve(), hands off every argument unchanged, and delegates all memtrust logic to the binary, pinning only the PyPI release to its own package version. Reading the whole file top to bottom takes about a minute, which was the point: we wanted a small bridge you can audit at a glance, one that gives you both determinism and simplicity without hiding anything.

We spent a good amount of time deciding where to actually source the uv binary from. Each platform package's prepack script fetches the matching uv release archive directly from Astral's GitHub releases, along with the per-archive .sha256 checksum file uv publishes alongside it, and the script requires an exact match before it will proceed. The maintainer running the release does this verification once, at npm publish time. We chose that tradeoff on purpose: a maintainer verifying the checksum with full visibility into the supply chain gives a stronger guarantee than repeating the check on every install, and downstream npm install runs simply rely on that already-verified state. An offline or network-restricted install never has to reach out to Astral's servers at all, since the uv binary already sits in node_modules ready to run.

uv tool run --from memtrust==<pinned version> memtrust <args> does the heavy lifting after that. It provisions a Python interpreter if the system doesn't have one, installs that pinned memtrust release from PyPI into an isolated tool environment on first use, then caches and runs the tool. An agent or a CI job with Node available ends up with a fully working Python benchmark tool, no extra setup required.

Node's footprint is big enough to justify closing this gap on its own. Sonatype's 2026 State of the Software Supply Chain Report puts npm at 7.97 trillion package downloads across 2025, up 65.43% year over year, against 804.97 billion for PyPI in the same period (Sonatype, 2026). Take that as a rough proxy for what's likely sitting in CI images: Node-based tooling reaches a huge share of environments by default, which raises the odds that Node is already present even where a curated Python toolchain isn't. A Node wrapper bridges that gap directly.

The uv tool itself has real adoption behind it. Astral's repository sits at 87,595 GitHub stars as of 2026-07-17 (GitHub API, astral-sh/uv). pypistats.org reports 165.67 million downloads in the trailing month. The Python Developers Survey run by the PSF and JetBrains found uv usage went from 0% to 11% adoption in the single year it was introduced, reported in JetBrains' "The State of Python 2025." Stack Overflow's 2025 Developer Survey puts uv at 9.5% adoption against pip's 40.9% and npm's 56.8%. Building a bootstrap step on top of uv was an easy call given those numbers.

Generating structured JSON output

Every memtrust run invocation writes a full JSON report to disk by default, saving to ./memtrust-report-<date>.json unless you point it elsewhere with --output. The command produces that JSON report on top of its console summary, so you never have to remember a --json switch. The report's results block breaks down per-backend, per-eval outcomes using the same signal taxonomy the underlying evals use internally. The memtrust run --eval command currently exposes seventeen registered evals, from longmemeval through temporal_kg_boundary. A program reading the file gets the same verdict a person reading the console table would get.

Here's the actual payload an agent or CI job parses. We pulled this directly from a real memtrust run --backends mempalace,mem0,zep,openviking --eval all invocation against memtrust 0.3.3. The run used zero configured credentials, and we left the output completely unedited:

{
  "mempalace": {
    "status": "skipped",
    "reason": "mempalace is not configured: environment variable MEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup instructions.",
    "missing_env_var": "MEMPALACE_STORAGE_PATH"
  },
  "mem0": {
    "status": "skipped",
    "reason": "mem0 is not configured: environment variable MEM0_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup instructions.",
    "missing_env_var": "MEM0_API_KEY"
  }
}

The output is strict key-value pairs. The status and missing_env_var fields give programs machine-checkable data to act on right away, which leads to a design choice worth calling out.

We added a memtrust report <path> subcommand for reading a prior run's JSON back and printing a formatted summary. That's useful when the agent running the eval and the person or system reviewing it are two separate processes.

There's an easy way for a parser to get this wrong, though. A backend missing a configured credential environment variable keeps the run alive: it prints SKIPPED in that backend's section and continues on. A CLI running unattended in CI needs to behave this way, since a missing API key for one backend shouldn't stop the rest of the eval suite from finishing. But that means a program parsing the JSON has to check each backend's own status field. If it relies only on the command's exit code, it'll wrongly assume a skipped backend actually passed.

The CLI also includes a --sign flag backed by an Ed25519 keypair generated with memtrust keygen. This writes a signed receipt alongside the JSON report, and the signature proves which key produced the file and that the contents haven't been altered. We built this for agent-to-agent consumption: structured output only means something if the agent reading it can verify where it came from, and an unsigned JSON file is trivial to spoof. Signing stays optional for standard runs, but it's there for reports that need to cross a trust boundary.

Current approaches to agent tool distribution

Several existing practices try to close the agent-tool distribution gap. It's worth being precise about what each one actually solves, and where each one stops short.

Protocol-level standards for agent-tool-calling give a direct answer for tool invocation. The Model Context Protocol, introduced in November 2024 and donated in December 2025 to the newly formed Agentic AI Foundation, a directed fund under the Linux Foundation, reported over 97 million monthly SDK downloads and more than 10,000 active public servers at the time of that announcement. A protocol standard solves the invocation contract: it defines how a tool describes its own capabilities and how an agent calls them in a predictable shape. But the underlying tool still bears full responsibility for returning truthful values. Standardizing the envelope leaves the contents completely unverified.

Traditional package-manager install flows, including pip and npm's default behavior, need a human reading terminal output to make judgment calls. The user has to accept prompts, notice warnings, and interpret stack traces. A default pip install or npm install outputs unstructured text, because the people who designed these tools built them for a person watching the terminal, not a program parsing it.

Benchmark and leaderboard self-reporting in the broader ML ecosystem has a well-documented gaming problem. A 2025 analysis found that one major model provider privately tested 27 model variants against a popular crowdsourced arena benchmark before release. Proprietary models received access to 54.3% to 70.1% of the benchmark's evaluation data, while openly available models got far less. That additional arena data access alone produced score gains of up to 112% relative to a baseline (Singh, Nan, Wang, et al., "The Leaderboard Illusion," NeurIPS 2025). A separate survey documented the broader pattern of train/test contamination undermining leaderboard comparisons across the field. A self-reported number that relies entirely on trust is easy to optimize toward instead of actually earning.

Detection-capable claims in vulnerability and bug-finding tooling frequently get conflated with live-reproduced claims. The industry has started to recognize this distinction directly. Recent work building automated exploit-verification pipelines highlights the difference: identifying a plausible flaw is a categorically easier claim than producing a working reproduction against a real running system, and treating the two as equivalent overstates what the detection tool actually found (a 2025 paper on automated exploit generation and validation makes this argument in those terms).

There's a gap running through all four of these categories. Each one lets a tool output well-formatted or plausible data without proving it's true. A protocol standardizes the call. A package manager gets the bits onto disk. A leaderboard reports a number. A detection tool flags a candidate. What none of them do is the adversarial re-verification this project treats as mandatory. An agent-native benchmark tool has to add that on top of everything those categories already handle well: install cleanly, call predictably, report a number, flag a candidate, and then prove that number or flag actually holds up when an independent engineer checks the work.

Verifying the tool's internal claims

The distribution mechanics only matter if the underlying tool is trustworthy. An agent calling memtrust run autonomously parses the JSON and decides what to do next, with no visibility into any silent failures behind it. Part 1 laid out exactly what's at stake here: a fictional API call inside that adapter forced every downstream consumer to act on claims nobody had actually verified.

The discipline from that re-verification keeps running on repeat. Every time the project's backlog surfaces a missing capability, an engineer looks at the actual gap, builds the smallest piece of it that can genuinely be closed, and re-verifies the result independently before calling it done. We scope partial fixes to their real boundaries on purpose, to keep the backlog honest rather than padded. A recent knowledge-graph boundary bug is a good example of this loop in action, and the memtrust eval suite missed it entirely until a few days ago.

Fixing the temporal-KG boundary bug

The bug lives upstream, inside the vendor's own knowledge-graph store. Issue MemPalace/mempalace#1913 reported a race condition: querying a knowledge-graph fact with an as_of point-in-time filter at the exact instant that fact was invalidated returned both the old value and the new value at once. The code used a closed-interval SQL comparison (valid_to >= as_of) where it needed a half-open interval (valid_to > as_of). A fact ending exactly at the query instant still matched, and if its replacement started at that same instant, a single-valued predicate like uses_model reported two contradictory answers at once with no warning at all. The pattern that triggers this is invalidating a fact and adding its replacement at the same shared timestamp, and the vendor's own pre-fix onboarding guidance told every agent to use exactly that pattern. The maintainers fixed it in merged PR #1914, switching to the half-open interval and adding a new supersede() primitive as the preferred atomic replacement.

The memtrust backlog had flagged the capability to detect this class of bug months earlier. We'd bundled it together with a second, unrelated capability (drawer neighbor-expansion leak scoping) into one item, and that item sat open because it was too big for a single scoped fix. Forcing it through just to close the backlog line would have meant shipping something half-built, or quietly widening what counts as "done," and the project rules rule out both of those. The item stayed open until we could shrink it down to a manageable unit.

The scope-expanding investigation is where the real work happened. Splitting the bundled item meant reading the adapter code directly, and that reading turned up something useful: the wiring for kg_query(entity, as_of=..., direction="both") already existed post-rewrite and passed straight through to the confirmed tool_kg_query. The evals were simply passing a zero value for as_of. So the actual capability gap was building a classifier and a test to exercise the path that already worked, which was a much smaller job than the original bundled item made it look. You only find that kind of distinction by reading the actual code with the goal of shrinking the claim down to what's true. Re-estimating a vague item over and over just hides these details instead of surfacing them.

We built a new TemporalBoundarySignal enum (CLEAN, DOUBLE_COUNT, NOT_APPLICABLE) in adapters/base.py, and added a dedicated eval at evals/temporal_kg_boundary.py. This eval seeds a fact, invalidates it, and adds its replacement at the same boundary instant to match the shape of #1913. It queries the database at that instant and classifies the response two separate ways: first, it checks the adapter's own self-reported signal, and second, it computes an independent classification straight from the raw list of facts the query actually returned. That second check is what forces the eval to verify the raw data on its own rather than trusting the adapter's word for it. A disagreement between the two signals points to a bug in the adapter's own classification logic, which is a different failure from a backend double-count, and the eval catches both separately.

The test suite behind this holds to the same standard. It runs against two from-scratch fake implementations that faithfully reproduce the real pre-#1914 closed-interval comparison and the real post-#1914 half-open comparison, computing a real answer from real logic each time rather than leaning on hardcoded stubs.

Fifteen tests, all passing, against fakes that implement both the buggy closed-interval logic and the fixed half-open logic in full.

We'll lay out the limitations here as plainly as the eval's own docstring does. First, the real mempalace package is still absent from the build environment. PR #1914 is merged to the vendor's develop branch, but it's absent from the 3.5.0 release this adapter was built and live-verified against. That makes this a detection-capable harness: we proved it correct against faithful synthetic reproductions of both the buggy and fixed comparison logic, but it currently only runs against synthetic targets. The eval should flag #1913's failure shape the moment we point it at a pre-fix deployment, but we still need to actually run that live test before we call this done. Second, temporal_kg_boundary is now wired into the CLI's eval registry as the seventeenth entry memtrust run --eval can invoke by name. It reports not_applicable against any backend other than mempalace, since MemPalace is the only adapter implementing the knowledge-graph calls this eval needs.

Before this change merged, a second engineer independently re-ran the full suite and confirmed every test passed. ruff check, ruff format --check, and mypy --strict all ran clean against the actual merge commit. The suite has grown since then: we're at 590 tests passing and 8 skipped, at 93% overall coverage as of this writing. We applied the same discipline from Part 1's re-verification here, on a much smaller change, because the habit has to hold on boring changes just as much as it held on the dramatic one.

The drawer neighbor-expansion half of that original bundled backlog item remains open and untouched on purpose. Splitting the item let us close only the piece we could actually verify, and we left the other half exactly where it was.

Catching regressions in mem0 during a live run

The same habit produced a second example a few days later, from a real live benchmark run. The mem0_direct adapter uses self-hosted mem0ai running in-process against a local Qdrant instance and the OpenAI API. It ran contradiction detection, compression/round-trip fidelity, and extraction quality. Contradiction detection came back with 7 for 7 cases silently overwritten and 0% flagged: if you tell it your meeting moved from 2pm to 3pm, it stores the new fact and reports zero changes. Compression/round-trip fidelity landed at a 31.6% mean literal fidelity across 5 cases, which we expected, since mem0's design goal is semantic extraction, not literal fidelity. Extraction quality ran 15 cases, 12 deliberately junk inputs and 3 deliberately valid ones. The system correctly rejected all 12 junk inputs, but lost all 3 valid ones on retrieval. That's a small sample, and we're treating it as a signal worth investigating further rather than a conclusion.

Getting any of those numbers required fixing a bug first: the gpt-5-mini/gpt-5o-mini single-character mismatch described in Part 1, which mem0ai/mem0#6085 had already documented independently, 16 days earlier. That's the value of an automated npm install path in practice: a CI job that pulls this tool in on a schedule, with almost no setup effort, catches real regressions automatically.

Unifying distribution and verification

An agent that installs your CLI automatically but gets back false data hasn't actually solved anything. It still needs a human to check the result behind it, which means a benchmark tool with airtight verification discipline that only runs for a person with a curated Python environment never reaches the audience it's meant for. Agents and CI pipelines make decisions with no human supervision in the loop, so a tool built for autonomous execution has to hold its own claims to the same evidentiary bar it applies to the systems it's benchmarking.

We still have real limitations here. We've wired temporal_kg_boundary into the CLI, but we've only pointed it at the synthetic fakes described above, and we still need to test it against a live, pre-fix MemPalace deployment before it's more than a detection-capable claim. A Kuzu graph-database driver stays unbuilt on purpose, since the upstream project backing it formally deprecated that path and dropped default test coverage for it, and building against an abandoned dependency is a bad trade regardless of what it might buy us. "Detection-capable" and "field-tested against a live deployment of the exact bug" are two different claims, and security and eval tooling generally blur that line. We're calling it out directly so the softer claim has to stand on its own instead of borrowing credibility from the stronger one.

Installation and verification both have to work, together, for any of this to mean something. An agent that can install your tool in one command but can't trust what it prints back hasn't gained much over the tools that came before it.

Repo: github.com/RudrenduPaul/memtrust.

References

Sonatype. "2026 State of the Software Supply Chain." 2026.
GitHub API, astral-sh/uv repository metadata, fetched 2026-07-17.
pypistats.org: uv package download statistics. Fetched 2026-07-17.
JetBrains. "The State of Python 2025." August 2025, based on the Python Developers Survey (PSF + JetBrains).
Stack Overflow. 2025 Developer Survey, Technology section. 2025.
MemPalace. mempalace#1913, "kg query --as-of returns two values for a single-valued fact at a supersession boundary."
MemPalace. mempalace#1914, "fix: half-open as-of interval + supersede() to end KG boundary double-count." Merged.
Anthropic. "Donating the Model Context Protocol and Establishing the Agentic AI Foundation." December 2025.
Linux Foundation. "Linux Foundation Announces the Formation of the Agentic AI Foundation." December 2025.
Singh, Nan, Wang, et al. "The Leaderboard Illusion." NeurIPS 2025, arXiv:2504.20879.
"Benchmark Data Contamination of LLMs: A Survey." arXiv:2406.04244, June 2024.
"CVE-Genie: From CVE Entries to Verifiable Exploits." arXiv:2509.01835. 2025.

Co-authored by Rudrendu Paul and Sourav Nandy. Rudrendu is an engineer building agent-native open-source tooling, including memtrust and a companion suite of AI-agent infrastructure projects. Find the code at github.com/RudrenduPaul.

Would Your AI Benchmark Survive Being Audited Like the Vendors It Judges? Ours Didn't, at First

Rudrendu Paul — Tue, 28 Jul 2026 16:51:52 +0000

Part 1 of 2: how we found our own AI-memory benchmark tool calling a fictional API for one of the four vendors it tests, and the two PASS verdicts the honest fix forced us to take back

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/memtrust, an independent, reproducible benchmark harness for agent-memory backends. pip install memtrust-cli.

One of the four agent-memory vendors we benchmark, MemPalace, has two public GitHub issues about its own benchmark numbers. One has 233 thumbs-up reactions and 39 comments, and the other has 42 and 8 (both verified live via the GitHub API on 2026-07-17). They document a headline 100% LongMemEval score measured with Haiku reranking that was never reproducible from the repo's own benchmark scripts, and it was eventually pulled from the README as unverifiable. A separate 96.6% figure that gets quoted everywhere turns out to be mostly the default embedding model doing the work in raw mode. A "lossless" compression claim drops that same 96.6% to 84.2% once you actually measure it, a 12.4 percentage point gap. Two pull requests attempting to fix the reporting, MemPalace/mempalace#433 and MemPalace/mempalace#729, were both closed unmerged the same day, April 12, 2026. #729 was closed within seven minutes of being opened (verified against the PR's own timestamps on GitHub).

This isn't an isolated case. Stanford's 2026 AI Index Report has a line in its technical performance chapter that should worry anyone citing a benchmark number in a sales deck. On SWE-bench Verified, model performance went from 60% to near 100% in a single year (Stanford HAI, 2026 AI Index Report). A jump like that signals a benchmark saturating faster than anyone can redesign it, which means the number stopped measuring the thing it was built to measure. This problem has real precedent: a 2019 study built fresh test sets for ImageNet and CIFAR-10 matching the original collection methodology as closely as possible, and found accuracy dropped 11 to 14 percentage points on ImageNet and 3 to 15 points on CIFAR-10 the moment the benchmark stopped being one everybody had already been implicitly tuning against for years (Recht et al., ICML 2019).

We built memtrust to stop guessing which vendor's number to trust. We run LongMemEval, LoCoMo, and a growing set of evals we wrote ourselves against all four backends the same way, and we publish the raw logs so the numbers can be whatever they actually are. While doing that work, we found something uncomfortable: our own tool had been making the same kind of unverified claim about itself, in its own code, since the very first version.

A fresh clone, zero credentials, memtrust run --backends mempalace,mem0,zep,openviking --eval all. Every backend reports a real SKIPPED status, and the run still exits cleanly with a real report. That's the honesty standard the tool is supposed to hold itself to, and it matters for what comes next.

Our MemPalace Adapter Called a Fictional Class

The adapter talking to it is a normal piece of glue code. memtrust calls store(), query(), update(), and delete(), and the adapter translates those into whatever the vendor's real API expects. Every version of that adapter, until this rewrite, called mempalace.Palace(storage_path=...), a class with .remember(), .recall(), and .invalidate() methods.

We ran one line to confirm it before starting the rewrite:

$ python3 -c "import mempalace; hasattr(mempalace, 'Palace')"
False

The exact terminal check that surfaced this. Palace is not an attribute of the installed mempalace package, version 3.5.0. Every store/query/update call this adapter ever made against it ran through a class that doesn't exist.

We grepped every ^class definition across the entire installed package to make sure we weren't missing a re-export somewhere. mempalace/palace.py, the file the class name implies it should live in, defines exactly two things: MineAlreadyRunning and MineValidationError, both exceptions. There is no Palace class anywhere in the package.

That means store(), query(), and update() had never worked against the real vendor package in this project's entire history. Every test that appeared to pass was exercising a hand-written fake standing in for a guess about an API that was never confirmed, and that guess was wrong. The adapter's own pre-rewrite documentation had already flagged this honestly as a low-confidence guess: "Confidence: MEDIUM on product behavior, LOW on exact Python method names." Low confidence turned out to mean zero confidence, because the class was fictional.

Here's the part that made this more than an embarrassing bug: memtrust exists to catch a vendor claiming something its product doesn't actually do, and we'd just found our own tool doing precisely that about its own capability, unnoticed the whole time.

Rewriting the Adapter Against the Real MCP Server

The fix required calling the vendor's real MCP server directly. Its tool calls dispatch to plain, real, module-level functions: tool_status, tool_list_wings, tool_list_rooms, tool_add_drawer, tool_search, tool_update_drawer, tool_delete_drawer, tool_kg_add, tool_kg_invalidate, tool_kg_query. Those functions had already been in limited, correct use in one narrow corner of the same adapter for metadata calls. The rewrite made them the only way the adapter talks to it for everything.

We verified the vendor's docstrings by running the code. Every response shape documented in the rewritten adapter was captured by calling the real functions live against a real local, chromadb-backed palace. We ran pip install mempalace, pointed MEMPALACE_PALACE_PATH at a temp directory, called mempalace.mcp_server.tool_add_drawer(...), and read what actually came back. Live-verification caught behaviors that a docs read alone would have missed or gotten wrong:

tool_search results carry no per-record ID field at all. There's no id, no drawer_id, nothing, just content and scoring fields. We considered recomputing an ID client-side from the content, then rejected it: the vendor's update_drawer() keeps a drawer's original ID stable across a content edit on purpose. We verified this ourselves by updating a drawer and confirming the pre-update ID still comes back. Recomputing from a query response's current content would silently produce the wrong ID for anything that had ever been edited. The adapter now reports memory_id="", which is an honest admission that the real vendor response has nothing to put there.
The knowledge-graph subsystem ignores MEMPALACE_PALACE_PATH entirely when called as a library. We traced this to a module-level flag that only ever gets set to True when the process's own command-line arguments carry a --palace flag. A library caller never passes that argument. The practical effect is bad: two MemPalaceAdapter instances pointed at two different storage paths in the same process both read and write the exact same physical knowledge-graph file on disk. That's a real vendor bug we found by exercising the adapter's own test setup, and it was an unreported issue, so we wrote it up.

One finding cut the other way. The vendor's own docstring for tool_add_drawer claims that content chunked across multiple physical rows can't be updated or deleted by its logical ID. It states, "tool_get_drawer(drawer_id) and tool_delete_drawer(drawer_id) report 'not found' on the chunked path." We read the actual function that resolves those calls, and it handles the chunked case correctly. Live-testing confirmed it: storing 2,000 characters of content, well over the 800-character default chunk size, then updating or deleting it by its logical ID worked right every time we tried it. An earlier draft of this rewrite had trusted the vendor's docstring text here without checking it live, which would have shipped a false "confirmed limitation" in the other direction. Both kinds of error, trusting a fictional API and trusting stale documentation, come from the same root cause: trusting written claims over executable code.

Re-Verifying Our Own Prior Verdicts

A rewrite fixing the fictional API is still just a fix made and validated by the person who wrote it. The standard this project applies to every vendor is that a self-report requires independent verification against reality, so we applied that same standard to ourselves.

Thirteen rows in memtrust's issue-validation log, our internal record of every real GitHub issue we've investigated against these four backends, had a verdict on this vendor's repo sitting on PASS or deferred. They were built and tested against the old, fictional adapter. Once the real rewrite existed, every one of those thirteen went back through independent, freshly spawned reviewers who had no memory of building the original fix. They checked each claim against the real, working code from scratch.

Eleven of the thirteen held up. Two of those eleven had real, working mechanisms that the rewrite had accidentally dropped along the way, ordinary collateral damage from restructuring a file: an authored_at ranking fallback and a docstring section distinguishing "no API key required" from "no network access required" both got restored, and we wrote new tests proving them against the real adapter this time. One row was confirmed correctly still failing, matching how it had been marked before. The remaining two rows turned out to be wrong and got downgraded.

Downgrading PR #1005 for Unshipped Capabilities

A prior version of memtrust claimed it could detect when the vendor's search silently degrades, falling back and reporting a partial result. It stated the claim was "confirmed against the real, merged PR diff" for MemPalace/mempalace#1005. Two things turned out to be false in that sentence. First, PR #1005 was never merged. GitHub shows it closed unmerged by its own author as self-superseded. We confirmed this directly against the live PR. The retrieval-fallback mechanism it proposed did land eventually through separate PRs, but the observability fields memtrust's code was reading, warnings and available_in_scope, were never shipped in any of them. Second, we grepped the real, installed package's source for those exact field names and found zero matches anywhere. The code path that would populate them does not exist in the shipping product.

The parsing logic itself is fine and won't crash or misread a response if that shape ever does appear. It simply never fires because the vendor never sends that response. We had correctly implemented a check against a capability the vendor hasn't built yet, and that's a weaker claim than the one we had been making. The verdict moved from PASS to PARTIAL to reflect that gap.

Downgrading Issue #1733 for Measuring the Wrong Code Path

The second downgrade cuts deeper. A contributor named Kartalops filed a real, well-diagnosed bug. The vendor's memory "wake-up" sort was supposed to prioritize high-importance, recent memories. Nothing in the real ingest path ever wrote the importance field it sorted by. Every memory defaulted to the same value, and the sort silently degenerated into plain insertion order. A prior memtrust fix built a ranking-quality signal and marked this PASS on the theory that it could now detect that exact failure shape.

Once we traced the real code paths after the rewrite, the fix and the bug turned out to live in two different parts of the vendor's own codebase. Kartalops's bug is in mempalace/layers.py, in a function called Layer1.generate(), the actual "wake-up" sort. memtrust's adapter has never called that function in any version, including the fictional one. What memtrust actually calls is tool_search, a structurally separate code path that sorts purely by vector similarity. The ranking-quality signal we built is real and does catch a real class of ranking degradation, just a different one. It was checking a field that belongs to a method this adapter has never invoked.

A capability gap sits a category above a parsing bug or a missed edge case. There is currently no way for memtrust to reach, measure, or detect the bug Kartalops reported through any code path this adapter has, so the verdict moved from PASS to FAIL, capability gap. That's the honest label for a simple fact: right now, we can't do this.

The Final Validation Numbers Across All Backends

Across all four backends, 197 real GitHub issues and PRs have been investigated this way. Each one asks the same question: given the code as it actually exists today, can this tool diagnose or resolve the exact failure that was reported? Here are the results after this re-verification and the two downgrades above:

Verdict	Count	Share
PASS	55	28%
PARTIAL	16	8%
FAIL, capability gap	42	21%
FAIL, not applicable	84	43%

197 issues investigated in total.

55 out of 197 is a sobering result compared to a vendor's 96.6%, but it's the number that was still standing after we went looking on purpose for the kind of overclaiming this project exists to catch, and we found it in our own code before anyone else did. A PASS rate that survives someone actively trying to break it carries weight that an unchallenged PASS rate lacks.

Known Gaps in the Remaining Adapters

This adapter is the one we can currently stand behind at this level of confidence, because it's the one we rewrote against real, live-verified functions and then independently re-checked. The other three adapters are still works in progress. memtrust's own methodology documentation grades adapter confidence on a scale from High to Low, and today that grade actually varies quite a bit from one adapter to the next. We assign high confidence where source code or a documented SDK was read directly and exercised against real installed packages. We assign low confidence anywhere a method signature is still a best-effort reconstruction from documentation that lacked full availability during the build. OpenViking's adapter is flagged as the one most likely to need correction against a live instance. Its public documentation covers resource and skill ingestion in detail while leaving out a confirmed endpoint for writing or querying a conversational memory entry.

This fix settles something narrower than every number in this project. One previously false claim is now either true or honestly labeled as pending. The process used to get there involved independent re-verification by reviewers separate from the original fix, and we intend to keep pointing that same process at the rest of the codebase.

Finding a Vendor Bug During Our First Live Run

Everything above covers a bug in our own MemPalace adapter. While we were fixing it, memtrust also produced its first live benchmark result against a different backend, and getting that result turned up a second bug, this one belonging to the vendor.

mem0_direct runs the actual, self-hosted mem0ai open-source library in-process, backed by a local Qdrant instance and OpenAI for embeddings and extraction. This targets the self-hosted library only, not the commercial product built on top of it, and that's an important distinction to keep in mind for everything below.

A clean mem0ai install with nothing set but OPENAI_API_KEY fails every single LLM-based extraction call out of the box, for anyone. mem0's own default model, hardcoded in mem0/llms/openai.py as "gpt-5-mini", is a reasoning-tier model that only accepts the API's default temperature. mem0's own reasoning-model detection in mem0/llms/base.py checks its model-name set for the string "gpt-5o-mini", one character off from the model it actually ships, so the check never matches, and mem0 sends temperature=0.1 on every call regardless. Every extraction call comes back a 400 error for anyone who just sets an API key and runs it. Our Mem0DirectAdapter now works around it by passing is_reasoning_model=True, mem0's own documented override for this exact situation. mem0ai/mem0#6085 documents the same failure independently; a different user filed it on 2026-07-04, 16 days before we hit it ourselves.

With that fixed, we pointed it at contradiction detection, compression/round-trip fidelity, and extraction quality. Contradiction detection ran seven cases: we told it a fact, then contradicted it later, and mem0 silently overwrote the old value every single time, yielding a 100% silent-overwrite rate and 0% flagged as a conflict. Compression/round-trip fidelity ran five cases and came back with 31.6% mean literal fidelity. That low score is expected, since mem0 performs semantic extraction only, so a literal-reconstruction score was always going to be low; it just quantifies the cost of that design choice. Extraction quality ran fifteen cases, twelve deliberately junk and three deliberately valid. All twelve junk inputs were correctly rejected, and all three valid ones were lost on retrieval. That valid-case sample is only three, small enough that it's worth digging into further rather than treating as conclusive.

We wanted to include this here right away. The same standard that caught our own fictional API in the MemPalace adapter caught a real bug in a vendor's default configuration on the very first live run against a second backend, which tells us the methodology holds up against everyone's code, including our own.

Existing Methods for Verifying Benchmark Claims

Set aside memtrust for a moment. If you're evaluating an AI-agent memory backend, or any AI system whose vendor publishes its own accuracy claims, you have roughly three existing paths to figuring out whether a number is real, and each one leaves a gap.

General-purpose LLM and RAG evaluation frameworks give you real, usable metrics. They offer faithfulness, answer relevancy, hallucination detection, and custom LLM-as-judge scoring built for evaluating a single model's output against a single-turn or single-session query. We checked the documentation of RAGAS, DeepEval, and TruLens, the three most widely used open-source frameworks in this category, directly. None of them ship a first-class way to test memory. They lack tests for recall across separate sessions days apart, what happens when a new fact contradicts an old one, or how retrieval quality decays as the knowledge base ages. The closest built-in metric tracks retained facts "throughout a conversation," by its own documentation's wording, which means one continuous session, stopping short of the multi-session boundary an agent-memory product is actually supposed to solve. You can build that evaluation on top of these frameworks' general primitives, but nobody ships it out of the box.

Vendor self-reported benchmarks are the default path because they're the ones already sitting in the product's own documentation. Setting dishonesty aside, the structural problem is that self-grading and public grading face different incentives, and there's a documented asymmetry in how freely that grading gets checked. A 2024 industry analysis found that four of thirteen vector-database vendors it reviewed contractually prohibited customers from publishing independent benchmark results against them (BenchANT, "To Benchmark Vector Databases or to Get Sued for Breaching a DeWitt Clause?", 2024). Vendors remain free to publish comparative numbers about themselves; whether someone else can check that number against reality is a decision the vendor's own contract makes for you, in a documented number of real cases.

Academic benchmark suites are the strongest, most rigorous foundation available, and also the ones with the least reason to overstate a result. LongMemEval, from researchers at UCLA, Tencent AI Lab, and UC San Diego, tests five distinct long-term memory abilities across curated question sets embedded in chat histories that scale up to 500 sessions and roughly 1.5 million tokens. Its own headline finding is highly critical of the systems it tested: ChatGPT, running on GPT-4o, answered 91.8% of questions correctly when given the full conversation directly, and only 57.7% once its own memory retrieval sat between the model and the answer (Wu et al., "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory," ICLR 2025). LoCoMo, from UNC Chapel Hill, USC, and Snap Research, built very long synthetic conversations and found an even starker gap. Human accuracy on its question-answering task measured 87.9 F1, while the best model tested, GPT-4-turbo, reached 32.4 (Maharana et al., "Evaluating Very Long-Term Conversational Memory of LLM Agents," ACL 2024). Both are the kind of rigorous, adversarial ground truth a vendor benchmark should be measured against, but they still leave an operational gap: you still need to know whether a specific vendor's product, wired up the way you'd actually deploy it, behaves the way the vendor's marketing page says it does, and whether it catches the specific failure a real user already hit in production.

A persistent gap runs through all three of these approaches. None of them makes it someone else's ongoing job to point an adversarial standard at real, reported production failures against a specific vendor's actual shipped product, or to republish the raw evidence and take back a result once it's shown to be wrong. That's the job memtrust is built to do, and it treats its own prior verdicts as requiring the same fresh, adversarial check as a vendor's claim.

Process Changes Driven by the Fictional API Bug

The surface lesson is: verify vendor APIs before you ship an adapter. The narrower, less comfortable lesson underneath it is that a tool built to catch overclaiming is just as capable of producing it. The only thing that caught it here was treating our own PASS verdicts with the same suspicion we point at everyone else's, on a fixed schedule rather than only when something goes wrong.

The test suite behind all of this currently runs 590 passing tests, 8 skipped, entirely offline, with zero live vendor credentials required. That number proves the eval logic is internally consistent. It doesn't prove any single adapter's wire format matches a live vendor's real API, and that's why the re-verification step above exists as a separate, independent check.

Part 2 of this series covers what happens when you combine this verification discipline with a distribution layer that lets an agent or CI job install and run the tool with no human in the loop.

Repo: github.com/RudrenduPaul/memtrust. The full 197-row validation log, commit by commit, is in the repo if you want to check any of the numbers above yourself. That transparency is the entire point of publishing them this way.

References

Stanford Institute for Human-Centered Artificial Intelligence. The 2026 AI Index Report. 2026.
MemPalace/mempalace#27: "Multiple issues between README claims and codebase." Opened April 7, 2026.
MemPalace/mempalace#29: "Multiple issues with benchmark methodology and scoring." Opened April 7, 2026.
MemPalace/mempalace#433: "feat: rule-based contradiction detection (issue #27)." Closed unmerged April 12, 2026.
MemPalace/mempalace#729: "fix: Multiple issues between README claims and codebase (closes #27)." Closed unmerged April 12, 2026, seven minutes after opening.
BenchANT. "To Benchmark Vector Databases or to Get Sued for Breaching a DeWitt Clause?" 2024.
Wu, Wang, Yu, Zhang, Chang, Yu. "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory." ICLR 2025, arXiv:2410.10813.
Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang. "Evaluating Very Long-Term Conversational Memory of LLM Agents." ACL 2024, arXiv:2402.17753.
Recht, Roelofs, Schmidt, Shankar. "Do ImageNet Classifiers Generalize to ImageNet?" ICML 2019, arXiv:1902.10811.

How to Check If AI Systems Can Find and Cite Your Site (in 5 Minutes)

Rudrendu Paul — Tue, 28 Jul 2026 16:51:42 +0000

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/LLMScout, a zero-dependency, cross-platform CLI that runs 21 GEO/AEO checks against any website. npm install -g llmscout-cli or pip install llmscout-cli.

Position-1 click-through rate on keywords that trigger an AI Overview fell from 7.3% in December 2023 to 1.6% in December 2025, per Ahrefs. If an AI system can't find or cite your site, that traffic doesn't come back on its own. Here's how to check where your own site actually stands, in under five minutes.

What LLMScout checks

LLMScout is a CLI that runs 21 checks against any site: 12 covering technical SEO fundamentals, 9 covering GEO signals specifically for AI systems. Every check reports PASS, WARN, or FAIL, and the whole run exits with a machine-readable code, so it drops straight into CI. It ships as two distributions: an npm package with two runtime dependencies, and a Python package with zero. Neither one needs a headless browser.

One check worth knowing about before you run it is ai-crawler-directives. It reports on all 7 major AI bots separately, because blocking GPTBot (which trains models) does nothing to stop OAI-SearchBot (which does live retrieval for ChatGPT) from fetching and citing your page. They're independently configurable in robots.txt, and most tools treat them as a single signal.

Install

There are two ways to install it. If you're not sure which one you want, try npm first:

npm install -g llmscout-cli

This needs Node 18+, and it pulls in exactly two runtime dependencies: cheerio and commander. No Python, no browser download, no subprocess calls.

If you'd rather use Python:

pip install llmscout-cli

This version has zero runtime dependencies; it's pure standard library. It runs the same 21 checks and gives the same PASS/WARN/FAIL verdicts as the npm version.

Both install in under a minute on Windows, macOS, and Linux.

Run your first check

Once installed, point it at a site:

llmscout init ./my-site --site-url https://yourdomain.com
llmscout check ./my-site

init writes an llmscout.json config file into ./my-site. check fetches the URL in that config and runs all 21 checks against it.

Here's real output from running check against example.com:

LLMScout check -- https://example.com

[PASS] (technical) Title tag
  Title "Example Domain" is 14 characters, within the recommended 10-60 range.

[WARN] (technical) Meta description
  No meta description found.
  Fix: Add <meta name="description" content="..."> with 50-160 characters summarizing the page.

[WARN] (technical) Canonical tag
  No <link rel="canonical"> tag found.
  Fix: Add a canonical link tag pointing at the preferred URL for this page.

[FAIL] (technical) robots.txt
  robots.txt was not reachable at https://example.com/robots.txt (HTTP 404).
  Fix: Add a robots.txt file at your site root, even a permissive one, so crawlers
  and agents have explicit directives.

[WARN] (technical) sitemap.xml
  No sitemap was reachable (tried: https://example.com/sitemap.xml).
  Fix: Add a sitemap.xml at your site root, or point to one with a Sitemap: directive
  in robots.txt, to help search engines discover pages.

[PASS] (technical) Heading structure
  Exactly one <h1> and no skipped heading levels detected.

[WARN] (geo) Structured data (JSON-LD)
  No JSON-LD structured data found.
  Fix: Add schema.org JSON-LD markup (e.g. Organization, WebSite, or Article) so
  generative engines can understand the page's entities.

[WARN] (geo) AI crawler directives
  robots.txt is unreachable, so AI-crawler directives could not be determined.
  Fix: Add a reachable robots.txt if you want to state an explicit policy for AI
  crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot,
  Google-Extended, Applebot-Extended).

Summary: 6 PASS, 14 WARN, 1 FAIL (21 checks)

There are three things worth understanding about that output.

FAIL exits with code 1. At least one check failed outright, and that's the signal a CI pipeline should gate on. WARN is a missed optimization; it won't fail the run on its own. PASS means the check came back clean.

Notice the dependency between the last two checks: because robots.txt is unreachable, ai-crawler-directives can't determine any policy, so it reports WARN instead of FAIL. An unreachable robots.txt and one that explicitly blocks every AI bot are different problems with different fixes, and the tool keeps that distinction explicit instead of collapsing both into one warning.

Every WARN and FAIL comes with a fix suggestion built in. You don't need to look up what a canonical tag is or how to add one; the fix line tells you.

JSON output and CI

For CI, add --json and the output becomes something a script can actually parse:

llmscout --json check ./my-site

Exit codes: 0 for a clean run, 1 if at least one check FAILed, 2 for a usage or config error. That's enough to fail a build on a new FAIL without writing your own parser logic.

The JSON includes per-check id, status, message, and fix fields, plus a summary object with the totals. Wire it into a pre-merge check and a PR that regresses on structured data or robots.txt breaks the build instead of shipping quietly. The full schema is in the README.

Fleet mode for multiple sites

If you manage more than one site, llmscout fleet takes a JSON manifest of site paths and runs the full 21-check suite against every one of them in a single command. --out-dir writes one auto-named report file per site instead of dumping everything to stdout at once.

llmscout fleet ./fleet.json --out-dir ./reports

The manifest:

{
  "sites": [
    { "name": "client-a", "path": "./clients/client-a" },
    { "name": "client-b", "path": "./clients/client-b" }
  ]
}

That produces client-a.txt and client-b.txt (or .json with --json), one report per site, named straight from the manifest's name field. That gives an agency a separate, named report file per client that you can open, send, or archive, instead of scrolling back through one combined terminal dump.

Wrap up

The checks cover the signals that matter most for AI-system citability: structured data, crawler access policies, content extractability, entity markup. Contested signals, like llms.txt, are flagged as informational since the evidence for their impact is still developing. Run it against your own site and see where you actually stand.

GitHub: https://github.com/RudrenduPaul/LLMScout
npm: https://www.npmjs.com/package/llmscout-cli
PyPI: https://pypi.org/project/llmscout-cli/

Co-authored by Rudrendu Paul and Sourav Nandy. Rudrendu is an AI/ML builder and published researcher (ICML, Springer Nature, Elsevier, IEEE) with 15+ years of experience at Fortune 50 companies, building open-source tools for the AI development ecosystem. Find the code at github.com/RudrenduPaul.