AI agents are becoming increasingly capable. But every external tool we connect to them introduces a new trust boundary.
The Model Context Protocol (MCP) makes connecting tools to AI agents easier than ever. Yet an important question remains:
How do we verify what an MCP server actually does before trusting it with our files, credentials, and infrastructure?
At XData, we're researching this through an open-source project called XData AgentHub.
Beyond a list of MCP servers
Most MCP directories help developers discover servers.
We wanted to go further.
Instead of simply listing available integrations, we designed a verification framework that evaluates identity, software artifacts, security characteristics, and observable runtime behavior.
Our approach uses progressive verification levels:
- L0 — Catalogued: A server is listed, but sufficient verification evidence is not yet available.
- L1 — Identity verified: Publisher identity, repository ownership, and artifact integrity are checked.
- L2 — Security inspected: A pinned server version undergoes isolated inspection, dependency scanning, secret detection, and tool metadata analysis.
- L3 — Runtime tested: The server is executed inside a sandbox with a defined test plan and observable permission boundaries.
The goal is not to declare software perfectly safe. It's to provide reproducible evidence about the exact versions being evaluated.
Our first results: 23 MCP servers
In our November 1, 2026 snapshot, we evaluated 23 MCP servers.
| Verification level | Servers |
|---|---|
| L3 — Runtime tested | 1 |
| L1 — Identity verified | 15 |
| L0 — Catalogued | 7 |
Only Playwright reached L3 in this initial dataset.
That doesn't mean the remaining servers are malicious. It means they haven't satisfied the evidence requirements for higher levels.
The surprising part: runtime network behavior
We tested Playwright MCP inside a gVisor sandbox.
The environment restricted filesystem access, network destinations, privileges, and credentials.
Our predefined test plan included both expected operations and negative cases.
All six test cases produced the expected results.
During execution, we observed:
- Filesystem writes remained under
/tmp. - The observed child processes were Chromium and its crash handler.
- Allowed test-site connections succeeded.
- Chromium also attempted connections to external Google DNS endpoints.
- The sandbox blocked those external attempts.
The important distinction is that a blocked connection is not a successful compromise.
However, this experiment demonstrated why runtime observation is valuable: software behavior can include network activity that isn't immediately obvious from its exposed MCP tools.
Why this matters for AI agents
MCP servers often run with access to developer environments, local files, API keys, and internal services.
Trusting a server should therefore involve more than recognizing its publisher or inspecting its documentation.
We believe a practical verification framework should answer several questions:
- Who published this server?
- Can we verify the exact artifact?
- What vulnerabilities and suspicious patterns are present?
- What tools and permissions does it expose?
- What does it actually do when executed?
- Can someone reproduce the results?
These questions form the foundation of our Verified MCP work.
Introducing XData AgentHub
Verified MCP is one part of a broader open-source project.
XData AgentHub also provides 104 security-reviewed AI agent skills for Claude Code, Codex, and OpenCode.
The catalog covers engineering workflows, security reviews, DevSecOps, infrastructure, RAG, and AI agent development.
Our philosophy is simple:
AI agents should be powerful, but their capabilities should also be transparent, verifiable, and controllable.
The project is free and MIT-licensed.
GitHub: https://github.com/XData-si/xdata-agenthub
Documentation and methodology: https://github.com/XData-si/xdata-agenthub/blob/main/docs/VERIFIED_MCP.md
Interactive catalog: https://xdata-si.github.io/xdata-agenthub/
What's next?
We're expanding runtime verification, improving permission profiles, and exploring continuous re-verification when MCP server versions change.
Our research also has limitations. Every result applies to a specific artifact, environment, and test plan. Passing our checks is not a universal security guarantee.
We're publishing the methodology and evidence because we want others to challenge and improve the approach.
If you're building or maintaining MCP servers, what evidence would convince you that a server is safe enough to connect to a production AI agent?
I'd love to hear your thoughts.
I'm an AI engineer and founder of XData and CognitioLabs in Slovenia. We work on open-source AI infrastructure, agent security, and digital sovereignty.
Top comments (2)
The gap I'd want filled in first is L2. If the levels are progressive, Playwright at L3 must also have cleared L2, but the table reads L3: 1, L1: 15, L0: 7 with no L2 row — so is that zero servers at L2, or does L3 imply it? It matters because the honest reading of the table is "almost everything is L1", and whether the bottleneck is inspection effort or inspection findings changes what a user should do with an L1 server.
On the DNS observation: the sandbox sees the whole process tree, and Chromium reaching for Google DNS looks like the bundled runtime's own behaviour rather than the MCP server's. Without a per-tool egress baseline for the runtime itself, every runtime test will flag the runtime's normal noise next to the tool's actual behaviour, and that noise compounds as you add servers. Can the sandbox attribute observed egress to the tool's own code, or is the report at process-tree level?
On your closing question: the evidence that would convince me is the pinned artifact plus a test plan someone else can re-run — and re-run on every version bump, which is the continuous re-verification you flag as next. A pass at one version says nothing about the next, and that's the part most directories never handle.
Thanks — that clears up all three points.
The L2 explanation makes sense. If the table represents the highest level reached rather than independent counts, adding the empty L2 row should remove the ambiguity. More importantly, having a concrete “why not L2” for every L1 server makes the badge much more useful than a simple pass/fail label.
Process-level attribution plus a clean runtime baseline sounds like exactly the right next step for L3. The delta is what I'd want to see: what the same Chromium/Node runtime does on its own versus what appears only when the MCP server is running. That should make the sandbox report much easier to interpret and avoid treating expected runtime behaviour as tool behaviour.
The version-bump behaviour is also stronger than I assumed. Resetting to L0 when the artifact changes, pinning by SHA-256, and keeping the test plan/workflows reproducible gives the verification a clear scope: this exact artifact passed these exact tests. Automating the already-demonstrated 0.0.82 → 0.0.84 re-verification loop feels like the natural next step.
That combination — artifact-bound verification, explicit failure reasons, runtime-delta attribution, and automatic re-verification on release — would address the main trust problem I have with most MCP directories.