We Sandbox-Tested 23 MCP Servers. Here's What We Found.

John Aleksander Jazbec — Thu, 05 Nov 2026 11:45:47 +0000

AI agents are becoming increasingly capable. But every external tool we connect to them introduces a new trust boundary.

The Model Context Protocol (MCP) makes connecting tools to AI agents easier than ever. Yet an important question remains:

How do we verify what an MCP server actually does before trusting it with our files, credentials, and infrastructure?

At XData, we're researching this through an open-source project called XData AgentHub.

Beyond a list of MCP servers

Most MCP directories help developers discover servers.

We wanted to go further.

Instead of simply listing available integrations, we designed a verification framework that evaluates identity, software artifacts, security characteristics, and observable runtime behavior.

Our approach uses progressive verification levels:

L0 — Catalogued: A server is listed, but sufficient verification evidence is not yet available.
L1 — Identity verified: Publisher identity, repository ownership, and artifact integrity are checked.
L2 — Security inspected: A pinned server version undergoes isolated inspection, dependency scanning, secret detection, and tool metadata analysis.
L3 — Runtime tested: The server is executed inside a sandbox with a defined test plan and observable permission boundaries.

The goal is not to declare software perfectly safe. It's to provide reproducible evidence about the exact versions being evaluated.

Our first results: 23 MCP servers

In our November 1, 2026 snapshot, we evaluated 23 MCP servers.

Verification level	Servers
L3 — Runtime tested	1
L1 — Identity verified	15
L0 — Catalogued	7

Only Playwright reached L3 in this initial dataset.

That doesn't mean the remaining servers are malicious. It means they haven't satisfied the evidence requirements for higher levels.

The surprising part: runtime network behavior

We tested Playwright MCP inside a gVisor sandbox.

The environment restricted filesystem access, network destinations, privileges, and credentials.

Our predefined test plan included both expected operations and negative cases.

All six test cases produced the expected results.

During execution, we observed:

Filesystem writes remained under /tmp.
The observed child processes were Chromium and its crash handler.
Allowed test-site connections succeeded.
Chromium also attempted connections to external Google DNS endpoints.
The sandbox blocked those external attempts.

The important distinction is that a blocked connection is not a successful compromise.

However, this experiment demonstrated why runtime observation is valuable: software behavior can include network activity that isn't immediately obvious from its exposed MCP tools.

Why this matters for AI agents

MCP servers often run with access to developer environments, local files, API keys, and internal services.

Trusting a server should therefore involve more than recognizing its publisher or inspecting its documentation.

We believe a practical verification framework should answer several questions:

Who published this server?
Can we verify the exact artifact?
What vulnerabilities and suspicious patterns are present?
What tools and permissions does it expose?
What does it actually do when executed?
Can someone reproduce the results?

These questions form the foundation of our Verified MCP work.

Introducing XData AgentHub

Verified MCP is one part of a broader open-source project.

XData AgentHub also provides 104 security-reviewed AI agent skills for Claude Code, Codex, and OpenCode.

The catalog covers engineering workflows, security reviews, DevSecOps, infrastructure, RAG, and AI agent development.

Our philosophy is simple:

AI agents should be powerful, but their capabilities should also be transparent, verifiable, and controllable.

The project is free and MIT-licensed.

GitHub: https://github.com/XData-si/xdata-agenthub

Documentation and methodology: https://github.com/XData-si/xdata-agenthub/blob/main/docs/VERIFIED_MCP.md

Interactive catalog: https://xdata-si.github.io/xdata-agenthub/

What's next?

We're expanding runtime verification, improving permission profiles, and exploring continuous re-verification when MCP server versions change.

Our research also has limitations. Every result applies to a specific artifact, environment, and test plan. Passing our checks is not a universal security guarantee.

We're publishing the methodology and evidence because we want others to challenge and improve the approach.

If you're building or maintaining MCP servers, what evidence would convince you that a server is safe enough to connect to a production AI agent?

I'd love to hear your thoughts.

I'm an AI engineer and founder of XData and CognitioLabs in Slovenia. We work on open-source AI infrastructure, agent security, and digital sovereignty.