DEV Community

Cover image for How to Evaluate an MCP Server Before You Connect It to Your Agents
Waxell profile image Logan
Logan for Waxell

Posted on • Originally published at waxell.ai

How to Evaluate an MCP Server Before You Connect It to Your Agents

#security #mcp #devops #ai

Most teams install MCP servers the same way they used to install npm packages: find one that does what you need, copy the config, move on. That was already a risky habit in the npm ecosystem. With MCP, the consequences are different in kind, not just in degree. A bad npm package breaks your build. A bad MCP server runs inside your agents — with your credentials, your data, and your users on the other end.

MCP server evaluation is the process of assessing a third-party server's authentication model, tool definition integrity, permission scope, and governance compatibility before connecting it to your agents — not after. It's the step between "this server does what I need" and "this server is safe to run." Most teams skip it. Most teams will eventually wish they hadn't.

Why MCP Is a Different Kind of Supply Chain Risk

The npm comparison isn't a metaphor. Security researchers watching the MCP ecosystem describe it explicitly: rapid adoption, minimal vetting, growing enterprise dependency, and the same underlying pattern that made npm a regular source of supply chain incidents.

But there's a meaningful difference. When a malicious npm package runs, it runs in a build environment — usually isolated, usually reviewable, with blast radius limited to what the build process touches. When a malicious MCP server runs, it runs inside an agent that's connected to your email, your database, your file system, your GitHub. The agent doesn't know the difference between a tool doing what you expected and a tool doing something it was quietly updated to do.

The numbers from early security assessments make the scale of the problem concrete. Equixly's offensive security analysis of MCP servers in the wild — a commercial assessment, not a peer-reviewed study — found that 43% are vulnerable to command injection. Endor Labs' research across 2,614 MCP implementations found that 82% use file operations prone to path traversal. Astrix Security's audit of 5,200+ public MCP server repos found that 53% rely on static API keys or personal access tokens with no expiry and no scope limit. And the risks compound: Enkrypt AI's scan of 1,000 MCP servers found that 33% had critical vulnerabilities, and Pynt's research showed that chaining just ten MCP plugins pushes the probability of a successful exploit to 92%.

Those numbers aren't theoretical. In a widely documented incident researchers called "NeighborJack," hundreds of MCP servers were found running exposed to the public internet, bound to 0.0.0.0 without a firewall. The servers were operational. They were doing their jobs. They were also completely accessible to anyone who thought to look.

The rug pull attack is the most dramatic version of this failure mode — a server that behaves correctly for weeks, then changes. But the rug pull requires an attacker. Most of the risk in MCP deployments today doesn't require anyone malicious. It requires teams that never evaluated the server they connected.

The Five Checks Before You Connect

This isn't a comprehensive security audit. It's the minimum viable evaluation — the five questions that surface the most risk before you've written a single line of agent code.

1. What's the authentication model?

The first thing to check is how the server handles credentials. Does it require a long-lived static API key? A personal access token? Or OAuth 2.1 with proper scopes and refresh flow?

More than half of MCP servers in the wild still use long-lived static credentials — a single key that, once compromised, grants persistent access with no expiry and no scope limit. That's not the server's fault; it reflects where the ecosystem is. But it's your call whether to accept that model in your stack. If you're connecting to a server that asks for a broad API key "to get started," ask what it would take to use scoped, rotatable credentials instead. If the answer is "it doesn't support that," weight it accordingly.

2. Can you read the tool definitions before you connect?

MCP servers expose tools to your agent. The tool descriptions — what each tool is named, what parameters it accepts, what it claims to do — are the interface your agent reasons about. If you can't read those descriptions before connecting, you can't know what your agent is being asked to trust.

This matters because tool descriptions are also an attack surface. Tool poisoning embeds malicious instructions directly in tool descriptions, exploiting the fact that agents process them as context. Before you connect any MCP server, open the source, find the tool definitions, and read them. Not skim — read. If the server is closed-source or the definitions aren't accessible before installation, that's a reason to pause.

3. What permissions does it actually need — and what does it ask for?

Most MCP servers ask for more access than the job requires. This isn't necessarily malicious — it's the path of least resistance at setup time — but it's worth examining. If a documentation server needs read access to your GitHub repos, that's reasonable. If it also requests write access because "some users find it useful," that's scope creep baked into the default config.

Before connecting, map the tasks your agent will actually perform with this server against the permissions it requests. Any gap between what the task needs and what the server asks for is a scope you're accepting by default. The narrower you can make that gap upfront, the less you'll need to fix later. This is the foundational principle of MCP least privilege, and it starts at evaluation, not after deployment.

4. What's the version history and update behavior?

A server that's been through twenty stable releases with documented changes is a different risk profile than one published three weeks ago by an account with no other activity. Check both.

Version history tells you how the maintainer handles changes — whether updates are documented, whether breaking changes are flagged, whether the tool definitions have been stable or volatile. An MCP server that silently updates its tool definitions is exactly the pre-condition for a rug pull. If you can't see a clear history of what changed and when, you have no baseline to detect when something changes after you've connected.

5. What's the multi-server blast radius?

If you're running multiple MCP servers — which most non-trivial agent deployments do — evaluate each one in context of the others. A server with modest permissions on its own may compound significantly with another server that shares authentication context or accesses the same underlying system.

The compounding exploit probability cited above comes from exactly this scenario: each additional server widens the attack surface, and a compromise in one can propagate because the adjacent servers were never evaluated for combined risk. Before connecting a new server, ask: if this server were compromised tomorrow, what else in our current setup would it have reach toward?

Red Flags Worth Stopping For

Some things are blockers. If you see these, the evaluation isn't over — it just became a longer conversation.

The server requires broad OAuth scopes and frames them as necessary for basic functionality. A calendar server that needs access to your entire Google account to read calendar events isn't scoped correctly. If the documentation doesn't explain why a given scope is required, that's the question to ask before connecting.

The server has no mechanism for you to be notified when tool definitions change. This is the structural condition that makes rug pulls possible. If the server doesn't version its tools or notify connected clients of definition changes, you're accepting continuous invisible exposure to whatever the maintainer ships next.

The maintainer is unknown, the package is recent, and the download counts spiked suddenly. This is the typosquatting / name-squatting pattern that the npm ecosystem learned to recognize. The MCP ecosystem is still learning.

There's no security contact, no disclosure policy, and no issue history. Open-source doesn't automatically mean trustworthy, but a project with years of public issue history and an active maintainer has a track record you can read. A project with none of that is asking you to trust without evidence.

Evaluation Is Necessary. It's Not Sufficient.

The five checks above tell you whether a server is safe to connect. They don't tell you whether it stays safe after you've connected it.

Tool definitions change. Credentials rotate — or don't. Servers that were scoped correctly at evaluation drift over time as maintainers add features and teams change. The evaluation gets you to first connection; it doesn't govern what happens in every session after that.

What comes after evaluation is a registry — a governed record of which MCP servers are approved, what tools each one is permitted to expose to each agent, and what policies govern those tool calls at runtime. And critically, change detection: when a tool definition updates server-side, execution against that definition should pause until the change is reviewed, not continue silently because the server still responds.

This is the thing evaluation alone can't provide. You can vet a server thoroughly at connection time and still have no way to know when it changes on Tuesday. The controlled interface pattern — where tool call boundaries are explicit and validated at runtime — is what closes that gap.

Evaluation answers "is this safe to connect?" Governance answers "is this still safe, right now, in this session?"

How Waxell handles this: Waxell's Registry maintains versioned identities for every connected MCP server and its exposed tools. When a tool definition changes server-side, execution blocks until the change is acknowledged — which means the rug pull attack doesn't survive contact with a governed deployment. Policies define what each agent is permitted to call, independent of what the server exposes. And because every execution is logged against the policy that governed it, you can answer any question about what your agents were allowed to do and what they actually did. Get early access →

Frequently Asked Questions

What is MCP server evaluation?
MCP server evaluation is the process of assessing a third-party MCP server's authentication model, tool definition transparency, permission requirements, maintainer credibility, and multi-server blast radius before connecting it to your agents. It's the due diligence step between finding a server that does what you need and confirming it's safe to run in your stack. Most teams skip it because there's no immediate failure signal when a server is over-privileged or inadequately maintained — the risk is invisible until it isn't.

How do I vet an MCP server before connecting it?
Five checks cover the most common failure modes: verify the authentication model (prefer OAuth 2.1 over long-lived static tokens), read the tool definitions before connecting (not after), audit whether the permissions requested match the permissions the task actually requires, review the version history for how changes are handled, and assess the combined blast radius with other MCP servers already in your environment. These don't replace a full security audit, but they surface the risks that cause the most incidents.

What makes an MCP server trustworthy?
Open-source code you can read, tool definitions that are versioned and auditable, an authentication model that supports scoped and rotatable credentials, a maintainer with a verifiable track record, and a clear process for security disclosure and change notification. Trustworthiness isn't binary — it's a risk profile you assess relative to what the server has access to and what your agents will do with it.

What are red flags in an MCP server?
Requests for broad OAuth scopes framed as necessary for basic functionality, no versioning or notification mechanism for tool definition changes, an unknown or recently created maintainer account with a sudden download spike, and no security contact or public issue history. Any of these individually warrants a longer look. Multiple of them together is a reason to evaluate alternatives.

Is evaluating MCP servers the same as having MCP governance?
No. Evaluation is the pre-adoption step — it determines whether a server is safe to connect. Governance is the runtime layer — it determines what agents are allowed to do with connected servers in every session after that. A server that passes evaluation can still expose you to risk if tool definitions drift, permissions expand, or your agents operate under policies that were never formally defined. Evaluation and governance are both necessary; they answer different questions.

Does every MCP server need to go through this evaluation?
Yes, including internal ones. First-party MCP servers you build yourself still need tool definitions reviewed, permission scopes audited, and policies defined for which agents can call them. The failure modes from over-privileged internal servers are the same as for third-party ones — and internal servers often get less scrutiny precisely because they feel more trusted.

Sources

Top comments (0)