DEV Community

Cover image for Are your MCP server's safety labels honest? A one-command check (and what it found on 31 popular servers)
Formael profile image Dilan Srilal (dilansri)
Dilan Srilal (dilansri) for Formael

Posted on • Originally published at formael.com

Are your MCP server's safety labels honest? A one-command check (and what it found on 31 popular servers)

#mcp #security #devtools #opensource

You wrote this once, on the day the tool was born:

{
  "name": "search_docs",
  "annotations": {
    "readOnlyHint": true,
    "destructiveHint": false,
    "openWorldHint": false
  }
}
Enter fullscreen mode Exit fullscreen mode

Then the tool kept changing. A parameter got added. A backend call got wired in. The schema grew.

The annotations block stayed exactly as you left it.

Why the drift matters

The MCP spec calls these annotations hints and tells clients not to trust them blindly. Plenty of clients use them anyway, because they arrive at tool-list time and they are cheap to act on. A common pattern: readOnlyHint: true means skip the confirmation prompt.

So a stale label costs more than a confused reviewer. It can quietly remove a safety step that would otherwise have fired.

The protocol has no step that re-checks the label against the schema as the tool evolves. That is the gap actlint fills. It reads what a server advertises over tools/list, works out what each tool most likely does from the name, description, and input schema, and reports where that disagrees with what the tool declares. It never calls a tool.

We pointed it at 541 tools across 31 widely used servers, including official ones from Stripe, GitHub, Microsoft, Google, MongoDB, and PayPal. Eleven came back completely clean. Four carry a label that denies something the tool's own schema says it can do. Full scorecard, per-server grades, methodology, and the two findings that did not survive our own hand review: MCP Safety Labels Run on the Honor System.

This post is about running it on yours.

Ways to point it at a server

Local stdio server, launched by actlint:

npx actlint npx -y @modelcontextprotocol/server-filesystem .
Enter fullscreen mode Exit fullscreen mode

Hosted server over HTTP, with a credential if it needs one to list its tools:

npx actlint --http https://mcp.example.com/mcp --header "Authorization: Bearer ${MCP_TOKEN}"
Enter fullscreen mode Exit fullscreen mode

Server that needs an environment variable to start.

npx actlint --env STRIPE_SECRET_KEY npx -y @stripe/mcp --tools=all
Enter fullscreen mode Exit fullscreen mode

Capture once, check many times

Talking to the server is the only step that touches the network. Everything after it is pure, so you can save the manifest and replay it as often as you like with identical results:

# Once, where the credential lives
npx actlint --capture tools.json --http https://mcp.example.com/mcp --header "Authorization: Bearer ${MCP_TOKEN}"

# Every run after this, offline, no credential, no socket
npx actlint --manifest tools.json --fail-on high
npx actlint --manifest tools.json --json  -o report.json
npx actlint --manifest tools.json --sarif -o results.sarif
Enter fullscreen mode Exit fullscreen mode

That split is also the right shape for CI: capture in a job with network access, gate in one without.

In GitHub Actions 

- uses: formael/actlint/packages/github-action@df516d81ba3719687d1d72227737887c3534e1db # v0.1.0
  with:
    args: --manifest manifest.json --fail-on high
Enter fullscreen mode Exit fullscreen mode

Turning it on for a server that already has findings, without a red build on day one:

npx actlint --manifest tools.json --write-baseline baseline.json
npx actlint --manifest tools.json --baseline baseline.json --fail-on low
Enter fullscreen mode Exit fullscreen mode

Baseline entries are keyed by fingerprint, so they survive re-ordering and unrelated edits.

Reading the scorecard 

  actlint  ▸  stdio server                                      honesty grade: A
  ──────────────────────────────────────────────────────────────────────────────
  14 tools scanned · 0 under-declared · 0 undeclared · 0 over-declared

  advisories — capability hygiene, not honesty verdicts
  ◐ ADV   move_file  no-scope-constraint
      This is a sensitive action whose input schema carries no narrowing constraint …
      ↳ OWASP ASI02:2026 · OWASP MCP02:2025 · CoSAI MCP-T3 · EU AI Act Art.15 · NIST …
Enter fullscreen mode Exit fullscreen mode

Three verdicts, in order of how much they should worry you:

Verdict Meaning
under-declared The tool claims less risk than it has. This is the one that matters.
undeclared Says nothing either way. The spec default already protects the client, so this is a nudge.
over-declared Honest excess caution, paid for in alarm fatigue.

Advisories are capability notes rather than honesty verdicts, so they never move the grade and never fail a build.

The grade is an honesty grade, not a safety score. A server full of honestly declared destructive tools earns an A. Any under-declared finding caps it at B.

Every finding carries a plain-English rationale and a mapping to OWASP ASI, the OWASP MCP Top 10, CoSAI, the EU AI Act, and NIST. Any rule id explains itself offline:

npx actlint explain write-as-readonly
Enter fullscreen mode Exit fullscreen mode

How it decides

Two design choices are worth knowing before you trust the output.

It doesn't look at your annotations while deriving. The derivation step reads only the name, description, and input schema. Your declared hints are the thing being audited, not an input to the reasoning, so you can't nudge the linter toward a friendlier answer by editing the label it is grading.

The schema is read structurally, not as text. A format: "uri" field nested a few objects deep gets found. The substring "url" sitting inside a description does not create a signal. Schema shape is weighted more heavily than the name and description signals, which are natural language and therefore ambiguous.

Where signals conflict or are missing, the result resolves toward concern with lowered confidence. uncertain is a real, non-failing outcome. Under-claiming a risk beats crying wolf, because a linter people stop believing is not worth running.

What it isn't

It is a linter, in the lineage of eslint, tflint, and hadolint. It never runs, routes, or blocks anything. It reads labels and tells you whether they hold up.

Tool poisoning, runtime over-privilege, and auth are real problems, and they belong to other tools.

A clean grade is not a safety certificate. It means the server told the truth about what its tools can do. Those tools can still be dangerous, and truthful labels are what let a client handle that properly.

Go check yours 

npx actlint <command-to-launch-your-server>
npx actlint --http https://your-mcp-host/mcp
Enter fullscreen mode Exit fullscreen mode

If it gets your server wrong, that report is the most useful thing you can send us. A false positive caught by the maintainer who knows the tool is how the rules improve.

Repo: github.com/formael/actlint

What does your server grade?

Top comments (0)