DEV Community

Cover image for I Scanned 1,200 MCP Configs From GitHub. Here's What I Found.
AD
AD

Posted on

I Scanned 1,200 MCP Configs From GitHub. Here's What I Found.

#security #ai #opensource #python

A deep-dive into the security posture of real-world AI agent deployments — and the open-source tool I built to fix it.

TL;DR

I collected 1,200 real MCP (Model Context Protocol) configuration files from public GitHub repositories, scanned them with an open-source security tool I built, and found that:

  • 100% had security gaps
  • 20.7% had CRITICAL or HIGH issues (hardcoded secrets, unauthenticated endpoints, unrestricted shell execution)
  • 0 of 1,200 configured response limits or session caps
  • The 9 most popular MCP servers (240K+ combined stars) all had findings — 5 were CRITICAL

The tool is Pluto AgentGuard. It's free, runs locally, and takes about 3 minutes to scan 1,200 configs.

The Problem: We Secured LLM Outputs but Forgot About Agent Actions

The AI security conversation has focused heavily on what LLMs say — hallucinations, jailbreaks, harmful content. Entire product categories exist for prompt filtering and output guardrails.

But the attack surface has shifted. Modern AI agents don't just generate text — they do things: browse the web, execute shell commands, query databases, push code, trigger CI/CD pipelines. The Model Context Protocol (MCP) is the dominant standard for connecting these capabilities to LLMs.

Here's the disconnect: nobody is auditing the configuration layer that determines what agents can actually do. The MCP config file — usually claude_desktop_config.json or .mcp.json — is the security boundary between "an AI assistant that helps me code" and "an AI assistant that can run arbitrary commands on my machine."

I wanted to know: how secure are these configurations in the real world?

Methodology: How I Collected 1,200 Configs

Collection

I used the GitHub Code Search API to find real MCP configuration files across public repositories. The search targeted:

  • claude_desktop_config.json files containing mcpServers
  • .mcp.json files with MCP server definitions
  • mcp_config.json and similar variants

Collection rules:

  • Maximum 3 configs per repository (avoid bias from monorepos)
  • Deduplicated by SHA-256 content hash (identical configs counted once)
  • Only files that parsed as valid JSON with MCP server definitions
  • 7-second delay between API pages (GitHub rate limit: 10 requests/minute)

Result: 1,200 valid configs from 1,159 unique repositories, collected June 25, 2026.

Scanning

Each config was scanned using Pluto AgentGuard's scan_mcp_config function, which checks for:

  1. Dangerous server packages — a curated database of 13+ MCP servers known to grant high-risk capabilities (shell execution, browser control, database write, source control write)
  2. Missing authentication — remote MCP endpoints (http:// or https:// URLs) without auth headers or tokens
  3. Insecure transport — HTTP instead of HTTPS for remote connections
  4. Hardcoded secrets — 18+ regex patterns matching API keys, tokens, passwords, and private keys embedded in config values
  5. Context safety gaps — missing response size limits (max_tokens, max_response_length) and session caps (max_turns, session_timeout)
  6. Human-in-the-loop (HITL) absence — high-risk servers configured without approval gates

Each finding is assigned a severity (CRITICAL / HIGH / MEDIUM / LOW / INFO) and mapped to OWASP Agentic AI threat categories.

The entire scan ran locally in ~3 minutes. No API keys. No cloud. No LLM calls.

Results: The Numbers

Overall (1,200 configs)

Metric Value
Total configs scanned 1,200
Unique repositories 1,159
Total findings 2,904
🔴 CRITICAL 88 (3.0%)
🟠 HIGH 280 (9.6%)
🟡 MEDIUM 2,536 (87.3%)
Configs with CRITICAL or HIGH 20.7%
Configs with any finding 100%

Every single config had at least a MEDIUM finding. One in five had a CRITICAL or HIGH issue.

The Popular Servers (9 configs, 240K+ combined GitHub stars)

I also separately scanned some of the highest-starred MCP servers to see how the most popular, most copied configs look:

Server Stars Max Severity Key Finding
Context7 58K 🔴 CRITICAL No authentication on remote endpoint
Chrome DevTools MCP 44K 🔴 CRITICAL Full Chrome DevTools Protocol access, no HITL
Serena 26K 🔴 CRITICAL Unrestricted shell execution, no HITL
FastMCP 26K 🟡 MEDIUM Context safety gaps
Activepieces 23K 🔴 CRITICAL No authentication on remote endpoint
n8n MCP 22K 🟠 HIGH Arbitrary code execution via workflows, no HITL
Google MCP Toolbox 16K 🟠 HIGH Unrestricted SQL (supports 20+ databases), no HITL
Figma MCP 15K 🟡 MEDIUM External content injection risk
mcp-chrome 12K 🔴 CRITICAL No auth + insecure HTTP transport

5 CRITICAL. 2 HIGH. 0 of 9 had response limits or session caps.

I've filed security issues on the CRITICAL repos: Context7, Chrome DevTools, Serena, Activepieces, mcp-chrome.

The 4 Most Common Risks (With Examples)

1. Browser Control Without Approval (CRITICAL)

Chrome DevTools MCP (44K★) gives the agent full Chrome DevTools Protocol access. That includes:

• Attach to existing Chrome sessions
• Execute JavaScript in page context
• Capture network response bodies (which may contain tokens or PII)
• Read cookies and local storage
• Intercept and modify requests
Enter fullscreen mode Exit fullscreen mode

Without an approval gate, a prompt injection could instruct the agent to access session data from authenticated browser tabs.

Note: Some clients like Claude Desktop prompt for approval before tool execution. However, the config itself doesn't enforce this — and not all MCP clients provide client-side HITL.

2. Shell Execution Without Sandboxing (CRITICAL)

Serena (26K★) provides shell execution capabilities to the agent. When enabled, the agent can run commands with the process user's permissions, which could include reading sensitive files, accessing credentials, or modifying the system.

The Serena team notes that the shell tool is disabled in most contexts and that Claude Desktop requires approval by default — a fair point. The risk surfaces when the config is used with clients that don't enforce HITL, or when users explicitly enable the shell tool without understanding the implications.

3. Unauthenticated Remote Endpoints (CRITICAL)

Context7 (58K★) and Activepieces (23K★) expose remote MCP endpoints over HTTPS with no authentication. Anyone who knows the URL can connect.

The typical config looks like:

{
  "mcpServers": {
    "context7": {
      "url": "https://mcp.context7.com/mcp"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

No API key. No OAuth. No mTLS. A remote API endpoint without client authentication.

4. Missing Response Limits (MEDIUM — but universal)

Zero of 1,200 configs set max_response_length or max_tokens on their MCP servers. This enables context stuffing attacks: a malicious tool returns an oversized response that pushes the agent's system prompt and safety instructions out of the context window.

This is the lowest-effort fix imaginable — add two lines to your config — and nobody does it.

Why This Matters: The "Left of Boom" Gap

The current AI security stack looks like this:

[Prompt Filters] → [LLM] → [Output Guardrails] → [Agent Actions]
     ✅ covered         ✅ covered            ❌ unmonitored
Enter fullscreen mode Exit fullscreen mode

Teams invest in prompt injection detection and output filtering. But the agent action layer — what the LLM actually does through MCP tools — is a blind spot. There's no "firewall" between the LLM's tool-use decision and the actual execution.

This is the "left of boom" problem. By the time a guardrail catches something, the agent may have already:

  • Executed a shell command
  • Queried a database
  • Triggered a CI/CD pipeline
  • Accessed authenticated sessions

You need to catch the risk before the agent gets access to these capabilities. That means auditing the configuration layer.

Introducing Pluto AgentGuard

I built Pluto AgentGuard to fill this gap. It's a security launch gate for AI agents — you run it before deploying, not after something breaks.

What it does (7 commands)

Command What it does
aguard scan Static analysis of MCP configs, secrets, permissions
aguard test 22 attack scenarios across 6 packs test your policy's coverage
aguard whatif Simulate policy changes and see risk delta before applying
aguard owasp Map findings to 20 OWASP-inspired controls
aguard evidence Generate launch readiness evidence packets
aguard baseline Create baselines, detect configuration drift over time
aguard monitor Replay agent traces, detect unauthorized tool calls

Quick start 

pip install pluto-aguard

# Scan your MCP config
aguard scan ./your-project/

# Test your policy against attack scenarios
aguard test --policy ./policy.yaml --attack-pack all

# See what happens if you add a new server
aguard whatif --config ./config.yaml

# Map to OWASP controls
aguard owasp ./your-project/
Enter fullscreen mode Exit fullscreen mode

What makes it different

Most MCP security tools do config scanning. AgentGuard adds three things I haven't seen elsewhere:

  1. Policy testing (aguard test): Instead of "does your config have issues?", it asks "does your policy actually stop attacks?" — 22 scenarios covering prompt injection, data exfiltration, privilege escalation, context manipulation, supply chain, and social engineering.

  2. What-if simulation (aguard whatif): Before you add a new MCP server or change a policy rule, simulate the impact. See the risk score delta. Catch regressions before they ship.

  3. Evidence generation (aguard evidence): Produces a structured evidence packet (scan results + test results + OWASP mapping + risk score) for security review sign-off. Useful for enterprise teams that need launch gates with artifacts.

CI/CD integration

AgentGuard ships as a GitHub Action:

- uses: arpitha-dhanapathi/pluto-aguard@v0.9.2
  with:
    scan-path: ./
    fail-on: high  # Block PR if HIGH or CRITICAL found
    format: sarif   # Upload to GitHub Security tab
Enter fullscreen mode Exit fullscreen mode

It also supports JSON, Markdown, HTML, and SARIF output formats.

What You Should Do Right Now

If you're using MCP servers in any AI agent setup, here's a 5-minute security checklist:

1. Run a scan (30 seconds) 

pip install pluto-aguard
aguard scan ./your-project/
Enter fullscreen mode Exit fullscreen mode

2. Add response limits (1 minute)

Add to every MCP server in your config:

{
  "max_response_length": 8000,
  "max_turns": 20,
  "session_timeout": 3600
}
Enter fullscreen mode Exit fullscreen mode

3. Add HITL for dangerous servers (2 minutes)

If you use Chrome DevTools, Playwright, Serena, filesystem, or any shell-capable server — enable human-in-the-loop approval. The exact mechanism depends on your client (Claude Desktop, Cursor, VS Code, etc.), but the principle is: the agent should ask before executing destructive operations.

4. Authenticate remote endpoints (2 minutes)

If your MCP server is remote (HTTPS URL instead of stdio), add auth:

{
  "mcpServers": {
    "my-server": {
      "url": "https://my-server.com/mcp",
      "headers": {
        "Authorization": "Bearer ${MCP_API_KEY}"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

5. Add AgentGuard to your CI (bonus)

Block PRs that introduce MCP misconfigurations:

- uses: arpitha-dhanapathi/pluto-aguard@v0.9.2
  with:
    scan-path: ./
    fail-on: high
Enter fullscreen mode Exit fullscreen mode

The Bigger Picture

MCP is 18 months old and already the de facto standard for agent-to-tool communication. The ecosystem is moving fast — 90K+ stars on awesome-mcp-servers, thousands of servers, and major platforms (Claude, Cursor, VS Code, Windsurf) supporting it natively.

But the security tooling hasn't kept pace. We're in the "move fast and break things" phase of agent infrastructure, and the configs people are shipping to production look like the web in 2005 — no auth, no limits, full trust.

The good news: the fixes are simple. Auth headers, response limits, HITL approval, and a scan in CI. None of this requires new technology — just applying existing security principles to a new surface.

The bad news: right now, almost nobody is doing it.

Let's fix that.

Pluto AgentGuard is open-source (Apache 2.0), written in Python, and runs entirely locally. Star it on GitHub if this was useful.

Have questions or findings to share? Open an issue or find me on LinkedIn.

Top comments (0)