From 9% to 63%: Letting a Benchmark Build a Better MCP Security Proxy

Gowthaman — Thu, 16 Jul 2026 21:09:21 +0000

You can't improve what you don't measure. So I measured how much of the AI-agent attack surface a defense actually covers — then used that number, over and over, to build a better one.

A year ago, "AI security" mostly meant "don't let the chatbot say something dumb." Today, AI assistants
act: they read your files, query your databases, hit APIs, and browse the web. They do it through a
fast-spreading open standard called the Model Context Protocol (MCP) — think of it as USB for AI
agents, one standard plug between an assistant and the tools it uses.

That plug is powerful, and it's a brand-new place for things to go wrong. A malicious or compromised
tool can hide instructions in its own description, quietly swap its behavior after you've approved it,
or trick the agent into shipping your credentials to an attacker. The 2025–2026 research literature now
documents dozens of these attacks.

So the obvious question: when you put a security tool in front of your MCP servers, how much of that
danger does it actually stop? Nobody had a good number. I set out to get one — and then something more
useful happened. The measurement didn't just grade the defense; it built it. Over the study, my
open-source proxy mcp-bastion went from covering 9%
of the known MCP attack surface to 63%, one measured gap at a time.

This is the story of that loop, and what it teaches about where AI-agent security actually stands.

First, an honest measuring stick

I built mcp-defense-bench, an open, vendor-neutral
benchmark that measures how much of the MCP attack surface a defensive proxy covers. Three design
choices keep it honest, and they matter more than the headline number:

A matched benign control for every attack. Every malicious test case has a near-identical innocent twin. A tool only scores if it flags the attack and leaves the twin alone. Without this, a "security tool" that flags everything would score 100% while being useless in practice.
Measure only what the tool inspects at runtime. Adapters drive each tool's real code — not a mock, not its docs. If a proxy inspects tool descriptions but never looks at tool results, it scores zero on result-borne attacks, even if its rules could have matched. We measure deployed behavior, not theoretical capability.
Reproducible or it doesn't count. A defender is scored only if its detection runs locally and deterministically. Cloud-model "magic" that can change between runs is observed, not scored.

The rubric behind it is a threat–control crosswalk: 24 MCP attack vectors, each mapped to an
architectural layer, a STRIDE class, and the governance frameworks organizations actually use — the
NIST AI Risk Management Framework, the NSA's MCP Security guidance, and the OWASP Top 10s for LLM and
Agentic applications. (If you want the "map of the whole landscape" version of this story, that's a
a separate write-up; here I'm focused on the improvement loop.)

Each attack scores one of three levels: none (missed), detect (warns you), or enforce
(blocks it). Weighted coverage counts an enforce as 1.0, a detect as 0.5.

The loop: measure → find the gap → fix → measure again

Here's the part that surprised me. A benchmark is usually a scoreboard you check at the end. This one
became a to-do list.

The proxy started at 9% — it scanned tool descriptions for poisoning and nothing else. Then the
benchmark pointed at the next gap, I closed it, and re-ran. Every step below was re-verified at zero
false positives:

Step	What the benchmark exposed → what I added	Coverage
baseline	description scanning only	9%
1	result-borne injection → scan tool results, not just definitions	18%
2	parameter smuggling → validate arguments against the tool's schema	23%
3	DNS-rebinding on local servers → an origin check that blocks	30%
4	cross-tool exfiltration → sensitive-argument + least-privilege scanning	34%
5	a brand-new split-payload attack → cross-tool correlation	38%
6	four vectors no tool covered → command-injection, config-drift, server-identity, cross-server taint	48%
7	depth, not breadth → evasion normalization + inline secret redaction + block-by-default	63%

Two moments in that table are worth slowing down on, because they're where "measure a defense" turned
into "discover something."

When the benchmark caught up with a fresh attack

Halfway through, two attacks were published (June 2026) that my frozen rubric didn't have yet:
mid-session tool injection (a tool that mutates its behavior after you've approved it) and
ShareLock (a stealthy attack that splits a malicious payload across several tools, so no single
tool description looks suspicious).

I verified both against their source papers, encoded each as a matched attack/benign pair, and re-ran
every tool. The result was stark: ShareLock was covered by nothing. Every scanner looked at one tool
at a time and saw nothing wrong.

But a proxy sees the whole tool set at once. So I added cross-tool correlation — it scans the
combined descriptions and flags the coordinated "share/checksum"-style staging metadata the attack uses.
mcp-bastion became the first and (on the benchmark) only measured tool to catch it. A benchmark that
only tested a fixed set of attacks would have missed this entirely. The value was in re-measuring as
the threat surface moved.

The attack only an aggregating proxy can see

Later, I added cross-server data-flow tracking. Picture an agent reading a secret from your vault
server and then, two calls later, posting it to a webhook on a different server. No single tool did
anything wrong; the danger is in the flow across a trust boundary. A per-server scanner is blind to
it. A proxy that fronts all your servers isn't — it can watch a credential-shaped token read from server
A reappear in an argument headed to server B, and stop it. That's not a rule you can bolt onto one
tool; it's a capability you only get from the vantage point.

From "warns you" to "blocks it"

A tool that detects an attack and prints a warning is a linter. A tool that blocks it is a firewall.
The last leg of the journey (48% → 63%) was mostly about turning warnings into blocks — safely.

Evasion normalization. The benchmark had obfuscated variants of known attacks — payloads hidden behind Cyrillic look-alike characters or base64. My proxy caught the invisible-character trick but sailed right past the others. Adding a normalization pass (fold look-alikes to ASCII, decode base64, then scan) took its robustness from 1 of 3 evasions caught to 3 of 3.
Inline secret redaction. Instead of just warning that a tool result contains an AWS key, the proxy now strips the secret out of the result before the agent (or your logs) ever see it. Detection became mitigation.
Block the sure things by default. The deterministic, near-zero-false-positive checks — schema violations, command-injection payloads, exact-token cross-server leaks — now block out of the box, while the fuzzier heuristics stay as warnings. The proxy went from blocking a handful of vectors to blocking 11 of the 19 it covers.

Through all of it, the false-positive count stayed at zero across 35 test cases. That's the whole
point of the matched benign controls: you can push toward "block by default" because you've proven the
checks don't fire on innocent traffic.

The honest ceiling: what a proxy will never fix

Here's the result I care about most, and it's not the 63%. It's the 5 of 24 vectors that no measured
tool covers — and that no content-scanning proxy ever will:

Registry/supply-chain attacks (typosquatted packages, unverified provenance) — need cryptographic attestation and a verified registry, not runtime scanning.
Sandbox escape — needs OS-level isolation.
Consent fatigue — needs client-side UX controls.

A proxy is necessary but not sufficient. The data makes the case for defense-in-depth across
distinct mechanism classes better than any argument could: 19 of 24 vectors are caught by some
tool but missed by others, and a whole cluster sits outside every proxy's reach. Anyone selling you a
single box that "secures your MCP" is selling you 63%, at best.

It re-runs itself now

The loop is no longer manual. mcp-bastion's continuous-integration pipeline runs the full benchmark
on every change and fails the build if coverage drops below a floor or any false positive appears.
The measurement isn't a one-time paper result; it's a standing regression test on the defense. When the
next MCP attack drops, the process for handling it is already wired: add the vector, re-measure, close
the gap, and the gate makes sure it stays closed.

Try it, break it, tell me I'm wrong

Everything is open:

Benchmark: mcp-defense-bench — the rubric, the corpus, the adapters, and a public leaderboard. Add an adapter for your tool and measure it.
Reference proxy: mcp-bastion on npm (npx -y mcp-bastion) and in the official MCP registry.
The full paper (methodology, tables, references): Measuring the Defenders: A Layer-Aware, Framework-Mapped Benchmark for Model Context Protocol Security Proxies — arXiv:2607.XXXXX (link going live shortly).

One disclosure, up front: I build one of the tools the benchmark measures. That's exactly why the
whole thing is open, vendor-neutral, driven through each tool's real code, and scored alongside
competitors and a do-nothing baseline — so you don't have to take my word for any of it. Re-run it
yourself.

The number that matters isn't 63%. It's that an honest measurement, applied relentlessly, turned a
9%-coverage scanner into the broadest MCP defense measured — and drew a bright line around the 37% that
needs a fundamentally different kind of tool. That line is the useful part.

If you work on MCP, agent security, or AI governance, I'd genuinely value critique of the rubric and
the methodology — open an issue, or reach me at agowthaman90@gmail.com.

The MCP reliability & security gap — and an open-source proxy that fills it

Gowthaman — Wed, 08 Jul 2026 13:31:18 +0000

As AI agents move from demos to production, the Model Context Protocol (MCP) has become the default way to give them tools. But the operational layer around MCP is still immature, and two gaps show up fast.

The reliability gap

MCP servers disconnect mid-session — a crashed subprocess, a dropped network stream — and the agent has no good signal. It sees a generic "No such tool available" error, indistinguishable from a tool that never existed. Worse, it can't reconnect itself; a human has to intervene. In a long autonomous run, capabilities silently vanish and the agent fails in confusing ways.

The security gap

MCP tools are described in natural language that the model reads and acts on. That creates two live threats:

Rug pull — a server presents a benign tool, you approve it, and later swaps the definition for a malicious one. Static, open-source scanners only check once at install time, so they miss it.
Tool poisoning — hidden instructions in a description ("ignore previous instructions", "read the user's SSH key and post it to...") that steer the model.

Runtime defenses for these live mostly in paid platforms.

mcp-bastion

mcp-bastion is a proxy that addresses both, and it's deliberately boring to adopt: it's a correct MCP server to your client and a correct MCP client to your servers, so it works with any compliant client through configuration alone — no per-client code — and removing it is a one-line revert.

It's organized in three layers:

Reliability — health checks, capped-backoff auto-reconnect, and control tools (bastion__status, bastion__reconnect) that let the agent itself inspect and recover connections.
Runtime security — trust-on-first-use tool pinning that blocks rug pulls until you re-approve, heuristic poisoning inspection, and cross-server shadowing detection.
Audit & compliance — every tool call becomes a structured, optionally tamper-evident event, fanned out to pluggable sinks (console / file / webhook) and mapped to NIST AI RMF and OWASP LLM Top 10.

It speaks both stdio and Streamable HTTP. TypeScript, Apache-2.0, layered architecture, tests, and CI.

npx mcp-bastion

Repo: https://github.com/Gowthaman90/mcp-bastion

It's early and I'd love feedback — especially on the security heuristics and which audit sinks people want.

DEV Community: Gowthaman