DEV Community

Cover image for Why You Need MCP Governance (And Why Observability Isn't the Same Thing)
Logan for Waxell

Posted on • Originally published at waxell.ai

Why You Need MCP Governance (And Why Observability Isn't the Same Thing)

Most teams shipping MCP-connected agents have spent real time on observability. They can see which tools are being called, track latency, catch errors. They have dashboards. They have alerts. They feel covered.

They're not covered. They're informed.

There's a difference, and it matters more than most teams realize until something goes wrong.


What Is MCP Governance?

MCP governance is the set of runtime policies and enforcement mechanisms that determine what an AI agent is permitted to do with MCP tools — independent of what the model decides to do. Where observability tells you what your agent did, governance determines what it's allowed to do. You can have complete observability and zero governance. Most teams with MCP deployments do.

The distinction isn't subtle once you see it. Observability is a window. Governance is a wall. Watching an agent call a tool you didn't intend it to call is observability working correctly. Preventing that call from happening is governance.


Why Isn't Observability Enough?

Because watching something happen and controlling whether it happens are different problems.

This is the most common misunderstanding in MCP deployments right now. Teams invest in tracing, build alert pipelines, instrument their tool calls — and conclude they've handled the hard part. What they've actually built is a very good record of what went wrong. That's genuinely useful. It's not the same as preventing it.

Consider what observability can and can't do:

Observability can tell you that your agent called a database write tool 847 times in a single session. It cannot stop that session from running. Observability can show you that sensitive data appeared in a tool call payload that should have been masked. It cannot block that payload from being sent. Observability can surface that a tool your agent has been calling for three weeks silently updated its definition last Tuesday. It cannot prevent your agent from running the new definition until you've reviewed it.

In every one of those cases, observability is the incident report. Governance is the control that makes the incident not happen.

The cost asymmetry here is real. Catching a problem after the fact — after data has moved, after a bill has arrived, after a tool has run a version you didn't approve — is categorically more expensive than preventing it. Observability is necessary. It's just not sufficient.


What Actually Goes Wrong Without It?

Not hypothetically — here's what the failure modes look like in practice.

The agent does something nobody authorized. Your agent has access to an email tool for a narrow set of notification purposes. One session, reasoning through an unusual user request, it decides sending an email is the right move — and it is, from the model's perspective, given what it knows. No policy existed saying "only send emails under these conditions." The model's judgment was your policy. In most cases this is fine. In some cases it isn't. Governance lets you define the conditions explicitly; without it, you're relying on the model to stay inside lines you never drew.

Costs spiral before you know it's happening. An agent running a loop over an expensive tool call can generate a bill that bears no relationship to what you'd expect. Observability shows you this after the session ends. A per-session cost cap stops the session before it gets there. These aren't the same thing, and the difference shows up on your invoice. Teams frequently describe this as the moment governance moved from "nice to have" to "obviously necessary."

A tool changes after you approved it. In September 2025, a package called postmark-mcp built trust over fifteen versions of clean, functional behavior — then silently added a single line of code that BCC'd every outgoing email to an attacker-controlled address. Koi Security estimated roughly 300 organizations may have been exposed, though Postmark itself reported awareness of only one affected customer — and for approximately eight days, every metric said the integration was healthy. The tool responded normally. API calls succeeded. Nothing in the monitoring layer surfaced the change. That's the rug pull attack, and the MCPTox benchmark found that 5.5% of public MCP servers already contain tool poisoning vulnerabilities. Governance that validates tool definitions at execution time — not just at install — catches this. Observability catches the aftermath.

An audit asks what your agent was permitted to do. This one is underestimated until it isn't. When a compliance team, a customer, or a regulator asks whether your agent operated within defined policy during a specific period, the answer requires evidence of governance having existed — not just logs of what the agent did. A complete execution trace with no underlying policy record doesn't demonstrate governance. It demonstrates that you watched. These are different things to an auditor.


Isn't This What My System Prompt Is For?

This comes up a lot, and it's worth addressing directly.

System prompts can instruct an agent to stay within certain behaviors. They are not governance. A system prompt is a soft constraint — the model reads it and attempts to follow it, and in the overwhelming majority of cases it does. But "the model attempts to follow it" and "the behavior is enforced" are not the same guarantee. A sufficiently unusual input, a prompt injection via tool result, or simply a long context where earlier instructions get underweighted can all produce behavior outside what your system prompt intended. (OWASP ranks prompt injection as the #1 vulnerability in its Top 10 for LLM Applications.)

Governance enforces constraints at the execution layer, independent of what the model decides. It doesn't ask the model to comply. It prevents non-compliant executions from proceeding. That's a different mechanism, and it's the one that matters when the model's judgment fails — which it will, given enough sessions.


When Does This Actually Become Urgent?

Honestly? Earlier than most teams act on it.

The intuition is that governance is an enterprise concern — something you bolt on when you have a compliance team asking questions or a customer requiring it. That intuition gets teams into trouble because the failure modes above don't wait for scale. A single agent session can generate an unauthorized action, a cost spike, or a compromised tool result. The blast radius is bounded by session; the risk exists from session one.

The teams that retrofit governance after an incident consistently describe the same experience: the gap was obvious in hindsight, the work to close it was harder than it would have been upfront because they had to calibrate policies against live production traffic, and the incident that prompted it was more expensive than the governance implementation would have been.

The right time to add governance is when you're adding MCP tools, not after you've learned why you should have.


How Waxell handles this: Waxell's policy engine sits above the execution layer — tools are registered with explicit versioned identities, and execution validates against registered policy before each run, not just at install. Per-session cost caps halt runaway sessions before costs accrue. Tool result inspection scans every MCP response for injection patterns and schema anomalies before they enter agent context. And because Waxell's telemetry is immutable, you can produce a governance record — not just an execution log — for any session, any tool, any time period. Learn more → · Get started →


Frequently Asked Questions

Why do you need MCP governance?
Because observability and governance solve different problems. Observability tells you what your agent did — which tools it called, what they returned, what the session cost. Governance determines what your agent is permitted to do, enforced at runtime independent of the model's judgment. You can have complete observability and zero governance, and most MCP deployments do. The gap shows up when an agent takes an action nobody explicitly authorized, when a cost cap should have halted a session but didn't exist, or when an audit asks for evidence that agents operated within defined policy.

What's the difference between MCP observability and MCP governance?
Observability is the window; governance is the wall. Observability records what happened — tool calls, latency, errors, costs — and surfaces it for review. Governance enforces what's allowed to happen, before execution, independent of the model's reasoning. Catching a problem in your observability dashboard means the problem already occurred. A governance policy that blocks non-compliant execution means the problem doesn't occur.

Can't a system prompt handle this?
Not reliably. A system prompt instructs the model to stay within certain behaviors, and most of the time the model complies. But compliance is a soft guarantee: unusual inputs, prompt injection via tool result, or long contexts where earlier instructions get underweighted can all produce out-of-policy behavior. Governance enforces constraints at the execution layer, independently of what the model decides. It doesn't ask for compliance — it prevents non-compliant executions from proceeding.

When should you add MCP governance?
When you're connecting MCP tools, not after your first incident. The instinct to defer governance until scale or compliance pressure arrives is common and consistently leads to harder retrofitting work. Policies that would take an afternoon to define on a new deployment can take weeks to calibrate safely against live production traffic. The failure modes governance prevents — unauthorized actions, cost spirals, compromised tool definitions, compliance gaps — exist from the first production session.

Does MCP governance require rewriting my agent code?
No. Governance operates as a layer above your agents and tools — policies are defined in the governance plane and enforced at execution time without modifying agent logic or tool implementations. This is by design: governance that requires code changes to implement will be skipped when shipping pressure is high. A governance layer that sits independently of your code can be updated, tightened, or audited without touching the agents it governs.

Top comments (0)