AI Agent Capability Diffs: Review New Powers Before They Reach Production

An agent can gain a dangerous new power in a change that looks harmless in a code review. A new MCP server, a broader OAuth scope, or one extra tool field may let it read a customer record and send it somewhere else. The diff says “add connector.” It does not say “new data-exfiltration path.”

That is the gap an AI agent capability diff closes. Instead of reviewing only source files, record the operational powers available to each agent, compare the new snapshot with the approved one, and require a deliberate decision for meaningful additions. This is not a replacement for IAM, runtime policy, or human approval. It is the release-time evidence that tells those controls where the risk changed.

This guide builds a small, vendor-neutral pattern you can use with MCP servers, function tools, SDK-based agents, and internal APIs.

Why ordinary code review misses agent authority

Traditional reviews are good at showing changed lines. They are weaker at showing the effect of a change spread across a tool manifest, an OAuth configuration, a prompt package, and a deployment environment.

Consider this change:

{
  "name": "search_customer_notes",
  "endpoint": "/notes/search",
  "scopes": ["notes.read"]
}

It looks read-only. But if the same agent already has post_webhook, an untrusted document can steer it to copy retrieved notes into an external request. Neither tool alone explains the risky path. A capability diff gives reviewers a compact statement: this release adds access to sensitive notes and the agent already has external egress.

The idea works because it asks a stable question on every release:

What can this agent do now that it could not do in the last approved release?

That question is more useful than “did the tool list change?” A renamed read tool may not change authority. A new parameter that broadens a filesystem path can. So the unit of review is a normalized capability, not a tool’s friendly description.

Model capabilities as facts, not prose

Start with a manifest generated from source-controlled tool definitions and deployment configuration. Do not ask an LLM to infer permissions from descriptions at review time. Descriptions are helpful context, but enforcement evidence should be deterministic.

Here is a deliberately small TypeScript shape:

type Effect = "read" | "write" | "execute" | "network" | "unknown";
type Sensitivity = "public" | "internal" | "customer" | "secret" | "unknown";

type Capability = {
  agent: string;
  id: string;                 // stable: "crm.customer.write"
  effect: Effect;
  resource: string;           // "crm:customer" or "filesystem:/work/*"
  sensitivity: Sensitivity;
  environment: "dev" | "staging" | "production";
  boundary: "service-token" | "user-delegated" | "human-approved";
  source: string;             // tool/server/config location
};

type Snapshot = {
  version: 1;
  generatedAt: string;
  capabilities: Capability[];
};

Useful fields answer five review questions:

Field	Reviewer question
effect	Can it observe, change, run, or send?
resource	Which records, paths, endpoints, or infrastructure?
sensitivity	How harmful is unintended access?
boundary	Which identity or approval boundary protects it?
environment	Is the power present where real users and data exist?

Avoid a single “risk score” as the source of truth. Scores can prioritize attention, but a reviewer needs the underlying facts. filesystem.write on /tmp/agent and filesystem.write on a production deployment manifest are not equivalent.

Generate a conservative snapshot

Every adapter should emit explicit facts it knows. If a tool’s side effect or resource scope cannot be determined, emit unknown; never silently label it read-only.

function classifyTool(tool: {
  name: string;
  server: string;
  annotation?: { readOnly?: boolean; destructive?: boolean };
  scopes?: string[];
}): Capability[] {
  const effect: Effect = tool.annotation?.destructive
    ? "write"
    : tool.annotation?.readOnly
      ? "read"
      : "unknown";

  return [{
    agent: "support-triage",
    id: `${tool.server}.${tool.name}`,
    effect,
    resource: tool.scopes?.join(",") || "unknown",
    sensitivity: "unknown",
    environment: "production",
    boundary: "service-token",
    source: `mcp:${tool.server}/${tool.name}`
  }];
}

Your adapters may read MCP tool metadata, OpenAPI operations, cloud IAM policy, filesystem allowlists, or custom function definitions. Keep their output separate from policy. The collector says what it observed or declared; policy decides whether that state may ship.

Also save the raw discovery artifact and a hash of the normalized snapshot. When an adapter changes, you need to distinguish a real authority change from a classifier upgrade.

Diff the powers, not just tool names

Compare capability identity plus its meaningful fields. A tool with the same name but a broader resource selector is a modification, not a no-op.

const fingerprint = (c: Capability) =>
  [c.id, c.effect, c.resource, c.sensitivity, c.environment, c.boundary].join("|");

function addedCapabilities(before: Snapshot, after: Snapshot) {
  const old = new Set(before.capabilities.map(fingerprint));
  return after.capabilities.filter(c => !old.has(fingerprint(c)));
}

function classifyChange(c: Capability) {
  if (c.effect === "unknown" || c.resource === "unknown") return "review";
  if (c.environment !== "production") return "notice";
  if (["write", "execute", "network"].includes(c.effect)) return "block";
  if (c.sensitivity === "secret" || c.sensitivity === "customer") return "review";
  return "notice";
}

The policy above is intentionally strict about unknowns and production mutations. Tune it to your delivery process, but do not solve noisy findings by converting everything to a notice. First improve the classifier, add an expiry-bound exception, or narrow the resource field.

Detect risky combinations

The most useful findings involve two capabilities that are individually expected. Add simple relationship rules before reaching for complex graph analysis.

function has(caps: Capability[], effect: Effect, sensitivity?: Sensitivity) {
  return caps.some(c => c.effect === effect &&
    (!sensitivity || c.sensitivity === sensitivity));
}

function crossCapabilityFindings(caps: Capability[]) {
  const findings = [];
  if (has(caps, "read", "customer") && has(caps, "network")) {
    findings.push("Customer-data read access can combine with external network access");
  }
  if (has(caps, "execute") && has(caps, "read", "secret")) {
    findings.push("Command execution can combine with secret access");
  }
  return findings;
}

This does not prove exfiltration or compromise. It identifies a path that needs an owner, a resource boundary, and a runtime control. Make that distinction explicit in the CI output; security findings should be evidence, not drama.

Put the diff in the pull request

A useful check gives reviewers three things: additions, meaningful scope changes, and the decision that allowed them.

- run: npm run capabilities:generate -- --out current.json
- run: npm run capabilities:diff -- approved.json current.json --format markdown > capability-diff.md
- run: npm run capabilities:policy -- approved.json current.json

For every block or review change, require a compact approval record:

change: crm.customer.write
owner: customer-platform
reason: "Agent creates a draft case after a verified support request"
controls:
  - user-delegated token
  - schema validation
  - approval for production writes
rollback: disable tool registration
expires: 2027-02-01

The expiry matters. Permanent exceptions quietly become undocumented authority. When the date arrives, the same team must renew, narrow, or remove the capability.

Separate declared, observed, and allowed access

One snapshot is not enough in a mature system. Keep three views:

1. Declared: what manifests and configuration say the agent can use.
2. Allowed: what policy permits for this agent, environment, and release.
3. Observed: what runtime telemetry shows it actually tried or used.

These views uncover different failures. A declared-but-unallowed tool may mean a bad rollout. An observed-but-undeclared tool may mean shadow configuration. An allowed-but-never-used power is an opportunity to reduce reach.

Do not turn the release diff into a runtime authorization system. Keep runtime policy as the final decision point, because tool schemas and configuration can drift after a build. The diff makes that drift visible; it does not eliminate the need to deny unsafe calls when they occur.

Make scope changes visible to humans

The hardest bugs are often not new tool names. They are small expansions hidden inside an existing capability. A filesystem tool may change from one repository to a parent directory. A CRM query may add a tenant=* selector. A network client may accept arbitrary URLs instead of an allowlist. If the snapshot stores those selectors as a single opaque string, reviewers must mentally parse a noisy diff.

Normalize scopes where practical, then render a sentence beside the machine comparison:

MODIFIED  github.pull_request.write
resource  repo:acme/help-center  ->  repo:acme/*
decision  BLOCK: production write scope broadened
owner     developer-experience

This output does not need to be clever. It needs to be precise enough for someone unfamiliar with the implementation to decide whether the new reach is intentional. Keep stable IDs for correlation, but show the human meaning of a selector, environment, and boundary in the pull-request comment.

Resource scope also needs a vocabulary. Define how your system represents a single record, a tenant, a repository, a directory tree, an API host, and a wildcard. Without it, one adapter might emit all while another emits *, making semantically identical states appear different—or worse, making a broadening look harmless. Add contract fixtures for that vocabulary before you start enforcing diffs.

Be especially careful with credentials. A tool’s input schema might look restricted while its service token has account-wide access. The snapshot should record the effective credential boundary when it is known, and produce unknown when it is not. Reviewers should never confuse an agent’s intended scope with the access the downstream service will actually honor.

Roll out without blocking every release

Start in report-only mode for two or three releases. Measure unknown classifications, repeated additions, and reviewers’ response time. Then block only new production write, execute, and broad network capabilities. Add sensitive read paths next. This sequence creates space to repair adapters without normalizing exceptions.

Test the workflow like production code:

• A rename with identical authority should be clearly explained, not silently ignored.
• A path change from /workspace/project/* to /** must be a blocking scope expansion.
• Missing side-effect metadata must yield unknown and require review.
• A sensitive read plus new egress path must produce a combination finding.
• An expired exception must fail the release check.

Finally, make ownership visible. Every production capability needs a system owner, not just the person who added a tool. Reviewers can approve a change; owners maintain the answer when the tool changes six months later.

A release checklist

Before you ship an agent-tool change, confirm:

• The snapshot was generated from the same revision and deployment configuration being released.
• Every new capability has a stable ID, effect, resource scope, sensitivity, and environment.
• Unknown fields received a review rather than a friendly default.
• Production writes, execution, and external network paths have an explicit approval record.
• Compound sensitive-read-to-egress and secret-to-execution paths were reviewed.
• Runtime policy still enforces the final decision after deployment.
• The raw discovery evidence, normalized diff, decision, and expiration date are retained.

The payoff is modest but important: reviewers stop guessing whether an agent “has more access.” They can see what changed, why it exists, who owns it, and which control keeps it bounded.

FAQ

Is an AI agent capability diff the same as an IAM policy diff?

No. IAM is a crucial input, but an agent capability diff combines identity scope with tool effects, resource selectors, environment, and agent-to-tool wiring. It answers the operational question of what the agent can do through its complete tool surface.

Should every new read-only tool block a release?

Usually not. Read-only tools can still expose sensitive data, so classify their data scope and use review for customer or secret access. Block only where your risk policy calls for it; noisy checks are ignored checks.

How do I handle tools with incomplete MCP metadata?

Emit unknown for effect or scope and require review. Add an adapter override with an owner and tests rather than assuming the description is accurate. Unknown is a data-quality task, not proof of safety.

Can a capability diff replace runtime tool authorization?

No. It runs before release and detects changed authority. Runtime authorization evaluates the actual call, user, resource, and context after deployment. Use both layers.

What is the smallest useful first version?

Inventory production tools, normalize their effect and resource scope, store one approved JSON snapshot, and fail CI on newly added write, execute, or external-network powers. Add sensitive-data and relationship checks after the basic signal is trustworthy.

How often should approved capability snapshots be refreshed?

Generate them on every pull request and release candidate. Also run scheduled discovery against deployed environments so you can find drift introduced by configuration, dependencies, or manually connected servers.