DEV Community

Devam Parikh
Devam Parikh

Posted on

From Tool Discovery to Real Execution: Verifying a Multi-Cluster MCP Path

#kubernetes #mcp #sre #devops

Seeing tools in an agent UI is not the same as proving the platform works.

In a multi-cluster MCP setup, there are many layers between a user prompt and a Kubernetes API response:

Prompt
  -> agent
  -> MCP client
  -> gateway route
  -> federated backend
  -> remote MCP server
  -> Kubernetes API
  -> tool result
  -> agent response
Enter fullscreen mode Exit fullscreen mode

If any layer is misconfigured, the final experience may fail in confusing ways.

This post is a verification checklist for multi-cluster MCP paths.

Start with the backend tools

Before involving the agent, verify the application-cluster MCP server directly.

Questions to answer:

  • Is the MCP server running?
  • Is it reachable over the expected network path?
  • Can it list tools?
  • Can it execute a simple read-only tool?
  • Does its Kubernetes service account have the intended RBAC?

Do not start debugging the agent until the backend tool server works independently.

Verify the gateway backend

If Agentgateway or another gateway is federating MCP servers, verify the backend object next.

For each target:

  • is the host correct?
  • is the port correct?
  • is the MCP path correct?
  • is the protocol correct?
  • is the target name stable?

Target names often become tool prefixes.

If the target is named cluster-a, a backend tool like:

k8s_get_resources
Enter fullscreen mode Exit fullscreen mode

may appear to the agent as:

cluster-a_k8s_get_resources
Enter fullscreen mode Exit fullscreen mode

That prefix is part of the operator experience. Choose names humans can recognize.

Verify routing

Next, confirm the route that publishes the federated MCP endpoint.

Check:

  • the route is accepted
  • backend references are resolved
  • the path matches the URL configured in the agent
  • the gateway listener is attached
  • network policy or service mesh rules are not blocking traffic

This is where many failures hide. The backend may be healthy, but the route may point at the wrong path or namespace.

List tools from the federated endpoint

Now connect directly to the federated MCP endpoint and list tools.

You want to see tools from every expected cluster with the right prefixes:

cluster-a_k8s_get_resources
cluster-a_k8s_describe_resource
cluster-b_k8s_get_resources
cluster-b_k8s_describe_resource
Enter fullscreen mode Exit fullscreen mode

If tools are missing, the problem is likely gateway federation, backend reachability, or tool server discovery.

If tools are present, you have proven discovery, but not execution.

Execute a safe read-only tool

Pick a boring command:

list namespaces
list pods in a non-sensitive namespace
describe a known deployment
Enter fullscreen mode Exit fullscreen mode

The goal is not to show off. The goal is to prove:

  • the tool call reaches the correct cluster
  • RBAC allows the action
  • the response returns through the gateway
  • the MCP client receives structured output

Keep the first test read-only.

Attach the tool to the agent

Only after direct MCP verification should you attach the tools to the agent.

In the agent definition, explicitly list allowed tools:

toolNames:
  - cluster-a_k8s_get_resources
  - cluster-a_k8s_describe_resource
  - cluster-b_k8s_get_resources
  - cluster-b_k8s_describe_resource
Enter fullscreen mode Exit fullscreen mode

Explicit tool lists are easier to review than broad access.

They also make it obvious which clusters and actions the agent can use.

Test the agent with specific prompts

Avoid vague prompts at first.

Use prompts that force the intended route:

List pods in Cluster A in the default namespace.
Enter fullscreen mode Exit fullscreen mode 
Describe the frontend deployment in Cluster B.
Enter fullscreen mode Exit fullscreen mode 
Compare pod readiness between Cluster A and Cluster B for this namespace.
Enter fullscreen mode Exit fullscreen mode

The agent should select the correctly prefixed tool.

If it chooses the wrong cluster, fix the system prompt or tool descriptions before adding more tools.

Watch for false success

An agent can produce a confident answer even when the tool failed.

For verification, inspect the actual tool calls and tool outputs.

Look for:

  • wrong tool name
  • wrong namespace
  • RBAC denial
  • network timeout
  • empty result interpreted as success
  • model explanation that does not match tool stderr

The agent's natural-language answer is not enough. The tool trace is the source of truth.

Add write access last

Once read-only federation works, write tools can be added carefully:

  • separate service account
  • smaller RBAC surface
  • explicit tool list
  • human approval
  • risk classification
  • audit logging
  • namespace restrictions

Do not mix early read-only validation with write-tool debugging. You want a clean baseline first.

Final checklist

Use this rollout order:

  1. Direct MCP server health.
  2. Direct read-only tool call.
  3. Gateway backend accepted.
  4. Route accepted.
  5. Federated endpoint lists tools.
  6. Federated endpoint executes a read-only tool.
  7. Agent discovers tools.
  8. Agent selects the correct prefixed tool.
  9. Agent response matches real tool output.
  10. Approval-gated write tools added later.

The important idea:

Tool discovery proves the menu exists. Real execution proves the kitchen works.

Top comments (0)