Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
⚡ Byte Size Summary
- See why we shipped an OpenShift diagnostic MCP server as read-only by design, and the RBAC wall that made write access harder than it looks
- Walk through a real failed remediation test where an agent recommended a correct-looking fix built on stale, deprecated config — and what that failure mode actually is
- Get the maturity-gated approval architecture we designed for write access — and why it's still sitting on paper, not in production
The Story
We built mcp-sre-tools — an MCP server that exposes OpenShift and Kubernetes diagnostics to an LLM, wired into Claude Desktop and n8n, covering ARO, ROSA HCP, OSD-GCP, and generic clusters. Nine diagnostic tools: get_cluster_health, diagnose_crashloop, get_failing_pods, and others in that family. READ_ONLY_MODE is on by default, and there are no write tools in the codebase at all. That part shipped clean.
The friction started when we scoped what came next: a remediation mode, where the agent wouldn't just diagnose a broken deployment — it would patch it.
That's where the story stopped being a build story and became an organizational one.
The Problem
Platform engineers and developers landed on opposite sides of the same question almost immediately, and for reasons that turned out to be more substantial than the usual risk-aversion reflex.
Developers were comfortable trusting agent-proposed changes roughly the way they'd trust a colleague's pull request — read the diff, sanity-check it, merge it. Platform engineers pushed back hard, and their objection wasn't reflexive. It was specific: a PR from a colleague comes with inspectable reasoning. You can ask them why. An LLM's proposed patch doesn't carry that same trail — the "why" is buried in a forward pass, not a code review comment.
Business stakeholders, meanwhile, were worried about something simpler and more immediate: an autonomous agent breaking a critical application in production.
Three legitimate concerns, three different vocabularies for the same underlying question — how much do we trust a system whose reasoning we can't fully inspect, applied to infrastructure we can't afford to break?
Why Existing Approaches Fall Short
The instinct is to reach for RBAC and call it solved. Scope the agent's service account to a namespace, give it patch permissions on Deployments and nothing else, and let it operate inside a fence.
That fence has a hole in it. Meaningful remediation almost always eventually needs to touch Secrets or environment variables — a misconfigured database connection string, an expired credential reference, a missing env var causing a crash loop. The moment your remediation scope includes Secrets, "namespace-scoped RBAC" stops being clean sandboxing and starts being a much bigger trust surface than the phrase implies.
We didn't have a way around that with RBAC alone. So we fell back to a narrower, honest justification for read-only: even without write access, a diagnostic agent cuts human mean-time-to-resolution. It's a smaller value proposition than full self-healing, but it's a real one — and it's the one we could actually defend without hand-waving.
The Architecture
Diagram 1 — as built: the shipped, read-only MCP architecture.
The controls we designed the server to work with — note the repo intentionally ships without a default rbac.yaml, to stay adaptable across cluster types and org policies. Deployment teams are expected to write their own scoped ClusterRole/RoleBinding tailored to their access model; the sample below shows the shape we recommend, not a default that ships:
-
Namespace-scoped RBAC (recommended, not shipped) — bind the MCP server's service account to
Role/RoleBindingresources scoped per-namespace, not a cluster-wideClusterRole - Service-account-based access — no static kubeconfig or personal credentials in the agent's execution path
- NetworkPolicy egress/ingress restriction — the MCP server's pod network is fenced to only the cluster API and the LLM endpoint it needs to reach
- Logging and observability as non-functional requirements — every tool call is logged, not bolted on after the fact
# Recommended shape, not a shipped default — deployment teams write their own
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: mcp-sre-tools-reader
namespace: <target-namespace>
rules:
- apiGroups: [""]
resources: ["pods", "events"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch"]
Diagram 2 — proposed, never implemented: the maturity-gated write-access flow.
⚠️ This is a design artifact, not a shipped system. The kill switch and rollback automation shown below were never built.
┌─────────────────────┐
│ Anomaly detected │
└──────────┬───────────┘
│
▼
┌─────────────────────────────┐
│ Agent proposes RBAC-scoped │
│ remediation action │
└──────────┬───────────────────┘
│
▼
┌─────────────────────────────┐
│ Risk-based routing │
│ low risk high risk │
└──────┬───────────────┬───────┘
│ │
▼ ▼
┌─────────────┐ ┌─────────────────────┐
│ Light review │ │ Human approval gate │
└──────┬───────┘ └──────────┬───────────┘
│ │
└─────────┬──────────┘
▼
┌───────────────────────┐
│ Kill-switch check │ ◄── [NEVER BUILT]
└───────────┬─────────────┘
│ pass
▼
┌───────────────────────┐
│ Execute + full audit │
│ record │
└───────────┬─────────────┘
│
▼
┌───────────────────────┐
│ Pre-captured rollback │ ◄── [NEVER BUILT]
│ plan (ready before │
│ execution) │
└───────────┬─────────────┘
│
▼
┌───────────────────────┐
│ Outcome logged │
└───────────────────────┘
The flow we designed, on paper, for a future write-capable mode:
- Anomaly detected
- Agent proposes an RBAC-scoped remediation action
- Risk-based routing — low-risk actions get lighter review, high-risk actions route to human approval
- Kill-switch check before execution
- Execution, with a full audit record
- Pre-captured rollback plan available before the action runs, not written after
- Outcome logged
Two components in that list were never built: the kill switch and the rollback automation. They exist as a design, not as shipped capability. Worth saying plainly, because it's the honest state of the project, not a gap we're hiding.
How It Works: Step by Step
For the part that is live — the read-only diagnostic path:
- An alert or a manual query triggers the agent via Claude Desktop or an n8n workflow
- The agent calls one or more of the nine MCP tools (e.g.,
diagnose_crashloop) against the target cluster - The MCP server's service account, scoped by namespace RBAC, executes the read-only API calls
- Results return to the LLM, which synthesizes a diagnosis
- A human reads the diagnosis and decides what to do next — the agent stops there
For the part that stayed on paper — the proposed remediation path — see the seven-step flow above. It never advanced past step 2 in production; steps 3 through 7 are design artifacts.
Security and Operational Considerations
RBAC and least privilege. The repo ships without a default rbac.yaml by design, to stay adaptable across cluster types — deployment teams write their own scoped ClusterRole/RoleBinding for their environment. That's also where the write-mode proposal broke down: remediation that needs to touch Secrets can't stay inside a tidy read-only-style RBAC boundary no matter who authors it.
Secrets exposure. The live tool set never reads or writes Secret contents. The unbuilt remediation mode is exactly where that boundary would have been tested, and wasn't.
Blast radius. Contained by design in the shipped version — nine read-only tools can't mutate cluster state, full stop.
Rollback strategy. For the live tools: not applicable, nothing is mutated. For the proposed write mode: rollback was designed as a pre-captured plan, generated before execution rather than reconstructed after a failure — but this was never implemented, and the one real recovery event we had (below) had to be handled manually.
Auditability. Every diagnostic tool call is logged. The proposed write-mode flow adds a full audit record as one of its seven stages — again, unbuilt.
What Breaks at Scale
We ran a private test — outside the public repo — with write/patch/upgrade-capable variants of the agent, specifically to see whether the self-healing story held up. It didn't, and the way it failed is the most important finding in this whole project.
The agent correctly diagnosed a real problem. Then it recommended a fix built on fluentd-era OpenShift Logging configuration — the collector layer — against a cluster already running the current Vector-and-Loki stack. The recommendation looked plausible. It targeted a collector and CRD shape that the cluster had already moved past.
At close to the same time, in the same test, the agent made a second, unrelated bad call: a recommendation to change the cluster's node-level hardware settings via the worker MachineSet. That recommendation, applied, provisioned the wrong instance type.
Two independent, unrelated bad recommendations landing at the same time — one on the logging stack, one on node hardware — made it look, in the moment, like a single cascading failure. It wasn't. Untangling that during recovery took longer than fixing either problem alone would have, precisely because the two failures got conflated. Recovery meant two separate fixes, not one clean GitOps revert: an updated ClusterLogging CR to throttle ingress at the collector layer and redirect output pipelines directly to the Loki backend, and a reconciliation of the worker MachineSet back to its approved providerSpec baseline — scaling down the misconfigured nodes and letting the Machine Operator provision replacements.
The label that matters here isn't "hallucination" in the dramatic sense people usually mean. The logging recommendation wasn't nonsense — it was training-data staleness masquerading as competence: correct-sounding reasoning built on a collector and config shape the cluster had already moved past. That's a more insidious failure mode than the black-box framing usually implies, because the output looks exactly as confident whether the underlying knowledge is current or stale.
The instance-type recommendation was a different kind of mistake — not stale knowledge, just a bad call. What made the incident harder to diagnose wasn't either failure alone, but the two landing together and looking, briefly, like one problem instead of two.
Scaling this up doesn't just mean more permissions — though it does mean that, and the platform team's original objection was concrete and correct on that point alone. It also means the bottleneck moves. Once you gate every write action behind human approval, the approval step itself becomes the constraint, and approval fatigue doesn't scale linearly with the number of things an agent proposes.
Quick Recap
- Read-only shipped, write-mode didn't — and the RBAC wall around Secrets access is a real reason, not a cautious excuse
- Two unrelated bad calls can look like one big failure — a stale-knowledge fix and a separate bad hardware recommendation landed together and were harder to untangle than either would've been alone
- Governance architecture is easy to design and hard to ship — the kill switch and rollback automation are still just a diagram
GitHub Repo
Full implementation: mcp-sre-tools
What's Next?
The open question this leaves for the series: what does a safe path to write access actually look like, given that the blocker wasn't RBAC mechanics but the reliability of the reasoning behind each proposed action? That's worth its own piece.
Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
Top comments (0)