Guardrails Before Write Access: Building Agentic Kubernetes Operations with Human Approval

Agentic operations for Kubernetes sound exciting until the agent asks to run a production command.

That is the moment the architecture stops being a demo and starts becoming a platform problem.

It is not enough to connect an LLM to Kubernetes tools. A real platform needs boundaries:

• What can the agent read?
• What can it change?
• Who approves risky actions?
• Can the agent retrieve secrets?
• How do we stop a chat interface from becoming a cluster-admin shortcut?
• How do we prove the request actually went through the intended path?

This post walks through a practical pattern for building a guarded Kubernetes agent workflow using chat-based invocation, MCP tools, risk classification, RBAC, and human approval.

The examples are intentionally anonymized and use generic names, but the lessons come from real implementation work.

The platform shape

The architecture had three layers:

Chat surface
  |
  | user asks for inspection/remediation
  v
Agent runtime
  |
  | calls approved tools through MCP
  v
Kubernetes tool server
  |
  | uses a scoped Kubernetes service account
  v
Application cluster

The chat surface can be Slack, Microsoft Teams, or another collaboration tool. The important part is that the agent is invoked where incident response already happens.

The agent runtime is responsible for interpreting the request, selecting tools, summarizing results, and asking for approval when needed.

The Kubernetes tool server is the dangerous part. It is where natural language becomes Kubernetes API access, so it must be designed as a platform boundary rather than a convenience script.

Start with read-only operations

The safest starting point is read-only inspection:

• list pods
• describe deployments
• read events
• fetch logs
• inspect services, ingresses, gateways, and routes
• compare resource state across namespaces or clusters

This already supports many incident workflows.

During an incident, a useful agent does not need to delete a pod immediately. It first needs to gather context quickly:

• What changed?
• Which pods are unhealthy?
• Are there recent restarts?
• Are events showing image pull errors, scheduling failures, or failed probes?
• Is the service routing to ready endpoints?
• Is this isolated to one namespace, one cluster, or many clusters?

Read-only tools let the team gain speed without handing the agent mutation rights on day one.

Add write tools behind approval

Eventually, teams will want remediation actions:

• restart a deployment
• scale a workload
• patch a resource
• execute a command in a container
• apply a manifest
• delete a stuck pod

These tools should not be exposed as ordinary chat commands. They should be protected tools with explicit approval.

A good flow looks like this:

User:
  Run a diagnostic command in this pod.

Agent:
  Risk: Yellow
  Operation: pod exec
  Target: namespace/app-pod
  Reason: exec is read-only in intent, but the target namespace is sensitive.

Approval gate:
  Approve / Reject

Agent:
  Runs the tool only after approval and returns the result.

The agent should explain the risk before the tool runs, but the actual enforcement should live in the platform approval mechanism.

That distinction matters. A model politely asking "Do you approve?" is not the same as an approval gate that pauses the tool call.

Use simple risk classes

You do not need a perfect policy engine on the first day. A simple risk model is already useful:

Green   Low-risk read-only inspection
Yellow  Sensitive read-only action or limited operational change
Red     Destructive, secret-adjacent, broad, or hard-to-reverse action

Example classification:

Request	Risk	Why
List pods in a namespace	Green	Read-only inspection
Get logs from an application pod	Green/Yellow	Depends on log sensitivity
Exec uname -a in a pod	Yellow	Exec is powerful even if command is harmless
Scale a deployment	Yellow	Mutates live workload capacity
Delete a pod	Red or Yellow	Depends on namespace, controller, and blast radius
Print environment variables	Red	Secret-adjacent
Read mounted service account token	Red	Secret exposure

The goal is not to make the agent "smart" in a vague way. The goal is to make risk visible before execution.

Treat exec as secret-adjacent

One of the most important lessons: blocking Kubernetes Secret API access is not enough.

A service account may have:

get/list/watch secrets: no
create pods/exec: yes

At first glance, that sounds safe. The agent cannot call kubectl get secret.

But exec can still enter a running container. From inside that container, a user might read:

• mounted secret files
• service account tokens
• environment variables
• app config files
• credentials written to disk

So the real security posture is:

Kubernetes Secret API access: blocked
Secret exposure through pod runtime/exec: still possible

That does not mean exec must be forbidden everywhere. It means exec deserves special handling:

• require approval
• classify suspicious commands as Red
• block obvious secret-reading patterns
• scope RBAC carefully
• prefer diagnostic commands with predictable output
• avoid exec into sensitive namespaces unless there is a clear reason

Good guardrails understand that Kubernetes permissions and runtime access are related but not identical.

RBAC is part of the product experience

When an agent fails to execute a command, the model may guess why.

For example, it might say:

The operation may have been blocked by policy or GitOps drift prevention.

But the actual reason could simply be Kubernetes RBAC:

cannot patch resource "deployments/scale"

That distinction matters. If the agent guesses, users lose trust.

The platform should capture and surface the real tool error:

• RBAC denied
• resource not found
• container binary not found
• namespace blocked by tool guardrail
• admission denied
• network timeout
• invalid command arguments

The agent can summarize, but it should not invent the root cause when the tool already returned one.

Distroless containers change your demo plan

Another practical lesson: many containers do not have a shell.

Commands like these may fail:

sh -c "..."
bash
ps
curl

That does not mean exec is broken. It may just mean the target image is distroless or minimal.

For a reliable demo or diagnostic workflow:

• pick commands that exist in the target image
• do not assume a shell is available
• test against known-good pods
• treat "executable file not found" as a container/runtime fact, not an agent failure

This is mundane, but it saves a lot of confusion.

Chat permissions need their own boundary

If an agent is available from a collaboration tool, the chat surface also needs access control.

There are two separate layers:

1. Where the app can be installed or invoked.
2. Which teams, channels, or conversations the bot will actually serve.

The first layer is app configuration. The second layer is runtime enforcement.

For sensitive operational agents, runtime allowlists are useful:

ALLOWED_TEAM_IDS
ALLOWED_CHANNEL_IDS
ALLOWED_CONVERSATION_IDS

The agent should reject messages and approval actions from unapproved locations.

This matters because an approval button is also an operational action. It should not be usable from a random personal chat or copied into an uncontrolled place.

Thread context is subtle

Chat threads look obvious to humans, but bot frameworks may not automatically provide the full thread history to the bot.

If the agent is mentioned halfway through a long incident thread, it may only receive the message where it was tagged, not every previous reply.

That means the platform has to choose:

• require users to include relevant context in the prompt
• fetch thread history through the chat provider API
• summarize thread history before passing it to the agent
• map each chat thread to a separate agent context

For incident response, the safest model is:

one chat thread = one agent context

That prevents unrelated incidents from accidentally sharing memory.

What I would build first

For an early production version, I would keep the first iteration intentionally conservative:

• read-only Kubernetes tools enabled by default
• write tools disabled until RBAC and approvals are ready
• exec treated as approval-required
• secret-looking commands classified as Red
• chat channel allowlists enabled
• one chat thread mapped to one agent context
• real Kubernetes errors surfaced clearly
• all tool calls logged with actor, target, risk, and decision

Then I would add automation gradually:

1. Read-only triage.
2. Approved exec for diagnostic commands.
3. Approved rollout restart or scale operations.
4. Policy-backed remediation.
5. Pull-request based fixes for configuration changes.

Final thought

Agentic Kubernetes operations should not start with "let the AI fix production."

They should start with:

Can we help humans gather context faster, classify risk clearly, and execute approved actions through a controlled platform path?

That framing is less flashy, but it is much closer to how platform teams can actually adopt agentic systems.

The interesting work is not just connecting an LLM to Kubernetes.

The interesting work is building the boundaries around it.