The fastest way to make people nervous about AI agents is to start with:
Let the agent fix production.
For platform teams, a better starting point is:
Let the agent inspect production safely and help humans make better decisions faster.
That is the read-only-first model.
Why read-only first works
Most incidents do not begin with a fix. They begin with questions:
- What changed?
- Which pods are failing?
- Are events showing scheduling issues?
- Did a rollout just happen?
- Are logs showing one error or many?
- Is the issue isolated to one namespace or many?
- Is the service routing to ready endpoints?
These are inspection tasks.
An agent that answers them quickly is already useful, even if it cannot mutate anything.
Read-only is not useless
A read-only Kubernetes agent can:
- list pods, deployments, services, ingresses, gateways, and routes
- summarize events
- inspect rollout state
- fetch logs
- compare resources across clusters
- identify missing endpoints
- explain likely failure modes
- generate a human-readable incident summary
That can reduce time-to-context without increasing blast radius.
The first adoption goal should be:
Can the agent help a human understand the system faster?
Not:
Can the agent autonomously remediate production?
Make read-only tools boring
Read-only tools should be boring in the best way:
- declarative
- versioned
- reviewed
- scoped by RBAC
- observable
- easy to disable
- attached explicitly to agents
Avoid giving the agent a giant toolbox and hoping the prompt keeps it safe.
Prefer explicit tool lists:
toolNames:
- k8s_get_resources
- k8s_describe_resource
- k8s_get_events
- k8s_get_logs
The smaller the first tool surface, the easier it is to reason about.
Add governance before mutation
Before adding write tools, answer these questions:
- Who can invoke the agent?
- Where can it be invoked from?
- Which clusters can it reach?
- Which namespaces are allowed?
- Which tools require approval?
- How are approvals logged?
- What counts as a high-risk request?
- Can the agent access secrets indirectly?
- How are failed tool calls surfaced?
If those answers are unclear, write access is premature.
A staged rollout model
I like this progression:
Stage 1: Read-only single-cluster inspection
Stage 2: Read-only multi-cluster inspection
Stage 3: ChatOps integration with approved channels
Stage 4: Approval-gated diagnostic exec
Stage 5: Approval-gated low-risk remediation
Stage 6: Pull-request based config fixes
Stage 7: Policy-backed automated remediation
Each stage earns the next one.
The platform should prove reliability, auditability, and user trust before gaining more power.
Read-only still needs guardrails
Read-only does not mean zero risk.
Logs can contain sensitive data. Resource names can reveal architecture. Events can expose internal details. Cross-cluster access can broaden visibility.
So even read-only agents need:
- authentication
- authorization
- audit logs
- namespace scoping
- rate limits
- clear data handling rules
- private or gateway-controlled model options
The difference is that read-only failures usually have a smaller blast radius than write failures.
When to introduce write tools
Add write tools when:
- users trust read-only responses
- tool traces are observable
- RBAC is scoped
- approval gates work
- risk classification is understandable
- the team has tested failure modes
Start with reversible operations:
- rollout restart
- scale within safe bounds
- patch a known annotation
- run a narrow diagnostic exec
Avoid starting with:
- delete arbitrary resources
- apply arbitrary manifests
- broad shell execution
- secret inspection
- cluster-wide mutation
The best first demo
The best first demo is not deleting a pod.
A better demo:
- Ask the agent why a deployment is unhealthy.
- Agent gathers pods, events, rollout, and logs.
- Agent summarizes the likely cause.
- Agent proposes a safe next step.
- Human decides.
That demonstrates value without pretending the system is ready for autonomy.
Final thought
Read-only-first is not a timid approach. It is a platform engineering approach.
It lets teams introduce agentic workflows while preserving the trust model that production systems need.
The goal is not to make the agent powerless.
The goal is to make each new capability earned, observable, and reversible.
Top comments (0)