Originally published on Naveed Ahmed's Engineering Blog.
DevOps teams have been automating infrastructure for years.
We have Terraform, Kubernetes, CI/CD pipelines, and dozens of monitoring tools.
Yet engineers still spend a surprising amount of time investigating repetitive alerts, reviewing configuration changes, and collecting context during incidents.
AI agents may help reduce that workload.
But giving an AI agent access to production infrastructure introduces a different problem: how do you make sure it acts safely?
I recently explored this question while writing about scaling DevOps in the age of AI.
Here are the most important architectural lessons.
1. From Manual Operations to AI-Assisted Engineering
Traditional automation works well when the steps are known.
AI can help with tasks where engineers first need to collect and interpret context.
For example, an AI-assisted incident workflow might:
- Collect Kubernetes Pod events
- Examine container restart reasons
- Query Prometheus metrics
- Compare recent deployments
- Summarize likely causes
- Recommend a relevant runbook
That can reduce investigation effort, provided the output is checked against actual system evidence.
The important distinction is that proposing a fix and safely executing a fix are separate responsibilities.
2. MCP Tools Instead of Unrestricted Shell Access
Model Context Protocol (MCP) provides a standardized way for AI applications to work with external tools.
For DevOps teams, that might mean exposing narrowly scoped diagnostic operations instead of a general-purpose production terminal.
Examples include:
get_pod_events(namespace, pod)
get_deployment_history(namespace, deployment)
query_prometheus(metric, time_range)
get_recent_application_errors(service)
However, MCP doesn't automatically enforce security.
The server implementing these tools must apply authentication, authorization, parameter validation, rate limits, and auditing.
A read-only Kubernetes service account is a better starting point than cluster-admin credentials.
3. Use Three Levels of Production Guardrails
Not every infrastructure action carries the same risk.
I prefer separating automation into three levels.
| Level | Example | Control |
|---|---|---|
| Read-only | Fetch Pod logs and metrics | Authenticated automation |
| Bounded change | Adjust replicas within approved limits | Deterministic policy and recovery checks |
| High-risk mutation | Delete data or alter IAM permissions | Explicit human approval |
These controls must be enforced outside the language model.
You can use tools such as Open Policy Agent to evaluate structured requests, but a policy decision must still be enforced by the execution service.
Even a Pod restart can cause problems, so production change automation needs workload-aware safeguards.
4. AI-Assisted Kubernetes Incident Triage
Imagine an application starts returning 5xx errors immediately after deployment.
A diagnostic agent could gather:
kubectl get pods -n production
kubectl describe deployment api -n production
kubectl rollout history deployment/api -n production
kubectl get events -n production --sort-by=.lastTimestamp
It could then correlate that information with application logs, traces, and recent code changes.
The result might be a concise incident briefing containing:
- What changed
- Which services are affected
- Relevant evidence
- Possible causes
- Suggested verification steps
That's useful even if the agent never executes a production mutation.
5. Platform Engineering Makes AI More Reliable
AI-generated infrastructure is only as useful as the standards around it.
Without reusable patterns, different services can end up with inconsistent security settings, readiness probes, resource limits, and monitoring.
An internal developer platform can provide approved templates, reusable modules, and controlled self-service workflows.
AI agents can work within those boundaries instead of generating a completely new infrastructure design for every request.
This is where platform engineering becomes important: it creates reliable building blocks for engineers and automation alike.
Read the Complete Guide
My full article covers:
- Agentic platform engineering
- MCP-based DevOps tools
- Three-tier production guardrails
- Autonomous Kubernetes incident triage
- Self-healing workflow risks
- Internal developer platforms
- Practical career development for DevOps engineers
Read Scaling DevOps in the Age of AI: Full Architecture Guide
Final Thoughts
AI may change how DevOps work gets done, but it doesn't remove the need for engineering judgment.
The opportunity is to automate repetitive investigation, create reliable self-service systems, and reserve human attention for decisions that truly need it.
Would you allow AI to execute production changes, or keep it restricted to diagnosis and recommendations?
Written by Naveed Ahmed — Lead DevOps & Platform Architect
Top comments (0)