Kubernetes incidents move fast - a pod crash loop at 2am doesn't wait for a human to read five dashboards. An autonomous SRE agent can triage, diagnose, and propose fixes while keeping a human in the approval loop for anything destructive.
The Core Pattern: Investigate First, Act With Permission
The safe design uses a two-phase loop. The agent gets read-only tools for observation - kubectl describe, log fetching, metrics queries - and a separate, gated set of write tools for remediation like scaling deployments or restarting pods. Every write action pauses and waits for explicit human approval before execution.
This maps cleanly onto LangGraph's interrupt mechanism, which lets a node in the agent graph pause execution, surface a structured decision to a human reviewer, and resume only on confirmation. The agent can reason over what it finds and generate a targeted fix rather than simply running a fixed runbook - but the blast radius is controlled by that approval gate.
Tracing is strongly recommended for this kind of agent. Without full visibility into which tool calls the agent made and what it decided at each step, debugging a bad remediation in production is nearly impossible. LangSmith (LangChain's observability layer) or any equivalent tracing setup captures the full execution trace so you can replay what happened.
Real Example
Here's a simplified LangGraph node that fetches pod status and flags for human review before restarting:
from langgraph.graph import StateGraph
from langgraph.checkpoint.memory import MemorySaver
def investigate(state):
pod_status = kubectl_describe(state["pod_name"]) # read-only
state["diagnosis"] = parse_crash_reason(pod_status)
return state
def request_approval(state):
# interrupt() pauses here; human sees diagnosis + proposed action
human_decision = interrupt({
"diagnosis": state["diagnosis"],
"proposed_action": f"Restart pod {state['pod_name']}"
})
state["approved"] = human_decision["approved"]
return state
def remediate(state):
if state["approved"]:
kubectl_rollout_restart(state["pod_name"]) # write action
return state
graph = StateGraph(dict)
graph.add_node("investigate", investigate)
graph.add_node("request_approval", request_approval)
graph.add_node("remediate", remediate)
graph.set_entry_point("investigate")
graph.add_edge("investigate", "request_approval")
graph.add_edge("request_approval", "remediate")
The interrupt() call is what separates an autonomous agent from a fully automated one - it lets the agent do the cognitive heavy lifting while keeping a human accountable for state changes in production.
Key Takeaways
- Separate read tools from write tools in your agent's toolkit - investigation should never require destructive permissions
- Use an interrupt/approval gate before any action that modifies cluster state, not just the riskiest ones
- Full execution tracing isn't optional for production agents; you need to reconstruct exactly what the agent reasoned and did
Have you found a reliable way to set the right granularity for approval gates - per action, per incident, or based on severity thresholds?
Sources referenced: LangChain Blog - "How we built an autonomous SRE agent for Kubernetes"
Top comments (0)