DEV Community

mohsen
mohsen

Posted on

I recorded my Kubernetes AI agent failing, on purpose

Most demos of AI-for-infrastructure tools show a clean run. I think that is exactly backwards, and I want to explain why by walking through the thirty seconds of my own demo that a normal product video would have cut.

The scene

A deployment is crash-looping. The agent has already diagnosed it. I ask it to restart the deployment.

It stops and asks for approval. I approve. The restart runs, and the command succeeds.

Then I ask the follow-up question: did the restart change anything?

No. The pods fail with the same error. A restart was never going to supply a missing environment variable. The agent re-reads the pods, re-reads the events, and names the same root cause it found at the very beginning: DATABASE_URL is not set, so the container exits 1.

The restart was a reasonable thing to try and the wrong thing to fix it, and the system is the one saying so.

Why that scene is the product

The worst failure for an incident-response tool is not being unhelpful. It is emitting something that looks like a real diagnosis and is not. At 3am a confident wrong answer costs more than no answer, because it sends a tired human down a wrong path with false authority.

So the design follows from that.

The approval gate is at the tool boundary, not in the prompt. A guardrail written into a system prompt is a suggestion, and models can be argued out of suggestions. Instead every mutating operation passes through one chokepoint that returns exactly three answers: do it, ask a human, or refuse. The model does not get a vote, so there is nothing to talk around.

Detectors are compiled predicates, not model calls. They are always on, they run on every observation, and they cost zero tokens. The LLM is only invoked once a detector actually fires. Watching your cluster is free; thinking about it is the expensive part, so it happens last.

Every decision is appended to a hash-chained log. A run can be replayed afterwards instead of remembered. If you are going to let software touch production, "what exactly did it do, and why" has to be answerable after the fact.

Roles are real. readonly, operator, admin and superadmin, enforced per key.

What it reads

kubectl for cluster state, Prometheus for metrics via PromQL, and Loki for logs via LogQL. It answers in plain English and quotes the evidence it actually read, so you can check it rather than trust it.

The video

Eight minutes, every terminal scene a verbatim recording against a live cluster. Nothing typed by hand, nothing reconstructed. The approval-gate segment starts at 2:27.

Try it

The v1 architecture is written up in the Journal of Grid Computing (https://doi.org/10.1007/s10723-026-09837-6), with a preprint at https://arxiv.org/abs/2509.02449. The repo is several generations past that now.

If you run Kubernetes in production, the thing I most want to hear is which failure you would throw at it that I have not handled.

Top comments (1)

Collapse
 
deanlee profile image
Dean Lee

The 3am cost function has an extreme convexity that clean demos work hard to hide.

In incident response, a false negative is linear. An agent saying it cannot diagnose the crash leaves the on-call engineer at baseline, spending the usual twenty minutes reading logs. A confident false positive carries unbounded downside. Once an automated suggestion convinces a sleep-deprived human to bounce a database or purge a cache, it injects noise directly into the triage state space and multiplies mean time to recovery.

Putting the approval gate at the tool boundary rather than inside the system prompt changes the financial structure of the tool. A prompt guardrail is an unhedged short on model compliance. You collect small convenience gains on normal runs while underwriting the tail risk that a subtle context shift bypasses the restriction. Enforcing the gate at the foreign function interface converts that short into a defined-loss contract. The model acts as an untrusted proposal generator, and the runtime enforces the boundary regardless of what the prompt argued.

Treating compiled predicates as the always-on filter also fixes the inference accounting. Running frontier tokens across an idle cluster to verify that steady-state metrics remain steady is pure burn. Compiling detectors into zero-marginal-cost predicates keeps the token bill bounded by real state changes rather than polling intervals.