DEV Community

Devam Parikh
Devam Parikh

Posted on

When the Agent Is Wrong: Surface Real Kubernetes Errors, Not Model Guesses

#ai #sre #kubernetes #devops

AI agents are good at producing explanations.

That is useful until the explanation is wrong.

In Kubernetes operations, a failed tool call often already contains the truth:

Error from server (Forbidden): cannot patch resource "deployments/scale"
Enter fullscreen mode Exit fullscreen mode

But if the agent summarizes too aggressively, it may say something like:

This may have been blocked by a policy engine or GitOps drift prevention.
Enter fullscreen mode Exit fullscreen mode

That might sound plausible. It might also be completely wrong.

For operational agents, tool output must be treated as evidence. The model's explanation is secondary.

The source of truth

When a Kubernetes tool fails, preserve the real error:

  • command or tool name
  • target cluster
  • namespace
  • resource
  • Kubernetes API error
  • stderr
  • exit code
  • admission response if present
  • RBAC denial if present

The agent can summarize this, but it should not replace it with a guess.

Good:

The scale operation failed because Kubernetes RBAC denied patch access to deployments/scale for the tool service account.
Enter fullscreen mode Exit fullscreen mode

Bad:

The scale operation likely failed because GitOps reverted it.
Enter fullscreen mode Exit fullscreen mode

Unless the tool output proves GitOps was involved, that is speculation.

Common Kubernetes failures agents misread

RBAC denial 

cannot patch resource "deployments/scale"
Enter fullscreen mode Exit fullscreen mode

This is not an application failure. It is not a model failure. It is not necessarily a policy failure.

It means the service account does not have the requested permission.

Missing binary in a container 

executable file not found in $PATH
Enter fullscreen mode Exit fullscreen mode

This often happens with distroless or minimal images.

The exec path may be working perfectly. The target container just does not have sh, bash, curl, or ps.

Namespace or tool guardrail

Some Kubernetes tools intentionally block sensitive namespaces.

If a tool refuses to operate in kube-system, that may be a built-in guardrail rather than an RBAC issue.

Resource not found 

deployments.apps "api" not found
Enter fullscreen mode Exit fullscreen mode

This might mean:

  • wrong namespace
  • wrong cluster
  • stale incident context
  • typo
  • resource renamed

The agent should verify before inventing a root cause.

Show evidence in the response

For incident workflows, I prefer this response shape:

Result: Failed
Operation: scale deployment
Target: namespace/app
Reason: RBAC denied
Evidence: service account cannot patch deployments/scale
Next step: grant scoped RBAC if this operation should be allowed
Enter fullscreen mode Exit fullscreen mode

This is much better than a paragraph of uncertain explanation.

It gives the human a clear path:

  • fix RBAC
  • choose a different action
  • reject the operation
  • escalate to someone with permissions

Distinguish diagnosis from speculation

Agents should use language carefully.

Evidence-backed:

The API server denied this request because...
Enter fullscreen mode Exit fullscreen mode

Speculative:

This could be related to...
Enter fullscreen mode Exit fullscreen mode

Speculation is allowed, but it must be labeled as speculation.

In production operations, confident wrong answers are worse than cautious incomplete answers.

Use tool traces for debugging the agent itself

When testing an operational agent, inspect the tool trace:

  • Did it choose the right tool?
  • Did it choose the right cluster?
  • Did it pass the right namespace?
  • Did it ask for approval when required?
  • Did it preserve stderr?
  • Did it summarize the actual error?

If the model explanation does not match the tool trace, fix the prompt, tool description, or response formatter.

Do not train users to trust polished guesses.

Improve prompts, but do not rely only on prompts

Prompts can help:

When a tool call fails, quote the concrete error category and do not invent causes that are not present in the tool output.
Enter fullscreen mode Exit fullscreen mode

But prompts are not enough.

The platform should structure tool results so the model receives:

  • status
  • error category
  • raw error
  • target metadata
  • suggested remediation

Structured evidence gives the agent less room to improvise.

Final thought

The best Kubernetes agents are not the ones that sound most confident.

They are the ones that preserve operational truth.

Let the model explain, summarize, and guide. But keep Kubernetes errors visible, structured, and auditable.

When the system fails, the human should be able to answer:

What exactly happened, and what evidence do we have?

Top comments (0)