Building a read-only Kubernetes troubleshooting CLI in Go

Iman Tabatabaei — Mon, 20 Jul 2026 06:55:37 +0000

Every Kubernetes incident I've handled starts the same way: kubectl get pods, spot the sad one, kubectl describe, scroll the events, check the node, guess. It works, but it's slow, and it's the same detective work every single time. I wanted a command that does that sweep for me and just tells me why something is broken. That turned into kubeagent.

kubeagent scan reads a whole cluster and, for each unhealthy pod, node, or Service, names the root cause — CrashLoopBackOff, ImagePullBackOff, OOMKilled, Pending/Unschedulable, Multi-Attach volume errors, failing probes, stale kubelet leases, Services with zero ready endpoints, and more. It's read-only, works offline, and ships as a single Go binary.

Three design decisions are worth walking through.

1. Deterministic first, AI optional

It would have been easy to pipe everything to an LLM and call it a day. I didn't want that. The core is deterministic: each failure mode is detected by explicit rules against the cluster state, so it works offline, gives the same answer every time, and I can unit-test it.

There's an optional --explain flag that makes a single Claude API call to summarize the findings in plain English — but it only ever sees a structured summary (namespace, kind, status, restart counts, the detector's finding), never raw pod specs, env vars, or secrets. The tool is fully useful with no API key at all.

2. Read-only by default (the guardrails)

A troubleshooting tool you're afraid to run in prod is useless. kubeagent issues only get/list/watch calls by default — nothing else.

There's an opt-in --fix for a couple of reversible remediations (roll back a Deployment stuck on a bad image; uncordon an accidentally-cordoned node), but every write is guard-railed: a fixed allowlist of actions, never in kube-system, each behind a [y/N] confirm, preconditions re-checked against live state, and the result re-verified afterward. Crucially, no LLM ever decides a write — the remediations are hardcoded and deterministic.

3. Detectors as pure functions

This is the part I'm happiest with as a Go design. Each detector is a pure function: it takes a snapshot of cluster objects (pods, nodes, events…) and returns findings. No I/O, no client calls inside them. That buys a lot:

They unit-test against fake objects — construct a pod struct in a CrashLoopBackOff state, assert the detector flags it. No cluster, no mock-of-a-mock gymnastics.
Adding a new failure mode is cheap and isolated: write the failing test, write the function, done.
The I/O layer that actually lists objects is separated out and tested against client-go's fake clientset.

I built the whole thing TDD — failing test first — and this pure-function shape is what made that pleasant instead of painful.

The Go stack

Built directly on client-go, the same library kubectl and operators use. No framework.
The CLI uses the standard-library flag package — no Cobra. For a focused tool it's plenty.
Single static binary (CGO_ENABLED=0); install with go install github.com/imantaba/kubeagent@latest.
There's also a watch daemon that runs in-cluster: informers + a heartbeat ticker + an HTTP server, re-running the same deterministic diagnosis on change and exposing findings as Prometheus metrics — with hand-rolled metric output and zero extra dependencies (informers ship with client-go).

Try it

go install github.com/imantaba/kubeagent@latest
kubeagent scan

It's MIT-licensed and read-only, so it's safe to point at any cluster you can already reach.

Repo: https://github.com/imantaba/kubeagent
Docs: https://k8sproject.top