I Built a Kubernetes Incident Manager That Explains Broken Pods With a Local AI Model

It's 3 AM. Your phone buzzes. A pod is in CrashLoopBackOff.

You open your laptop, run kubectl describe, scroll through events, pull the logs from the crashed container, and try to work out what actually went wrong. Ten minutes later you find it: a missing Secret. Or a memory limit. Or a database that isn't reachable.

AI is good at exactly this kind of reading. But most teams can't paste production logs into a hosted AI chat. Those logs are full of connection strings, tokens, customer emails and internal IPs.

So I built a small incident manager that runs inside the cluster and explains broken pods with an open-weight model running on my own machine. Nothing leaves the infrastructure, and it costs nothing per incident.

Code: github.com/farooqhussain82/incident-explainer

What it does

1. Every 15 seconds it lists the pods in the cluster.
2. When one breaks (crash loop, OOMKilled, image pull error, missing Secret or ConfigMap, failing readiness probe, unschedulable), it opens an incident.
3. It collects the evidence: a kubectl describe-style summary, events, and logs, including the logs of the previous crashed run.
4. It redacts secrets from that text.
5. It asks a local model in Ollama (Qwen 3 8B by default) what went wrong and what to check next.
6. When the pod is healthy for four checks in a row, the incident resolves itself.

There is a small web page for the incidents, and it can post new incidents to Slack, Teams or Mattermost through an incoming webhook.

The demo: four pods, four different failures

To test it, I made a throwaway kind cluster and deployed four pods that break in four common ways. All four were detected and explained on my laptop (Apple M4, 16 GB RAM) in 20 to 45 seconds each.

Pod	What I broke	What the model said
payments-api	Refers to a Secret that doesn't exist	The pod can't start because Secret payments-secrets is missing. Check it exists in the namespace.
report-worker	Memory limit of 64Mi, process uses far more	OOMKilled, exit code 137, because usage goes over the 64Mi limit.
checkout-web	App can't reach its database	Connection refused to PostgreSQL on port 5432. Check the database is running and reachable.
orders-db	Postgres started without a password	Postgres won't initialise because POSTGRES_PASSWORD is not set.

Each answer quotes the exact log or event lines it is based on, so you can check it in seconds instead of trusting it blindly:

Then I created the missing Secret for payments-api. About a minute later the incident moved to "Recently resolved" on its own.

One detail I'm happy with: an out-of-memory container also shows CrashLoopBackOff in Kubernetes, because it keeps restarting. The detector looks at the last termination reason and reports it as OOMKilled, which is the real cause and the thing you need to fix.

Why an open, local model

This was the main reason for the project, not an extra.

• The data stays inside. The model runs next to the cluster. Logs are never sent to a third party.
• No bill per incident. A crash loop that fires 200 times a night doesn't turn into an API invoice.
• You choose the model. Qwen 3 8B runs fine on a laptop. Use a bigger one on a GPU node, or one tuned on your own runbooks. It's one environment variable.

I also made the model answer in a fixed JSON format (summary, likely cause, evidence, next steps, confidence). That keeps the output predictable enough to show in a UI and send to chat.

Redaction, even though the model is local

"Local" should not mean "careless". Before anything reaches the model, the text is cleaned:

• private keys, AWS keys, JWTs and bearer tokens
• passwords in URLs like postgres://user:password@host
• values of password=, token=, api_key= and similar
• email addresses and single IP addresses

In the checkout-web demo, the log contained a database URL with a password, a JWT and IP addresses. The model only saw this:

connecting to postgres://[REDACTED-URL-CREDS]@[REDACTED-IP]:5432/checkout
Authorization: Bearer [REDACTED-JWT]

Each incident page has a "Show exactly what the model saw" section, so anyone can check.

One thing I had to fix during testing: my first version also masked 0.0.0.0/0 in an infrastructure change. That hid the most important fact, which was a database being opened to the whole internet. Now CIDR ranges and 0.0.0.0 are kept, and only single host IPs are masked.

Security of the thing itself

A tool that reads every pod in your cluster has to be boring and safe:

• Read-only RBAC: only get, list and watch on pods, pod logs and events. I checked it with kubectl auth can-i: it cannot read Secrets and cannot delete pods.
• Env var values are never collected, only their names and where they come from (for example, "comes from Secret payments-secrets").
• A 13 MB FROM scratch image: no shell, no package manager, runs as non-root with a read-only filesystem and all Linux capabilities dropped.

It's written in Go with only the standard library, so there are no third-party packages to keep patched.

Honest limits

• It runs as one replica and stores incidents in a JSON file. Fine for a team cluster, not for a huge fleet.
• It explains, it does not fix. Any action is still a human decision.
• An 8B model gets the common failures right, but small details can be off. In one run it said the OOM test "writes to /dev/zero", when tail /dev/zero actually reads from it. The quoted evidence makes slips like that easy to spot.
• Redaction is pattern based. It catches the common shapes of secrets, not every possible one. There's a -dry-run mode that prints exactly what would be sent.

Try it

You need Docker, kind, kubectl, Go and Ollama.

ollama pull qwen3:8b
git clone https://github.com/farooqhussain82/incident-explainer
cd incident-explainer
make kind-up    # local cluster
make deploy     # build the image, load it, apply the manifests
make demo       # four broken pods
make ui         # open http://localhost:8080

It also works without a cluster, on a single log file or an infrastructure change plan:

kubectl logs my-pod --previous | ./bin/incident-explainer explain -

What I'd add next

• A Helm chart and an option to run Ollama inside the cluster.
• Grouping, so ten replicas of one broken Deployment become one incident, not ten.
• Using node events and recent rollouts as extra evidence.

If you run Kubernetes and you've ever read crash loop logs at 3 AM, I'd love to hear what failures you'd want this to understand. And if you try it, tell me what it got wrong.