Building KubeTokenWatch

#signoz #wemakedevs #ai #agents

I built an AI agent that debugs my Kubernetes cluster — and traces its own bill

As a devops learner we see lots of error in the cluster when we deploy any kubernetes object . A pod is stuck. kubectl get pods shows a status you don't fully trust. We start by running describe,Check logs, Try to connect pieces together. It works, eventually. It's just slow, and it happens at the worst possible time.

Here's the second scenario: you've built an AI agent that gonna find out the issues for you and listed them down one by one. It works great, until the bill arrives, and you realize you have no idea which request, which user, or which loop actually caused it.

I built KubeTokenWatch to fix both problems with one agent. It looks at your cluster, tells you what's broken and how to fix it that you can follow to resolve easily. Isn't it save you extra hour?

What it actually does

You point it at a namespace. It scans for anything unhealthy, pods, deployments, statefulsets, daemonsets, jobs, PVCs, and give you the overview what causes these with list of improvement.

The clever part isn't the LLM call, it's what the agent looks at before making it. Kubernetes has 30-plus built-in resource kinds, plus an unbounded number of custom resources. I could have written a check for each one. I didn't, because that list never actually ends, and it breaks the moment someone's cluster has a CRD I never heard of.

Since it is an ai agent. It will handed you a big bill that you can scared of. So I also implemented the cost tracer that you will find it in the signoz dashboard. Oh I just need to let you know that I instrumented this ai agent with opentelemetry so that I can trace the budget along with related metrics and logs.

Here singoz plays a big role. Previously what we need to do? We have to instrument our application using opentelemetry and then need separate backend along with visualization tools like grafana. Now singoz give you all at once. You don't need another backend and then another dashboard. You can just use signoz. So why any delay? Just follow along with my github repo. You will do it.

Everything, setup, usage, project structure, lives in the repo:

👉 github.com/iamridoydey/KubeTokenWatch

Architecture

CLI (cli.py)  ─┐
               ├──▶  agent.py (shared core)
API (app.py)  ─┘            │
                             ▼
              Groq LLM decides to call get_failed_resources
                             │
                             ▼
         Kubernetes API (via active kubeconfig — any cluster type)
                             │
                             ▼
         Real object status + events returned to the LLM
                             │
                             ▼
         LLM writes diagnosis + fix list ── OTel span: model, user, tokens, cost
                             │
                             ▼
              OTel Collector (part of self-hosted SigNoz)
                             │
                             ▼
              SigNoz — dashboards + budget-spike alert

cli.py and app.py are thin wrappers — all the real logic (calling the LLM, running the Kubernetes tool, tracing cost) lives once in agent.py.

Setup

1. Prerequisites

Python 3.10+
A running self-hosted SigNoz instance (OTel collector reachable at localhost:4317)
A free Groq API key from console.groq.com — no credit card required
Access to a Kubernetes cluster via a working kubeconfig (kubectl get pods should already work — kind, minikube, or any cloud cluster all work identically)

2. Install dependencies

pip install -r requirements.txt

3. Configure environment

cp .env.example .env
# edit .env and add your GROQ_API_KEY
export $(cat .env | xargs)

SigNoz deployment

This repo includes casting.yaml and casting.yaml.lock, the Foundry configuration used to deploy SigNoz for this project. To reproduce the exact same self-hosted SigNoz setup on another machine:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

Usage

CLI

python cli.py diagnose --namespace default
python cli.py diagnose --namespace kube-system --user-id alice

HTTP API

python app.py

curl -X POST http://localhost:5000/diagnose \
  -H "Content-Type: application/json" \
  -d '{"user_id": "u1", "namespace": "default"}'

View traces in SigNoz

Open your SigNoz dashboard (typically http://localhost:8080)

Let's build A pannel that will give you the idea how easy it is to use signoz as your observability platform

On the signoz left bar click on dashboards

Click on `new dashboard` and then click on `create dashboard`

Set the dasboard title (ex: KubeTokenWatch Cost Tracer)

Click on `new pannel` and select `time series`

In the input write filter and aggregation type as per image below

Give this pannel a name and Save it.

Now create an alert based on threshold you define.

Click on three dot to edit panel that you created now. Scroll to the right of your panel and you will see alert option. Click on that. Then set the threshold to 0.0005. After that you will see send to. You can configure it through documentation. Easy enough.