DEV Community

Muhtalip Dede
Muhtalip Dede

Posted on

Break a kind cluster on purpose, then watch an Observe agent

#kubernetes #ai #devops #sre

One command from kprompt-examples: kind up, seven failure scenarios, verify they actually broke, then run the Observe agent offline in heuristic mode — $0, no silent auto-heal.

Originally published at https://kprompt.ai/blog/observe-agent-kind-demo.

v0.5 shipped the optional Observe agent: always-on watch → correlated Incident → gated Slack/webhook, with Autopilot still propose-only. The missing piece for a live walkthrough was a payments namespace that actually misbehaves — not a slide claiming CrashLoopBackOff.

kprompt-examples is that fixture set. Break the cluster on purpose, then watch the agent react without spending an LLM token.

One command

Needs Docker, kind, kubectl, and kprompt with the agent subcommand. Heuristic mode — no API key, no spend. DEMO_SECONDS=60 stretches the agent window for recordings.

git clone https://github.com/kprompt/kprompt-examples.git
cd kprompt-examples
make walkthrough   # up → break-all → verify → agent-full (~45s)
Enter fullscreen mode Exit fullscreen mode

Or step by step when you want to film each failure:

make up
make break SCENARIO=01-crashloop
make verify
kprompt agent run -n payments \
  --emit-initial --analyze --fetch-logs --health --heuristic \
  --memory --patterns --autopilot-propose
Enter fullscreen mode Exit fullscreen mode

--emit-initial matters: a live watch stays quiet until new Pod/Event traffic. Already-broken workloads may not re-emit until the next BackOff. Emitting current state first makes demos and CI deterministic.

What you should see

Seven scenarios in one namespace, with a healthy baseline so the health score has something Ready to weigh against:

Scenario Rough signal
CrashLoop Restarting container / BackOff
ImagePull ImagePullBackOff / ErrImagePull
OOM OOMKilled / Exit 137
Stalled rollout Progress deadline / unavailable replicas
Unbound PVC Pending Pod waiting on volume
Failing CronJob Job failures / backoff
Missing Redis hostname DNS / connection errors to a stub Service name

Expect:

  • Health score moves while baseline web stays Ready
  • Correlated Incidents and gated alerts — not one Slack message per kubelet Event
  • With --autopilot-propose: a propose-only rollback suggestion on the stalled checkout rollout (Applied stays false)

That last point is the product claim: Observe never applies, patches, or deletes by default (ADR-0013). Autopilot emits PlanResult-shaped proposals (ADR-0015); apply stays gated.

Why this is not “AI auto-heal”

Surface Job
Laptop CLI Reactive: NL → PlanResult → approve → apply
Observe agent Always-on: watch → Incident → severity/confidence gate → notify
Autopilot Opt-in propose-only; apply needs policy + explicit approve

K8sGPT-style tools are great at on-demand scan → explain. Observe is a different job: continuous watch with threaded alerts from live Events/Pods. Do not expect fleet-scanner or multi-agent-framework parity — we ship one kprompt-native pipeline.

Honest limits

  • Heuristic analysis is deterministic and offline — useful for demos and CI, not a substitute for a real BYOK LLM run
  • Redis/Postgres in the dependency scenario are busybox stubs for Service-name discovery, not databases
  • Full caveats live in the examples README
  • Experimental — prefer kind / non-prod first

Next

Try: kprompt.ai · GitHub · kprompt-examples
Muhtalip

Top comments (0)