An AI Agent Found the Outage. I Still Wouldn't Give It kubectl.

Shaurya — Sun, 26 Jul 2026 16:13:13 +0000

An AI Agent Found the Outage. I Still Wouldn't Give It `kubectl`.

A bad payment release was breaking checkout. The incident agent found the
failed requests, connected them to the release, and proposed the right fix: roll
back payment-service.

I still did not want the agent to have a Kubernetes credential.

That tension became MANDATE, the project I built for the Agents of SigNoz
hackathon. It is an authority layer between an AI agent and the tools that can
change a real system. SigNoz is not just where I send traces after the fact. It
detects the incident, supplies bounded evidence for the decision, records the
approval and execution path, and verifies whether the system actually
recovered.

The missing part of agent observability

Normal agent tracing can answer useful questions: Which model ran? Which tool
did it call? How long did it take? How many tokens did it use?

For an action-taking agent, I needed another set of answers:

Who gave this agent authority?
Which tools and resources were in scope?
What live evidence justified the action?
Did policy allow, deny, or pause it for approval?
What exact arguments did the operator approve?
Did the action work in the real system?

MANDATE keeps these decisions outside the model. An agent may propose a useful
next step, but a deterministic gateway derives identity, resource scope, risk,
budget, evidence requirements, and approval policy from a signed mandate and
operator-owned configuration.

The reference stack uses a Python/FastAPI gateway, a Next.js operator console,
Slack approvals, a Kind Kubernetes workload, OpenTelemetry, and a self-hosted
SigNoz deployment provisioned through Foundry. Agents connect through MCP or
REST, while Kubernetes credentials and SigNoz service-account keys remain on
the gateway side.

The incident I used to test it

The demo is deliberately small: checkout-service calls payment-service.
A controlled release changes the payment deployment to a regressed image and
generates failed checkouts.

One native SigNoz metric alert watches the five-minute checkout failure rate:

name: checkout SLO breach
condition: >
  100 * sum(rate({"checkout_requests_failed_total"}[5m]))
  / clamp_min(sum(rate({"checkout_requests_total"}[5m])), 1) > 5
severity: critical
channels: [automation]

A second, trace-based rule detects a span named mandate.release.deploy with
mandate.release.regressed = true. This gives me both customer impact and a
release marker instead of asking the agent to guess from a red graph.

The MANDATE Incident Command dashboard combines the checkout error rate,
payment latency, failed-request traces, release markers, Kubernetes state, and
recovery evidence.

When either service/release alert fires, SigNoz sends a webhook to MANDATE. The
incident coordinator receives a short-lived root mandate, then delegates
read-only investigation to a SigNoz investigator. That child can inspect the
payment deployment and query a bounded SigNoz evidence bundle, but it cannot
inherit new tools, a larger budget, or a wider resource scope.

The proposed rollback is governed independently:

k8s.rollback_deployment:
  enabled: true
  source: kubernetes
  cost_usd: 0.02
  policy:
    mutating: true
    denial_only: false
    approval_required: true
    risk_level: critical
    evidence_required: required

The gateway records an immutable action plan containing the target, canonical
arguments, evidence digest, policy version, blast radius, and expiry. The
operator sees that exact plan in Slack or the MANDATE console—not a vague
“approve the agent” button.

Approval only advances the action to another checkpoint. Immediately before
execution, the gateway revalidates the mandate, argument digest, expiry,
budget, audit chain, dependency health, and live evidence. Only then does the
gateway perform the rollback. It queries the post-state afterward and records
verification as a separate phase.

How I used SigNoz

I ended up with two correlated telemetry lanes:

Agent-runtime telemetry explains model calls, subagents, tool calls, latency, token use, evaluation score, and estimated cost.
mandate.* telemetry explains capability validation, delegation, policy, planning, approval, execution, verification, revocation, and audit health.

Both lanes use low-cardinality identifiers such as mandate.id,
mandate.parent.id, mandate.root.id, and mandate.action.id. Full prompts,
capabilities, authorization headers, and API keys are intentionally excluded.

The repository reconciles three dashboards, seven saved investigation views,
and eighteen metric-, log-, and trace-based alerts:

Authority Overview for policy, approval, delegation, execution, verification, budgets, and audit integrity.
Incident Command for service impact, release correlation, infrastructure, and recovery.
Agent & LLM Cost for model latency, errors, tokens, cost, tool calls, and evaluations.

SigNoz Query Builder is important here because the evidence collector needs
structured, reproducible queries across metrics and traces. The SigNoz MCP
server exposes those observability capabilities to the gateway, but MANDATE
proxies only the read-only subset granted to the current agent. The agent never
receives the underlying SigNoz key.

The bugs that changed the design

The most useful lessons came from failures, not the happy-path demo.

At one point, a child remediation mandate accidentally included a resource
outside its parent's scope. The gateway correctly rejected the delegation, but
the webhook handler returned HTTP 502. SigNoz retried the alert, and every retry
spawned another incident coordinator. I fixed both sides of the problem: child
authority cannot expand beyond the parent, and a failed incident workflow is
recorded and acknowledged idempotently instead of inviting a retry storm.

I also approved a rollback after manually restoring the healthy deployment.
The action moved to executing, then stopped with evidence_unavailable.
Initially that looked like a broken rollback. It was actually the safety model
working: the release marker and failure evidence that justified the plan were
no longer live. An approval is consent for one current plan, not permission to
execute it later under different conditions.

Finally, an overly broad infrastructure query hit ClickHouse's memory
OvercommitTracker. That pushed me toward short evidence windows, bounded
aggregates, and low-cardinality metric labels. Observability queries are part
of the operational path; they need budgets too.

Reproducing the flow

The repository includes casting.yaml and casting.yaml.lock so the SigNoz
deployment can be reproduced with Foundry. For my local setup:

uv run mandate setup
uv run mandate start --profile lab --signoz
make kind-up kind-build kind-deploy
make demo-trigger-payment-regression

The final command prepares observable signals, verifies the SigNoz assets,
deploys the controlled payment regression, and generates checkout failures.
From there, I watch Incident Command, inspect the evidence-backed rollback, and
approve the exact plan in Slack. I do not manually reset the workload while
that approval is pending, because doing so intentionally makes the evidence
stale.

The demo remains deterministic even without a live LLM call. A capped live
model path exists for experimentation, but model output never controls a
critical policy branch.

What I would keep

The main lesson is simple: seeing an agent's reasoning is not the same as
controlling its authority.

SigNoz gave MANDATE a shared evidence plane for the incident, the agent, the
policy decision, the human checkpoint, the real execution, and the recovery
proof. That made the interesting question visible end to end:

Not merely “What did the agent do?” but “Why was it allowed to do it, and how
do we know it worked?”

The source, reproducible setup, dashboards, alerts, and runbooks are available
in the MANDATE repository.

DEV Community: Shaurya

An AI Agent Found the Outage. I Still Wouldn't Give It kubectl.

An AI Agent Found the Outage. I Still Wouldn't Give It kubectl.

The missing part of agent observability

The incident I used to test it

How I used SigNoz

The bugs that changed the design

Reproducing the flow

What I would keep

An AI Agent Found the Outage. I Still Wouldn't Give It `kubectl`.