Our incident-response agent got the root cause wrong 7 times out of 12. It still never made a bad rollback.

We ran Agent K against 12 seeded production incidents. It named the correct root cause 5 times out of 12 — 3 out of 9 if you discard the runs we couldn't cleanly attribute. Over the same 12 runs it produced the correct rollback / no-rollback verdict 10 times, published zero claims without a SigNoz link behind them, and executed zero actions outside its policy gate.

That gap between "diagnosed correctly" and "behaved correctly" is the entire project.

Context

Agent K was our entry for Agents of SigNoz (WeMakeDevs × SigNoz), Track 01. The setup: a FastAPI RAG support service — 72 synthetic help-centre docs in Postgres + pgvector, local sentence-transformers embeddings, free-tier models behind one OpenAI-compatible client (Groq by default; Cerebras, Gemini and OpenRouter swap in via one env var) — instrumented with OpenTelemetry, shipping traces, metrics and logs to SigNoz over OTLP. We installed SigNoz with Foundry, so casting.yaml/casting.yaml.lock are committed and the stack is reproducible; pointing the same exporter at SigNoz Cloud later took no code change.

When an alert fires, Agent K investigates through the SigNoz MCP server, publishes only claims carrying a resolvable SigNoz link, and rolls the service back only if a piece of plain Python says it may.

We started from an assumption we never tried to engineer away: the model will be wrong. So correctness can't be the safety mechanism. The safety has to be code.

Four ways to break the app, two of which deserve a rollback

Everything hangs off an in-process flag store with four seeded failures, and one deliberate asymmetry:

# app/flags.py
FLAG_NAMES: tuple[str, ...] = (
    "prompt_regression", "retry_storm", "retrieval_latency", "db_pool_exhaustion",
)

# The scenarios that represent a "deployment" (a code/prompt change) and therefore
# get a deployment.marker span; the other two are runtime/infra faults with no marker.
DEPLOYMENT_CLASS_FLAGS: set[str] = {"prompt_regression", "retry_storm"}

SigNoz has no native deployment-marker API (we chased SigNoz/signoz#6162, closed unshipped), so we emit our own deployment.marker OTel span through the pipeline that already exists. Two scenarios emit one; two don't. That present/absent asymmetry is the queryable signal — the absence of a marker is positive evidence that a rollback would not help.

One POST /ask in SigNoz: the rag.retrieval, rag.prompt_construction and chat spans, with the free SQLAlchemy DB span nested under retrieval.

The gate that is not allowed to call an LLM

Six deterministic checks, all of which must pass:

# app/policy.py — THIS MODULE MUST NEVER CALL AN LLM.
ACTION_ALLOWLIST: tuple[str, ...] = ("rollback",)   # exactly one entry
SANDBOX_SERVICES  = frozenset({"agent-k-rag-service"})
BURN_RATE_THRESHOLD  = 1.0
CONFIDENCE_THRESHOLD = 0.70
COOLDOWN_SECONDS     = 600.0

slo_breach, allowlist, cooldown, confidence, deployment_related, sandbox_scope. Every unknown fails closed — an alert we can't read a burn rate from parses to 0.0, which fails the SLO check and denies the action.

The "no LLM" part isn't a comment we trust ourselves to honour. One test walks the module's import graph with ast and fails the build if app.llm ever appears in it; another monkeypatches llm.generate to raise on any call and then runs a full evaluation, which catches an indirect call a grep would miss. Both live in the 219-test suite.

The bug our own eval caught: trust-the-model sneaking back in

This is the part worth reading twice.

The deployment_related check needs to know which scenario it's looking at, and the cheapest source was the model's claim text — our system prompt forces it to start its answer with one of the four scenario names. Fine, until the first 12-run eval printed this: two db_pool_exhaustion incidents narrated as retry_storm, each approving a rollback that could not possibly have fixed a database pool problem.

The gate wasn't broken. It was doing exactly what we wrote — and what we'd written was "the model may authorise a rollback by writing the word retry_storm in a sentence." Every safety property we were claiming had a hole in it shaped like a string match.

The fix was to make the check require corroboration from telemetry, not narration:

# app/policy.py — check 5
observed = frozenset(deployment_markers or ())
is_deployment_class = incident_type in DEPLOYMENT_CLASS_FLAGS
marker_seen         = incident_type in observed
deployment_ok       = is_deployment_class and marker_seen

And deployment_markers is only ever populated from a tool result, never from a claim:

# app/investigation.py — inside the evidence-gathering step
if ev_type == "deployment":
    inv.deployment_markers_seen.update(
        name for name in FLAG_NAMES if f'"{name}"' in content_text
    )

We also made the ordering load-bearing — an investigation that stopped before the marker query could never approve anything — and asserted it at import rather than leaving it in a comment:

assert _DEPLOYMENT_QUERY_INDEX < MIN_EVIDENCE_ITERATIONS, (
    "the deployment-marker query must fall within MIN_EVIDENCE_ITERATIONS, or the "
    "policy gate can never corroborate a deployment-class claim"
)

After that fix, both bad approvals disappeared. The agent still misdiagnoses — it called retrieval_latency "retry_storm" in all three of those runs — but a misdiagnosis now produces a denial, not a rollback.

The incident report's policy decision: five checks pass, deployment_related fails, and the action is denied with a next step for the human.

Five things that only broke once it was actually running

1. SigNoz's OTLP receivers don't bind until you finish first-run setup. A freshly cast stack looks healthy — every container up, curl localhost:8080 returns 200 — while curl localhost:4318/v1/traces gives connection reset by peer, which reads exactly like a broken exporter. The collector is OpAMP-managed and can't register an agent without an org, so the backend loops on cannot create agent without orgId. Check it directly:

curl -s http://localhost:8080/api/v1/version   # "setupCompleted":false

Register an org (UI or POST /api/v1/register) and the receivers come up. We lost real time believing app/telemetry.py was wrong. It wasn't.

2. We invented MCP tool names. We'd written query_traces, query_logs, query_metrics against a mental model of the API. The real inventory in signoz-mcp-server v0.9.0 is signoz_search_traces, signoz_search_logs, signoz_aggregate_traces. Every evidence query failed, stripping every claim and escalating every investigation — a total failure that looked like an agent problem. Two more traps followed: no Windows binary ships in the releases (we cross-compiled in WSL), and SigNoz v0.134 rejects JWTs sent via SIGNOZ-API-KEY, wanting Authorization: Bearer — with a token that expires every 30 minutes, which is a delightful thing to discover mid-demo.

3. The free tier is a real design constraint. A 20-row trace search measured 8,424 tokens and came back HTTP 413 before a hypothesis existed. SigNoz returns every possible span attribute per row and most are null for this app, so we strip nulls rather than truncate — same signal, an order of magnitude fewer tokens. Then the model began diagnosing from the query engine's own statistics: "db_pool_exhaustion is likely due to an extremely high number of rows scanned (1018) and bytes scanned (10434)". Those are the stats for Agent K's own query. Hence:

_ENGINE_NOISE_KEYS = frozenset({
    "meta", "rowsScanned", "bytesScanned", "durationMs", "stepIntervals",
    "nextCursor", "queryName", "columnType", "aggregationIndex", "signal",
    "fieldContext", "fieldDataType", "id", "warnings",
})

Evidence must describe the incident, never the act of looking at it.

4. The agent investigated our test suite. Our tests import app.main, which calls setup_telemetry(), which shipped every span pytest produced into the real SigNoz. A live run read deployment-marker counts of retry_storm=7 vs prompt_regression=3 when only prompt_regression had been injected — the retry_storm markers were pytest's — and misdiagnosed on that basis. AGENT_K_DISABLE_OTLP_EXPORT now suppresses export while still building all three providers.

5. A 200 from a single-page app is not a resolving link. We built evidence links to /deployments, a route that does not exist in SigNoz. Our link checker passed anyway — SigNoz is an SPA and answers 200 for every path, so a status check can't tell a real route from a typo. A human clicking the link landed on nothing. Deployment markers are spans, so the honest destination was the traces explorer. (Our default base URL was wrong too: 3301, SigNoz's older port, where Foundry serves on 8080.)

Where the rollback actually happens

Agent K never holds the Docker socket. A separate deployer sidecar does, and it exposes exactly one mutating endpoint that takes an empty body — the caller names no image, no service, no command. A caller that cannot name an image cannot be tricked into deploying an attacker's image.

deployer:
  volumes:
    # The privilege boundary, in one line. Nothing else in this project gets it.
    - /var/run/docker.sock:/var/run/docker.sock

One rule in there took a failed rollback to learn: the command is docker compose up -d --force-recreate, never restart. restart brings the container back on its current image and reports success while changing nothing — the rollback appears to work and the incident continues.

The Agent K dashboard in SigNoz: policy verdicts, why actions were denied, and recovery-verified counts side by side.

What we'd tell ourselves on day one

• Grade verdicts, not just diagnoses. Our accuracy number is mediocre and our verdict number is good, and the second one decides whether a service gets restarted at 3am. Both verdict misses were in the safe direction: it declined to act when it could have.
• Never let a confidence score reach 1.0. An early run rendered "Confidence 100%" on a diagnosis that was flatly wrong — the model proposed 0.9 and our boosts clamped to 1.0. We now cap recalibrated confidence at 0.95, permanently.
• Assume every input is attacker-shaped, including your own telemetry. Two of the five failures above were the agent reasoning over data about itself: its own query statistics, and its own test suite's spans.

The honest summary: the model is the least trustworthy component in the system, and designing around that made everything else simpler. All of this worked for our setup — one machine, one service, four failure modes we wrote ourselves. We haven't tested any of it against an incident we didn't author.

Code: github.com/SujalXplores/Agent-K · Built for Agents of SigNoz, Track 01. Built with AI coding assistance (Claude Code), disclosed in the repo; the shipped agent itself runs on free-tier providers and local embeddings only.