DEV Community

Kalash Poddar
Kalash Poddar

Posted on

How I Built an AI Agent That Autonomously Investigates Production Incidents Using SigNoz MCP

#signoz #opentelemetry #ai #python

At 3am, an alert fires. Your app's error rate is at 40%. You open SigNoz, find the broken traces, correlate them with logs, form a hypothesis, write a runbook, create a new alert so it doesn't happen silently again. Forty-five minutes later, you go back to bed.

I wanted to eliminate that forty-five minutes. Not with a better dashboard — with an agent that does the investigation itself.

This is what I built for the Agents of SigNoz Hackathon: ARIA (Autonomous Root-cause Investigation Agent). Here's exactly how it works, what broke along the way, and what I learned about using SigNoz's MCP server for things it wasn't obviously designed for.

The Setup

Three processes running together:

  • target-app — a FastAPI service with intentional fault injection (two endpoints that cascade-fail and hit a fake DB timeout)
  • SigNoz MCP Server — the official @signoz/signoz-mcp-server running as an HTTP process on port 8002, pointed at SigNoz Cloud
  • ARIA — a FastAPI agent on port 8001 that talks to the MCP server, calls an LLM, and serves a real-time incident dashboard

Both services send OpenTelemetry traces, metrics, and logs to SigNoz Cloud via gRPC/TLS. The MCP server talks to the same SigNoz Cloud instance via the REST API.

The Part Nobody Talks About: MCP Write Tools

Every demo I found of the SigNoz MCP server was someone asking it questions: "show me error logs", "what's the p99 latency". That's the read path, and it works great.

What I wanted was the write path. After generating an RCA, ARIA should automatically create a dashboard and an alert rule in SigNoz. The tools exist in the MCP server: signoz_create_dashboard, signoz_create_alert, signoz_create_notification_channel. But nobody had working example code for them.

The signoz_create_alert tool was the hardest part of the entire project. The v2alpha1 schema requires alertType set to the string "TRACES_BASED_ALERT", ruleType set to "traces", and — the one that took two hours to find — the condition.op field must be a string ("3" for greater-than), not an integer. When you get it wrong the response is a generic 400 with no explanation.

The working payload:

await mcp_client.call_tool("signoz_create_alert", {
    "alert": f"[AUTO] {incident_title}",
    "alertType": "TRACES_BASED_ALERT",
    "ruleType": "traces",
    "condition": {
        "op": "3",          # must be a string, not int
        "target": 0.1,
        "compositeQuery": {
            "queryType": "builder",
            "builderQueries": {
                "A": {
                    "dataSource": "traces",
                    "filters": {
                        "items": [{
                            "key": {"key": "hasError"},
                            "op": "=",
                            "value": True
                        }]
                    },
                    "aggregateOperator": "rate",
                }
            }
        }
    },
    "severity": "high",
    "receivers": ["sentinel-webhook"]
})
Enter fullscreen mode Exit fullscreen mode

Once this worked, the alert appeared in SigNoz Cloud within a second of the RCA completing. Watching a dashboard and an alert rule appear in SigNoz that the agent created autonomously — that was the moment the project felt real.

How the Autonomous Loop Works

Every 90 seconds, ARIA's ContinuousMonitor polls SigNoz via MCP:

top_ops = await mcp.call_tool("signoz_get_service_top_operations", {
    "service": service, "limit": 10
})
error_traces = await mcp.call_tool("signoz_search_traces", {
    "filters": {"hasError": True, "serviceName": service},
    "limit": 5
})
Enter fullscreen mode Exit fullscreen mode

If P99 latency exceeds 2000ms or error rate exceeds 10%, it kicks off a full RCA. Three MCP calls run in parallel:

traces, logs, top_ops = await asyncio.gather(
    mcp.call_tool("signoz_search_traces", {
        "filters": {"hasError": True, "serviceName": service},
        "timeRange": {"from": start_time, "to": end_time}
    }),
    mcp.call_tool("signoz_search_logs", {
        "query": "severity_text IN ['ERROR','WARN'] AND service_name='" + service + "'",
        "limit": 20
    }),
    mcp.call_tool("signoz_get_service_top_operations", {
        "service": service, "limit": 10
    }),
)
Enter fullscreen mode Exit fullscreen mode

All three results go into one prompt. The LLM sees real trace IDs, actual error messages from logs, and real latency percentiles per endpoint — not a description of the system, the live data itself.

That distinction matters more than I expected. Early versions of the prompt were elaborate: chain-of-thought instructions, reasoning frameworks, output format guidance. The runbook output was generic. When I stripped it down to just the raw data and said "analyze this and return JSON", the runbook steps started referencing specific endpoints and actual trace IDs. Data quality matters more than prompt engineering.

The LLM returns structured JSON:

{
  "severity": "HIGH",
  "incident_title": "Cascade failure on /api/orders/cascade",
  "root_cause": "P99 latency at 4200ms — downstream dependency timeout cascading...",
  "affected_endpoints": ["/api/orders/cascade", "/api/orders/process"],
  "immediate_actions": ["Check downstream DB connection pool"],
  "runbook_steps": ["1. Open trace ID abc123 in SigNoz..."],
  "create_dashboard": true,
  "create_alert": true
}
Enter fullscreen mode Exit fullscreen mode

When create_dashboard and create_alert are true, ARIA calls the MCP write tools. Dashboard and alert rule appear in SigNoz Cloud automatically.

The Closed Feedback Loop

The alert ARIA creates in SigNoz points at a webhook on ARIA itself: http://your-host:8001/webhook/alert.

When SigNoz's alertmanager fires — because the error threshold is still being breached — it hits that endpoint. ARIA receives the alert, runs another full RCA cycle, and escalates severity if the service is still broken:

Cycle 1: Anomaly detected → RCA → severity MEDIUM → signoz_create_alert
         [Service still failing 90s later]
Cycle 2: SigNoz fires alert → /webhook/alert → re-analyze → severity HIGH
         [Still breaking]
Cycle 3: severity CRITICAL → escalated notification
Enter fullscreen mode Exit fullscreen mode

SigNoz isn't just storing telemetry — it's actively driving ARIA's behavior through its alerting system. That's a genuine control loop, not a one-shot analyzer.

ARIA Observing Itself

ARIA instruments itself with OpenTelemetry and exports to the same SigNoz Cloud instance it's protecting:

with tracer.start_as_current_span("rca.full_pipeline") as span:
    span.set_attribute("rca.service", service)

    with tracer.start_as_current_span("mcp.gather_evidence"):
        traces, logs, top_ops = await asyncio.gather(...)

    with tracer.start_as_current_span("llm.generate_rca") as llm_span:
        response = await openrouter_client.chat.completions.create(...)
        llm_span.set_attribute("llm.input_tokens", response.usage.prompt_tokens)
        llm_span.set_attribute("llm.output_tokens", response.usage.completion_tokens)

    with tracer.start_as_current_span("mcp.write_remediation"):
        await mcp.call_tool("signoz_create_dashboard", {...})
        await mcp.call_tool("signoz_create_alert", {...})
Enter fullscreen mode Exit fullscreen mode

In SigNoz Cloud, filter traces by service.name = sentinel-agent. Every RCA shows as a distributed trace — MCP query duration, LLM call latency, dashboard creation time, all visible in the waterfall. During development this was genuinely useful: I could see my MCP calls were completing in ~200ms but the LLM was taking 7–8 seconds, which told me exactly where the latency budget was going.

The Bugs That Took the Most Time

The API key with escaped quotes.

In my startup script I had:

export SIGNOZ_API_KEY=\"AJDiq+...\"
Enter fullscreen mode Exit fullscreen mode

The backslash-quotes were being passed as literal characters. The agent started fine, the read-path MCP tools worked, but every write call returned 401. The key being sent included actual \" characters. Twenty minutes staring at the wrong thing before I spotted it.

The LLM that returned nothing.

nvidia/nemotron-3-ultra-550b-a55b:free sometimes returns choices: [] when rate-limited — not a 429 error, a valid 200 with an empty choices list. My original code was:

raw = response.choices[0].message.content.strip()
Enter fullscreen mode Exit fullscreen mode

This crashed silently. The monitor would detect an anomaly, start the RCA, hit the IndexError, swallow it, and move on. No incident appeared in the dashboard. No log message. Nothing.

The fix:

if not response or not response.choices or response.choices[0].message is None:
    logger.warning("LLM returned empty response — rate limited, retrying next cycle")
    return {
        "severity": "UNKNOWN",
        "incident_title": f"LLM unavailable — {service} needs manual review",
        "root_cause": "Rate limit on free LLM tier. Autonomous monitoring continues.",
    }
Enter fullscreen mode Exit fullscreen mode

Now ARIA creates a visible LOW-severity incident instead of silently failing.

condition.op must be a string.

Already covered above, but worth repeating: "3" not 3. Two hours. No error message. Just 400.

Results

After firing 30 concurrent cascade failures at the target service:

Metric Value
End-to-end RCA time ~9 seconds
Severity correctly identified HIGH ✅
SigNoz dashboard auto-created ✅
SigNoz alert rule auto-created ✅
Root cause identified correctly ✅
Runbook tied to real trace IDs ✅

Try It Yourself 

git clone https://github.com/kalash33/aria-signoz-agent
cd aria-signoz-agent
cp .env.example .env
# Fill in: SIGNOZ_API_KEY, SIGNOZ_API_URL, OPENROUTER_API_KEY,
#          OTEL_EXPORTER_OTLP_HEADERS (your SigNoz ingestion key)

# Start SigNoz MCP server
SIGNOZ_API_KEY=your_key \
SIGNOZ_URL=https://your-instance.us2.signoz.cloud \
npx @signoz/signoz-mcp-server@latest --transport http --port 8002

# Start agent + target app
docker compose up -d

# Trigger a manual RCA
curl -X POST http://localhost:8001/analyze/sync \
  -H "Content-Type: application/json" \
  -d '{"service":"target-app","lookback_minutes":15}'

# Open incident dashboard
open http://localhost:8001
Enter fullscreen mode Exit fullscreen mode

The MCP write tools are the interesting frontier. SigNoz handles the hard part — storing, indexing, and querying your telemetry at scale. Connecting that data to an autonomous agent that can create SigNoz resources in response to what it finds is where things get useful.

GitHub: https://github.com/kalash33/aria-signoz-agent

Top comments (0)