DEV Community

Cover image for I Built an AI SRE Agent That Doesn't Just Detect Incidents — It Investigates Them
BHUVAN
BHUVAN

Posted on

I Built an AI SRE Agent That Doesn't Just Detect Incidents — It Investigates Them

#signoz #hackathon #ai #opentelemetry

Most observability platforms tell you **what* broke. I wanted to build something that could figure out why it broke.*

During my first hackathon, I set myself a challenge that sounded simple but turned out to be incredibly interesting:

Can an AI investigate production issues the way an SRE would?

Not by reading one giant prompt containing thousands of logs.
Not by following a hardcoded script.
But by thinking through the investigation — deciding what information it needs next, collecting evidence from an observability platform, and finally arriving at a root-cause analysis.

That idea became AI SRE Agent, an autonomous incident investigator built on SigNoz, OpenTelemetry, FastAPI, and Gemini function calling.

The Problem

Modern observability tools are amazing at collecting telemetry. They tell us:

  • Which service is slow
  • Which requests are failing
  • Which traces have high latency
  • Which logs contain errors

But after that, the real work begins. An engineer still has to manually:

  • Open traces
  • Search logs
  • Compare services
  • Follow dependencies
  • Form a hypothesis
  • Validate it

The dashboard provides the clues. The engineer solves the mystery.

I wanted to see if an AI could perform that investigation automatically.

The Idea

Instead of giving an LLM every log and trace in one massive prompt, I built an AI that behaves like an engineer. It starts with almost no information, then it decides:

"What should I inspect first?"

After receiving that information, it decides:

"Now what should I inspect next?"

It keeps collecting evidence until it has enough confidence to produce a diagnosis. The important part: the investigation path is never hardcoded. Every decision is made dynamically by the model.

Architecture

The application is four microservices built with FastAPI:

                 ┌─────────────────────┐
                 │  frontend-service   │
                 │      Port 8001      │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │   order-service     │
                 │      Port 8002      │
                 └───────┬───────┬─────┘
                         │       │
                         ▼       ▼
           ┌─────────────────┐  ┌─────────────────┐
           │ inventory-service│ │  agent-service   │
           │    Port 8003     │ │    Port 8004     │
           └─────────────────┘  └─────────────────┘
Enter fullscreen mode Exit fullscreen mode

Every service is instrumented with OpenTelemetry and exports traces, logs, and metrics into a self-hosted SigNoz instance.

To make the environment realistic, each service intentionally injects production-like failures:

  • Random latency spikes
  • HTTP 500 and 502 failures
  • Slow database queries
  • Connection pool exhaustion
  • AI response failures
  • Simulated transaction deadlocks

The goal wasn't another e-commerce demo — it was a system that behaves like a real production environment, warts and all.

Live Observability with SigNoz

Once the services start running, SigNoz immediately begins collecting telemetry from every service. Instead of just seeing that requests are slow, I get visibility into P99 latency, error rates, operations per second, distributed traces, and structured logs — across every service at once.

SigNoz Services Dashboard — you can already spot the culprits before the AI even gets involved: frontend-service and order-service share the same 16.35% error rate, while diagnosis-agent (the AI itself) sits at 0% errors with a 10.6s P99 — a hint that it's doing multi-step reasoning, not just serving fast requests.

Building the AI Investigator

This is where the project gets interesting. The AI agent doesn't know the answer — it only has access to three tools:

get_slow_traces()
get_error_logs(service_name)
get_error_traces()
Enter fullscreen mode Exit fullscreen mode

That's all. The LLM decides which tool to call, which service deserves investigation, whether another query is necessary, and when enough evidence has been collected. This makes every investigation unique.

A Real Investigation

Here's one investigation session, end to end.

Step 1 — cast a wide net. The AI first requests the slowest traces:

get_slow_traces()
Enter fullscreen mode Exit fullscreen mode

It finds multiple slow requests, all pointing the same direction — frontend-service, GET /checkout, roughly 6 seconds each. Interesting, but slow doesn't automatically mean guilty.

Step 2 — narrow the search. Instead of checking every service one by one, the model narrows in:

get_error_logs(frontend-service)
Enter fullscreen mode Exit fullscreen mode

The logs repeatedly show the same failure: the order service failing with a 502. Now the picture clears up — the frontend wasn't the one failing, it was waiting on something downstream.

Step 3 — the diagnosis. The AI concludes:

  • Checkout latency is high
  • frontend-service is waiting on order-service
  • order-service is returning HTTP 500/502
  • The root cause likely lives inside order-service or one of its downstream dependencies

And instead of pretending it knows everything, it recommends the next investigation step.

The complete run — three agent steps, two tool calls, and a final diagnosis with a recommended next action, all generated without a hardcoded investigation script.

Making the AI Observable

While building this, I had another thought: if the AI is making decisions, how do I debug the AI itself?

So I instrumented the agent exactly like any other microservice. Every investigation creates a distributed trace:

agent.investigation
│
├── agent.step.1
│      ├── Gemini API
│      └── get_slow_traces()
│
├── agent.step.2
│      ├── Gemini API
│      └── get_error_logs()
│
└── agent.step.3
       └── Final diagnosis
Enter fullscreen mode Exit fullscreen mode

Every LLM request records prompt tokens, completion tokens, total tokens, response latency, and tool execution timing. The same platform that debugs my application also debugs my AI — that turned into my favorite part of the project.

A full 10.6-second investigation broken into its three reasoning steps. You can see exactly where the time goes — agent.step.3, the final synthesis step, took 4.33s (40.78% of total execution), while the earlier tool-calling steps were faster. That's the kind of insight you just don't get from a chat transcript alone.

Simulating Production Traffic

To properly evaluate the system, I built a load generator that continuously sends checkout requests — some succeed, some fail, some become extremely slow. This keeps realistic traces, logs, and metrics flowing into SigNoz, so the AI investigator always has real production-shaped telemetry to analyze instead of synthetic examples.

Four services and a load generator running side by side. The summary panel (195 attempts, 158 successful, 37 failed, 81.03% success rate) is exactly the kind of noisy, partially-degraded signal a real on-call engineer has to make sense of — and what the agent has to reason through.

The Biggest Challenges

Ironically, most of the hackathon wasn't spent writing AI code — it was spent solving infrastructure problems. Some memorable ones:

  • Repairing a corrupted WSL installation
  • Migrating to SigNoz's new Foundry installation
  • Docker Desktop vs. native WSL engine issues
  • ClickHouse startup problems
  • OpenTelemetry configuration
  • Auth issues with SigNoz's new Roles Beta API

These slowed development, but honestly they made the project more realistic. Building observability tooling means spending as much time on infrastructure as on application code.

What I Learned

The biggest lesson wasn't about AI — it was about observability.

Dashboards answer questions you already know to ask. Investigations begin when you don't know what question to ask next. That's exactly where AI becomes valuable — not by replacing engineers, but by automating the repetitive first stage of incident investigation.

What's Next?

Given more time, I'd love to add:

  • Continuous monitoring that automatically triggers investigations when error rates exceed thresholds
  • More realistic production failure scenarios
  • Automatic incident reports generated after every investigation
  • Support for the SigNoz MCP Server instead of handwritten API tools
  • Multi-agent collaboration, where specialized agents investigate logs, metrics, and traces independently before combining findings

Final Thoughts

This was my first hackathon, and I built the entire project solo. I intentionally kept the microservices simple because the real focus wasn't the application — it was the investigation process.

What I'm most proud of isn't that I built an AI capable of analyzing telemetry. It's that the AI itself became observable. Every decision, every tool call, every LLM request, every reasoning step — all of it shows up inside SigNoz as a distributed trace.

We're entering a future where we won't just observe our applications — we'll observe the AI systems responsible for operating them, too. I think that's a pretty exciting direction for modern SRE tooling.

GitHub Repository

🔗 github.com/polikeybhuvan/signoz-ai-sre-agent

If you have suggestions, ideas, or feedback, I'd love to hear them!

Top comments (0)