Beyond Dashboards: Building an AI SRE Agent That Understands SigNoz Telemetry

Suhas — Sat, 25 Jul 2026 17:19:10 +0000

«Observability tools are excellent at collecting telemetry. But after the data is collected, a developer still has to inspect logs, identify the root cause, and decide what to fix. I wanted to explore a different question: Can an AI agent understand observability data and help developers reach the next step faster?»

The beginning

When I started this hackathon, I didn't plan to build an AI SRE agent.

My first challenge was much simpler—I needed an application that could generate telemetry for SigNoz.

Instead of creating a new microservice from scratch, I reused an old PlantNet plant identification backend that I had built earlier. The goal was to instrument it, send telemetry to SigNoz, and understand how observability actually works.

At first I experimented in Google Colab, but I quickly realized it wasn't the right environment for running SigNoz and continuously testing a backend service. I migrated everything to GitHub Codespaces and modified my PlantNet backend so it could work there.

Learning SigNoz

Getting SigNoz running was not the difficult part.

Getting meaningful telemetry into SigNoz was.

For nearly a week I focused on:

setting up SigNoz
modifying my backend
understanding how telemetry reaches the dashboard
debugging why no data was appearing

There were several occasions where I thought something was wrong with my code, only to discover that SigNoz services had stopped and simply needed to be restarted.

Using command

docker restart signoz-signoz-0

Changing Direction

Initially my plan was similar to many observability projects:

monitor latency
inspect failed requests
visualize telemetry

After spending time exploring SigNoz, I realized I was mainly consuming the information that SigNoz already provides.

That raised a different question:

_«What if the telemetry could be consumed by an AI agent instead of only being displayed on a dashboard?»
_
That question completely changed the direction of the project.

Instead of building another monitoring dashboard, I started building an AI assistant capable of reading telemetry, explaining failures, and suggesting fixes.

Building the AI SRE Agent

The final workflow became:

The user starts a diagnostic scan.
The agent requests telemetry from SigNoz.
SigNoz returns telemetry in JSON format.
The JSON is sent to Groq running the Llama 3.1 model.
The model performs root cause analysis.
The diagnostic report is displayed.
The user can apply the generated code fix and test the updated service.

*Making the Agent Observable
*
_One design decision I liked was instrumenting the AI agent itself.
_
The agent isn't just consuming observability data—it also creates its own OpenTelemetry spans during operations such as:

fetching telemetry
reasoning over failures
invoking the LLM

This means the diagnostic agent itself can also be monitored inside SigNoz.

That felt like a natural extension of the observability workflow.

Auto-Fix Workflow

After the AI completes its analysis, it generates a Python code block containing a suggested remediation.

Instead of copying the response manually, the application extracts the generated code and appends it to the target service.

The interface also provides:

a button to restore the original service
a button to execute the updated service
a live code viewer for verification

The objective wasn't to replace engineers but to shorten the path from identifying a problem to testing a possible solution.

Engineering Challenges

Building the project involved several iterations.

Some of the most significant challenges were:

moving from Google Colab to GitHub Codespaces
configuring SigNoz correctly
debugging missing telemetry
handling repeated service restarts
managing multiple terminal processes inside Codespaces
improving prompts so the LLM produced useful remediation instead of generic advice
preventing duplicate patches from repeatedly modifying the target service

Each issue changed the implementation and helped shape the final workflow.

Repository Structure

The project is divided into a few major components:

agent2.py — fetches telemetry, builds prompts, and communicates with Groq.
app.py — Streamlit interface for diagnostics and code execution.
ai_agent_tracer.py — OpenTelemetry instrumentation for the AI agent.
service.py — target service used during auto-fix demonstrations.

🔗 GitHub Repository: https://github.com/suhaskamde09-lgtm/signoz

Current Limitations

This project is still an early prototype.

Currently it:

analyzes telemetry after a manual scan
generates remediation suggestions
allows the generated code to be applied locally

It does not automatically deploy fixes or verify that the generated patch is correct. Human review is still required before trusting any generated code.

Future Work

There are several directions I want to explore next:

analyze multiple services instead of a single backend
trigger diagnostics automatically from alerts
create GitHub pull requests instead of editing local files
improve prompt engineering for more reliable remediation
perform validation before applying generated fixes

*Final Thoughts
*
This project started as an attempt to learn SigNoz by instrumenting an existing backend.

Along the way it evolved into something different: an AI-assisted SRE workflow that transforms observability data into actionable recommendations.

Rather than replacing observability platforms, the goal is to build on top of them—using AI to help developers understand failures faster while keeping engineers in control of the final decision.

DEV Community: Suhas

Beyond Dashboards: Building an AI SRE Agent That Understands SigNoz Telemetry