«Observability tools are excellent at collecting telemetry. But after the data is collected, a developer still has to inspect logs, identify the root cause, and decide what to fix. I wanted to explore a different question: Can an AI agent understand observability data and help developers reach the next step faster?»
The beginning
When I started this hackathon, I didn't plan to build an AI SRE agent.
My first challenge was much simpler—I needed an application that could generate telemetry for SigNoz.
Instead of creating a new microservice from scratch, I reused an old PlantNet plant identification backend that I had built earlier. The goal was to instrument it, send telemetry to SigNoz, and understand how observability actually works.
At first I experimented in Google Colab, but I quickly realized it wasn't the right environment for running SigNoz and continuously testing a backend service. I migrated everything to GitHub Codespaces and modified my PlantNet backend so it could work there.
Learning SigNoz
Getting SigNoz running was not the difficult part.
Getting meaningful telemetry into SigNoz was.
For nearly a week I focused on:
- setting up SigNoz
- modifying my backend
- understanding how telemetry reaches the dashboard
- debugging why no data was appearing
There were several occasions where I thought something was wrong with my code, only to discover that SigNoz services had stopped and simply needed to be restarted.
Using command
docker restart signoz-signoz-0
Changing Direction
Initially my plan was similar to many observability projects:
- monitor latency
- inspect failed requests
- visualize telemetry
After spending time exploring SigNoz, I realized I was mainly consuming the information that SigNoz already provides.
That raised a different question:
_«What if the telemetry could be consumed by an AI agent instead of only being displayed on a dashboard?»
_
That question completely changed the direction of the project.
Instead of building another monitoring dashboard, I started building an AI assistant capable of reading telemetry, explaining failures, and suggesting fixes.
Building the AI SRE Agent
The final workflow became:
- The user starts a diagnostic scan.
- The agent requests telemetry from SigNoz.
- SigNoz returns telemetry in JSON format.
- The JSON is sent to Groq running the Llama 3.1 model.
- The model performs root cause analysis.
- The diagnostic report is displayed.
- The user can apply the generated code fix and test the updated service.
*Making the Agent Observable
*
_One design decision I liked was instrumenting the AI agent itself.
_
The agent isn't just consuming observability data—it also creates its own OpenTelemetry spans during operations such as:
- fetching telemetry
- reasoning over failures
- invoking the LLM
This means the diagnostic agent itself can also be monitored inside SigNoz.
That felt like a natural extension of the observability workflow.
Auto-Fix Workflow
After the AI completes its analysis, it generates a Python code block containing a suggested remediation.
Instead of copying the response manually, the application extracts the generated code and appends it to the target service.
The interface also provides:
- a button to restore the original service
- a button to execute the updated service
- a live code viewer for verification
The objective wasn't to replace engineers but to shorten the path from identifying a problem to testing a possible solution.
Engineering Challenges
Building the project involved several iterations.
Some of the most significant challenges were:
- moving from Google Colab to GitHub Codespaces
- configuring SigNoz correctly
- debugging missing telemetry
- handling repeated service restarts
- managing multiple terminal processes inside Codespaces
- improving prompts so the LLM produced useful remediation instead of generic advice
- preventing duplicate patches from repeatedly modifying the target service
Each issue changed the implementation and helped shape the final workflow.
Repository Structure
The project is divided into a few major components:
- agent2.py — fetches telemetry, builds prompts, and communicates with Groq.
- app.py — Streamlit interface for diagnostics and code execution.
- ai_agent_tracer.py — OpenTelemetry instrumentation for the AI agent.
- service.py — target service used during auto-fix demonstrations.
🔗 GitHub Repository: https://github.com/suhaskamde09-lgtm/signoz
Current Limitations
This project is still an early prototype.
Currently it:
- analyzes telemetry after a manual scan
- generates remediation suggestions
- allows the generated code to be applied locally
It does not automatically deploy fixes or verify that the generated patch is correct. Human review is still required before trusting any generated code.
Future Work
There are several directions I want to explore next:
- analyze multiple services instead of a single backend
- trigger diagnostics automatically from alerts
- create GitHub pull requests instead of editing local files
- improve prompt engineering for more reliable remediation
- perform validation before applying generated fixes
*Final Thoughts
*
This project started as an attempt to learn SigNoz by instrumenting an existing backend.
Along the way it evolved into something different: an AI-assisted SRE workflow that transforms observability data into actionable recommendations.
Rather than replacing observability platforms, the goal is to build on top of them—using AI to help developers understand failures faster while keeping engineers in control of the final decision.
Top comments (0)