I built AI SRE Copilot for the WeMakeDevs "Agents of SigNoz" hackathon — a Streamlit app that reads raw application logs and tells you the root cause, severity, and fix, using Groq's Llama 3.3 70B model instead of manually scrolling through stack traces.
That solves the log-debugging problem. But it creates a new one: an AI tool that just hands you an answer is a black box. If the model is slow, or fails silently, or behaves worse on certain logs, you'd never know unless you happened to be watching at that exact moment. So alongside the AI feature, I instrumented the AI call itself with OpenTelemetry and shipped the traces to SigNoz — treating the Groq call as a real, observable dependency, the same way I'd treat a database query.
How it's wired up
Every time a log gets analyzed, I wrap the request in an OpenTelemetry span and attach custom attributes to it:
-
ai.model— which model handled the request -
ai.response_latency_ms— how long Groq took to respond -
log.size_bytes— size of the input log -
analysis.status— success or failure -
analysis.type— what kind of analysis ran -
user.action— what triggered the trace
These traces get exported through the OpenTelemetry SDK to SigNoz, where they show up as searchable, queryable data instead of disappearing into a terminal log.
What SigNoz actually showed me
To watch this over time instead of one trace at a time, I built a custom panel in SigNoz using the Query Builder, plotting signoz_latency.count over the last three days:
![SigNoz custom panel plotting signoz_latency.count over the last 3 days, showing a sharp spike around 7/24 that settles back down]
There's a clear latency spike around 7/24 that I never would have caught by just watching the app — I wasn't looking at that exact moment. But because the trace data was sitting in SigNoz, I could go back afterward and actually see it happened, and roughly when.
That's the real value of this setup: the AI's behavior doesn't just vanish after each request. It's there, on a dashboard, whenever I want to check it.
Takeaway
The AI half of this project — prompting Groq, formatting a response — was the easy part. The useful lesson was realizing observability doesn't stop at your own app code. If your app calls out to an AI model, that call is a dependency like any other, and it deserves to be traced and measured, not left invisible. SigNoz made that possible without me having to build any tracing infrastructure myself.
Built for the WeMakeDevs "Agents of SigNoz" hackathon.
🌐 Live App: https://ai-sre-copilot-pusprhbhw2vtaa62ojxnjo.streamlit.app/
💻 GitHub Repository:https://github.com/kusumaranikusumarani3232-hub/AI-SRE-Copilot
Top comments (0)