A week ago, if you'd asked me what observability meant, I would've said: "some graphs that tell you stuff is broken." That's about it. I didn't know what a trace was. I didn't know metrics and logs were different things that needed to be stitched together to actually mean something. I'd never touched OpenTelemetry.
Then I signed up for the SigNoz hackathon.
I didn't want to build something that just sat on top of a dashboard and looked nice. Every observability tool I'd seen felt the same: alert fires, you open ten tabs, you manually piece together what happened, and by the time you've figured it out, you've already lost twenty minutes. I wanted to build something that closed that gap. Something that could go from "something is wrong" to "here's what broke, why, and what to do about it" without me doing all the detective work myself.
So over the next week I built Aro, an AI SRE agent that sits on top of SigNoz. And along the way I ended up learning more about traces, metrics, logs, and how real observability systems work than I expected to learn in a single week.
This is what I built, what actually happened while building it, and what I got wrong before I got it right.
The problem I was actually trying to solve
Most observability tools are great at telling you that something is wrong, but they leave the "why" and "what now" entirely up to you, and that's the part that always felt broken to me.
An alert fires, and suddenly you're the one jumping between a traces dashboard, a metrics panel, and a logs viewer, trying to mentally stitch together three different views of the same fifteen minutes, while the incident is still live and people are asking for updates. The tools aren't wrong, they're just incomplete, because they hand you data and expect you to be the one doing the correlation, the reasoning, and the decision-making, every single time.
That's the gap I wanted to close. Not another dashboard with prettier charts, but something that sits a layer above the dashboards and actually does the correlating for you, so that by the time you look at it, you're not starting from "here's a graph," you're starting from "here's what broke, here's why, and here's what you can do about it."
The difference matters more than it sounds, because…
a dashboard assumes you already know how to read it under pressure, and an incident is exactly the moment when that's hardest to do well.
So the goal was never to build a better SigNoz. It was to build something that used SigNoz as the source of truth, and then did the thinking on top of it, the way an experienced on-call engineer would, except available the second the alert fires, not five minutes after someone gets paged.
I picked SigNoz for this specifically because of two things that lined up well with what I was trying to build.
- First, it's built on OpenTelemetry from the ground up, which meant I could instrument a demo app once and have traces, metrics, and logs all land in the same place, instead of duct-taping three separate vendors together.
- Second, and more importantly for what I was building, SigNoz ships an MCP server, which meant my agent could actually query SigNoz directly as a tool, asking it real questions like "what changed in this service in the last ten minutes," instead of me writing a scraper or hitting some half-documented API and hoping the data held together.
That combination, OTel on one side and MCP on the other, is really what made this project possible in a week, and it's also where most of what I learned actually happened.
The build
Everything I'm about to walk through is the actual pipeline I built, in the order it happens when a real incident occurs, not a curated list of features. I'll go through each piece the way I built it, including the parts I got wrong before I got them right.
1. Setting up SigNoz with Foundry
The first real step wasn't building anything, it was getting SigNoz running, and I went with SigNoz Foundry from the start, and the documentation made it look straightforward.
It only took a few setup calls to get everything running, and along the way it also set up the SigNoz MCP server for me, without any extra configuration on my end. Within a short while, I had SigNoz running locally on localhost:8080, fully working, with MCP already wired in and ready to be called as a tool later.
Looking back, this is the one part of the whole build that went almost exactly the way the docs said it would, and it's a big part of why I had enough time left over to focus on the agent logic instead of fighting infrastructure for days.
2. Wiring up telemetry
With SigNoz running, the next step was getting a dashboard in place, so I imported the Agno AI agent dashboard template, since it matched the kind of service I was planning to monitor. And this is where I made the mistake I mentioned earlier. I assumed that once the dashboard was imported, data would just start showing up on its own, the same way the panels looked ready and waiting. It didn't, because a dashboard template is just a set of empty panels shaped for a particular kind of app, and it stays empty until an actual instrumented service starts sending it data.
I only figured out what was missing after going through the Agno dashboard documentation, which links directly to their observability guide for actually instrumenting an app so it generates the right signals for that dashboard to read.
Once I followed that guide and instrumented my demo app properly with OpenTelemetry, traces, metrics, and logs started flowing in, and the dashboard I'd imported earlier finally started showing real data instead of empty panels.
That mistake ended up teaching me the actual relationship between a dashboard and telemetry, which is that a dashboard is only ever a view, and it's worthless until something real is feeding it.
With SigNoz set up and real telemetry flowing in, the next problem was what to do the moment something in that data actually looked wrong, which is where alerts and incidents come in.
3. Alerts → incidents
This is where the system starts behaving like something closer to a real SRE workflow, rather than a demo. SigNoz is the first system to see trouble. It's already evaluating alerts against the traces and metrics coming in from the demo app, and when something crosses a threshold, like latency spiking or errors climbing past a set limit, an alert fires.
Aro listens for that alert, and the moment it comes in, two things happen at once. Aro creates an incident inside the dashboard, and it posts a focused message straight into Slack, so nobody has to be staring at a screen to know something just broke.
After this, I could build the actual investigation on top of it with some confidence that the data feeding it was correct.
Here's a tighter version:
4. Investigation via SigNoz MCP
This is the piece I'd point to if someone asked what actually makes Aro different from a bot that just forwards alerts into Slack. Once an incident is created, Aro doesn't guess, it asks SigNoz directly through the SigNoz MCP server, pulling traces around the incident window, fetching relevant metrics, and reading the logs tied to that service, the same way I would if I were investigating manually.
The investigation isn't a black box either. There's a visible timeline showing every step, checking traces, checking logs, checking metrics, along with whether each one succeeded or failed.
For a few specific views, I also used SigNoz's Query Builder to shape custom queries the MCP tools could call into, instead of relying only on default dashboard queries.
Up to this point, everything Aro knows comes from SigNoz alone. That was intentional, I wanted the core investigation grounded in real telemetry before letting it reach outside that system.
5. Going beyond SigNoz, with Composio
Once the SigNoz based investigation is done, there's an option in the UI to click "Investigate deeper," and this is where Aro is allowed to step outside SigNoz entirely, reaching into other tools like GitHub, Linear, and Notion.
For this part, I used Composio instead of writing integration logic for each tool myself, mainly because I didn't want to spend the limited time I had writing near identical auth and API handling code for every single service I planned to connect. Composio already handled that layer, which meant I could focus on what Aro actually does with those tools, rather than how it talks to them.
With that wired up, Aro can check recent GitHub commits against the affected service, pull related Linear issues, or surface a relevant Notion runbook, all tied to the same incident. I can ask it to create a Linear issue and assign it, or prepare a PR if there's a safe rollback.
One thing I was deliberate about: nothing auto merges, nothing auto executes. Aro prepares the action, a draft PR, a ticket, a suggested fix, but a human still clicks to make it real. Not because I couldn't automate it further, but because giving an agent write access to production adjacent systems without a human in the loop felt like the wrong trade for a tool meant to help during incidents, not add risk to them.
6. Memory layer
The last piece is memory, and I kept this part intentionally lighter than the rest, since it's more about where this goes next than what's fully proven out today. Every incident, every investigation step, and every decision made along the way gets stored, and I can also feed it more directly, runbooks, PDFs, old incident notes, even rough text.
Over time, the idea is that Aro stops treating every incident as the first one it's ever seen, and can instead say something like "this looks similar to what happened last month, here's what fixed it then," because it's drawing on our actual history, not just what's on the dashboard right now.
7. Team workspace and the Slack bot
That same memory makes the team workspace more than a list of names. Inviting teammates gives Aro context on who they are, so it can assign an issue to the right person, flag who's more senior on a service, or point out that someone already fixed this exact problem last month.
All of that carries into the Slack bot too, a full chat interface on the same memory, so you're never forced into the dashboard. Chats can stay private, or open up to the whole team when you want multiple people working an incident together instead of duplicating effort in separate DMs.
Want to see it without setting anything up? The production version runs on dummy data by default, so you can just click through.
That's the full pipeline, telemetry in, agent investigates, reaches into other tools, remembers. None of it worked cleanly on the first try, and that's really what the next part is about.
What I actually learned
Coming into this, observability meant graphs to me. That was about it.
Now I think of traces, metrics, and logs as three different questions, not three different tools.
- A trace tells you the path a single request took, and where it slowed down.
- A metric tells you whether that's a one-off or part of a pattern building over hours.
- A log tells you the actual detail at that one point, the "why" the other two can't give you on their own.
Before this, I'd have reached for one of these and called it observability. Now I don't think you can really understand an incident without all three at once, which is basically the whole premise Aro is built on.
The unglamorous part was the hardest part
Getting SigNoz, Slack, GitHub, Linear, Notion, and the memory layer to all sit inside one monorepo and actually talk to each other cleanly ate up more of the week than I expected, and it wasn't the fun part.
But it was the part that decided whether anything else worked at all.
Every new integration added more surface area for something to quietly break. More than once, I lost hours chasing a bug that had nothing to do with agent logic, it was two services passing context wrong between each other.
If this project taught me one real skill, it's not agent design. It's this: keeping a multi-service system coherent enough to reason about under a deadline.
What I'd do differently
If I rebuilt this, I'd move the memory layer much earlier.
I added it near the end, after the core incident pipeline was already working. Which meant it never saw enough real incidents to actually prove itself, whether "this looks like incident X from last month" holds up, or whether it just sounds good in a demo.
Next time, memory goes in early, even rough, so it has time to earn its place instead of being the least battle-tested part of the system by the time the deadline hits.
Closing the loop
A week ago, I didn't know what a trace was, or why metrics and logs needed to be separate things.
Now I've got an agent that pulls all three together during an incident, figures out what likely broke, and checks its own memory to see if something similar happened before.
That gap is the whole story of this hackathon. Everything I covered above, the mistakes with the dashboard template, switching off Docker, building memory too late, is how I got from one side of that gap to the other.
If you want to see it running or dig into the code:
Live demo: helloaro.vercel.app
GitHub repo: github.com/Shivam-Katare/signoz-hackathon
Top comments (0)