A week ago, if you'd asked me what observability meant, I would've said: "some graphs that tell you stuff is broken." That's about it. I didn't know what a trace was. I didn't know metrics and logs were different things that needed to be stitched together to actually mean something. I'd never touched OpenTelemetry.
Then I signed up for the SigNoz hackathon.
I didn't want to build something that just sat on top of a dashboard and looked nice. Every observability tool I'd seen felt the same: alert fires, you open ten tabs, you manually piece together what happened, and by the time you've figured it out, you've already lost twenty minutes. I wanted to build something that closed that gap. Something that could go from "something is wrong" to "here's what broke, why, and what to do about it" without me doing all the detective work myself.
So over the next week I built Aro, an AI SRE agent that sits on top of SigNoz. And along the way I ended up learning more about traces, metrics, logs, and how real observability systems work than I expected to learn in a single week.
This is what I built, what actually happened while building it, and what I got wrong before I got it right.
The problem I was actually trying to solve
Most observability tools are great at telling you that something is wrong, but they leave the "why" and "what now" entirely up to you, and that's the part that always felt broken to me.
An alert fires, and suddenly you're the one jumping between a traces dashboard, a metrics panel, and a logs viewer, trying to mentally stitch together three different views of the same fifteen minutes, while the incident is still live and people are asking for updates. The tools aren't wrong, they're just incomplete, because they hand you data and expect you to be the one doing the correlation, the reasoning, and the decision-making, every single time.
That's the gap I wanted to close. Something above the dashboards that does the correlating for you, so you land on "here's what broke and why" instead of "here's a graph, good luck."
The goal was never a better SigNoz. It was using SigNoz as the source of truth and doing the thinking on top of it, like an experienced on-call engineer, minus the five-minute page delay.
SigNoz helped me in 2 ways in this project:
- It's built on OpenTelemetry, so one instrumentation pass gave me traces, metrics, and logs in the same place instead of three separate vendors.
- It ships an MCP server, so my agent could just ask it things like "what changed in this service in the last ten minutes" instead of me scraping a half-documented API.
That combo, OTel plus MCP, is what made a one-week build possible, and it's where most of what I actually learned happened.
The build
Everything I'm about to walk through is the actual pipeline I built, in the order it happens when a real incident occurs, not a curated list of features. I'll go through each piece the way I built it, including the parts I got wrong before I got them right.
1. Setting up SigNoz with Foundry
The first real step wasn't building anything, it was getting SigNoz running, and I went with SigNoz Foundry from the start, and the documentation made it look straightforward.
It only took a few setup calls to get everything running, and along the way it also set up the SigNoz MCP server for me, without any extra configuration on my end. Within a short while, I had SigNoz running locally on localhost:8080, fully working, with MCP already wired in and ready to be called as a tool later.
Looking back, this is the one part of the whole build that went almost exactly the way the docs said it would, and it's a big part of why I had enough time left over to focus on the agent logic instead of fighting infrastructure for days.
2. Wiring up telemetry
With SigNoz running, I imported the Agno AI agent dashboard template, since it matched the kind of service I was monitoring. This is where I made the mistake I mentioned earlier: I assumed data would show up on its own once the dashboard was imported. It didn't, because a template is just empty panels shaped for a particular app, and it stays empty until an instrumented service actually sends it data.
I only figured out what was missing by going through the Agno dashboard docs, which link to their observability guide for instrumenting an app so it produces the right signals.
Once I followed that guide and instrumented my demo app properly with OpenTelemetry, traces, metrics, and logs started flowing in, and the dashboard I'd imported earlier finally started showing real data instead of empty panels.
Here's the env pointing my load generator at the local collector (telemetry-lab/run_load.sh):
export OTEL_RESOURCE_ATTRIBUTES="service.name=telemetry-lab-agno"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_INSECURE=true
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
And the SDK bootstrap that actually installs the exporter (telemetry-lab/load_generator.py):
resource = Resource.create({
"service.name": SERVICE_NAME, # telemetry-lab-agno
"service.namespace": "signoz-hackathon",
"deployment.environment": DEPLOYMENT,
})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(endpoint=OTLP_ENDPOINT, insecure=True) # :4317
)
)
trace.set_tracer_provider(tracer_provider)
metrics.set_meter_provider(
MeterProvider(
resource=resource,
metric_readers=[
PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint=OTLP_ENDPOINT, insecure=True),
export_interval_millis=5000,
)
],
)
)
That mistake ended up teaching me the actual relationship between a dashboard and telemetry: a dashboard is only ever a view, and it's worthless until something real is feeding it.
3. Alerts → incidents
This is where the system starts behaving like something closer to a real SRE workflow, rather than a demo. SigNoz is the first system to see trouble. It's already evaluating alerts against the traces and metrics coming in from the demo app, and when something crosses a threshold, like latency spiking or errors climbing past a set limit, an alert fires.
Here's the actual rule for trace error rate above 5% (telemetry-lab/setup_signoz_alerts.py):
{
"alert": f"{SERVICE} trace error rate",
"alertType": "TRACES_BASED_ALERT",
"ruleType": "threshold_rule",
"condition": {
"compositeQuery": {
"queryType": "builder",
"queries": [
# A = error spans, B = all spans for service.name
{"type": "builder_formula", "spec": {
"name": "F1",
"expression": "(A / B) * 100",
"legend": "error rate %",
}},
],
},
"selectedQueryName": "F1",
"thresholds": {
"kind": "basic",
"spec": [{
"name": "critical",
"op": "above",
"matchType": "at_least_once",
"target": 5, # fire when error rate > 5%
"channels": [CHANNEL],
}],
},
},
"evaluation": {
"kind": "rolling",
"spec": {"evalWindow": "5m", "frequency": "1m"},
},
"labels": {"severity": "critical", "service": SERVICE, "signal": "traces"},
}
There's also a logs rule in the same file for ERROR/FATAL counts above zero, and a slow-traces p99 rule alongside it.
Aro listens for that alert, and the moment it comes in, two things happen at once. Aro creates an incident inside the dashboard, and it posts a focused message straight into Slack, so nobody has to be staring at a screen to know something just broke.
After this, I could build the actual investigation on top of it with some confidence that the data feeding it was correct.
4. Investigation via SigNoz MCP
This is the piece I'd point to if someone asked what actually makes Aro different from a bot that just forwards alerts into Slack. Once an incident is created, Aro doesn't guess, it asks SigNoz directly through the SigNoz MCP server, pulling traces around the incident window, fetching relevant metrics, and reading the logs tied to that service, the same way I would if I were investigating manually.
// actual tool call Aro makes (`aro/lib/signoz/mcp-client.ts + aro/lib/signoz/tools.ts`)
// Aro → SigNoz MCP
await client.callTool("signoz_search_traces", {
timeRange: "24h",
limit: 10,
filter: `service.name = 'telemetry-lab-agno' AND has_error = true`,
})
Which goes over the wire as a JSON-RPC call to the MCP server:
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "signoz_search_traces",
"arguments": {
"timeRange": "24h",
"limit": 10,
"filter": "service.name = 'telemetry-lab-agno' AND has_error = true"
}
}
}
The investigation isn't a black box either. There's a visible timeline showing every step, checking traces, checking logs, checking metrics, along with whether each one succeeded or failed. That trail gets built right alongside the calls above (aro/lib/dashboard/orchestrator.ts):
trail.push(startActivity({
kind: "tool",
title: "Searching error traces",
tool: "signoz_search_traces",
provider: "signoz",
}))
// same pattern for signoz_get_service_top_operations + signoz_search_logs
For a few specific views, I also used SigNoz's Query Builder to shape custom queries the MCP tools could call into, instead of relying only on default dashboard queries.
Up to this point, everything Aro knows comes from SigNoz alone. That was intentional, I wanted the core investigation grounded in real telemetry before letting it reach outside that system.
5. Going beyond SigNoz, with Composio
Once the SigNoz investigation finishes, there's an "Investigate deeper" option that lets Aro step outside SigNoz entirely, into GitHub, Linear, and Notion.
I used Composio here instead of writing auth and API handling for each tool myself, mainly because I didn't want to spend the limited time I had writing near-identical integration code for every service I planned to connect. Every Composio call runs through one shared executor (aro/lib/integrations/composio.ts):
const result = await composio.tools.execute(input.toolSlug, {
userId: account.userId,
connectedAccountId: account.connectedAccountId,
dangerouslySkipVersionCheck: true,
arguments: input.arguments,
})
And here's what Aro actually fires for GitHub and Linear (aro/lib/agent-tools/github.ts, linear.ts):
// Recent code / commits for the affected service
await executeIntegrationTool({
toolkit: "github",
toolSlug: "GITHUB_SEARCH_CODE",
arguments: { q: `repo:owner/repo path:telemetry-lab TaxClient`, per_page: 10 },
})
// Create + assign a Linear issue from the same incident context
await executeIntegrationTool({
toolkit: "linear",
toolSlug: "LINEAR_CREATE_LINEAR_ISSUE",
arguments: {
title: "Restore TaxClient 150ms timeout",
description: "...",
team_id: teamId,
assignee_id: assigneeId,
priority: 2,
},
})
Raising a PR uses the same path with GITHUB_CREATE_A_PULL_REQUEST. With that wired up, Aro can check recent commits against the affected service, pull related Linear issues, or surface a relevant Notion runbook, all tied to the same incident.
One thing I was deliberate about: nothing auto-merges, nothing auto-executes. Aro prepares the action, a human clicks to make it real. Giving an agent write access to production-adjacent systems without a human in the loop felt like the wrong trade for a tool meant to help during incidents, not add risk to them.
6. Memory layer
I kept this part intentionally lighter than the rest, it's more about where this goes next than what's fully proven out today. Every incident, investigation step, and decision gets stored. I can also feed it directly: runbooks, PDFs, old incident notes, rough text.
The idea is that over time Aro stops treating every incident as the first one it's seen. It can say "this looks like what happened last month, here's what fixed it," drawing on real history instead of just the current dashboard.
7. Team workspace and the Slack bot
That same memory makes the team workspace more than a name list. Inviting teammates gives Aro context on who they are, so it can assign an issue to the right person or flag that someone already fixed this exact problem last month.
The Slack bot runs on the same memory, so you're never forced into the dashboard. Chats stay private, or open up to the whole team when several people need to work on an incident together instead of duplicating effort in separate DMs.
If you want to see it without setting anything up? The production version runs on dummy data by default, so you can just click through.
That's the full pipeline: telemetry in, agent investigates, reaches into other tools, remembers. None of it worked cleanly on the first try, and that's really what the next part is about.
What I actually learned
Coming into this, observability meant graphs to me. That was about it.
Now I think of traces, metrics, and logs as three different questions, not three different tools.
- A trace tells you the path a single request took, and where it slowed down.
- A metric tells you whether that's a one-off or part of a pattern building over hours.
- A log tells you the actual detail at that one point, the "why" the other two can't give you on their own.
Before this, I'd have reached for one of these and called it observability. Now I don't think you can really understand an incident without all three at once, which is basically the whole premise Aro is built on.
The unglamorous part was the hardest part
Getting SigNoz, Slack, GitHub, Linear, Notion, and the memory layer to all sit inside one monorepo and actually talk to each other cleanly ate up more of the week than I expected, and it wasn't the fun part.
But it was the part that decided whether anything else worked at all.
Every new integration added more surface area for something to quietly break. More than once, I lost hours chasing a bug that had nothing to do with agent logic, it was two services passing context wrong between each other.
If this project taught me one real skill, it's not agent design. It's this:
keeping a multi-service system coherent enough to reason about under a deadline.
What I'd do differently
I'd move the memory layer much earlier. I added it near the end, after the core pipeline was already working, so it never saw enough real incidents to prove whether "this looks like incident X from last month" actually holds up, or just sounds good in a demo. Next time it goes in early, even rough, so it earns its place instead of being the least battle-tested part of the system by the deadline.
Closing the loop
A week ago, I didn't know what a trace was, or why metrics and logs needed to be separate things.
Now I've got an agent that pulls all three together during an incident, figures out what likely broke, and checks its own memory to see if something similar happened before.
That's the whole story of this hackathon: the dashboard-template mistake and the late memory layer are how I got from one side of that gap to the other.
If you want to see it running or dig into the code:
Live demo: helloaro.vercel.app
GitHub repo: github.com/Shivam-Katare/signoz-hackathon
Top comments (0)