Team: The Badminton Philosophers
The three of us met on a college badminton court 6 months ago, started talking about AI, Cryptography and security between games, and never really stopped. That is where the name comes from. We are building an AI agent project for the Agents of SigNoz hackathon and we gave ourselves one day to explore SigNoz properly before the sprint started.
We built a small agent to explore with. It was supposed to answer questions by searching a knowledge base, doing a quick calculation if numbers were involved, and returning a clean answer. Simple enough that we could focus on the observability rather than the agent itself.
It ran. It returned answers. Everything looked fine.
Everything was not fine.
Setting up SigNoz first
Before we ran anything we needed SigNoz up. The current self-host path goes through Foundry, their CLI that provisions the whole stack from one config file.
curl -fsSL https://signoz.io/foundry.sh | bash
Create casting.yaml:
apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment:
flavor: compose
mode: docker
Deploy:
foundryctl cast -f casting.yaml
Verify:
docker ps
Five containers need to show healthy. ClickHouse is the telemetry store. Postgres is the metastore. There is a ClickHouse keeper, the OTel collector, and the SigNoz app. Then http://localhost:8080 opens the UI.
One warning the documentation person caught before the setup person made this mistake: Foundry generates compose files into a folder called pours and owns them completely. Do not edit those files by hand. The next cast run overwrites every change. All configuration goes in casting.yaml.
We instrumented the agent with OpenTelemetry and OpenInference, sent it twenty test queries, and watched the spans land in SigNoz. The Services page showed our agent. Response times looked reasonable. No errors flagged anywhere.
We thought we were done with setup and about to move on to exploring features.
Then we opened the first trace.
The trace told a different story
The agent was taking between four and nine seconds to respond. We had assumed that was the LLM being slow. Reasonable assumption. LLMs are slow sometimes.
The spans told a different story. The agent.run span showed 188ms total. The llm.call underneath it took 188ms as well, meaning almost all the time was in the LLM. The tool.calculator span was 0.1ms, barely registering. But it was being called repeatedly and silently failing on malformed inputs.
The LLM call was taking about 800 milliseconds. Fast, actually. The thing eating all the time was the calculator tool. It was being called three and four times per query in some cases, each call taking over a second, and it was failing silently about half the time and just retrying without logging anything visible.
We would never have seen this from the outside. The agent returned an answer every time. The answers looked plausible. The response time was annoying but not obviously broken. From the surface everything looked like a slow but working system.
The trace showed a broken system that happened to produce output anyway.
Opening the logs made it worse
The setup person clicked through to the logs correlated with one of those traces. In SigNoz you can jump from a trace directly to the log lines that fired during that exact span. Same interface, one click, no timestamp hunting across separate tools.
The logs showed the calculator tool receiving inputs like "what is 4 times the result of the previous step" as literal strings. The agent was not resolving the reference before passing it. The tool was trying to parse "the result of the previous step" as a number, failing, and retrying with a slightly different phrasing that also failed, and eventually either getting lucky or returning a wrong answer confidently.
The LLM was fine. The tool was receiving garbage and trying to work with it anyway.
Pranay Prateek, co-founder and CEO of SigNoz, wrote in their funding announcement that many teams are stuck using only basic logs and metrics and cannot reach modern tooling like distributed tracing, often because of expensive pricing and compatibility constraints. That sentence landed differently after seeing what a trace revealed that logs alone never would have.
Without the trace we would have spent time tuning the LLM prompt trying to get better answers. The prompt was not the problem. The trace showed us that in about thirty seconds.
Metrics showed us the pattern across all twenty runs
We had been looking at individual traces. The metrics view showed us the same failure happening systematically.
The calculator tool was erroring about forty percent of the time across the full run. Not randomly. It clustered on queries that involved referencing a previous result. The LLM was producing a specific type of malformed input consistently and the tool was consistently failing on it.
The metrics made the pattern visible. The trace had shown us one instance. The metrics showed us it was not an edge case.
Dashboards and alerts: catching this next time before it happens
We built a dashboard tracking tool error rate and agent response time together so we could see them move in relation to each other.
Then we set an alert for tool error rate above twenty percent sustained for two minutes. If this happens again in a real deployment we will know about it before users start noticing the slow and wrong answers.
Ankit Nayan, co-founder and CTO, has been driving OpAMP support into SigNoz, a newer OpenTelemetry sub-protocol that lets you reconfigure the Collector dynamically without container restarts. The direction the platform is heading is toward an observability stack that adapts at runtime rather than at redeploy time, which matters more as the shape of what you are monitoring changes constantly.
The thing we found while looking for something else
While the setup person was running all of this, the person reading documentation found something in the Foundry config reference that made us stop what we were doing.
SigNoz ships an optional MCP server. MCP stands for Model Context Protocol. It lets an AI agent query SigNoz directly in plain language and get answers from real telemetry. Enable it by updating casting.yaml and rerunning cast:
apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment:
flavor: compose
mode: docker
mcp:
spec:
enabled: true
foundryctl cast -f casting.yaml
Check it is alive:
curl -fsS localhost:8000/livez && echo " OK"
We had been planning to write a polling loop that called the SigNoz REST API every sixty seconds to pull span data into our detection agent. The documentation person sent a message to the group that said something like "wait, does this mean we do not need any of that."
We do not need any of that. Our agent can ask SigNoz what it needs to know directly. The two-day integration layer we had planned does not exist anymore.
The person watching the SigNoz YouTube channel found this working in their Noz AI debugging assistant. A recent video showed Noz generating a plain language reliability report from a full week of telemetry automatically. Instead of manually comparing dashboards across time ranges, you ask a question and get an answer.
For a hackathon building AI agents on top of SigNoz, that is worth knowing before you plan your architecture.
What we would have built without traces
We would have spent the first day tuning the LLM prompt. We would have made the system message more explicit about not referencing previous results. We would have iterated on a few variations and run evaluations. Eventually we might have stumbled onto the fact that the calculator was receiving bad input, but we would have arrived there through guesswork, not evidence.
The trace showed us the actual failure in thirty seconds. Not approximately where things were slow. Not that errors were happening. Exactly which tool, exactly which span, exactly what input it received, exactly how many times it retried.
Hiro Tamada, a Founding Engineer at Kernel building AI agent infrastructure, made a point in a video that we kept coming back to during this day. He said that teams make the mistake of leaving metrics and dashboards until the end of a feature cycle and that the ones who start instrumentation on day one get to use traces as a continuous debugging tool rather than scrambling to add visibility after something breaks in production.
We built a broken agent and SigNoz showed us it was broken in the first thirty minutes. If we had shipped this to real users first we would have gotten confused complaints about wrong answers and spent days guessing at the cause.
One technical thing worth knowing
We initially sent data as unstructured log messages attached to spans rather than structured attributes with typed values. Querying unstructured text through the SigNoz Query Builder is painful compared to filtering on clean typed fields. Use structured span attributes from the start. Type your keys and values. The Query Builder is significantly more useful when the data underneath it is clean.
What we are building with all of this
We are building an agent that watches other AI agents for behavioral drift. Not resource spikes. Not error rates. Whether the agent is still producing semantically similar outputs week over week, because model updates and RAG database changes can shift how your agent behaves without changing a single line of code or triggering a single infrastructure alert.
The detection agent asks SigNoz which spans look semantically different from baseline. SigNoz answers through MCP. The agent scores the drift and pushes it back as a custom metric. When the score drops below threshold an alert fires.
We found the approach because we spent a day inside SigNoz before writing project code. The broken calculator agent is what gave us the idea. Watching traces reveal a failure that looked like success from the outside is what made us want to build a system that does that watching continuously and automatically.
We will show you what we built after the sprint.
ā The Badminton Philosophers
Resources: SigNoz docs at signoz.io/docs, OpenTelemetry Python SDK at opentelemetry.io, SigNoz YouTube at youtube.com/@signoz
Top comments (4)
Great write-up! I really liked how you showed that an AI agent can appear to work well while still having hidden issues that only proper observability can reveal. Very insightful
Thank you :)
Really enjoyed reading this! The progression from "everything looks fine" to discovering the actual bottleneck through traces was engaging and easy to follow. Wishing your team the best in the SigNoz hackathon!
Really impressive idea. Best of luck in the hackathon!