By Sagar Maurya & Disha Sonowal — WeMakeDevs SigNoz Hackathon
Before this, we wrote a first post about setting up SigNoz for the very first time — installing it, sending it some data, and figuring out what all this "tracing" stuff actually means. You can also check that out on LinkedIn. If you're new to any of this, that post explains the basics in a simple way, so it's worth a quick look first.
This post is us going one step further as a team. We wanted to see if SigNoz could actually help us discover something, not just show us pretty charts. It did, and here's exactly what happened, explained simply.
First, two words you need to know
- Trace — a record of everything that happened during one single request, from the moment it starts to the moment it finishes.
- Span — one step inside that trace. If a request does 4 things, it has 4 spans, one for each step.
That's really all you need to follow along.
What we built
A small fake AI assistant. Instead of doing just one thing, it does four things every time someone asks it a question:
- Look up some documents — pretend to search for information related to the question (a random number, somewhere between 1 and 8 documents)
- Ask the AI — pretend to send the question and those documents to an AI model and get an answer back
- Use a tool — pretend to use something like a calculator, a calendar, or a search tool
- Clean up the answer — tidy up the final response before sending it back
We wrapped each of these four steps in code that tells SigNoz "hey, this step just started" and "hey, this step just finished." That's what creates the spans.
Looking at just one question it answered
We ran our fake assistant 30 times, and then picked one single run to look at closely in SigNoz. Here's what it showed us:
The whole thing took 1.72 seconds. Almost all of that — 1.43 seconds — was spent in the "ask the AI" step. Everything else (looking up documents, using a tool, cleaning up) barely took any time at all in comparison. You can literally see this with your eyes on the chart, no reading through code needed.
Then we asked a bigger question
Looking at one run is useful, but we wanted to know something bigger: does the AI step get slower when it has more documents to look through?
In real life, this makes sense — if you ask an AI to read through 8 documents instead of 1, it usually takes longer to answer, because there's more for it to process.
So instead of checking each of our 30 runs one by one, we asked SigNoz to do the work for us:
- We told it: "only show me the AI-call steps, ignore the other three"
- We told it: "now group these by how many documents each one used"
- We told it: "show me the average time for each group"
And within seconds, SigNoz gave us this table:
| Documents used | Average time taken |
|---|---|
| 1 | 568ms |
| 2 | 658ms |
| 3 | 825ms |
| 4 | 999ms |
| 5 | 1147ms |
| 6 | 1333ms |
| 7 | 1488ms |
| 8 | 1614ms |
Look at that — it climbs almost perfectly evenly. Every extra document adds roughly 150 milliseconds to the response time. We built this rule into our code, so finding it wasn't a total surprise, but what genuinely impressed us was how easily SigNoz found and displayed it. We didn't write any extra code, we didn't open Excel, we didn't do any manual math. Just a couple of clicks, and the whole pattern was right there in a table.
We also checked the logs
Traces are great for timing, but sometimes you just want to read a plain sentence about what happened. That's what logs are for. We turned those on too, and got results like this:
Now every request has two views: the trace shows how long things took and in what order, while the logs show what actually happened, in plain readable sentences. Small tip if you try this yourself — just printing messages to your terminal doesn't automatically send them to SigNoz. You need a couple of extra lines of setup code to actually connect them, or nothing will show up in the Logs tab.
What we're taking away from this
The biggest thing we learned: a trace waterfall is great for spotting what's slow. But the real magic happened when we tagged our own extra detail onto a span — in our case, "how many documents did this use." That one small addition is what let us ask SigNoz a real question and get a real answer back, across all 30 runs at once, instead of guessing.
For the actual hackathon build, this is exactly the habit we want to carry forward — don't just time your code, tag it with details that'll actually matter later when something's slow and you're trying to figure out why.
Top comments (0)