I Traced My First AI Agent With SigNoz — and Caught a Bug Hiding in Plain Sight

S.SKANDHA ROSHAN — Sun, 19 Jul 2026 02:01:10 +0000

I Traced My First AI Agent With SigNoz — and Caught a Bug Hiding in Plain Sight

I filtered my traces for has_error = true. No results found.

But I was staring right at an exception — Simulated weather API timeout — sitting inside a span, plain as day. It felt like finding the rat had been in the trap the whole time, except the trap said "empty." That mismatch ended up being the most useful thing I learned tonight, and I only found it because I built something real enough to break.

Why I was doing this

I'm prepping for the Agents of SigNoz hackathon, and before building the actual project, I wanted to get my hands dirty with SigNoz first — not just skim the docs. So I set myself one goal for the night: build a small AI agent, trace it end to end with OpenTelemetry, and break it on purpose to see what SigNoz actually shows me.

Getting SigNoz running wasn't as simple as I expected

I'm on WSL (Ubuntu on Windows). The guide I'd bookmarked earlier that week used the old docker-compose setup, but SigNoz deprecated that as of v0.130.0 in favor of a new installer called Foundry — so half of what I'd read was already out of date.

Their current docs are also explicit about one thing: on Windows, run Docker natively inside WSL 2, not Docker Desktop, because ClickHouse Keeper is known to segfault under Docker Desktop's virtualization layer. I installed Docker directly in WSL with get.docker.com, which matched this recommendation.

Once Docker was running, I installed Foundry and deployed:

curl -fsSL https://signoz.io/foundry.sh | bash

cat > casting.yaml << 'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

foundryctl cast -f casting.yaml

It pulled all four images fine — ClickHouse, Postgres, the OTel collector, SigNoz itself — then died mid-startup with signal: killed. Turned out my WSL instance only had 1.8GB of RAM allocated by default, well under SigNoz's documented 4GB minimum — and Docker only reported signal: killed, which made the memory issue hard to recognize at first. The fix was a .wslconfig file on the Windows side:

[wsl2]
memory=6GB
processors=4

followed by wsl --shutdown and restarting. After that, foundryctl cast succeeded and I had a working SigNoz UI at localhost:8080.

Building the agent

Once SigNoz was up, I built a small LangGraph agent with two tools and one LLM call:

User query → weather_tool → calculator_tool → llm_call → Response

weather_tool — fetches (fake) weather data, with a simulated chance of timing out
calculator_tool — converts the temperature, always succeeds
llm_call — combines both results and answers the user via Gemini

Each node is wrapped in its own OpenTelemetry span:

def weather_tool_node(state):
    with tracer.start_as_current_span("weather_tool") as span:
        span.set_attribute("tool.name", "weather_tool")
        span.set_attribute("tool.input", "Coimbatore")

        should_fail = FORCE_FAIL or (random.random() < 0.3)
        if should_fail:
            span.set_attribute("tool.status", "failed")
            span.record_exception(Exception("Simulated weather API timeout"))
            state["weather_result"] = "ERROR: weather API timed out"
        else:
            span.set_attribute("tool.status", "success")
            state["weather_result"] = "32°C, clear skies"

        return state

Getting the traces to actually reach SigNoz took another round of trial and error. I first pointed the exporter at the gRPC endpoint on port 4317, and it kept failing with a TLS handshake error — SSL_ERROR_SSL: WRONG_VERSION_NUMBER. Switching to the HTTP exporter on port 4318 instead fixed it immediately:

exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")

I also hit two separate 404s from the Gemini API because gemini-1.5-flash and then gemini-2.5-flash had both been deprecated since I last checked model names. Switching to the gemini-flash-latest alias solved it — pointing at "latest" instead of pinning a specific version turned out to be the safer move.

Watching it actually work

With all of that sorted, running the script sent real traces into SigNoz, and the waterfall view showed exactly what I'd hoped: agent_run as the parent span, with weather_tool, calculator_tool, and llm_call nested underneath, each with its own duration.

llm_call took up about 64% of the total execution time. I'd assumed the LLM call would be the slowest step, but seeing the actual percentage broken out was more convincing than just guessing.

Breaking it on purpose

Next I forced the weather_tool to fail, to see how SigNoz would surface it. The span showed up with the exception recorded under its Events tab — exception.message: Simulated weather API timeout, full type and stack trace, timestamped to the millisecond.

Exactly what I expected. So I turned on SigNoz's "Highlight errors" toggle and filtered by has_error = true, expecting this trace to show up immediately.

Zero results.

At first I thought I'd broken something on my end. Then I realized what was actually going on: span.record_exception() logs the exception as an event attached to the span — it does not automatically change the span's status to error. In my setup, the has_error = true filter didn't match spans where I'd only called record_exception(). Nothing in the quick-start docs I'd read made that distinction obvious ahead of time.

Interestingly, even with that gap, the agent itself handled the failure gracefully — when I asked it for the weather, it told me the weather API had timed out but still gave me the temperature conversion from the calculator tool. The failure was real, but the agent degraded instead of crashing outright.

Closing the loop

Before fixing anything, I wanted a real number, not just one missed trace. I forced weather_tool to fail 8 times with the original code, then filtered has_error = true across all of them.

0 out of 8 showed up.

I then added the missing line to weather_tool_node:

from opentelemetry.trace import Status, StatusCode

# inside the failure branch, alongside record_exception():
span.set_status(Status(StatusCode.ERROR, "Simulated weather API timeout"))

Same test, 5 more forced failures, same filter.

5 out of 5 showed up this time, each one correctly listed with its timestamp and duration.

That's the actual size of the gap: recording an exception alone gave me a 0% detection rate under the error filter; adding one explicit status call brought it to 100%, on the exact same underlying failure.

Comparing steps with a quick dashboard

Once I had a few runs recorded, I built a small dashboard panel to compare average duration across all four spans instead of reading it off one trace at a time:

Across multiple runs, weather_tool and llm_call were consistently the two heaviest steps, with calculator_tool barely registering by comparison. That matched what I'd noticed in individual traces, but seeing it aggregated across runs made the pattern obvious in a way a single waterfall view doesn't.

Setting up one alert

The last SigNoz feature I wanted to actually touch, not just read about, was alerting. I set up a traces-based alert rule with the same filter as before — service.name = 'hackathon-agent' AND has_error = true, using count() — configured to fire when the count goes above 0 at least once in a rolling 5-minute window.

I didn't wire up a real notification channel for this local demo, so the rule exists and evaluates correctly, but nothing actually gets delivered anywhere yet. Looking at the rule's own history chart, I could see it clearly picking up the spike from my earlier forced-failure runs and crossing the threshold line exactly when expected.

What I actually learned

Recording an exception on a span and marking a span as errored are two separate things in OpenTelemetry — if you want your errors to show up in error-based filters or alerts, you need to explicitly set the span status.
WSL's default memory allocation (1.8GB) is nowhere near SigNoz's documented 4GB minimum, and Docker fails silently with signal: killed rather than telling you it's a memory problem.
Model names for hosted LLMs go stale fast. Pointing at -latest aliases instead of specific version strings saved me from repeating the same 404 error twice.
The waterfall view genuinely changes how you reason about where time goes in a multi-step agent — my assumption about which step was slowest and the actual measured percentage weren't the same thing.

Takeaway

Most of tonight was small, unglamorous debugging — a memory limit, a deprecated model, a wrong port — and none of that is what I expected to be writing about going in. But the one thing that actually taught me something about observability was the smallest, quietest bug of the night: a filter that should have caught my error, and silently didn't. That gap between "the data exists" and "the data is queryable the way you'd assume" is, I think, the entire reason to build something small and break it yourself instead of just reading the docs.

I started the evening trying to learn SigNoz. I ended it with a much better understanding of observability — and a working prototype I'll keep building on for the actual hackathon project.

DEV Community: S.SKANDHA ROSHAN

I Traced My First AI Agent With SigNoz — and Caught a Bug Hiding in Plain Sight