I Traced a Multi-Step LLM Agent With Self-Hosted SigNoz. One Feature Sold Me.

Multi-step LLM agents fail in a way normal backends don't. Nothing crashes. The pipeline "works", the answer is just bad, slow or three times more expensive than yesterday. print() debugging tells you nothing, because the interesting question is never "did step 3 run". It is "what did step 3 see, which model actually answered and what did it cost".

So I self-hosted SigNoz and pointed a simulated agent pipeline at it: a four-step research assistant (plan, retrieve, generate, synthesize) instrumented with OpenTelemetry, emitting traces, metrics and logs, with GenAI semantic-convention attributes (gen_ai.request.model, gen_ai.usage.input_tokens and friends) on every LLM call.

This post is about the feature that turned out to be the most useful. It was not the one I expected.

Setup: one CLI, eight lines of YAML

Self-hosting used to mean wrangling a long docker-compose file. SigNoz now ships Foundry, a small CLI that casts the whole stack:

curl -fsSL https://signoz.io/foundry.sh | bash   # installs foundryctl (checksum-verified)
foundryctl cast -f casting.yaml                  # deploys the full stack on Docker

with a casting.yaml that is all of eight lines:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

A few minutes of image pulls later, ClickHouse, Postgres, the SigNoz backend and an OTel collector were running. The UI is on localhost:8080 and the collector listens on 4317 (gRPC) and 4318 (HTTP).

One gotcha worth knowing: telemetry is rejected until you create the admin account in the UI. The collector registers with the backend over OpAMP, and until an organization exists the backend answers "cannot create agent without orgId" and the OTLP ports reset every connection. If your exporter logs Connection reset by peer on a fresh install, you haven't finished the two-minute signup at localhost:8080 yet. Create the account and ingestion starts working within about thirty seconds, no restarts needed.

My demo app needed zero SigNoz-specific code: the stock OpenTelemetry SDK exporting OTLP to localhost:4318. That is the point of an OTel-native backend. There is no vendor agent, so nothing about the app knows SigNoz exists.

The pipeline being observed

Each simulated request produces one trace:

agent.request                    (root, 2.02s)
├── agent.plan                   (850ms)
│   └── gen_ai.generate plan     (696ms, gen_ai.* attributes)
├── agent.retrieve               (461ms, sometimes errors: vector store timeout)
├── agent.generate               (564ms)
│   └── gen_ai.generate answer   (564ms, gen_ai.* attributes)
└── agent.synthesize             (139ms, sometimes errors: citation validation)

Every gen_ai.generate span carries the GenAI semantic conventions: operation name, provider, model, input tokens and output tokens. Alongside the traces the app emits counters for token usage (gen_ai.client.token.usage) and estimated spend (agent.llm.cost), both tagged by model and provider, a request-duration histogram and structured logs that inherit the active trace context automatically.

Sixty simulated requests later: 420 spans, 134 logs and a few hundred metric samples, all visible in the UI. The Services page picked up research-assistant on its own with RED metrics already computed (p99 latency, error rate, throughput). I wrote no configuration for that.

The feature I expected to love: the trace waterfall

And it is genuinely good. The trace detail view renders a flame graph and waterfall that read exactly like the agent's mental model: plan, then retrieve, then generate, then synthesize, with the LLM calls nested inside the steps that made them. Clicking any gen_ai.generate span opens a details panel with every attribute I set: model llama-4-maverick, provider meta, 1501 input tokens, 90 output tokens, plus a percentile badge telling me this span sat at p10 of its peers. "Why did this request take four seconds" stops being a mystery and becomes a picture.

But a pretty waterfall is table stakes for a tracing tool. The thing that sold me was what happens around it.

The feature I actually loved: your attributes become the query language

Here is the moment it clicked. In the Trace Explorer I typed:

gen_ai.request.model = 'qwen3-coder-plus'

Two things happened. First, the autocomplete suggested qwen3-coder-plus before I finished typing, because SigNoz had already indexed the values of an attribute I invented an hour earlier. Second, the results came back instantly: only the LLM spans that were served by that model, across every trace in the system.

Stop and consider what that means for agent debugging. I never told SigNoz what gen_ai.request.model is. There is no schema registration, no field mapping, no config file. Any attribute your instrumentation emits is immediately a first-class, autocompleted, indexed query dimension. Your instrumentation vocabulary becomes the product's vocabulary. For LLM systems, where all the interesting facts live in custom attributes (model, provider, token counts, agent role, tool name), this is the difference between an observability tool that fits and one you fight with.

And the same query keeps working as you move across signals:

• Span to logs, one click. From any span's details panel, the Logs tab jumps into the Logs Explorer with trace_id = '<this trace>' pre-filled and the time window auto-scoped. I got back exactly the two log lines belonging to that request, correlated purely by the trace context the OTel logging handler injects. Nobody parses log lines to find a request id. The correlation is structural.
• Query to dashboard or alert, two clicks. Every explorer view has "Add to Dashboard" and "Create an Alert" buttons at the bottom. The query I used to investigate becomes the panel that monitors and the alert that pages, without re-expressing it in a different query language. The investigate-then-monitor loop is one surface, not two tools.
• Metrics inherit the same attributes. The Metrics Explorer summary listed my custom metrics with their types and units the moment they arrived: gen_ai.client.token.usage showing 6 time series (3 models times 2 token types), agent.llm.cost in usd showing 3. Cost per model is a group-by away, using the exact attribute names from my instrumentation.

Most observability stacks treat custom attributes as second-class blobs that need schema work before they are queryable. SigNoz treats them as the whole point. For agent systems, they are.

What else is in the box

Going deep on one feature meant walking past a lot of others. Quick notes from the tour, agent-flavored:

• Services (APM) pages, zero config. Any service that sends spans gets RED metrics automatically: request rate, error rate, latency percentiles. In a multi-agent system every agent that traces becomes its own monitored service for free.
• Dashboards. Panel grids built with the same query builder, so traces, logs and metrics coexist on one board, importable and exportable as JSON. A "cost per model per agent" board takes minutes because the query language is the one you already know.
• Alerts with notification channels. Threshold or anomaly rules on any query, delivered to Slack, PagerDuty or a plain webhook. That webhook option is quietly powerful for agent systems: an alert can call your own service and close the loop from observability back into behavior, e.g. demoting a flaky model.
• Trace Funnels (beta). Define a sequence of span conditions and measure conversion and latency between the steps across all traces. For a pipeline like mine that is literally a success funnel: how many requests that planned also retrieved, generated and synthesized cleanly.
• Exceptions. Recurring span errors cluster into groups instead of being scattered across individual traces, so "vector store timeout" is one row with a count, not forty needles in a haystack.
• Logs Pipelines. Collector-side log pre-processing (parse JSON, extract or drop fields) before storage, for when you can't change the code that emits the logs.
• Metrics summary as a cardinality watchdog. The same inventory view that showed my metric types also shows time-series counts per metric, which is where you catch a label-explosion mistake before it hurts.

Each of these uses the attribute vocabulary your instrumentation defines. That is the theme of the whole product, and it is why the one feature I picked is really the foundation the rest stand on.

What this means if you're building agents

The uncomfortable truth about multi-agent LLM systems is that their most important behavior (which model ran, what it consumed, what it cost, why it was retried) is invisible to conventional monitoring. It lives entirely in domain-specific span attributes. A backend that makes those attributes instantly queryable, correlatable across traces and logs, and promotable into dashboards and alerts is not a nice-to-have there. It is the debugger.

The whole experiment cost me an afternoon: one CLI install, eight lines of YAML, a stock OTel SDK and zero vendor code in the app. I'm taking this setup into the Agents of SigNoz hackathon (July 20 to 26), where the plan is considerably less simulated. If you're building anything agent-shaped, self-host SigNoz and type one of your own attribute names into the Trace Explorer. That autocomplete dropdown is the moment you'll get it too.

---

The demo app is a 170-line Python script using only opentelemetry-sdk and the OTLP HTTP exporter. Stack: SigNoz self-hosted via Foundry on Docker (ClickHouse + Postgres + OTel collector), macOS host.

Demo code, self-host config and blog source: github.com/himanshu748/signoz-agent-observability. Team 404Found.