Debugging the Black Box: My First Week With SigNoz

I spent weeks building a 4-step LLM agent (plan → retrieve → generate → judge) and then realized I had zero visibility into why it was failing or costing money. I had been building this agent for a little while before I really understood why "just add logging" doesn't cut it. When something went wrong, I had no way to answer the most basic question: which step actually broke? Was the LLM slow, or was my retrieval step the bottleneck? Was I paying for one LLM call or six because of a silent retry loop? I was flying blind in exactly the way observability tools exist to fix; I just hadn't set one up yet.

So, before starting my actual hackathon build, I spent an afternoon self-hosting SigNoz and wiring it into a small FastAPI service, mostly to answer one question: is this actually going to help me see inside my agent, or is it just another dashboard I'll ignore?

Setting it up (and hitting the first wall immediately)

I went with self-hosted SigNoz—no cloud dependency, full control, and I wanted to be sure I could reproduce my own setup later. My first mistake was following an old mental model of the install process. As of v0.130.0, the old docker-compose.yaml paths are deprecated in favor of a new CLI called Foundry.

The actual process is now much cleaner:

curl -fsSL [https://signoz.io/foundry.sh](https://signoz.io/foundry.sh) | sh

cat > casting.yaml <<'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

foundryctl cast -f casting.yaml

A couple minutes later: five containers were up, and the UI was live at localhost:8080.

Wiring up the first trace
Instrumenting a FastAPI service turned out to be genuinely three pieces of code, not the sprawling configuration I expected:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(tracer_provider)
FastAPIInstrumentor.instrument_app(app)

Every request my app handled immediately showed up as a trace, with zero manual span creation for the HTTP layer. That part was easy. The part that took real debugging was getting cost to show up as a first-class metric, not just a log line.

Where it actually got interesting: custom attributes
Auto-instrumentation gives you HTTP-level visibility for free, but it doesn't know about LLM costs. The moment I cared about was adding a manual span around my agent's decision logic and attaching custom attributes:

with tracer.start_as_current_span("agent.generate") as span:
    span.set_attribute("gen_ai.usage.cost_usd", cost)
    span.set_attribute("gen_ai.usage.input_tokens", usage.input_tokens)
    span.set_attribute("gen_ai.request.model", model_name)

Caption: Full trace view with custom gen_ai attributes and hallucination score.

Watching gen_ai.usage.cost_usd show up as a searchable, filterable field on the span—not buried in a log string—was the moment this stopped feeling like logging and started feeling like observability. I could click into any single request and see exactly what it cost, which model handled it, and how long each step took, all correlated by one trace_id.

The dumb bug that taught me the most
While building a dashboard panel to chart that cost metric over time, I hit something that looked broken but wasn't. My cost values are tiny—around $0.0001 per request—and SigNoz's panel decimal precision only goes up to three decimal places. 0.0001 rounded to 0.000 on the chart, so the line just sat flat at zero. The fix wasn't a SigNoz bug, it was a unit-scaling problem: I added a formula (A * 1000000) to the query builder to convert the metric into millionths of a dollar before charting it.

Caption: SigNoz Time Series showing agent step latencies.

I also learned that similarity_search_with_relevance_scores in my retriever doesn't always return values between 0 and 1; I saw negative numbers in my logs. It was harmless for my use case, but it served as a reminder to actually read library warnings instead of assuming output is always well-behaved.

Setting Up Alerts for the Ops Console & Self-Healing
This initial exploration was just the warm-up. For the main hackathon build, I focused on building a highly reactive self-healing loop. I created a custom alert rule targeting the gen_ai.usage.cost_usd attribute. When the cost breached the threshold, SigNoz triggers a webhook to my FastAPI backend.

@app.post("/webhooks/signoz-alert")
async def signoz_alert_webhook(request: Request):
    payload = await request.json()
    alerts = payload.get("alerts") or [payload]
    triggered = []
    for a in alerts:
        labels = a.get("labels", {}) if isinstance(a, dict) else {}
        alert_name = labels.get("alertname") or a.get("ruleId") or str(a)[:100]
        self_healing.handle_alert(alert_name)
        triggered.append(alert_name)
    return {"status": "ok", "triggered": triggered, "state": asdict(self_healing.get_state())}

Caption: Self-healing Ops Console after injecting cost chaos.

My ultimate goal is for the agent to react seamlessly to its own telemetry — catching an unexpected latency spike or cost explosion and downgrading itself to a cheaper, lighter model mid-session without human intervention. Wrestling with webhook payload verification and avoiding cyclic retry loops was challenging, but seeing the event stream sync up proved the architecture worked.

What I'd tell someone setting this up today
Use Foundry: Do not use the old docker-compose path. Check the SigNoz repo's current docs.

Go beyond auto-instrumentation: The HTTP spans tell you that something happened; custom span attributes on your business logic (cost, tokens, model name) are what let you debug why.

Build dashboards early: Creating a panel early surfaces unit and precision issues that you won't notice just staring at raw trace data.

If you're setting up SigNoz for the first time, budget about 10 minutes from zero to your first trace. This project is part of my submission for the Agents of SigNoz Hackathon by WeMakeDevs. Observability-powered self-healing agents are incredibly powerful, and I'm excited to push this setup to its absolute limit.