DEV Community

Eshan Das
Eshan Das

Posted on

Observability should be a `git diff`, not a weekend: instrumenting an AI app with one command using SigNoz

#ai #opensource #monitoring #agents

Built for the Agents of SigNoz hackathon — Track 1, AI & Agent Observability.

Repo: https://github.com/Eshan276/signoz_hackathon
Install: curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz_hackathon/main/install.sh | sh

The problem: flying blind

The hackathon brief names it perfectly:

AI agents are chaining LLM calls, invoking tools, hitting vector DBs, and making decisions autonomously. But when latency spikes, costs explode, or an agent hallucinates in production, you're flying blind. You can't debug what you can't see.

Here's the thing though — the telemetry to not fly blind already exists. OpenTelemetry, OpenLLMetry, SigNoz's LLM dashboards: the pieces are all there. The reason most teams stay blind isn't that it's impossible. It's that wiring it all up is a weekend of yak-shaving: auto-instrumentation for each language, LLM and vector-DB spans, a collector, cost math, dashboards — all before the first span shows up.

So I built signoz-init: a Go CLI that turns that weekend into one command.

signoz-init init .
Enter fullscreen mode Exit fullscreen mode

Point it at any Docker Compose stack. It detects each service, wires up full OpenTelemetry into SigNoz — HTTP, database, LLM and vector-DB spans with token counts and per-request cost in dollars — and confirms the telemetry actually arrived. Without touching your docker-compose.yml or a single line of application code.

The pitch: observability should be a git diff, not a weekend.

What you get

Here's the trace from a single request through a RAG service, produced with zero application code changes:

web:POST /ask                      ← Node/Express gateway
└─ api:POST /ask                   ← FastAPI (root server span)
   ├─ api:qdrant.search            ← vector DB
   └─ api:chat gpt-4o-mini         ← LLM call
      ├─ gen_ai.usage.input_tokens : 77
      ├─ gen_ai.usage.output_tokens: 25
      └─ gen_ai.usage.cost_usd     : 0.0000856   ← in dollars, not tokens
Enter fullscreen mode Exit fullscreen mode

(Insert screenshot: the trace waterfall in SigNoz's Traces explorer, LLM span expanded.)

That last line is the whole idea. SigNoz tracks token counts natively, but it does not compute dollar cost — its own sample dashboards scale tokens by hand. signoz-init emits gen_ai.usage.cost_usd from an editable pricing table, filling a real gap.

How it works: five visible phases

The CLI runs five phases, and every one is something you can see and confirm:

Phase What happens
Detect Parses docker-compose.yml, classifies each service by layered signals — build-context manifests → image name → command → ports. Reports unknown honestly rather than guessing.
Confirm Shows a detection table. Low-confidence guesses are flagged, not stated as fact.
Generate Writes docker-compose.override.yml + .signoz/ assets. Shows a full diff first.
Apply Rebuilds and restarts the stack.
Verify Polls SigNoz until your services report, then prints what actually arrived.

That last phase is what makes it a product rather than a YAML generator. Anything can write config and hope. Confirming telemetry landed is the difference:

✓ 2 services reporting, 168 spans

  api  123 spans
    POST /ask, qdrant.search, chat gpt-4o-mini
  web  45 spans
    POST /ask, tcp.connect

  7 LLM spans with token counts
  7 spans with cost attribution
Enter fullscreen mode Exit fullscreen mode

The zero-touch injection trick

The magic is getting instrumentation into a running container without editing code:

  • Node is genuinely zero-touch: NODE_OPTIONS=--require /otel/otel-bootstrap.js plus env vars, all in the override file.
  • Python mounts a sitecustomize.py on PYTHONPATH. Python auto-imports sitecustomize at interpreter startup, before your app — the same mechanism the OpenTelemetry Operator uses for Kubernetes auto-injection. It installs the instrumentation and the cost processor.

Everything lands in docker-compose.override.yml, which Compose merges automatically. Reverting the entire thing is a single rm docker-compose.override.yml.

Where SigNoz comes in

SigNoz is the observability backend the whole tool targets, and I leaned on it at every layer:

Install via Foundry. SigNoz self-hosts through foundryctl and an 8-line casting.yaml (committed in the repo, so judges can re-run it). UI on :8080, OTLP on :4317/:4318, with its own bundled OTel collector.

Traces. Every instrumented request becomes a distributed trace in SigNoz spanning web → api → qdrant → LLM, viewable as a full waterfall in the Traces explorer.

Cost & tokens. By emitting canonical gen_ai.* attributes, SigNoz's built-in LLM views and Cost Meter light up for free — and I ship a custom 12-widget dashboard imported through SigNoz's dashboards API: cost over time, tokens by model, cost by service, LLM p95 latency, vector-search latency.

(Insert screenshot: the LLM Cost dashboard.)

Verification. The CLI queries SigNoz to prove spans landed — ClickHouse directly for self-hosted (no credentials needed), or the /api/v2/services API for SigNoz Cloud.

The first-run gotcha SigNoz taught me

The single most valuable thing I learned: "SigNoz is running" is not the same as "SigNoz can accept telemetry."

Freshly cast, SigNoz's ingester logs cannot create agent without orgId and never opens its OTLP ports — because the OpAMP server won't hand it a config until an organization exists, which only happens after first-run signup. Connections are refused, curl returns exit 56, and it looks exactly like a networking bug. It isn't. You just have to register the first admin, and OTLP opens within ~30 seconds.

This is a product requirement, not just my local workaround. So signoz-init detects the no-org state and guides you through it — because otherwise the first thing a new user sees is a silent black hole.

Going beyond cost: RAG quality and per-conversation rollups

Cost and latency tell you that something is wrong, not what. So I added two signals that directly attack the "an agent hallucinates in production" line from the brief — signals auto-instrumentation fundamentally cannot produce, because only the application knows them.

Groundedness — a cheap hallucination proxy

gen_ai.response.groundedness is a 0–1 score of how much of the model's answer actually came from the retrieved context. It's a deliberately cheap lexical heuristic — no second LLM call, no API key, no added latency — but it catches the failure that matters: an answer that wandered off the retrieved chunks.

def groundedness(answer, sources):
    answer_tokens = meaningful_words(answer)
    source_tokens = union(meaningful_words(s) for s in sources)
    return len(answer_tokens & source_tokens) / len(answer_tokens)
Enter fullscreen mode Exit fullscreen mode

A grounded answer scores ~1.0; an answer drawing on parametric knowledge scores ~0. It rides the same trace as the LLM span and rolls up natively in SigNoz — average groundedness, groundedness over time, the works.

Per-conversation rollups

session.id on every span means cost, latency, and groundedness aggregate per conversation, not just per call. That turns "this request cost $0.00004" into "this conversation cost $0.40" — the number a budget owner actually cares about.

This one taught me a subtle lesson (see gotcha #10 below): cost lives on the LLM span, but the session id starts on the request span. To make "cost by conversation" work, a span processor has to stamp the session onto every span in the request. I only caught that the dashboard widget was silently empty because I tested the actual query, not just the happy path.

Ten things that cost me real debugging time

This is the part I'm proudest of, and the best evidence the tool is worth building — because a normal developer hits these and gives up. Every one is verified, hands-on:

  1. TRACELOOP_BASE_URL routes traces, not the api_endpoint kwarg. Traceloop.init(api_endpoint=...) alone exports nowhere, silently.
  2. OpenLLMetry does NOT instrument web frameworks. It covers LLM providers, vector DBs, and HTTP clients — but not FastAPI. Without the framework instrumentor there's no root span and every LLM span is an orphan.
  3. Don't double-instrument. Re-instrumenting what Traceloop already patched breaks the existing patch — and silently killed my Qdrant spans.
  4. Traceloop's AI instrumentor init swallows every exception. Qdrant never got patched, with zero logging. Fix: call the instrumentors explicitly.
  5. async def + run_in_threadpool loses OTel context → orphaned spans. A plain def endpoint works, because Starlette copies contextvars. Counter-intuitive, verified both ways.
  6. Cost can't be attached in SpanProcessor.on_end — the exporter has already read the attributes, so it exports as 0. You have to wrap the exporter.
  7. BatchSpanProcessor.span_exporter is read-only. Assign through _batch_processor._exporter.
  8. Rebuild, don't restart. The Dockerfiles COPY source in, so docker compose restart runs stale code. Several confusing results traced back to this.
  9. OpenLLMetry's Qdrant instrumentor wraps search but NOT query_points — the modern API produces no span. Also: build clients lazily, or they capture unpatched methods.
  10. Traceloop never sets the global tracer provider. So trace.get_tracer() returns a proxy and every hand-written span is a NonRecordingSpan that silently vanishes. Instrumented libraries emit spans while your own code emits nothing, with no error.

Every one of these is now documented and handled in the tool, so its users never have to learn them.

Try it 

# install (no Go needed — prebuilt binary)
curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz_hackathon/main/install.sh | sh

# stand up SigNoz once (foundryctl), then:
signoz-init init ./demo
Enter fullscreen mode Exit fullscreen mode

The demo is a real RAG stack — Node gateway → FastAPI → Qdrant → LLM — and it runs with or without an API key. No key uses a mock LLM that still emits proper gen_ai.* spans with realistic tokens, so anyone can reproduce the full pipeline, cost included. With a key it calls a real model — and because it uses the OpenAI SDK, it works against any OpenAI-compatible endpoint (I verified it against Gemini via a single LLM_BASE_URL change).

~2,600 lines of Go, 16 tests green, one command from blind to observable.

Observability should be a git diff, not a weekend.

Built with SigNoz for the Agents of SigNoz hackathon. Repo: https://github.com/Eshan276/signoz_hackathon

Top comments (0)