DEV Community

Manpreet Singh
Manpreet Singh

Posted on

Agents of SigNoz

Section 1: The Observability Gap in Autonomous Frameworks

Agentic systems built on frameworks like LangChain or CrewAI do not execute as single linear requests. A single user prompt can trigger multiple asynchronous sub-processes: parallel tool calls, recursive self-correction loops, and delegation to sub-agents that each make independent LLM calls.

Traditional structured logging assumes a request maps to a single execution path with a predictable start and end. Agent execution violates this assumption in three specific ways:

  • Asynchronous execution loops. An agent can re-invoke a tool or re-query an LLM multiple times within a single logical "turn," with no fixed upper bound on iteration count. Logs record each call as an isolated event with no inherent link back to the originating decision.
  • Multi-hop sub-agent delegation. When one agent hands a task to another agent, the resulting call chain spans multiple processes or services. Without a shared execution context, there is no way to associate a sub-agent's actions with the parent task that spawned them.
  • Variable token cost and latency per step. LLM calls do not have fixed cost or duration. A retry, a longer context window, or a larger tool response can each independently spike cost or latency. Aggregate logs show total cost or total duration, but not which specific step in the chain caused the deviation.

The result: standard logs can confirm that an agent ran, but cannot answer which step failed, which step looped, or which step drove cost. This requires a data model built around causality and hierarchy, not just timestamped events. That data model is distributed tracing.

Section 2: System Architecture

The following flow shows how agent telemetry moves from generation to storage to query, using a single vertical execution path.

┌─────────────────────────────┐
│      Agent Runtime          │
│  (LangChain / CrewAI, etc.) │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│   OpenTelemetry SDK          │
│   (span creation, context     │
│    propagation, attributes)  │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│  OTLP Exporter (gRPC/HTTP)   │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│  OpenTelemetry Collector      │
│  (receive, batch, process)   │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│      ClickHouse               │
│  (columnar store: traces,     │
│   logs, metrics)               │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│     SigNoz Dashboards          │
│  Trace Explorer / Logs /       │
│  Metrics / Service Map          │
└─────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

OTLP collection layer. The OpenTelemetry Collector is a standalone process that receives telemetry over the OTLP protocol (gRPC or HTTP), batches it, and forwards it to a storage backend. It is decoupled from the application: the agent process only needs to know the Collector's endpoint, not any details of the downstream storage engine. In a self-hosted SigNoz deployment, the Collector listens on port 4317 for OTLP/gRPC and port 4318 for OTLP/HTTP.

ClickHouse as the storage engine. SigNoz uses ClickHouse, a columnar database, to store traces, logs, and metrics. Columnar storage is suited to observability workloads because queries typically aggregate over a small number of fields (duration, status, service name) across a large number of rows, rather than reading full rows. This gives SigNoz fast query performance on high-cardinality trace data, which is relevant given the volume of spans an agent with many sub-steps can generate.

Trace-to-log correlation. Every span generated by the OpenTelemetry SDK carries a trace ID and span ID. If the agent's logging output is also instrumented to include these IDs, SigNoz can associate log lines with the exact span during which they were emitted. This allows a query to start from a specific span in a trace (for example, a slow tool call) and retrieve only the log lines emitted during that span's execution window, rather than searching logs by timestamp across the entire trace.

Section 3: Setting Up Self-Hosted SigNoz

Self-hosted SigNoz is currently installed and managed through Foundry, a CLI (foundryctl) that generates and deploys the full stack from a single declarative configuration file. The previous install.sh script and the bundled deploy/ Docker Compose files are deprecated as of SigNoz v0.130.0 and are no longer maintained.

Install foundryctl:

curl -fsSL https://signoz.io/foundry.sh | bash
Enter fullscreen mode Exit fullscreen mode

Create a minimal casting.yaml for a single-machine Docker Compose deployment:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
Enter fullscreen mode Exit fullscreen mode

Deploy the stack:

foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

This single command validates prerequisites, renders the Docker Compose files into a pours/ directory, and starts the containers, which include SigNoz, the OpenTelemetry Collector, ClickHouse, ClickHouse Keeper, and PostgreSQL.

Once running, the deployment exposes:

  • Port 8080 — the SigNoz UI (Trace Explorer, Logs, Metrics, Service Map)
  • Port 4317 — OTLP gRPC ingestion
  • Port 4318 — OTLP HTTP ingestion

If you prefer to manage the containers directly instead of letting cast handle deployment, the equivalent two-step process is:

foundryctl gauge -f casting.yaml   # validate prerequisites
foundryctl forge -f casting.yaml   # generate compose files
cd pours/deployment && docker compose up -d
Enter fullscreen mode Exit fullscreen mode

At least 4GB of memory allocated to Docker is required. Once the containers are healthy, the UI is reachable at http://localhost:8080, and your agent's OTLP exporter should point at http://localhost:4317.

Section 4: Trace Tree Causality

A single agent turn produces one root span with child spans representing each sub-step. Below is the flat causal breakdown for one example execution.

Trace ID: 7f3a9c21
Root Span: user_prompt_handler          duration: 4.20s

Span: llm_call_planning                  duration: 0.82s
  attributes: tokens_in=512, tokens_out=128

Span: vector_db_search                   duration: 0.14s
  attributes: top_k=5, collection=docs_v2

Span: agent_tool_execution               duration: 2.10s
  attributes: tool_name=web_search, status=success

Span: http_request_external              duration: 1.90s
  parent_span: agent_tool_execution
  attributes: status_code=200

Span: llm_call_final_answer               duration: 1.10s
  attributes: tokens_in=2048, tokens_out=340
Enter fullscreen mode Exit fullscreen mode

Each span records its own duration and attributes independently. The parent_span field establishes the hierarchy without requiring nested formatting: http_request_external is a child of agent_tool_execution, meaning the external HTTP call occurred as part of that tool's execution and its duration is included within the tool span's total duration. This structure allows direct identification of which specific span accounts for the majority of total trace duration or cost, rather than inferring it from aggregate values.

Section 5: Conceptual OpenTelemetry Implementation

The example below initializes an OpenTelemetry tracer, configures it to export to a local SigNoz OTel Collector on port 4317, and wraps each sub-step of an agent's execution — retrieval, tool call, and generation — in its own span.

# filename: agent.py
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# 1. Setup OpenTelemetry Resource (The Service Name)
resource = Resource.create(attributes={
    "service.name": "agent-researcher-v1",
    "service.version": "1.0.0"
})

# 2. Configure the Tracer to send data to SigNoz (Localhost)
trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer("agent.main")

# Send traces to SigNoz OTel Collector on port 4317
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

def run_agent_task(user_prompt):
    # Start a root span for the entire user interaction
    with tracer.start_as_current_span("agent_execution_workflow") as span:
        span.set_attribute("user.prompt", user_prompt)

        print(f"Thinking about: {user_prompt}...")

        # Simulate a sub-step (Tool Call)
        with tracer.start_as_current_span("tool_search_vector_db"):
            # Your vector DB logic here
            span.add_event("Found 3 relevant documents")

        # Simulate LLM Generation
        with tracer.start_as_current_span("llm_generate_response"):
            span.set_attribute("llm.model", "gpt-4")
            span.set_attribute("llm.tokens_used", 150)

if __name__ == "__main__":
    run_agent_task("Why is my Kubernetes pod crashing?")
Enter fullscreen mode Exit fullscreen mode

Install the required packages before running this script:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install
Enter fullscreen mode Exit fullscreen mode

opentelemetry-distro provides the SDK and a mechanism to auto-configure common OpenTelemetry defaults. opentelemetry-exporter-otlp installs the OTLP exporters, including the gRPC exporter used in this script (opentelemetry-exporter-otlp-proto-grpc). opentelemetry-bootstrap inspects installed dependencies and adds any relevant auto-instrumentation packages for libraries already present in the environment.

Implementation notes:

  • Each tracer.start_as_current_span(...) call opens a new span under whatever span is currently active, which is what produces the parent-child hierarchy shown in Section 4.
  • span.set_attribute attaches structured, queryable metadata directly to a span — llm.model, llm.tokens_used, user.prompt — without needing to parse log text later.
  • span.add_event records a discrete point-in-time event inside a span, useful for marking a state change (such as "documents found") without opening a new span.
  • The same pattern extends to real tool dispatch logic, vector database clients, and LLM SDK calls: replace the placeholder logic inside each with block with the actual function call, and set attributes relevant to that step.
  • Frameworks such as LangChain and CrewAI expose callback interfaces that can invoke this instrumentation automatically at each execution step, without modifying core framework logic.

Section 6: Why This Matters for Agentic Workflows Specifically

SigNoz is built as an OpenTelemetry-native platform, meaning it consumes OTLP data directly with no proprietary translation layer. The core platform is open source under the MIT license, with only the ee/ directory (enterprise-only features) under a separate license. This has two direct implications for agent instrumentation:

  • No vendor lock-in. Instrumentation written with the standard OpenTelemetry SDK, as shown in Section 5, is portable to any OTLP-compatible backend. Switching backends later requires changing an exporter endpoint, not rewriting instrumentation code.
  • Full data ownership for sensitive agent data. Self-hosting keeps prompts, tool outputs, and retrieved documents inside your own infrastructure, since the Collector and ClickHouse both run on infrastructure you control.

SigNoz's more recent development has also moved directly toward agent-facing tooling: the project maintains an MCP (Model Context Protocol) server that exposes traces, logs, metrics, and service topology to coding agents such as Claude Code, allowing an agent to query production telemetry as part of its own debugging workflow. This closes the loop described in this post: not only can you trace what an AI agent does, you can also let an AI agent query that same trace data to diagnose issues directly.

Section 7: Building It in the Hackathon

Everything above is the blueprint: the trace shape, the Collector pipeline, the span design, and the exact install and instrumentation commands. The Agents of SigNoz hackathon main track is where this gets built for real — a live agent, fully instrumented, traces flowing into a self-hosted SigNoz instance, with dashboards showing exactly where latency and cost are going per tool call.

If your agents have ever run up a bill you couldn't explain, or failed quietly in a way you couldn't trace back, this is the architecture that answers those questions directly.

References

Top comments (0)