DEV Community: Uzair

Building an Automated Infrastructure Health & Monitoring Bot with OpenTelemetry and SigNoz

Uzair — Sun, 26 Jul 2026 09:59:30 +0000

Signoz track 3 submission: Build your own. Automated Infrastructure Health & Monitoring Bot 🤖

In production SRE workflows, static metric alerts often leave engineering teams scrambling to correlate spikes with root causes. Knowing that CPU hit 95% is only half the battle—you need distributed tracing to understand what process cycle triggered the spike and how the system responded in real time.

In this post, we’ll break down the Automated Infrastructure Health & Monitoring Bot, a lightweight, OpenTelemetry-native SRE monitoring agent built for Track 3: Build Your Own of the SigNoz Hackathon.

🌐 Live Demo & Repository

GitHub Repository: https://github.com/era651868-ctrl/Signoz.git
Live Deployment link: https://signoz-57js.onrender.com/
YouTube Demo: https://youtube.com/shorts/3iGRSQba2n8?si=1b0R1ZkqF9YjO6Dc
Local / Deployed Dashboard Endpoints:
- Health Endpoint: /health
- Live Metrics Stream: /metrics
- Incident Log Feed: /incidents

🏗️ System Architecture & Workflow

The bot is designed to simulate a real-world SRE monitoring agent operating in constrained environments. It continuously samples system telemetry, evaluates rule-based alert conditions, and streams OTLP traces directly to SigNoz.

+-------------------------------------------------------------------+
|               Infrastructure Health Agent (app.py)                |
|                                                                   |
|   +------------------+     +-------------------+                  |
|   | System Collector | --> | Anomaly Detection |                  |
|   | (CPU/Mem/Disk)   |     | (Threshold Engine)|                  |
|   +------------------+     +-------------------+                  |
|                                     |                             |
|                                     v                             |
|                           +-------------------+                   |
|                           |  OTLP Trace Engine|                   |
|                           +-------------------+                   |
+-------------------------------------|-----------------------------+
                                      | OTLP gRPC (Port 4317)
                                      v
                        +---------------------------+
                        |  SigNoz OTEL Collector    |
                        |  & Telemetry Store        |
                        +---------------------------+

💻 Core Implementation Deep Dive

1. Resilient OpenTelemetry Setup with Graceful Fallback (`app.py`)

To ensure the monitoring agent never crashes when network connectivity drops or during collector maintenance, the setup dynamically tests OTLP gRPC endpoint reachability before registering exporters.

import os
import socket
from urllib.parse import urlparse
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

SERVICE_NAME = os.getenv("SERVICE_NAME", "health-alert-bot")
OTLP_ENDPOINT = os.getenv("OTLP_ENDPOINT", "http://localhost:4317")

def is_otlp_endpoint_reachable(endpoint: str) -> bool:
    parsed = urlparse(endpoint)
    host = parsed.hostname or "localhost"
    port = parsed.port or 4317
    try:
        with socket.create_connection((host, port), timeout=1):
            return True
    except OSError:
        return False

def setup_tracing():
    resource = Resource.create(attributes={"service.name": SERVICE_NAME})
    provider = TracerProvider(resource=resource)

    if is_otlp_endpoint_reachable(OTLP_ENDPOINT):
        exporter = OTLPSpanExporter(endpoint=OTLP_ENDPOINT, insecure=True)
        provider.add_span_processor(BatchSpanProcessor(exporter))
        print(f"📡 Tracing enabled. Sending OTLP spans to {OTLP_ENDPOINT}")
    else:
        provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
        print(f"⚠️ OTLP endpoint unreachable. Falling back to console spans.")

    trace.set_tracer_provider(provider)
    return trace.get_tracer("bot-tracer")

tracer = setup_tracing()

2. Anomaly Evaluation & Span Attribute Injection

Every monitoring cycle evaluates resource thresholds and attaches rich contextual metadata directly onto OpenTelemetry spans. Critical breaches trigger nested alert child spans.

def main():
    threading.Thread(target=start_http_server, daemon=True).start()
    print("🚀 Health Alert Bot started running...")

    while True:
        metrics = collect_metrics()
        severity, reasons = evaluate_health(metrics)

        with tracer.start_as_current_span("monitor_cycle") as span:
            # Set high-cardinality telemetry as span attributes
            span.set_attribute("system.cpu.usage", metrics["cpu_percent"])
            span.set_attribute("system.memory.usage", metrics["memory_percent"])
            span.set_attribute("system.disk.usage", metrics["disk_percent"])
            span.set_attribute("alert.threshold.cpu", CPU_ALERT_THRESHOLD)
            span.set_attribute("alert.threshold.memory", MEMORY_ALERT_THRESHOLD)
            span.set_attribute("health.severity", severity)

            # Trigger sub-span drilldown on CRITICAL events
            if severity == "CRITICAL":
                with tracer.start_as_current_span("trigger_alert") as alert_span:
                    alert_span.set_attribute("alert.level", "CRITICAL")
                    alert_span.set_attribute("alert.reason", ",".join(reasons))
                    alert_span.add_event(
                        "cpu_threshold_exceeded", 
                        {"cpu_percent": metrics["cpu_percent"]}
                    )

            print(build_status_message(metrics, severity))

        record_incident(metrics, severity, reasons)
        time.sleep(CHECK_INTERVAL_SECONDS)

3. OpenTelemetry Collector Pipeline (`pours/deployment/ingester/ingester.yaml`)

The SigNoz collector configuration receives raw OTLP gRPC spans on port 4317 and processes them via ClickHouse batch exporters:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 50000
    send_batch_max_size: 55000
    timeout: 5s

exporters:
  clickhousetraces:
    datasource: tcp://signoz-telemetrystore-clickhouse-0-0:9000/signoz_traces
    use_new_schema: true
    timeout: 45s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousetraces]

⚡ Quickstart & Testing

1. Run Unit Tests

Verify anomaly detection rules and JSONL logging:

python -m unittest discover -s tests

2. Run Containerized Deployment

Deploy using Docker or Foundry:

docker build -t health-bot .
docker run -p 8000:8000 -e OTLP_ENDPOINT=http://host.docker.internal:4317 health-bot

📈 Key Learnings & SigNoz Value

Tracing Over Static Logging: Injecting CPU, memory, and disk utilization attributes into OpenTelemetry spans converts raw infrastructure spikes into searchable trace queries within SigNoz.
Resilient Instrumentation: Building automatic fallback mechanisms (OTLP to Console) guarantees that monitoring tools continue operating even during local collector outages.
Full Reproducibility: Providing valid casting.yaml and ClickHouse configurations allows teams to deploy self-hosted observability stacks in minutes.

SignalForge: Building a Real-Time Agent Control Tower with OpenTelemetry and SigNoz

Uzair — Sat, 25 Jul 2026 17:54:50 +0000

Building SignalForge: End-to-End Telemetry & Observability with OpenTelemetry and SigNoz

In autonomous AI agent systems and modern microservice architectures, knowing what happened isn't enough—you need to know when, why, and how fast it happened.

In this post, we'll walk through SignalForge Command Center, a lightweight Node.js/Express telemetry dashboard designed to simulate agent trace streams, calculate real-time Service Level Objectives (SLOs), and export standard OpenTelemetry (OTEL) data to SigNoz.

🌐 Live Demo & Deployment

The application is deployed and live! You can inspect the command center directly:

Deployed Web Dashboard: https://signoz-track2-production.up.railway.app/
YouTube Demo: https://youtube.com/shorts/5ffMjV5UgD0?si=-lJHE952wvnctBWk

🏗️ Architectural Overview

SignalForge consists of four core building blocks:

Express Server (server.js): Serves the REST API and dashboard UI, while maintaining a rolling in-memory telemetry store and simulating autonomous agent metrics.
OpenTelemetry SDK (otel-example.js): Configures auto-instrumentation for Express and HTTP requests, exporting traces via the OpenTelemetry Protocol (OTLP).
OpenTelemetry Collector (otel-config.yaml): Receives, processes, and batches trace/metric streams before routing them to SigNoz Cloud.
Interactive Dashboard (public/index.html & app.js): Polls backend APIs every 5 seconds, displaying real-time signal confidence, p95 latencies, and incident simulation controls.

💻 Key Implementation Files

1. OpenTelemetry SDK Setup (`otel-example.js`)

This script initializes the NodeTracerProvider and attaches auto-instrumentation modules for HTTP and Express.

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

// Configure resource attributes identifying this service
const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'signalforge-command-center',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
  }),
});

// Setup OTLP Exporter targeting your collector or SigNoz endpoint
const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

// Enable automatic HTTP & Express tracing
registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

console.log('OpenTelemetry instrumentation successfully initialized.');

2. Express Backend & Telemetry Simulator (`server.js`)

The main entry point manages telemetry aggregation, calculates active SLO metrics, and provides incident simulation endpoints.

// Enable OTEL auto-instrumentation if flag is set
if (process.env.ENABLE_OTEL === 'true') {
  require('./otel-example');
}

const express = require('express');
const path = require('path');

const app = express();
const PORT = process.env.PORT || 3000;

app.use(express.json());
app.use(express.static(path.join(__dirname, 'public')));

// In-memory store for simulated telemetry events
let telemetryStore = [];
let activeIncident = false;

// Generate simulated telemetry every 5 seconds
function buildTelemetryEvent() {
  const baseLatency = activeIncident ? Math.floor(Math.random() * 300) + 250 : Math.floor(Math.random() * 40) + 20;
  const status = activeIncident && Math.random() > 0.4 ? 'ERROR' : 'OK';

  const event = {
    id: `trace-${Date.now()}`,
    timestamp: new Date().toISOString(),
    latency: baseLatency,
    status: status,
    agentId: `agent-${Math.floor(Math.random() * 5) + 1}`,
  };

  telemetryStore.push(event);
  if (telemetryStore.length > 100) telemetryStore.shift();
}

setInterval(buildTelemetryEvent, 5000);

// Summarize Telemetry for Dashboard API
function summarizeTelemetry() {
  if (telemetryStore.length === 0) buildTelemetryEvent();

  const recent = telemetryStore.slice(-20);
  const total = recent.length || 1;
  const avgLatency = Math.round(recent.reduce((sum, item) => sum + item.latency, 0) / total);
  const errorCount = recent.filter(i => i.status === 'ERROR').length;
  const errorRate = ((errorCount / total) * 100).toFixed(1);
  const p95 = avgLatency + 140;

  return {
    totalEvents: telemetryStore.length,
    avgLatencyMs: avgLatency,
    p95LatencyMs: p95,
    errorRatePct: errorRate,
    signalConfidence: activeIncident ? 'CRITICAL' : 'OPTIMAL',
    sloStatus: errorRate > 5 ? 'BREACHED' : 'HEALTHY',
    activeIncident,
  };
}

// REST Endpoints
app.get('/api/telemetry/summary', (req, res) => {
  res.json(summarizeTelemetry());
});

app.post('/api/incident/toggle', (req, res) => {
  activeIncident = !activeIncident;
  res.json({ success: true, activeIncident });
});

app.listen(PORT, () => {
  console.log(`SignalForge Command Center running on port ${PORT}`);
});

3. OpenTelemetry Collector Pipeline (`otel-config.yaml`)

This configuration routes incoming telemetry from local applications over standard gRPC (4317) or HTTP (4318) receivers and forwards it directly to SigNoz Cloud.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  otlp/signoz:
    endpoint: "ingest.us.signoz.cloud:443" # Replace with your SigNoz region endpoint
    headers:
      "signoz-access-token": "${SIGNOZ_INGESTION_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/signoz]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/signoz]

⚡ Quickstart Guide

To run SignalForge locally:

Clone & Install Dependencies:

   git clone https://github.com/era651868-ctrl/Signoz-track2.git
   cd signalforge
   npm install

Run Standard Mode:

   npm start

Open http://localhost:3000 to view the dashboard interface.

Run with OpenTelemetry Export Enabled:

   ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces npm start

📈 Key Takeaways

Standardization First: By using OpenTelemetry SDKs, your application remains vendor-agnostic and can easily route metrics and traces to SigNoz, Jaeger, or Prometheus without changing code.
Proactive SLO Monitoring: Real-time aggregation of p95 latency and error rates allows teams to respond to degradation before SLOs are fully breached.
Simulated Chaos Engineering: Incorporating incident toggles directly into non-production environments aids in testing automated alert rules and dashboard visualization clarity.

Observability for Multi-Agent Workloads: Cost Tracking & Fault Tolerance with SigNoz

Uzair — Sat, 25 Jul 2026 07:26:32 +0000

Architecture Deep Dive: Defeating the AI Agent 'Black Box' with OpenTelemetry and SigNoz

As AI agents transition from simple single-prompt scripts into complex multi-step systems, observability becomes mission-critical. In production architectures, an unmonitored agent chaining successive LLM iterations, managing volatile context state, and invoking discrete microservices can silently burn thousands of dollars in API token consumption or crash under rate limits.

This technical guide details how we built and deployed the Agent Ops Command Center for Track 1 (AI & Agent Observability). Using Node.js, Express running on port 3000, OpenTelemetry, and SigNoz, we created a real-time control panel to track latency, token usage, sub-cent financial costs, and automated fault recovery.

🛠️ Infrastructure Assembly: Declarative Deployment via Foundry

Rather than manually configuring separate analytics layers, the architecture relies on a declarative deployment blueprint executed through Foundry. This wraps the underlying telemetry engine alongside a native Model Context Protocol (MCP) server instance to ensure the entire environment remains reproducible for verification.

Below is the structured installation configuration saved as casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: production-agent-sre
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    enabled: true

To deploy the ecosystem, trigger the deployment compiler:

foundryctl cast -f casting.yaml

🧱 Real-World Gotcha: Port 3000 Ingestion & Proxy Handshakes

During initial deployment, the Express server spins up on port 3000 (process.env.PORT || 3000). When verifying local connectivity across virtualized cloud environments, testing the backend directly on port 3000 ensures the health and telemetry probes are operational:

curl -v http://localhost:3000/api/health

The test suite returns a clean HTTP/1.1 200 OK along with runtime telemetry snapshot status. When routing through web previews, mapping the host variable explicitly grants direct access to the command center:

echo "https://3000-$WEB_HOST"

📡 Pipeline Architecture: End-to-End Tracer Instrumentation

To extract execution data cleanly without vendor lock-in, our Express backend (server.js) integrates directly with official OpenTelemetry GenAI Semantic Conventions (gen_ai.*). Every agent invocation is exported directly to SigNoz via OTLP/HTTP.

Below is the OpenTelemetry initialization and instrumentation setup used in Node.js:

const express = require('express');
const { trace, SpanStatusCode } = require('@opentelemetry/api');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { BasicTracerProvider, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const app = express();
const port = process.env.PORT || 3000;
const serviceName = process.env.OTEL_SERVICE_NAME || 'agent-ops-command-center';
const otlpEndpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318';

// Configure OTLP Exporter targeting SigNoz
const exporter = new OTLPTraceExporter({
  url: `${otlpEndpoint.replace(/\/$/, '')}/v1/traces`
});

const provider = new BasicTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
    'service.version': '1.0.0',
    'deployment.environment': process.env.NODE_ENV || 'development'
  })
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

const tracer = trace.getTracer(serviceName);

🔑 Key Features & Telemetry Design

1. Granular Token & Cost Attribution

Every agent execution creates an agent.run span. We attach custom Semantic Conventions to track exact token consumption and sub-cent financial expenditure across providers (OpenAI, Gemini):

span.setAttribute('agent.name', agentName);
span.setAttribute('task.type', taskType);
span.setAttribute('gen_ai.system', providerName);
span.setAttribute('gen_ai.request.model', modelName);
span.setAttribute('gen_ai.usage.input_tokens', inputTokens);
span.setAttribute('gen_ai.usage.output_tokens', outputTokens);
span.setAttribute('gen_ai.usage.total_tokens', totalTokens);
span.setAttribute('agent.cost.usd', costUsd);

2. Rate-Limit Resilience & Exponential Backoff

When API provider rate limits or quotas are hit (HTTP 429), the backend applies an exponential backoff mechanism. If the quota remains exhausted, the system automatically records a warning span in SigNoz and gracefully switches to a rate-limit fallback response instead of crashing:

// Catch rate limit errors and emit fallback trace to SigNoz
if (error.message.includes('Quota exceeded') || error.message.includes('429')) {
  span.setAttribute('gen_ai.request.model', `${modelName}-fallback`);
  span.setAttribute('gen_ai.usage.total_tokens', 150);
  span.setAttribute('agent.cost.usd', '0.000000');
  span.setStatus({ code: SpanStatusCode.OK });

  recordEvent('WARN', `Rate limit reached for ${providerName}. Used fallback.`, 'Agent run');
  span.end();

  return res.json({ run: fallbackRunData, live: true });
}

📊 Telemetry Design: Custom ClickHouse SRE Queries

Using the ClickHouse datastore underneath SigNoz, we can query LLM telemetry directly through the SigNoz Query Builder to monitor live system metrics.

📈 Panel 1: Real-Time Token Spend & Cost Tracking Metrics

Calculates financial accumulation across distinct model interactions:

SELECT
    attributes_string['gen_ai.request.model'] as Model,
    SUM(attributes_int['gen_ai.usage.input_tokens']) as TotalInputTokens,
    SUM(attributes_int['gen_ai.usage.output_tokens']) as TotalOutputTokens,
    SUM(toFloat64(attributes_string['agent.cost.usd'])) as TotalSpendUSD
FROM signoz_traces.signoz_index_v2
WHERE name = 'agent.run' AND timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY Model
ORDER BY TotalSpendUSD DESC

🔄 Panel 2: Agent High-Latency & Error Tracking

Identifies agent runs experiencing high latency or execution errors:

SELECT
    attributes_string['agent.name'] as AgentName,
    COUNT(*) as TotalRuns,
    AVG(durationNano) / 1000000 as AvgLatencyMs
FROM signoz_traces.signoz_index_v2
WHERE name = 'agent.run' AND timestamp >= NOW() - INTERVAL 30 MINUTE
GROUP BY AgentName
HAVING AvgLatencyMs > 250

🧠 Core Engineering Takeaways

Port 3000 Standardization: Aligning application routing and container proxy variables on port 3000 ensures clean local evaluation and deployment health checks.
Resilient Fallback Loops: Gracefully capturing rate limits inside OpenTelemetry spans prevents application downtime while preserving full trace visibility in SigNoz.
Declarative Reproducibility: Deploying through Foundry's casting.yaml ensures the underlying SigNoz telemetry collector stays reproducible for hackathon verification.

GitHub Repository: https://github.com/era651868-ctrl/Signoz-track1
Port: 3000
Live Deployment : https://signoz-track1-production.up.railway.app/
YouTube Live Demo : https://youtube.com/shorts/IFrIFgBu7AE?si=XxCj-26IKpFZ9sCB

Cracking the Black Box: How I Caught a Runaway AI Agent Using Self-Hosted SigNoz and OpenTelemetry

Uzair — Thu, 16 Jul 2026 10:09:04 +0000

Architecture Deep Dive: Defeating the AI Agent 'Black Box' with OpenTelemetry and SigNoz

Autonomous AI agents are incredibly powerful engineering solutions—until they hit an unhandled data schema mutation and spin out into a recursive execution loop. In production architectures, an unmonitored agent chaining successive LLM iterations, managing volatile context state, and invoking discrete microservices can become a massive liability, silently burning thousands of dollars in API token consumption.

If your operational visibility ends at the perimeter of the network proxy, you are operating blindly. This analytical guide details how to build a production-hardened observability foundation by self-hosting SigNoz via Foundry, configuring OTLP pipelines using standard semantic conventions, and assembling a custom ClickHouse-backed telemetry dashboard to detect cost runaway and agent loops dynamically.

🛠️ Infrastructure Assembly: Declarative Deployment via Foundry

Below is the structured installation configuration saved as casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: production-agent-sre
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    enabled: true

To deploy the ecosystem, trigger the deployment compiler:

foundryctl cast -f casting.yaml

🧱 Real-World Gotcha: Mitigating Web Preview Proxy Handshakes

During initial initialization, the container cluster successfully spun up all primary components (ClickHouse cluster, OTLP ingestion layer, and frontend gateway). However, accessing the dashboard via standard virtualized port configurations resulted in standard proxy handshaking timeouts.

To isolate the failure point, low-level testing directly on the local routing layer was used:

curl -v http://localhost:8080

The test suite immediately returned a clean HTTP/1.1 200 OK followed by the raw UI document structure, indicating a completely functional application layer. The underlying network proxy environment variables were caching dead routing handles. The workaround requires mapping the host address explicitly by querying the cluster's environment variables:

echo "https://8080-$WEB_HOST"

Pasting the output token directly into a clean browser instance completely bypassed the virtual network constraints, granting access to the SigNoz control console.

📡 Pipeline Architecture: End-to-End Tracer Instrumentation

To extract execution data cleanly without vendor lock-in, the agent framework integrates directly with official OpenTelemetry GenAI Semantic Conventions (gen_ai.*). This registers every microservice step—from LLM generation metrics to discrete utility routing—as structured OTLP data payloads.

Below is the initialization code for a Python-based multi-agent execution thread:

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Bind TracerProvider to the local SigNoz gRPC ingestion socket
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
telemetry_tracer = trace.get_tracer("agent-core-apm")

def execute_agent_pipeline(user_input, tracking_session_id):
    # Enclose the parent context inside a distinct trace scope
    with telemetry_tracer.start_as_current_span("orchestrate_agent_workflow") as master_span:
        master_span.set_attribute("gen_ai.workflow.name", "Enterprise-Data-Orchestrator")
        master_span.set_attribute("gen_ai.conversation.id", tracking_session_id)

        # Track individual LLM interaction sequences
        with telemetry_tracer.start_as_current_span("llm_chat_transaction") as inference_span:
            inference_span.set_attribute("gen_ai.system", "Anthropic")
            inference_span.set_attribute("gen_ai.request.model", "claude-3-5-sonnet")

            # Record downstream tools and structural database interactions
            with telemetry_tracer.start_as_current_span("dispatch_utility_tool") as service_span:
                service_span.set_attribute("gen_ai.tool.name", "query_analytical_db")
                # Emulate an active runtime schema violation to verify pipeline response
                service_span.set_status(trace.StatusCode.ERROR, "Unexpected data structure returned from storage layer")

📊 Telemetry Design: Custom ClickHouse SRE Query Dashboards

Standard system analytics checking generic memory metrics and HTTP statuses fail to surface internal logic errors within AI pipelines. By utilizing the raw performance of the ClickHouse datastore underneath SigNoz, we can engineer custom dashboards to track execution logic directly through the SigNoz Query Builder.

📈 Panel 1: Real-Time Token Spend & Cost Tracking Metrics

Calculates financial accumulation across distinct customer conversations to surface anomalous cost metrics:

SELECT
    attributes_string['gen_ai.conversation.id'] as TargetSession,
    SUM(attributes_int['gen_ai.usage.prompt_tokens']) as AggregatedInputTokens,
    SUM(attributes_int['gen_ai.usage.completion_tokens']) as AggregatedOutputTokens,
    (SUM(attributes_int['gen_ai.usage.prompt_tokens']) * 0.000003) + (SUM(attributes_int['gen_ai.usage.completion_tokens']) * 0.000015) as FinancialExposureUSD
FROM signoz_traces.signoz_index_v2
WHERE name = 'llm_chat_transaction' AND timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY TargetSession
ORDER BY FinancialExposureUSD DESC

🔄 Panel 2: Agent Recursion Loops & Storm Counter

Identifies broken logic configurations by flagging single transactions that issue more than ten nested tools inside a short temporal window:

SELECT
    attributes_string['gen_ai.conversation.id'] as TargetSession,
    COUNT(*) as DetectedToolExecutions
FROM signoz_traces.signoz_index_v2
WHERE name = 'dispatch_utility_tool' AND timestamp >= NOW() - INTERVAL 30 MINUTE
GROUP BY TargetSession
HAVING DetectedToolExecutions > 10

🔍 Panel 3: Runtime Error Analysis Matrix

Categorizes and aggregates explicit platform exceptions to identify if issues stem from third-party vendor slowdowns or local code exceptions:

SELECT
    attributes_string['error.type'] as ExceptionType,
    COUNT(*) as AggregateIncidents
FROM signoz_traces.signoz_index_v2
WHERE response_status_code = 'STATUS_CODE_ERROR'
GROUP BY ExceptionType
ORDER BY AggregateIncidents DESC

🧠 Core Engineering Takeaways

Context State Expands Multiplicatively: As multi-step reasoning models operate, prompt structures grow larger. OpenTelemetry tracing exposes this inflation clearly before total context limit saturation occurs.
Infrastructure-as-Code Prevents Divergence: Deploying via Foundry's casting.yaml ensures analytical pipelines remain completely consistent across isolated environments.
Semantic Standardization Eliminates Rewrite Overhead: Utilizing uniform standard semantic attributes ensures your tracking metrics adapt perfectly if you transition from Python routines to production Node.js frameworks.

By moving away from unstructured console outputs and anchoring our tracking to OpenTelemetry standards with SigNoz, we convert unpredictable AI processes into fully transparent, debuggable software architectures.

DEV Community: Uzair

Building an Automated Infrastructure Health & Monitoring Bot with OpenTelemetry and SigNoz

Signoz track 3 submission: Build your own. Automated Infrastructure Health & Monitoring Bot 🤖

🌐 Live Demo & Repository

🏗️ System Architecture & Workflow

💻 Core Implementation Deep Dive

1. Resilient OpenTelemetry Setup with Graceful Fallback (app.py)

2. Anomaly Evaluation & Span Attribute Injection

3. OpenTelemetry Collector Pipeline (pours/deployment/ingester/ingester.yaml)

⚡ Quickstart & Testing

1. Run Unit Tests

2. Run Containerized Deployment

📈 Key Learnings & SigNoz Value

SignalForge: Building a Real-Time Agent Control Tower with OpenTelemetry and SigNoz

Building SignalForge: End-to-End Telemetry & Observability with OpenTelemetry and SigNoz

🌐 Live Demo & Deployment

🏗️ Architectural Overview

💻 Key Implementation Files

1. OpenTelemetry SDK Setup (otel-example.js)

2. Express Backend & Telemetry Simulator (server.js)

3. OpenTelemetry Collector Pipeline (otel-config.yaml)

⚡ Quickstart Guide

📈 Key Takeaways

Observability for Multi-Agent Workloads: Cost Tracking & Fault Tolerance with SigNoz

Architecture Deep Dive: Defeating the AI Agent 'Black Box' with OpenTelemetry and SigNoz

🛠️ Infrastructure Assembly: Declarative Deployment via Foundry

🧱 Real-World Gotcha: Port 3000 Ingestion & Proxy Handshakes

📡 Pipeline Architecture: End-to-End Tracer Instrumentation

🔑 Key Features & Telemetry Design

1. Granular Token & Cost Attribution

2. Rate-Limit Resilience & Exponential Backoff

📊 Telemetry Design: Custom ClickHouse SRE Queries

📈 Panel 1: Real-Time Token Spend & Cost Tracking Metrics

🔄 Panel 2: Agent High-Latency & Error Tracking

🧠 Core Engineering Takeaways

Cracking the Black Box: How I Caught a Runaway AI Agent Using Self-Hosted SigNoz and OpenTelemetry

Architecture Deep Dive: Defeating the AI Agent 'Black Box' with OpenTelemetry and SigNoz

🛠️ Infrastructure Assembly: Declarative Deployment via Foundry

🧱 Real-World Gotcha: Mitigating Web Preview Proxy Handshakes

📡 Pipeline Architecture: End-to-End Tracer Instrumentation

📊 Telemetry Design: Custom ClickHouse SRE Query Dashboards

📈 Panel 1: Real-Time Token Spend & Cost Tracking Metrics

🔄 Panel 2: Agent Recursion Loops & Storm Counter

🔍 Panel 3: Runtime Error Analysis Matrix

🧠 Core Engineering Takeaways

1. Resilient OpenTelemetry Setup with Graceful Fallback (`app.py`)

3. OpenTelemetry Collector Pipeline (`pours/deployment/ingester/ingester.yaml`)

1. OpenTelemetry SDK Setup (`otel-example.js`)

2. Express Backend & Telemetry Simulator (`server.js`)

3. OpenTelemetry Collector Pipeline (`otel-config.yaml`)