SignalForge: Building a Real-Time Agent Control Tower with OpenTelemetry and SigNoz

Building SignalForge: End-to-End Telemetry & Observability with OpenTelemetry and SigNoz

In autonomous AI agent systems and modern microservice architectures, knowing what happened isn't enough—you need to know when, why, and how fast it happened.

In this post, we'll walk through SignalForge Command Center, a lightweight Node.js/Express telemetry dashboard designed to simulate agent trace streams, calculate real-time Service Level Objectives (SLOs), and export standard OpenTelemetry (OTEL) data to SigNoz.

---

🌐 Live Demo & Deployment

The application is deployed and live! You can inspect the command center directly:

• Deployed Web Dashboard: https://signoz-track2-production.up.railway.app/

• YouTube Demo: https://youtube.com/shorts/5ffMjV5UgD0?si=-lJHE952wvnctBWk

---

🏗️ Architectural Overview

SignalForge consists of four core building blocks:

1. Express Server (server.js): Serves the REST API and dashboard UI, while maintaining a rolling in-memory telemetry store and simulating autonomous agent metrics.
2. OpenTelemetry SDK (otel-example.js): Configures auto-instrumentation for Express and HTTP requests, exporting traces via the OpenTelemetry Protocol (OTLP).
3. OpenTelemetry Collector (otel-config.yaml): Receives, processes, and batches trace/metric streams before routing them to SigNoz Cloud.
4. Interactive Dashboard (public/index.html & app.js): Polls backend APIs every 5 seconds, displaying real-time signal confidence, p95 latencies, and incident simulation controls.

---

💻 Key Implementation Files

1. OpenTelemetry SDK Setup (otel-example.js)

This script initializes the NodeTracerProvider and attaches auto-instrumentation modules for HTTP and Express.

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

// Configure resource attributes identifying this service
const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'signalforge-command-center',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
  }),
});

// Setup OTLP Exporter targeting your collector or SigNoz endpoint
const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

// Enable automatic HTTP & Express tracing
registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

console.log('OpenTelemetry instrumentation successfully initialized.');

---

2. Express Backend & Telemetry Simulator (server.js)

The main entry point manages telemetry aggregation, calculates active SLO metrics, and provides incident simulation endpoints.

// Enable OTEL auto-instrumentation if flag is set
if (process.env.ENABLE_OTEL === 'true') {
  require('./otel-example');
}

const express = require('express');
const path = require('path');

const app = express();
const PORT = process.env.PORT || 3000;

app.use(express.json());
app.use(express.static(path.join(__dirname, 'public')));

// In-memory store for simulated telemetry events
let telemetryStore = [];
let activeIncident = false;

// Generate simulated telemetry every 5 seconds
function buildTelemetryEvent() {
  const baseLatency = activeIncident ? Math.floor(Math.random() * 300) + 250 : Math.floor(Math.random() * 40) + 20;
  const status = activeIncident && Math.random() > 0.4 ? 'ERROR' : 'OK';

  const event = {
    id: `trace-${Date.now()}`,
    timestamp: new Date().toISOString(),
    latency: baseLatency,
    status: status,
    agentId: `agent-${Math.floor(Math.random() * 5) + 1}`,
  };

  telemetryStore.push(event);
  if (telemetryStore.length > 100) telemetryStore.shift();
}

setInterval(buildTelemetryEvent, 5000);

// Summarize Telemetry for Dashboard API
function summarizeTelemetry() {
  if (telemetryStore.length === 0) buildTelemetryEvent();

  const recent = telemetryStore.slice(-20);
  const total = recent.length || 1;
  const avgLatency = Math.round(recent.reduce((sum, item) => sum + item.latency, 0) / total);
  const errorCount = recent.filter(i => i.status === 'ERROR').length;
  const errorRate = ((errorCount / total) * 100).toFixed(1);
  const p95 = avgLatency + 140;

  return {
    totalEvents: telemetryStore.length,
    avgLatencyMs: avgLatency,
    p95LatencyMs: p95,
    errorRatePct: errorRate,
    signalConfidence: activeIncident ? 'CRITICAL' : 'OPTIMAL',
    sloStatus: errorRate > 5 ? 'BREACHED' : 'HEALTHY',
    activeIncident,
  };
}

// REST Endpoints
app.get('/api/telemetry/summary', (req, res) => {
  res.json(summarizeTelemetry());
});

app.post('/api/incident/toggle', (req, res) => {
  activeIncident = !activeIncident;
  res.json({ success: true, activeIncident });
});

app.listen(PORT, () => {
  console.log(`SignalForge Command Center running on port ${PORT}`);
});

---

3. OpenTelemetry Collector Pipeline (otel-config.yaml)

This configuration routes incoming telemetry from local applications over standard gRPC (4317) or HTTP (4318) receivers and forwards it directly to SigNoz Cloud.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  otlp/signoz:
    endpoint: "ingest.us.signoz.cloud:443" # Replace with your SigNoz region endpoint
    headers:
      "signoz-access-token": "${SIGNOZ_INGESTION_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/signoz]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/signoz]

---

⚡ Quickstart Guide

To run SignalForge locally:

1. Clone & Install Dependencies:

   git clone https://github.com/era651868-ctrl/Signoz-track2.git
   cd signalforge
   npm install

1. Run Standard Mode:

   npm start

Open http://localhost:3000 to view the dashboard interface.

1. Run with OpenTelemetry Export Enabled:

   ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces npm start

---

📈 Key Takeaways

• Standardization First: By using OpenTelemetry SDKs, your application remains vendor-agnostic and can easily route metrics and traces to SigNoz, Jaeger, or Prometheus without changing code.
• Proactive SLO Monitoring: Real-time aggregation of p95 latency and error rates allows teams to respond to degradation before SLOs are fully breached.
• Simulated Chaos Engineering: Incorporating incident toggles directly into non-production environments aids in testing automated alert rules and dashboard visualization clarity.