DEV Community

Uzair
Uzair

Posted on

Building an Automated Infrastructure Health & Monitoring Bot with OpenTelemetry and SigNoz

#webdev #productivity #docker #python

Signoz track 3 submission: Build your own. Automated Infrastructure Health & Monitoring Bot 🤖

In production SRE workflows, static metric alerts often leave engineering teams scrambling to correlate spikes with root causes. Knowing that CPU hit 95% is only half the battle—you need distributed tracing to understand what process cycle triggered the spike and how the system responded in real time.

In this post, we’ll break down the Automated Infrastructure Health & Monitoring Bot, a lightweight, OpenTelemetry-native SRE monitoring agent built for Track 3: Build Your Own of the SigNoz Hackathon.

🌐 Live Demo & Repository

🏗️ System Architecture & Workflow

The bot is designed to simulate a real-world SRE monitoring agent operating in constrained environments. It continuously samples system telemetry, evaluates rule-based alert conditions, and streams OTLP traces directly to SigNoz.

+-------------------------------------------------------------------+
|               Infrastructure Health Agent (app.py)                |
|                                                                   |
|   +------------------+     +-------------------+                  |
|   | System Collector | --> | Anomaly Detection |                  |
|   | (CPU/Mem/Disk)   |     | (Threshold Engine)|                  |
|   +------------------+     +-------------------+                  |
|                                     |                             |
|                                     v                             |
|                           +-------------------+                   |
|                           |  OTLP Trace Engine|                   |
|                           +-------------------+                   |
+-------------------------------------|-----------------------------+
                                      | OTLP gRPC (Port 4317)
                                      v
                        +---------------------------+
                        |  SigNoz OTEL Collector    |
                        |  & Telemetry Store        |
                        +---------------------------+
Enter fullscreen mode Exit fullscreen mode

💻 Core Implementation Deep Dive

1. Resilient OpenTelemetry Setup with Graceful Fallback (app.py)

To ensure the monitoring agent never crashes when network connectivity drops or during collector maintenance, the setup dynamically tests OTLP gRPC endpoint reachability before registering exporters.

import os
import socket
from urllib.parse import urlparse
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

SERVICE_NAME = os.getenv("SERVICE_NAME", "health-alert-bot")
OTLP_ENDPOINT = os.getenv("OTLP_ENDPOINT", "http://localhost:4317")

def is_otlp_endpoint_reachable(endpoint: str) -> bool:
    parsed = urlparse(endpoint)
    host = parsed.hostname or "localhost"
    port = parsed.port or 4317
    try:
        with socket.create_connection((host, port), timeout=1):
            return True
    except OSError:
        return False

def setup_tracing():
    resource = Resource.create(attributes={"service.name": SERVICE_NAME})
    provider = TracerProvider(resource=resource)

    if is_otlp_endpoint_reachable(OTLP_ENDPOINT):
        exporter = OTLPSpanExporter(endpoint=OTLP_ENDPOINT, insecure=True)
        provider.add_span_processor(BatchSpanProcessor(exporter))
        print(f"📡 Tracing enabled. Sending OTLP spans to {OTLP_ENDPOINT}")
    else:
        provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
        print(f"⚠️ OTLP endpoint unreachable. Falling back to console spans.")

    trace.set_tracer_provider(provider)
    return trace.get_tracer("bot-tracer")

tracer = setup_tracing()
Enter fullscreen mode Exit fullscreen mode

2. Anomaly Evaluation & Span Attribute Injection

Every monitoring cycle evaluates resource thresholds and attaches rich contextual metadata directly onto OpenTelemetry spans. Critical breaches trigger nested alert child spans.

def main():
    threading.Thread(target=start_http_server, daemon=True).start()
    print("🚀 Health Alert Bot started running...")

    while True:
        metrics = collect_metrics()
        severity, reasons = evaluate_health(metrics)

        with tracer.start_as_current_span("monitor_cycle") as span:
            # Set high-cardinality telemetry as span attributes
            span.set_attribute("system.cpu.usage", metrics["cpu_percent"])
            span.set_attribute("system.memory.usage", metrics["memory_percent"])
            span.set_attribute("system.disk.usage", metrics["disk_percent"])
            span.set_attribute("alert.threshold.cpu", CPU_ALERT_THRESHOLD)
            span.set_attribute("alert.threshold.memory", MEMORY_ALERT_THRESHOLD)
            span.set_attribute("health.severity", severity)

            # Trigger sub-span drilldown on CRITICAL events
            if severity == "CRITICAL":
                with tracer.start_as_current_span("trigger_alert") as alert_span:
                    alert_span.set_attribute("alert.level", "CRITICAL")
                    alert_span.set_attribute("alert.reason", ",".join(reasons))
                    alert_span.add_event(
                        "cpu_threshold_exceeded", 
                        {"cpu_percent": metrics["cpu_percent"]}
                    )

            print(build_status_message(metrics, severity))

        record_incident(metrics, severity, reasons)
        time.sleep(CHECK_INTERVAL_SECONDS)
Enter fullscreen mode Exit fullscreen mode

3. OpenTelemetry Collector Pipeline (pours/deployment/ingester/ingester.yaml)

The SigNoz collector configuration receives raw OTLP gRPC spans on port 4317 and processes them via ClickHouse batch exporters:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 50000
    send_batch_max_size: 55000
    timeout: 5s

exporters:
  clickhousetraces:
    datasource: tcp://signoz-telemetrystore-clickhouse-0-0:9000/signoz_traces
    use_new_schema: true
    timeout: 45s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousetraces]
Enter fullscreen mode Exit fullscreen mode

⚡ Quickstart & Testing

1. Run Unit Tests

Verify anomaly detection rules and JSONL logging:

python -m unittest discover -s tests
Enter fullscreen mode Exit fullscreen mode

2. Run Containerized Deployment

Deploy using Docker or Foundry:

docker build -t health-bot .
docker run -p 8000:8000 -e OTLP_ENDPOINT=http://host.docker.internal:4317 health-bot
Enter fullscreen mode Exit fullscreen mode

📈 Key Learnings & SigNoz Value

  • Tracing Over Static Logging: Injecting CPU, memory, and disk utilization attributes into OpenTelemetry spans converts raw infrastructure spikes into searchable trace queries within SigNoz.
  • Resilient Instrumentation: Building automatic fallback mechanisms (OTLP to Console) guarantees that monitoring tools continue operating even during local collector outages.
  • Full Reproducibility: Providing valid casting.yaml and ClickHouse configurations allows teams to deploy self-hosted observability stacks in minutes.

Top comments (0)