Three days before the hackathon submission deadline, my local React dashboard looked absolutely flawless.
I would trigger a simulated brute-force attack, the UI would instantly flash a red alert, the background Python engine would spin up a Google Gemini instance to analyze the threat vector, calculate a 94% confidence score, and cleanly trigger a simulated firewall tool to drop the offending IP address.
From the outside, it was a software developer's dream. The frontend updated perfectly in real-time. But when I opened my self-hosted SigNoz instance to verify the backend data, the Trace Explorer was a complete ghost town.
That was the exact moment I realized a foundational truth of modern software engineering: building an autonomous AI agent is only half the problem—making its internal reasoning transparent, observable, and auditable is the actual challenge.
This is the raw engineering diary of how I built VigilTrace—an autonomous AI Incident Commander—and the exact telemetry roadblocks, Windows-specific traps, and OpenTelemetry architectural bugs I had to fight through to make it truly observable.
What I Was Building: VigilTrace
The design philosophy behind VigilTrace is simple: If you can't observe your AI agents, you don't own them.
Instead of creating a black-box AI script that blindly triggers a shell command when a log drops, I architected an interactive pipeline that treats AI analysis as a multi-stage, regulated security investigation. The workflow operates as follows:
Plaintext
Server Simulator ──> Access Logs ──> Incident Commander (Python)
│
SigNoz UI <── OTLP Collector <── OpenTelemetry SDK <── Gemini AI
│
[Blocklist.json] <── Firewall Simulator
Every single link in this chain—from the raw token usage inside the Gemini LLM step to the file I/O latency inside the FirewallSimulator—had to emit structured telemetry packets over gRPC to a local OpenTelemetry Collector monitored via SigNoz.
Mistake #1: The Phantom Metrics (Why create_gauge Left Me in the Dark)
My first major roadblock was system metrics. I wanted to capture real-time resource spikes (cpu_usage and memory_usage) on the machine while the AI engine was chewing through dense security logs.
I spun up a background metrics daemon in Python and wrote what I assumed was standard OpenTelemetry code:
Python
THE BUGGY CODE THAT PRODUCED ZERO DATA IN SIGNOZ
from opentelemetry import metrics
meter = metrics.get_meter("system-monitor")
cpu_gauge = meter.create_gauge(
name="system.cpu.utilization",
description="Tracks CPU load metrics"
)
Polling loop trying to push values directly
while True:
cpu_gauge.set(get_current_cpu_percent())
time.sleep(5)
The script executed without crashing. No console errors, no warnings. Yet, when I checked the Metrics Explorer inside SigNoz, the dropdown menu for system.cpu.utilization simply did not exist.
The Fix
After diving into the official OpenTelemetry documentation, I realized a crucial architectural detail I had missed: metrics in OTel are primarily driven by a pull-based or asynchronous callback architecture to avoid blocking runtime threads. Synchronous gauges are often mishandled or dropped depending on how the OTLP exporter state is initialized.
I had to completely refactor the metrics engine to use an Observable Gauge, passing a dedicated callback function that the SDK invokes automatically on its own collection cycle:
Python
THE WORKING FIX: Migrating to an Asynchronous Observable Gauge
from opentelemetry import metrics
meter = metrics.get_meter("system-monitor")
def get_cpu_callback(options):
# The SDK calls this function natively on every collection interval
return [metrics.Observation(get_current_cpu_percent())]
meter.create_observable_gauge(
name="system.cpu.utilization",
callbacks=[get_cpu_callback],
description="Tracks real-time system CPU load percentage"
)
As soon as I spun up the script with the callback logic, the OTLP exporter began generating beautiful time-series line charts directly within the SigNoz metrics dashboard. Lesson learned: don't push data when the OpenTelemetry SDK wants to pull it.
Mistake #2: The Broken Waterfall (Flat Spans vs. Nested Contexts)
When I finally got my first trace batches to stream into the SigNoz Trace Explorer, the structural layout looked entirely wrong. Instead of a single, coherent incident timeline showing a clean hierarchy, SigNoz rendered ten completely disconnected, flat timelines:
Plaintext
❌ WHAT I SAW ORIGINALLY:
Investigation (Trace ID: a1b2...)
Recommendation (Trace ID: c3d4...)
Firewall Execution (Trace ID: e5f6...)
Because every stage was being drawn as a root span, I couldn't track the lineage of an incident. If the execution failed at the firewall stage, I had no clean visual way to trace it back to the specific LLM confidence calculation that triggered it.
The Fix
The bug was a classic context propagation error. In Python, if you use a standard async loop or call distinct functions across different files, the active OpenTelemetry tracing context can easily drop out if you don't explicitly pass it or set it inside the current execution context thread.
I refactored the parent loop inside incident_commander.py to leverage the Python context manager, forcing child tasks to explicitly yield inside the parent scope:
Python
THE FIX: Ensuring tight nesting of distributed trace spans
from opentelemetry import trace
tracer = trace.get_tracer(name)
def run_incident_workflow(incident_data):
# Establish the overarching parent context span
with tracer.start_as_current_span("AI-Incident-Investigation") as parent_span:
parent_span.set_attribute("incident.id", incident_data["id"])
# Explicitly passing structural execution tokens down the pipeline
analyze_threat_vector(incident_data)
execute_mitigation_workflow(incident_data)
def execute_mitigation_workflow(incident_data):
# This automatically detects and nests itself under the active parent span
with tracer.start_as_current_span("FirewallSimulator.block_ip") as child_span:
child_span.set_attribute("tool.action", "drop_packet")
# Simulating file I/O blocklist updates
write_to_blocklist(incident_data["attacker_ip"])
This tiny adjustments paid off instantly. The next time I fired up the app, SigNoz rendered a gorgeous nested flamegraph waterfall:
Plaintext
✅ THE CORRECT HIERARCHY IN SIGNOZ:
AI-Incident-Investigation [Trace ID: 7fa98c...]
├── Analyze_Threat_Vector (LLM Ingestion)
└── Execute_Mitigation_Workflow
└── FirewallSimulator.block_ip
Mistake #3: The Windows Path Trap (spawn python3 ENOENT)
Because I develop across different operating environments, I cloned my Next.js/Node.js telemetry aggregator backend onto my secondary Windows workstation to test stability.
I hit npm run dev, clicked the execution button on the frontend interface, and my Node runtime exploded with this raw error block:
Plaintext
events.js:377
throw er; // Unhandled 'error' event
^
Error: spawn python3 ENOENT
at Process.ChildProcess._handle.onread (internal/child_process.js:245:12)
at onErrorNT (internal/child_process.js:482:16)
The issue was painfully obvious yet highly disruptive: my Node core layer was hardcoded to invoke the background telemetry workers via a direct shell invocation to python3. While Linux and macOS environments default to python3, standard Windows installations map the runtime binary path explicitly to python.
The Fix
I refactored the application initialization handler to evaluate the host operating system dynamically via process.platform before spinning up the background telemetry workers:
TypeScript
import { spawn } from 'child_process';
import os from 'os';
// Dynamically bridge OS path execution discrepancies
const pythonExecutable = process.platform === 'win32' ? 'python' : 'python3';
const runTelemetryWorker = () => {
const workerProcess = spawn(pythonExecutable, ['telemetry_daemon.py'], {
env: { ...process.env, PYTHONUNBUFFERED: '1' }
});
workerProcess.stderr.on('data', (data) => {
console.error([WORKER ERROR]: ${data.toString()});
});
};
This brought cross-platform parity to the project, allowing the Node orchestrator to coordinate cleanly with the Python OpenTelemetry SDK regardless of the host OS architecture.
Mistake #4: The React UI Illusion (The Dashboard That Lied)
The biggest psychological trap during development was the fidelity of my own React application UI.
Because I was using standard React state objects (useState) to pass event records locally between my components, the user interface looked like an absolute engineering marvel. Spans lit up sequentially, data tables populated flawlessly, and progress bars moved perfectly.
But my frontend state was completely insulated from reality. When I purposefully pulled down my Docker instances of SigNoz, my React app still looked like it was working perfectly. The frontend was happily lying to me, masking the fact that the underlying OTLP connection was completely severed and exporting absolutely zero enterprise telemetry data.
The Lesson Learned
A healthy user interface does not mean a healthy telemetry pipeline.
I fundamentally altered my testing validation loop. I decoupled my reliance on the client-side state. I wrote a dedicated integration command to hit the backend directly:
Bash
Explicitly force clean dependency instantiation and trigger an isolated log event
npm install
curl -X POST http://localhost:3000/api/simulate-attack -d '{"type": "brute_force"}'
From that point forward, my source of truth wasn't my shiny React UI—it was the SigNoz Services and Traces explorer panes. If a trace didn't resolve inside SigNoz, the feature was treated as broken, regardless of how beautiful the CSS animation looked on the screen.
The Ultimate Payoff: Total Transparency into AI
Once these technical integration battles were won, the true power of an OpenTelemetry-native platform like SigNoz clicked for me.
By injecting custom metadata attributes directly into the span context blocks during the Gemini LLM investigation phase, I transformed standard distributed traces into comprehensive, auditable incident records. When an alert triggers, an engineer doesn't have to guess why the AI took a specific action. They can open up SigNoz, click on the active span, and immediately inspect the exact logical reasoning parameters:
incident.threat_type: Credential Stuffing
agent.confidence_score: 96%
agent.evidence_summary: 850 failed logins from Tor Exit Node IP detected inside a 120-second window.
llm.tokens_used: 1452
Plaintext
[Insert Screenshot Here: SigNoz Trace Detail side-panel clearly rendering custom metadata attributes mapping out the agent reasoning chains and exact token costs]
What I Learned from the Terminal
Before this hackathon, I viewed observability as a utility you wrap around server infrastructure to check if your hard drives are full or your APIs are returning HTTP 500s.
Building VigilTrace changed my perspective entirely. As we move into an era dominated by autonomous AI agents, LLM tool-calling, and complex multi-step prompt chains, observability is no longer just an infrastructure requirement—it is the core framework for AI safety and explainability.
By incorporating OpenTelemetry and SigNoz from line one, I didn't just build a security system that works. I built a system that can explicitly explain why it works, when it fails, and how much it costs down to the millisecond.
Technology Stack Utilized
Frontend/Orchestration: React, TypeScript, Vite, Node.js, Express, Framer Motion, Recharts
AI & Telemetry Core: Python, Google Gemini API, OpenTelemetry SDK, OTLP/gRPC Exporters
Observability Backend: SigNoz (Self-Hosted via Docker / ClickHouse architecture)
Top comments (0)