DEV Community: Syed Kabeer Ali

How I Debugged an AI Incident Response Pipeline Using OpenTelemetry and SigNoz (And the 5 Mistakes That Nearly Broke My Hackathon Project)

Syed Kabeer Ali — Sat, 18 Jul 2026 11:46:36 +0000

Three days before the hackathon submission deadline, my local React dashboard looked absolutely flawless.

I would trigger a simulated brute-force attack, the UI would instantly flash a red alert, the background Python engine would spin up a Google Gemini instance to analyze the threat vector, calculate a 94% confidence score, and cleanly trigger a simulated firewall tool to drop the offending IP address.

From the outside, it was a software developer's dream. The frontend updated perfectly in real-time. But when I opened my self-hosted SigNoz instance to verify the backend data, the Trace Explorer was a complete ghost town.

That was the exact moment I realized a foundational truth of modern software engineering: building an autonomous AI agent is only half the problem—making its internal reasoning transparent, observable, and auditable is the actual challenge.

This is the raw engineering diary of how I built VigilTrace—an autonomous AI Incident Commander—and the exact telemetry roadblocks, Windows-specific traps, and OpenTelemetry architectural bugs I had to fight through to make it truly observable.

What I Was Building: VigilTrace
The design philosophy behind VigilTrace is simple: If you can't observe your AI agents, you don't own them.

Instead of creating a black-box AI script that blindly triggers a shell command when a log drops, I architected an interactive pipeline that treats AI analysis as a multi-stage, regulated security investigation. The workflow operates as follows:

Plaintext
Server Simulator ──> Access Logs ──> Incident Commander (Python)
│
SigNoz UI <── OTLP Collector <── OpenTelemetry SDK <── Gemini AI
│
[Blocklist.json] <── Firewall Simulator
Every single link in this chain—from the raw token usage inside the Gemini LLM step to the file I/O latency inside the FirewallSimulator—had to emit structured telemetry packets over gRPC to a local OpenTelemetry Collector monitored via SigNoz.

Mistake #1: The Phantom Metrics (Why create_gauge Left Me in the Dark)
My first major roadblock was system metrics. I wanted to capture real-time resource spikes (cpu_usage and memory_usage) on the machine while the AI engine was chewing through dense security logs.

I spun up a background metrics daemon in Python and wrote what I assumed was standard OpenTelemetry code:

Python

THE BUGGY CODE THAT PRODUCED ZERO DATA IN SIGNOZ

from opentelemetry import metrics

meter = metrics.get_meter("system-monitor")
cpu_gauge = meter.create_gauge(
name="system.cpu.utilization",
description="Tracks CPU load metrics"
)

Polling loop trying to push values directly

while True:
cpu_gauge.set(get_current_cpu_percent())
time.sleep(5)
The script executed without crashing. No console errors, no warnings. Yet, when I checked the Metrics Explorer inside SigNoz, the dropdown menu for system.cpu.utilization simply did not exist.

The Fix
After diving into the official OpenTelemetry documentation, I realized a crucial architectural detail I had missed: metrics in OTel are primarily driven by a pull-based or asynchronous callback architecture to avoid blocking runtime threads. Synchronous gauges are often mishandled or dropped depending on how the OTLP exporter state is initialized.

I had to completely refactor the metrics engine to use an Observable Gauge, passing a dedicated callback function that the SDK invokes automatically on its own collection cycle:

Python

THE WORKING FIX: Migrating to an Asynchronous Observable Gauge

from opentelemetry import metrics

meter = metrics.get_meter("system-monitor")

def get_cpu_callback(options):
# The SDK calls this function natively on every collection interval
return [metrics.Observation(get_current_cpu_percent())]

meter.create_observable_gauge(
name="system.cpu.utilization",
callbacks=[get_cpu_callback],
description="Tracks real-time system CPU load percentage"
)
As soon as I spun up the script with the callback logic, the OTLP exporter began generating beautiful time-series line charts directly within the SigNoz metrics dashboard. Lesson learned: don't push data when the OpenTelemetry SDK wants to pull it.

Mistake #2: The Broken Waterfall (Flat Spans vs. Nested Contexts)
When I finally got my first trace batches to stream into the SigNoz Trace Explorer, the structural layout looked entirely wrong. Instead of a single, coherent incident timeline showing a clean hierarchy, SigNoz rendered ten completely disconnected, flat timelines:

Plaintext
❌ WHAT I SAW ORIGINALLY:
Investigation (Trace ID: a1b2...)
Recommendation (Trace ID: c3d4...)
Firewall Execution (Trace ID: e5f6...)
Because every stage was being drawn as a root span, I couldn't track the lineage of an incident. If the execution failed at the firewall stage, I had no clean visual way to trace it back to the specific LLM confidence calculation that triggered it.

The Fix
The bug was a classic context propagation error. In Python, if you use a standard async loop or call distinct functions across different files, the active OpenTelemetry tracing context can easily drop out if you don't explicitly pass it or set it inside the current execution context thread.

I refactored the parent loop inside incident_commander.py to leverage the Python context manager, forcing child tasks to explicitly yield inside the parent scope:

Python

THE FIX: Ensuring tight nesting of distributed trace spans

from opentelemetry import trace

tracer = trace.get_tracer(name)

def run_incident_workflow(incident_data):
# Establish the overarching parent context span
with tracer.start_as_current_span("AI-Incident-Investigation") as parent_span:
parent_span.set_attribute("incident.id", incident_data["id"])

    # Explicitly passing structural execution tokens down the pipeline
    analyze_threat_vector(incident_data)
    execute_mitigation_workflow(incident_data)

def execute_mitigation_workflow(incident_data):
# This automatically detects and nests itself under the active parent span
with tracer.start_as_current_span("FirewallSimulator.block_ip") as child_span:
child_span.set_attribute("tool.action", "drop_packet")
# Simulating file I/O blocklist updates
write_to_blocklist(incident_data["attacker_ip"])
This tiny adjustments paid off instantly. The next time I fired up the app, SigNoz rendered a gorgeous nested flamegraph waterfall:

Plaintext
✅ THE CORRECT HIERARCHY IN SIGNOZ:
AI-Incident-Investigation [Trace ID: 7fa98c...]
├── Analyze_Threat_Vector (LLM Ingestion)
└── Execute_Mitigation_Workflow
└── FirewallSimulator.block_ip
Mistake #3: The Windows Path Trap (spawn python3 ENOENT)
Because I develop across different operating environments, I cloned my Next.js/Node.js telemetry aggregator backend onto my secondary Windows workstation to test stability.

I hit npm run dev, clicked the execution button on the frontend interface, and my Node runtime exploded with this raw error block:

Plaintext
events.js:377
throw er; // Unhandled 'error' event
^
Error: spawn python3 ENOENT
at Process.ChildProcess._handle.onread (internal/child_process.js:245:12)
at onErrorNT (internal/child_process.js:482:16)
The issue was painfully obvious yet highly disruptive: my Node core layer was hardcoded to invoke the background telemetry workers via a direct shell invocation to python3. While Linux and macOS environments default to python3, standard Windows installations map the runtime binary path explicitly to python.

The Fix
I refactored the application initialization handler to evaluate the host operating system dynamically via process.platform before spinning up the background telemetry workers:

TypeScript
import { spawn } from 'child_process';
import os from 'os';

// Dynamically bridge OS path execution discrepancies
const pythonExecutable = process.platform === 'win32' ? 'python' : 'python3';

const runTelemetryWorker = () => {
const workerProcess = spawn(pythonExecutable, ['telemetry_daemon.py'], {
env: { ...process.env, PYTHONUNBUFFERED: '1' }
});

workerProcess.stderr.on('data', (data) => {
console.error([WORKER ERROR]: ${data.toString()});
});
};
This brought cross-platform parity to the project, allowing the Node orchestrator to coordinate cleanly with the Python OpenTelemetry SDK regardless of the host OS architecture.

Mistake #4: The React UI Illusion (The Dashboard That Lied)
The biggest psychological trap during development was the fidelity of my own React application UI.

Because I was using standard React state objects (useState) to pass event records locally between my components, the user interface looked like an absolute engineering marvel. Spans lit up sequentially, data tables populated flawlessly, and progress bars moved perfectly.

But my frontend state was completely insulated from reality. When I purposefully pulled down my Docker instances of SigNoz, my React app still looked like it was working perfectly. The frontend was happily lying to me, masking the fact that the underlying OTLP connection was completely severed and exporting absolutely zero enterprise telemetry data.

The Lesson Learned
A healthy user interface does not mean a healthy telemetry pipeline.

I fundamentally altered my testing validation loop. I decoupled my reliance on the client-side state. I wrote a dedicated integration command to hit the backend directly:

Bash

Explicitly force clean dependency instantiation and trigger an isolated log event

npm install
curl -X POST http://localhost:3000/api/simulate-attack -d '{"type": "brute_force"}'
From that point forward, my source of truth wasn't my shiny React UI—it was the SigNoz Services and Traces explorer panes. If a trace didn't resolve inside SigNoz, the feature was treated as broken, regardless of how beautiful the CSS animation looked on the screen.

The Ultimate Payoff: Total Transparency into AI
Once these technical integration battles were won, the true power of an OpenTelemetry-native platform like SigNoz clicked for me.

By injecting custom metadata attributes directly into the span context blocks during the Gemini LLM investigation phase, I transformed standard distributed traces into comprehensive, auditable incident records. When an alert triggers, an engineer doesn't have to guess why the AI took a specific action. They can open up SigNoz, click on the active span, and immediately inspect the exact logical reasoning parameters:

incident.threat_type: Credential Stuffing

agent.confidence_score: 96%

agent.evidence_summary: 850 failed logins from Tor Exit Node IP detected inside a 120-second window.

llm.tokens_used: 1452

Plaintext
[Insert Screenshot Here: SigNoz Trace Detail side-panel clearly rendering custom metadata attributes mapping out the agent reasoning chains and exact token costs]
What I Learned from the Terminal
Before this hackathon, I viewed observability as a utility you wrap around server infrastructure to check if your hard drives are full or your APIs are returning HTTP 500s.

Building VigilTrace changed my perspective entirely. As we move into an era dominated by autonomous AI agents, LLM tool-calling, and complex multi-step prompt chains, observability is no longer just an infrastructure requirement—it is the core framework for AI safety and explainability.

By incorporating OpenTelemetry and SigNoz from line one, I didn't just build a security system that works. I built a system that can explicitly explain why it works, when it fails, and how much it costs down to the millisecond.

Technology Stack Utilized
Frontend/Orchestration: React, TypeScript, Vite, Node.js, Express, Framer Motion, Recharts

AI & Telemetry Core: Python, Google Gemini API, OpenTelemetry SDK, OTLP/gRPC Exporters

Observability Backend: SigNoz (Self-Hosted via Docker / ClickHouse architecture)

How to Build a Custom AI Security Agent with Coral & Gemini (My First Hackathon Build!)

Syed Kabeer Ali — Sun, 31 May 2026 13:15:17 +0000

Hey everyone! I just submitted my Track 2 project for the Pirates of the Coral-bean hackathon. I built The Coral Lookout, an autonomous AI agent that scans developer blogs (like Dev.to) to flag malicious npm packages and crypto scams.

Tbh, building this solo was a massive learning curve. I couldn't find a lot of tutorials on hooking up custom unmapped APIs to Coral, so I figured I’d write down exactly how I did it in case anyone else gets stuck on the same things I did.

Here is how to build your own custom AI agent from scratch!

The Stack
Data Pipeline: Coral CLI

LLM Engine: Gemini 2.5 Flash

Frontend UI: Python & Streamlit

Deployment: Docker + Railway

Step 1: Mapping the API with Coral (The Hard Part)
So, my initial plan was to just use Coral to query Dev.to articles. But I quickly realized that Dev.to isn't natively supported in Coral out of the box.

Instead of writing a wierd python scraping script, I learned you can just build a custom YAML source connector. You basically tell Coral how to read the JSON from the API.

Here is what my devto_guardian_connector.yaml looked like:

YAML
name: devto_agent
type: rest
base_url: "https://dev.to"
tables:

name: articles description: Latest articles pulled from Dev.to API request: method: GET path: /api/articles Pro tip: Make sure your indentation is perfect here. I spent like 45 minutes wondering why my SQL queries were failing only to realize I had an extra space in my YMAL file. 😅

Step 2: Querying the Data
Once the connector is linked, you can literally just use standard SQL to fetch internet data. In my app, I just ran a simple query to pull the titles and descriptions of the latest posts.

Python

just standard sql, no scraping required!

query = "SELECT title, description, url FROM devto_agent.articles LIMIT 15"
Step 3: Adding the Brains (Gemini 2.5 Flash)
Now that we have the raw text, we need to figure out if it's safe or if it's a scam. I used the Gemini API for this because its super fast.

I passed the title and description into Gemini and asked it to act like a cybersecurity expert (or in my case, a Pirate Oracle).
Wait, actually before you do this—make sure you load your API keys using dotenv in python, otherwise your app will crash instantly when you try to run it.

I wrote a prompt telling Gemini to look for high-risk signals, like links asking you to run suspicious npm install commands or random crypto airdrops. It returns a "confidence score" and a short explanation of why it flagged it.

Step 4: The Streamlit UI
I didn't want to build a boring corporate dashboard. I wanted this to feel like a real digital first mate!

I used Streamlit to build the UI. Streamlit is great, but styling it can be a pain. If you want to build a custom dark theme like I did, you have to inject CSS directly into the app using unsafe_allow_html=True.

Python
st.markdown("""

.threat-box { border-left: 5px solid red; background-color: #1a1a1a; }

""", unsafe_allow_html=True)
This let me create a really cool chronological timeline. When the agent finds a safe article, it logs it quietly. When it spots a threat, it flashes a red alert box and quarantines the link.

Step 5: Shipping it
Finally, I wrote a quick Dockerfile and pushed the whole thing to GitHub. I connected my repo to Railway, and it automatically built and deployed the app. (If your Railway build fails on the first try, just double check that your Streamlit port is set to 0.0.0.0 in your start script).

Conclusion
Building this agent definetly pushed me to my limits, but bridging an unstructured API to a SQL database and feeding it into an LLM is a superpower.

You can check out my full code here: https://github.com/MaskedMan-code/devto-guardian-agent