DEV Community: Syed Kabeer Ali

I Built NeuroGraph: What DevOps Agents Can Learn from the Human Nervous System

Syed Kabeer Ali — Sun, 18 Oct 2026 18:23:29 +0000

A service reports that it's unhealthy. What do you do next?

You might check the logs, look at its dependencies, find the owner, inspect the relevant policy, or search for an incident that looks familiar. You might discover that the last restart didn't actually fix the problem.

The hard part isn't necessarily finding one piece of information. It's connecting the pieces and deciding what they mean together.

That problem reminded me of something from human anatomy and physiology: the body doesn't work as a collection of completely isolated parts. Nervous signals, feedback mechanisms, and organ systems interact. A symptom alone often doesn't explain the entire picture.

I started building NeuroGraph around a similar engineering intuition: an agent should understand relationships between operational facts, remember previous outcomes, and verify whether an attempted fix worked.

Repository: https://github.com/MaskedMan-code/NeuroGraph

Why I wanted a graph-based approach

During an incident, the relevant context can be scattered across service inventories, alert descriptions, runbooks, team ownership, policies, and incident history.

A conventional assistant might receive an error and produce a plausible answer. But plausibility isn't enough for an operational workflow. I wanted the system to establish what it knows, explain where the evidence comes from, and acknowledge what it cannot establish.

NeuroGraph represents the operational environment as a connected graph in FalkorDB.

The graph includes services, employees, teams, projects, dependencies, incidents, policies, documents, actions, outcomes, and memories.

For example, Graph Explorer can find a path like this:

emp-bob → svc-payment-api-stg → pol-staging-restart

That path tells us more than three independent records. It connects a person to a service they own and to the policy governing a possible remediation.

The graph is doing actual work here. It's not just a visualisation placed on top of an otherwise disconnected application.

The stack behind NeuroGraph

The project uses:

Next.js for the user interface.

FastAPI for backend APIs.

LangGraph for workflow orchestration and session continuity.

FalkorDB for graph relationships, operational data, and explicit application memory.

The interface includes an operations dashboard, Graph Explorer, Incident Copilot, Memory Timeline, Audit Log, and demo settings.

The workflow follows graph-backed evidence to identify an affected service, retrieve its owner and dependencies, inspect the applicable policy, look at prior incidents, and decide what the current request permits.

I think of the graph as a bit like a network of connected pathways. The analogy isn't meant to suggest that a database is a biological nervous system. The engineering point is simpler: context becomes more useful when relationships can be traversed and inspected.

Three request modes

I separated three things that are often blurred together in agent demos.

Investigate only

The agent gathers evidence about the service, its dependencies, policy, ownership, incident history, and relevant memories.

This mode must not produce operational proposals or execute actions. It should still suggest sensible investigative steps if the incident remains unresolved.

Recommend action

The agent can prepare a proposed remediation and explain the relevant evidence and policy constraints, but it doesn't execute the operation.

That separation matters. A recommendation should remain inspectable rather than quietly becoming an action.

Request action

An explicit action request enters the policy-controlled workflow. Permissions, environment restrictions, policy decisions, approval requirements, and tool-level checks constrain what can happen.

In this prototype, operations are simulated. The production restart scenario requires human approval and remains pending rather than bypassing that requirement.

I also tested alert content that included malicious instructions. The workflow treats those instructions as untrusted data; they don't get to override the system's policy simply because they appear in an incident report.

A restart isn't the same as recovery

This became one of my favourite tests.

The staging transaction processor was degraded. NeuroGraph accepted a simulated restart, but the verification step still observed the service as degraded.

A less careful demo could mark the whole workflow successful because the restart command returned successfully.

NeuroGraph records the distinction between the accepted command and the failed verification. The Audit Log records the observed status, and the failed outcome is stored in application memory.

Then I ran another investigation.

The agent retrieved the previous failed outcome rather than treating the earlier attempt as a successful recovery.

That's where another analogy from physiology becomes useful: feedback matters. A response has to be evaluated against the state it was supposed to improve. Sending a signal is not proof that the desired result occurred.

For an operational agent, the equivalent is straightforward:

Action accepted does not mean service recovered.

Memory should preserve the outcome, not just the attempt

NeuroGraph stores application memories as graph records with types and verification statuses. These include episodic memories, procedural knowledge, decisions, outcomes, and session-related records.

That is deliberately separate from LangGraph's workflow checkpoints.

The distinction is useful because different memories answer different questions:

An episodic memory can describe a previous incident.

A decision memory can record which policy applied.

An outcome memory can say whether the remediation was verified.

Procedural memory can capture steps worth considering during a future investigation.

A previous fix should not automatically become today's answer. The present incident may have a different root cause, and a past remediation might only have mitigated the symptoms.

In one investigation, the agent correctly retrieved an earlier database connection-pool issue while noting that the current root cause had not been established.

I also found and corrected a response-quality problem: when no operational proposal existed, the workflow could say that no action was needed even though a service remained degraded. The updated response recommends reviewing evidence, escalating to the owner, and continuing the investigation before closing the incident.

That correction was small, but the distinction matters. An agent can refrain from an unsafe operation and still give useful guidance.

Testing the prototype

I ran the backend suite and got 45 passing tests. The frontend passed TypeScript type checking and its Next.js production build.

I also manually tested graph traversal, incident investigation, recommendations, production approval, prompt-injection handling, simulated action execution, failed verification, the Audit Log, cross-run memory retrieval, and the dashboard.

Those tests establish that the demonstrated workflows run in this prototype. They are not a comprehensive security audit or a claim of production readiness.

What isn't finished yet

NeuroGraph currently runs in deterministic demo mode. No live LLM provider is configured, so the demo doesn't claim to show live LLM reasoning. Its operational actions are simulated and don't modify real infrastructure.

Authentication is demo-grade, too. The current version isn't suitable for handling private production operational data without further work on identity, authorization, durable storage, deployment, and security.

I want that boundary to be explicit because an attractive demo isn't the same thing as a production-ready incident platform.

What I'd like to explore next

The next stage is to finish the deployment validation, improve investigation guidance, review dependency advisories, and explore adding a live LLM without letting it bypass deterministic policy and tool controls.

I'm also interested in the broader question of agent memory: how do we make sure an agent remembers a failed attempt as a failure, instead of repeatedly recommending the same action?

For now, NeuroGraph has been a useful way to explore connected operational data, persistent memory, and verification-driven workflows.

Try the code: https://github.com/MaskedMan-code/NeuroGraph

I'd be interested in hearing how others approach agent memory and failed-action recovery. When an automated fix doesn't work, what should the system preserve so the next investigation actually benefits from the experience?

How I Debugged an AI Incident Response Pipeline Using OpenTelemetry and SigNoz (And the 5 Mistakes That Nearly Broke My Hackathon Project)

Syed Kabeer Ali — Sat, 18 Jul 2026 11:46:36 +0000

Three days before the hackathon submission deadline, my local React dashboard looked absolutely flawless.

I would trigger a simulated brute-force attack, the UI would instantly flash a red alert, the background Python engine would spin up a Google Gemini instance to analyze the threat vector, calculate a 94% confidence score, and cleanly trigger a simulated firewall tool to drop the offending IP address.

From the outside, it was a software developer's dream. The frontend updated perfectly in real-time. But when I opened my self-hosted SigNoz instance to verify the backend data, the Trace Explorer was a complete ghost town.

That was the exact moment I realized a foundational truth of modern software engineering: building an autonomous AI agent is only half the problem—making its internal reasoning transparent, observable, and auditable is the actual challenge.

This is the raw engineering diary of how I built VigilTrace—an autonomous AI Incident Commander—and the exact telemetry roadblocks, Windows-specific traps, and OpenTelemetry architectural bugs I had to fight through to make it truly observable.

What I Was Building: VigilTrace
The design philosophy behind VigilTrace is simple: If you can't observe your AI agents, you don't own them.

Instead of creating a black-box AI script that blindly triggers a shell command when a log drops, I architected an interactive pipeline that treats AI analysis as a multi-stage, regulated security investigation. The workflow operates as follows:

Plaintext
Server Simulator ──> Access Logs ──> Incident Commander (Python)
│
SigNoz UI <── OTLP Collector <── OpenTelemetry SDK <── Gemini AI
│
[Blocklist.json] <── Firewall Simulator
Every single link in this chain—from the raw token usage inside the Gemini LLM step to the file I/O latency inside the FirewallSimulator—had to emit structured telemetry packets over gRPC to a local OpenTelemetry Collector monitored via SigNoz.

Mistake #1: The Phantom Metrics (Why create_gauge Left Me in the Dark)
My first major roadblock was system metrics. I wanted to capture real-time resource spikes (cpu_usage and memory_usage) on the machine while the AI engine was chewing through dense security logs.

I spun up a background metrics daemon in Python and wrote what I assumed was standard OpenTelemetry code:

Python

THE BUGGY CODE THAT PRODUCED ZERO DATA IN SIGNOZ

from opentelemetry import metrics

meter = metrics.get_meter("system-monitor")
cpu_gauge = meter.create_gauge(
name="system.cpu.utilization",
description="Tracks CPU load metrics"
)

Polling loop trying to push values directly

while True:
cpu_gauge.set(get_current_cpu_percent())
time.sleep(5)
The script executed without crashing. No console errors, no warnings. Yet, when I checked the Metrics Explorer inside SigNoz, the dropdown menu for system.cpu.utilization simply did not exist.

The Fix
After diving into the official OpenTelemetry documentation, I realized a crucial architectural detail I had missed: metrics in OTel are primarily driven by a pull-based or asynchronous callback architecture to avoid blocking runtime threads. Synchronous gauges are often mishandled or dropped depending on how the OTLP exporter state is initialized.

I had to completely refactor the metrics engine to use an Observable Gauge, passing a dedicated callback function that the SDK invokes automatically on its own collection cycle:

Python

THE WORKING FIX: Migrating to an Asynchronous Observable Gauge

from opentelemetry import metrics

meter = metrics.get_meter("system-monitor")

def get_cpu_callback(options):
# The SDK calls this function natively on every collection interval
return [metrics.Observation(get_current_cpu_percent())]

meter.create_observable_gauge(
name="system.cpu.utilization",
callbacks=[get_cpu_callback],
description="Tracks real-time system CPU load percentage"
)
As soon as I spun up the script with the callback logic, the OTLP exporter began generating beautiful time-series line charts directly within the SigNoz metrics dashboard. Lesson learned: don't push data when the OpenTelemetry SDK wants to pull it.

Mistake #2: The Broken Waterfall (Flat Spans vs. Nested Contexts)
When I finally got my first trace batches to stream into the SigNoz Trace Explorer, the structural layout looked entirely wrong. Instead of a single, coherent incident timeline showing a clean hierarchy, SigNoz rendered ten completely disconnected, flat timelines:

Plaintext
❌ WHAT I SAW ORIGINALLY:
Investigation (Trace ID: a1b2...)
Recommendation (Trace ID: c3d4...)
Firewall Execution (Trace ID: e5f6...)
Because every stage was being drawn as a root span, I couldn't track the lineage of an incident. If the execution failed at the firewall stage, I had no clean visual way to trace it back to the specific LLM confidence calculation that triggered it.

The Fix
The bug was a classic context propagation error. In Python, if you use a standard async loop or call distinct functions across different files, the active OpenTelemetry tracing context can easily drop out if you don't explicitly pass it or set it inside the current execution context thread.

I refactored the parent loop inside incident_commander.py to leverage the Python context manager, forcing child tasks to explicitly yield inside the parent scope:

Python

THE FIX: Ensuring tight nesting of distributed trace spans

from opentelemetry import trace

tracer = trace.get_tracer(name)

def run_incident_workflow(incident_data):
# Establish the overarching parent context span
with tracer.start_as_current_span("AI-Incident-Investigation") as parent_span:
parent_span.set_attribute("incident.id", incident_data["id"])

    # Explicitly passing structural execution tokens down the pipeline
    analyze_threat_vector(incident_data)
    execute_mitigation_workflow(incident_data)

def execute_mitigation_workflow(incident_data):
# This automatically detects and nests itself under the active parent span
with tracer.start_as_current_span("FirewallSimulator.block_ip") as child_span:
child_span.set_attribute("tool.action", "drop_packet")
# Simulating file I/O blocklist updates
write_to_blocklist(incident_data["attacker_ip"])
This tiny adjustments paid off instantly. The next time I fired up the app, SigNoz rendered a gorgeous nested flamegraph waterfall:

Plaintext
✅ THE CORRECT HIERARCHY IN SIGNOZ:
AI-Incident-Investigation [Trace ID: 7fa98c...]
├── Analyze_Threat_Vector (LLM Ingestion)
└── Execute_Mitigation_Workflow
└── FirewallSimulator.block_ip
Mistake #3: The Windows Path Trap (spawn python3 ENOENT)
Because I develop across different operating environments, I cloned my Next.js/Node.js telemetry aggregator backend onto my secondary Windows workstation to test stability.

I hit npm run dev, clicked the execution button on the frontend interface, and my Node runtime exploded with this raw error block:

Plaintext
events.js:377
throw er; // Unhandled 'error' event
^
Error: spawn python3 ENOENT
at Process.ChildProcess._handle.onread (internal/child_process.js:245:12)
at onErrorNT (internal/child_process.js:482:16)
The issue was painfully obvious yet highly disruptive: my Node core layer was hardcoded to invoke the background telemetry workers via a direct shell invocation to python3. While Linux and macOS environments default to python3, standard Windows installations map the runtime binary path explicitly to python.

The Fix
I refactored the application initialization handler to evaluate the host operating system dynamically via process.platform before spinning up the background telemetry workers:

TypeScript
import { spawn } from 'child_process';
import os from 'os';

// Dynamically bridge OS path execution discrepancies
const pythonExecutable = process.platform === 'win32' ? 'python' : 'python3';

const runTelemetryWorker = () => {
const workerProcess = spawn(pythonExecutable, ['telemetry_daemon.py'], {
env: { ...process.env, PYTHONUNBUFFERED: '1' }
});

workerProcess.stderr.on('data', (data) => {
console.error([WORKER ERROR]: ${data.toString()});
});
};
This brought cross-platform parity to the project, allowing the Node orchestrator to coordinate cleanly with the Python OpenTelemetry SDK regardless of the host OS architecture.

Mistake #4: The React UI Illusion (The Dashboard That Lied)
The biggest psychological trap during development was the fidelity of my own React application UI.

Because I was using standard React state objects (useState) to pass event records locally between my components, the user interface looked like an absolute engineering marvel. Spans lit up sequentially, data tables populated flawlessly, and progress bars moved perfectly.

But my frontend state was completely insulated from reality. When I purposefully pulled down my Docker instances of SigNoz, my React app still looked like it was working perfectly. The frontend was happily lying to me, masking the fact that the underlying OTLP connection was completely severed and exporting absolutely zero enterprise telemetry data.

The Lesson Learned
A healthy user interface does not mean a healthy telemetry pipeline.

I fundamentally altered my testing validation loop. I decoupled my reliance on the client-side state. I wrote a dedicated integration command to hit the backend directly:

Bash

Explicitly force clean dependency instantiation and trigger an isolated log event

npm install
curl -X POST http://localhost:3000/api/simulate-attack -d '{"type": "brute_force"}'
From that point forward, my source of truth wasn't my shiny React UI—it was the SigNoz Services and Traces explorer panes. If a trace didn't resolve inside SigNoz, the feature was treated as broken, regardless of how beautiful the CSS animation looked on the screen.

The Ultimate Payoff: Total Transparency into AI
Once these technical integration battles were won, the true power of an OpenTelemetry-native platform like SigNoz clicked for me.

By injecting custom metadata attributes directly into the span context blocks during the Gemini LLM investigation phase, I transformed standard distributed traces into comprehensive, auditable incident records. When an alert triggers, an engineer doesn't have to guess why the AI took a specific action. They can open up SigNoz, click on the active span, and immediately inspect the exact logical reasoning parameters:

incident.threat_type: Credential Stuffing

agent.confidence_score: 96%

agent.evidence_summary: 850 failed logins from Tor Exit Node IP detected inside a 120-second window.

llm.tokens_used: 1452

Plaintext
[Insert Screenshot Here: SigNoz Trace Detail side-panel clearly rendering custom metadata attributes mapping out the agent reasoning chains and exact token costs]
What I Learned from the Terminal
Before this hackathon, I viewed observability as a utility you wrap around server infrastructure to check if your hard drives are full or your APIs are returning HTTP 500s.

Building VigilTrace changed my perspective entirely. As we move into an era dominated by autonomous AI agents, LLM tool-calling, and complex multi-step prompt chains, observability is no longer just an infrastructure requirement—it is the core framework for AI safety and explainability.

By incorporating OpenTelemetry and SigNoz from line one, I didn't just build a security system that works. I built a system that can explicitly explain why it works, when it fails, and how much it costs down to the millisecond.

Technology Stack Utilized
Frontend/Orchestration: React, TypeScript, Vite, Node.js, Express, Framer Motion, Recharts

AI & Telemetry Core: Python, Google Gemini API, OpenTelemetry SDK, OTLP/gRPC Exporters

Observability Backend: SigNoz (Self-Hosted via Docker / ClickHouse architecture)

How to Build a Custom AI Security Agent with Coral & Gemini (My First Hackathon Build!)

Syed Kabeer Ali — Sun, 31 May 2026 13:15:17 +0000

Hey everyone! I just submitted my Track 2 project for the Pirates of the Coral-bean hackathon. I built The Coral Lookout, an autonomous AI agent that scans developer blogs (like Dev.to) to flag malicious npm packages and crypto scams.

Tbh, building this solo was a massive learning curve. I couldn't find a lot of tutorials on hooking up custom unmapped APIs to Coral, so I figured I’d write down exactly how I did it in case anyone else gets stuck on the same things I did.

Here is how to build your own custom AI agent from scratch!

The Stack
Data Pipeline: Coral CLI

LLM Engine: Gemini 2.5 Flash

Frontend UI: Python & Streamlit

Deployment: Docker + Railway

Step 1: Mapping the API with Coral (The Hard Part)
So, my initial plan was to just use Coral to query Dev.to articles. But I quickly realized that Dev.to isn't natively supported in Coral out of the box.

Instead of writing a wierd python scraping script, I learned you can just build a custom YAML source connector. You basically tell Coral how to read the JSON from the API.

Here is what my devto_guardian_connector.yaml looked like:

YAML
name: devto_agent
type: rest
base_url: "https://dev.to"
tables:

name: articles description: Latest articles pulled from Dev.to API request: method: GET path: /api/articles Pro tip: Make sure your indentation is perfect here. I spent like 45 minutes wondering why my SQL queries were failing only to realize I had an extra space in my YMAL file. 😅

Step 2: Querying the Data
Once the connector is linked, you can literally just use standard SQL to fetch internet data. In my app, I just ran a simple query to pull the titles and descriptions of the latest posts.

Python

just standard sql, no scraping required!

query = "SELECT title, description, url FROM devto_agent.articles LIMIT 15"
Step 3: Adding the Brains (Gemini 2.5 Flash)
Now that we have the raw text, we need to figure out if it's safe or if it's a scam. I used the Gemini API for this because its super fast.

I passed the title and description into Gemini and asked it to act like a cybersecurity expert (or in my case, a Pirate Oracle).
Wait, actually before you do this—make sure you load your API keys using dotenv in python, otherwise your app will crash instantly when you try to run it.

I wrote a prompt telling Gemini to look for high-risk signals, like links asking you to run suspicious npm install commands or random crypto airdrops. It returns a "confidence score" and a short explanation of why it flagged it.

Step 4: The Streamlit UI
I didn't want to build a boring corporate dashboard. I wanted this to feel like a real digital first mate!

I used Streamlit to build the UI. Streamlit is great, but styling it can be a pain. If you want to build a custom dark theme like I did, you have to inject CSS directly into the app using unsafe_allow_html=True.

Python
st.markdown("""

.threat-box { border-left: 5px solid red; background-color: #1a1a1a; }

""", unsafe_allow_html=True)
This let me create a really cool chronological timeline. When the agent finds a safe article, it logs it quietly. When it spots a threat, it flashes a red alert box and quarantines the link.

Step 5: Shipping it
Finally, I wrote a quick Dockerfile and pushed the whole thing to GitHub. I connected my repo to Railway, and it automatically built and deployed the app. (If your Railway build fails on the first try, just double check that your Streamlit port is set to 0.0.0.0 in your start script).

Conclusion
Building this agent definetly pushed me to my limits, but bridging an unstructured API to a SQL database and feeding it into an LLM is a superpower.

You can check out my full code here: https://github.com/MaskedMan-code/devto-guardian-agent