Building SentinelAI: How I Used SigNoz & OpenTelemetry to Make AI Agents Observable

Building SentinelAI: How I Used SigNoz & OpenTelemetry to Make AI Agents Observable

From black-box AI workflows to real-time traces, logs, metrics, security monitoring, and AI-powered root cause analysis.

Hook

AI agents are becoming increasingly capable—they can reason, call tools, retrieve knowledge, and automate complex workflows. But when an agent fails in production, the biggest challenge isn't fixing the bug—it's understanding why it happened.

During the Agents of SigNoz Hackathon by WeMakeDevs, I wanted to solve exactly this problem. Instead of building another AI chatbot, I built SentinelAI, an observability platform that gives developers complete visibility into AI agents using SigNoz and OpenTelemetry.

The Problem

While experimenting with AI applications, I realized that traditional logging wasn't enough.

Some common questions remained unanswered:

• Which tool call caused the failure?
• Why did latency suddenly increase?
• Which prompt consumed the most tokens?
• Why did the agent hallucinate?
• Which API became the bottleneck?

Without proper observability, AI systems become difficult to debug as they grow in complexity.

Why SigNoz?

I wanted an observability platform that could collect multiple signals in one place.

SigNoz stood out because it supports:

• OpenTelemetry-native instrumentation
• Distributed tracing
• Metrics
• Logs
• Dashboards
• Alerts

Instead of jumping between different monitoring tools, I could inspect an entire AI workflow from a single interface.

Project Architecture

Include architecture diagram here.

Explain:

       User
        ↓
  Frontend (React)
        ↓
   FastAPI Backend
        ↓
  OpenTelemetry SDK
        ↓
    SigNoz Cloud
        ↓
     Traces
    Metrics
      Logs
     Alerts
        ↓
SentinelAI Dashboard

Building SentinelAI

Instead of only monitoring servers, I wanted to monitor AI reasoning itself.

The application contains multiple modules.

Dashboard

The dashboard provides a quick overview of the system.

It displays:

• Agent Health Score
• Active Requests
• Average Latency
• Token Usage
• Estimated Cost
• Error Rate
• Live Alerts

The health score combines multiple signals into a single indicator, making it easier to identify unhealthy AI agents.

Live Agent Monitoring

Every AI request is tracked individually.

For each request I record:

• Prompt
• Response
• Execution time
• Token usage
• Model used
• Tool calls
• Status

Instead of manually searching logs, developers can inspect an individual execution from start to finish.

OpenTelemetry Integration

One of the most interesting parts of this project was instrumenting AI requests.

Every request generates telemetry that can later be visualized inside SigNoz.

This made it possible to understand where time was being spent:

• Prompt processing
• Retrieval
• LLM generation
• External API calls
• Database operations

Using SigNoz

SigNoz became the observability backbone of SentinelAI.

I used it for:

Distributed Traces

To visualize every stage of an AI request.

Metrics

To monitor:

• Request latency
• Error rate
• Token consumption

Logs

To inspect failures and debug issues.

Dashboards

To create an overview of system performance.

Alerts

To identify abnormal latency and failures quickly.

Security Monitoring

One feature I particularly enjoyed building was the Security Monitor.

It detects:

• Prompt Injection attempts
• Jailbreak prompts
• Suspicious tool calls
• High-risk requests

Each request receives a Security Risk Score, helping developers prioritize investigation.

AI Root Cause Analysis

Raw logs can still be difficult to understand.

So I added an AI-powered analysis engine.

Whenever an anomaly occurs, SentinelAI automatically generates:

• Incident summary
• Root cause
• Severity
• Suggested resolution
• Recommended next steps

This significantly reduces debugging time.

Analytics & Audit Export

To make the platform useful beyond debugging, I also implemented:

• Latency analytics
• Token usage trends
• Cost estimation
• JSON export
• CSV export

These exports make it easier to share telemetry with engineering teams and support compliance workflows.

Challenges I Faced

The biggest challenge wasn't building the UI.

The real challenge was understanding how observability works.

Initially, I assumed logs alone would be enough.

While integrating SigNoz, I realized that combining traces, metrics, and logs provides much richer context than relying on any single signal.

Another challenge was organizing AI telemetry in a way that remained easy to understand for developers.

What I Learned

This hackathon completely changed how I think about production AI.

Some key takeaways:

• Observability is as important as the AI model itself.
• Distributed tracing makes debugging much easier.
• Metrics help identify trends before users notice problems.
• AI applications require security monitoring in addition to performance monitoring.
• OpenTelemetry provides a flexible standard that works across different services.

Future Improvements

If I continue working on SentinelAI, I would like to add:

• Multi-agent workflow visualization
• Kubernetes monitoring
• Real-time anomaly detection
• Predictive incident alerts
• Slack & Microsoft Teams notifications
• Multi-tenant dashboards

Conclusion

Building SentinelAI taught me that successful AI applications need more than accurate responses—they need visibility, reliability, and security.

By combining OpenTelemetry with SigNoz, I was able to build a platform that helps developers understand every stage of an AI agent's execution instead of treating it as a black box.

This hackathon was a great opportunity to explore modern observability practices while building something practical for real-world AI systems.