Turning SigNoz Telemetry into AI Runtime Safety Intelligence
AI agents usually do not fail like traditional software. They often keep running while quietly getting worse: retrying the same tool call, looping through planning steps, using too many tokens, or stalling on a dependency that never responds. From the outside, nothing looks completely broken. From the inside, the workflow is becoming unsafe.
That gap is what Sentinel was built to close.
Sentinel is an AI Runtime Safety Supervisor built on top of SigNoz and OpenTelemetry. Instead of replacing observability, it adds a safety layer on top of it. SigNoz tells you what happened. Sentinel tells you whether the workflow is safe, why the problem happened, and what an engineer should do next.
I built Sentinel for the Agents of SigNoz hackathon as a way to explore a simple but important idea: telemetry is most valuable when it becomes evidence-backed action, not just another list of traces.
What Sentinel does
Sentinel turns runtime telemetry into explainable engineering intelligence.
At a high level, it:
ingests telemetry from a demo generator or a live SigNoz deployment,
converts traces, metrics, and logs into internal workflow evidence,
runs deterministic runtime safety checks,
produces a Runtime Safety Score,
explains the result with evidence references,
and recommends what the engineering team should review next.
The goal is not to create another observability dashboard. The goal is to help teams understand when an autonomous workflow is drifting into unsafe behavior.
Why I chose this problem
As AI systems become more autonomous, the failure modes become less obvious.
A workflow may still return responses while quietly consuming too many tokens, hammering the same tool, stalling on a blocked dependency, or entering a recursive loop. Traditional observability tools are essential, but they usually leave the final judgment to the engineer.
Sentinel adds a structured safety layer on top of that telemetry.
That is important because in real systems, the question is often not just “What happened?” but:
Is this workflow still healthy?
Why is the behavior risky?
What evidence supports that conclusion?
What should we do now?
Sentinel is my attempt to answer those questions in a repeatable, traceable way.
Architecture
Sentinel uses a straightforward pipeline:
AI Agent → OpenTelemetry → SigNoz → SigNoz MCP → Sentinel → Runtime Safety Score → Explanation → Recommendation
The important part is that Sentinel does not invent its own telemetry source. It consumes evidence from SigNoz and transforms it into runtime analysis.
The main layers are:
Telemetry ingestion: receives demo or live telemetry
Runtime correlation: links traces, metrics, and logs into coherent workflow evidence
Runtime assessment: computes a deterministic safety score
Explanation generation: turns the score into human-readable reasoning
Recommendation generation: suggests the next engineering action
Result publication: exposes the analysis in the dashboard and workflow pages
This structure keeps the system explainable and easy to debug.
How I used SigNoz
SigNoz is the foundation of the live telemetry path.
I used it in three places:
- OpenTelemetry export
Sentinel’s application can emit traces through OpenTelemetry. Those traces are exported to SigNoz, where they become part of the observable runtime data.
- SigNoz MCP server
Instead of directly scraping telemetry manually, Sentinel connects to the SigNoz MCP server using the Model Context Protocol. That lets Sentinel ask for traces, trace details, logs, and metrics in a structured way.
- Runtime analysis
The MCP results are mapped into Sentinel’s internal workflow model. From there, Sentinel computes a safety score and generates evidence-backed explanations and recommendations.
This was the key integration point in the project. Once the parser matched the actual MCP response shape, live traces started flowing into the dashboard correctly.
Demo mode and live mode
Sentinel supports two operating modes.
Demo mode
Demo mode ships with realistic AI workflow telemetry so the full product can be explored without requiring a live SigNoz setup. This makes it easy to demo the system and test the safety pipeline in a predictable environment.
Live SigNoz mode
When TELEMETRY_SOURCE=signoz, Sentinel connects to the local SigNoz MCP server, loads real traces, and shows the same runtime analysis pipeline on live data.
The best part is that the analysis path stays the same in both modes. Only the source of telemetry changes.
That made debugging much easier, because I could first validate the pipeline with demo data and then switch to live SigNoz once the MCP integration was ready.
A few things I learned while building it
The biggest lesson was that integration work is often about matching reality, not assumptions.
At first, I assumed the MCP response would be simple and flat. It was not. The actual response needed parsing from nested data, and I had to update the parser to map the real row structure into Sentinel’s workflow model.
I also learned that fallback behavior matters.
When the MCP server was not available, Sentinel should not crash or show a broken state. It should clearly show that it is using fallback data while keeping the user informed about what happened. That made the dashboard much more usable.
Another important lesson was around state.
I found a bug where the dashboard could show stale source information after refresh. That happened because cached evidence was being reused without refreshing the source status. Fixing that made the demo much more reliable.
Those issues were small individually, but together they made the difference between a prototype and a project that feels trustworthy.
What the dashboard shows
Sentinel’s dashboard is designed to answer the basic operational questions quickly.
You can see:
how many workflows are healthy, watch, warning, or critical,
the active telemetry source,
whether Sentinel is connected to SigNoz or falling back,
how many traces were loaded,
the Runtime Safety Score for each workflow,
the evidence chain behind each conclusion,
and the recommended next action.
The workflow details page goes one step further and shows the score formula, timeline, supporting evidence, and the reasoning behind the final recommendation.
That was important to me because I did not want Sentinel to feel like a black box.
Why evidence matters
A safety score is only useful if you can trust it.
That is why Sentinel keeps evidenceRefs attached to every assessment, explanation, and recommendation. Those references point back to specific telemetry items such as spans, metrics, and logs.
This gives the system a few important properties:
observable
explainable
traceable
verifiable
human-governed
Sentinel recommendations are advisory only. It does not terminate workflows or take automated control actions. It helps engineers make better decisions with better context.
Screenshots
Here are a few screenshots from the project:
Dashboard Overview
Demo Mode
Runtime Assessment
Evidence Traceability
Evidence Chain
Live SigNoz Integration
Tech stack
Sentinel is built with:
Next.js 15
TypeScript
Tailwind CSS
OpenTelemetry
SigNoz
MCP (Model Context Protocol)
deterministic heuristic analysis for runtime safety scoring
What I would improve next
If I continue working on Sentinel, the next steps would be:
richer historical analysis,
more advanced multi-agent correlation,
policy-based safety rules,
better workflow naming from live telemetry,
and stronger predictive risk scoring.
Those are all natural extensions, but the current version already demonstrates the core idea: observability can be turned into actionable runtime safety intelligence.
Final thoughts
Building Sentinel taught me that observability becomes much more powerful when it is tied to a clear decision-making layer.
SigNoz gives the raw visibility. Sentinel converts that visibility into a structured safety signal that engineers can act on.
For autonomous systems, that extra layer matters. It helps teams understand not just what the system did, but whether it is still safe to keep running.
Links
GitHub repository: https://github.com/25cs249-create/Sentinel-Runtime-Safety
Demo video: https://drive.google.com/file/d/1kgzIOSTd5pY-geeudHDrfz0KxJf2pJxD/view?usp=drivesdk
Top comments (0)