Building a Firewall and a Flight Recorder for AI Agent Tool Calls with SigNoz
How we built an OpenTelemetry-native policy engine that blocks malicious AI agent actions while creating a complete forensic trail.
Our on-call AI agent picked up a production incident, read the investigation notes, pulled a production database password, and then attempted to send it to an external URL.
Every individual tool call returned a perfectly normal success response.
Nothing on the dashboard looked suspicious.
The only reason we caught it was because the same OpenTelemetry span that SigNoz was visualizing was also the exact record our policy engine used to make its enforcement decision.
The enforcement decision and the forensic evidence were the same object.
That became the central idea behind MCP Sentinel.
What We Built
We built MCP Sentinel for the Agents of SigNoz Hackathon.
It's a lightweight proxy that sits between an AI agent and every tool it can invoke using the Model Context Protocol (MCP).
Every tool call passes through a single interception point.
At that point we:
- Start an OpenTelemetry span
- Evaluate security policies
- Either allow, redact, or block the request
- Record everything as telemetry
The architecture is intentionally simple.
AI Agent
│
▼
MCP Sentinel
│
┌──┴────────────┐
│ Policy Engine │
│ OpenTelemetry │
└──┬────────────┘
│
▼
SigNoz
│
▼
Actual Tool
Why We Built It
AI agents no longer wait for humans to tell them exactly which API to call.
They decide for themselves.
Give an agent production credentials and a single poisoned prompt can turn it into a data exfiltration engine—even if every API involved returns HTTP 200.
Traditional monitoring tells you that:
- the database query succeeded
- the HTTP request succeeded
- the filesystem read succeeded
It does not tell you:
"This agent just read a secret and immediately attempted to send it outside your network."
We wanted one place that both:
- enforced security policies
- recorded exactly what happened
SigNoz became the center of that design because it already provides:
- traces
- metrics
- logs
- dashboards
- alerts
from a single self-hosted OpenTelemetry backend.
One Span Per Tool Call
MCP Sentinel is built on FastMCP.
Every tool invocation passes through a single middleware hook:
on_call_tool
For every request we:
- Evaluate policies
- Start a span
- Attach metadata
- Record the decision
The core logic looks like this (trimmed):
decision = self._engine.evaluate(session_id, tool_name, arguments)
with self._tracer.start_as_current_span(
f"execute_tool {tool_name}",
kind=SpanKind.CLIENT
) as span:
span.set_attribute("mcp.tool.name", tool_name)
span.set_attribute("gen_ai.tool.name", tool_name)
span.set_attribute(
"gen_ai.tool.call.arguments",
_as_json(arguments)
)
span.set_attribute(
"sentinel.verdict",
decision.verdict
)
span.set_attribute(
"sentinel.risk_score",
decision.risk_score
)
if decision.rules:
span.set_attribute(
"sentinel.rules",
",".join(decision.rules)
)
if decision.reason:
span.set_attribute(
"sentinel.reason",
decision.reason
)
We reused the standard OpenTelemetry semantic conventions wherever possible:
mcp.*gen_ai.*
Everything specific to Sentinel lives under:
sentinel.verdictsentinel.rulessentinel.reasonsentinel.risk_score
One FastMCP detail turned out to be important:
To block a tool call you must return an error result from the middleware hook.
Raising an exception simply becomes a generic tool failure, meaning the agent cannot distinguish a policy decision from a runtime error.
The Policies
Every session is evaluated against five security policies.
The most interesting is risky_sequence.
It fires whenever an agent:
- reads sensitive information
- then attempts to send data outside the trusted environment
Another policy, pii_egress, scans outbound arguments for secrets and personally identifiable information.
In our attack scenario both policies triggered simultaneously on the same http_post request.
The request never reached the destination.
Caption
The live control plane. The agent reads incident notes, retrieves a production secret, then attempts an
http_post. Bothrisky_sequenceandpii_egresstrigger, and the request is blocked before leaving the system.
What SigNoz Gave Us
Once every tool call became a span, most of the remaining functionality came almost for free.
Every policy decision is also written as a structured log correlated by trace ID.
Opening the blocked request in SigNoz immediately answers:
- which policy fired
- why it fired
- which span recorded it
No custom audit UI required.
IMAGE 2
Caption
The blocked request appears as a correlated log record. The trace ID links directly back to the OpenTelemetry span that captured the decision.
Dashboards Directly from Traces
For the overview dashboard we deliberately skipped the metrics query builder.
Instead, we queried SigNoz's ClickHouse trace store directly using SQL.
The questions we cared about were already encoded as span attributes.
Counting calls by verdict became:
SELECT
attributes_string['sentinel.verdict'] AS verdict,
count() AS calls
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'mcp-sentinel'
AND name LIKE 'execute_tool%'
GROUP BY verdict
ORDER BY calls DESC
One dashboard answers the security question we cared about most:
Did any blocked request actually leave the system?
Because blocked requests are never forwarded, every blocked span is proof that nothing escaped.
IMAGE 3
Caption
MCP Sentinel Overview dashboard showing tool call activity, policy decisions, per-tool latency, and security insights, all generated directly from OpenTelemetry spans.
No additional data pipeline was required.
Everything came directly from telemetry.
For teams that prefer metrics, Sentinel also exports:
sentinel.tool_calls
tagged with:
- verdict
- tool
- rule
making it easy to build alerts and dashboards using standard metric workflows.
SigNoz Provisioned Itself (Almost)
One feature we're particularly happy with is that Sentinel provisions its own observability environment.
A setup script authenticates against the SigNoz REST API and automatically creates:
- dashboards
- notification channels
- alerts
The system that generates security events also installs the dashboards that monitor them.
What We Learned
Two observations stayed with us.
1. Agents Don't Stop After the First Block
In another experiment we blocked a destructive command.
Instead of giving up, the agent began searching for alternative ways to achieve the same goal.
It requested:
- an admin token
- a bypass key
It found neither.
Even if it had, any attempt to send them externally would have triggered the same sequence policy.
This behavior wasn't deterministic—we only observed it once—but it changed how we think about agent security.
Guardrails need to cover the entire sequence, not just individual API calls.
Every step needs to be recorded.
2. Security Decisions Are Just Telemetry
The second insight surprised us.
Treating policy decisions as telemetry worked because SigNoz is already built to process telemetry.
We never built:
- a separate audit database
- a custom event store
- a reconciliation service
The OpenTelemetry span itself became:
- the enforcement record
- the audit log
- the dashboard source
- the alert trigger
For a use case SigNoz wasn't explicitly designed for—AI agent security—it fit remarkably well.
Where We Want to Go Next
The next improvement is end-to-end trace context propagation.
Today the agent, Sentinel, and downstream tools generate related traces.
The next version will connect them into a single distributed trace so the complete execution path appears in one timeline.
The core idea, however, is already working.
One proxy.
One span per tool call.
One record that both stops an attack and documents exactly why it was stopped.
If you're giving AI agents access to real production systems, building that control plane early is worth the effort.
The repository includes:
- MCP proxy
- Policy engine
- Five built-in security policies
- Attack scenarios
- SigNoz provisioning scripts
- OpenTelemetry instrumentation
If you're experimenting with production AI agents, we'd love to hear your feedback and contributions.
Top comments (0)