DEV Community: Mukund B

Moving Grafana dashboards to SigNoz: simplified and accurate

Mukund B — Sun, 26 Jul 2026 20:59:16 +0000

How we built a Grafana to SigNoz compiler that checks every panel three ways.

We moved a Grafana dashboard to SigNoz, and instead of trusting that the panels came out right, we made the tool prove it, chart by chart.

One panel came back wrong, and the check told us exactly which layer was at fault—the translation, not the data.

That is the whole idea.

A migration you can measure beats a migration you have to hope about.

Why we built this

Teams stay on Prometheus and Grafana for one reason: the rewrite.

A real setup has hundreds of PromQL queries and panels behind it. SigNoz has an importer, so getting the dashboard across is not the hard part.

The hard part is trust.

After conversion, nobody can tell you whether panel 34 still means the same thing it did before. A chart can render perfectly and still show the wrong number, and you only find out when someone makes a decision based on bad data.

So we built Migration Proof for the Agents of SigNoz Hackathon.

It reads a Grafana dashboard JSON, extracts every panel and its PromQL, compiles each query into a SigNoz Query Builder query, and then verifies every panel against live data.

The result is a simple scorecard:

✅ Pass
❌ Mismatch
⚠️ Flagged for review

SigNoz sits at the center of this project in a role it wasn't specifically designed for.

We use it as a query engine and compare its answers directly against Prometheus.

The three-way check

This is the part that makes the verification honest.

If you compare only two things—our translated query against Prometheus—a mismatch has two possible causes:

the data never landed correctly in SigNoz
our translation is wrong

One diff hides two different bugs.

So instead we compare three systems.

Prometheus is the source of truth.
SigNoz PromQL proves that the data landed correctly.
SigNoz Query Builder proves that our translation is correct.

Now the failure modes separate naturally.

If:

Prometheus == SigNoz PromQL
but SigNoz Query Builder != Prometheus

then the compiler is wrong.

If:

SigNoz PromQL != Prometheus

then the ingestion pipeline is wrong.

Instead of simply telling you that a panel failed, the verification tells you which layer to fix.

This only works because SigNoz supports both native PromQL execution and Query Builder queries against the same underlying data.

That single capability makes fault isolation possible.

Prometheus (Truth)

SigNoz PromQL (Ingestion Validation)

SigNoz Query Builder (Translation Validation)

What the compiler actually does

The translator is written in Python.

We parse PromQL using promql-parser, a Rust-backed parser that produces a complete syntax tree.

That syntax tree becomes a typed intermediate representation before finally being emitted as SigNoz Query Builder JSON.

The compiler currently has 64 unit tests.

Instead of trying to support every corner of PromQL, we intentionally support a bounded subset based on what appears most often in production dashboards.

That includes:

label matchers
rate()
increase()
aggregations with by/without
histogram_quantile()
arithmetic expressions

Anything outside that subset is never guessed.

It is flagged for manual review instead.

This is one of the most important design decisions in the project.

A wrong conversion that appears successful is far worse than refusing to translate something.

Whenever the compiler is unsure, it tells you.

The demo, and the verification loop

On a clean run the parity scorecard reports:

6 out of 7 panels verified successfully.

For every successful panel, all three result lines overlap perfectly:

Prometheus
SigNoz PromQL
SigNoz Query Builder

The remaining panel is intentionally flagged.

It contains topk(), which is currently outside the supported subset.

Rather than generating an incorrect conversion, the compiler surfaces it for review.

Next we demonstrate the verification loop.

We intentionally inject a compiler bug.

Instead of translating:

rate(http_requests_total[5m])

the compiler accidentally emits a query against the raw counter.

The resulting chart still looks believable.

It is also completely wrong.

After re-running verification:

Prometheus still matches SigNoz PromQL.
The Query Builder result diverges.

The scorecard immediately localizes the problem.

The ingestion is correct.

The compiler is not.

We fix the bug, run verification again, and the panel returns to green.

Detect.

Localize.

Fix.

Verify.

Alerts: where importing really bites

Dashboards are only half of a migration.

Alerts are where things become dangerous.

SigNoz currently has a known issue where an alert rule can save successfully yet never actually fire.

That means:

"Alert imported successfully"

does not necessarily mean

"Alert actually works."

To verify alerts, we translate the same threshold alert into both systems and evaluate them against identical data.

We observe both state machines.

Prometheus transitions:

Pending → Firing

SigNoz transitions:

Pending → Firing

Both systems fire.

Not imported.

Proven.

What we learned about SigNoz by verifying against it

Building against a live SigNoz deployment surfaced issues that a simple converter would never notice.

We only discovered them because every translated panel was continuously compared against Prometheus.

OpenTelemetry relabeling

The OpenTelemetry Collector renames several Prometheus labels during ingestion.

Without explicitly mapping those names, comparisons never line up correctly.

Counter temporality

Counter metrics return empty results unless cumulative temporality is enabled.

Without that flag, the query looks like there is simply no data, which sends debugging in completely the wrong direction.

Rate differences

SigNoz Query Builder computes rate() slightly differently from native PromQL because of different window calculations.

The difference is only a few percent, but it is measurable.

Instead of pretending the values are identical, we measured the deviation and introduced a documented tolerance.

Eventual consistency

The hardest issue appeared while stabilizing the demo.

SigNoz Query Builder ingests the newest data roughly ninety seconds behind real time.

Under database load, queries against "now" can return incomplete points.

A parity check that compares the most recent timestamps will randomly fail even when both systems agree.

The correct solution is simple.

End the comparison window a few minutes in the past and compare only stable data.

That is how a production-grade parity checker should behave anyway.

None of these observations are complaints.

They are the kinds of engineering details you only discover by building real systems against production software and validating every number.

For SigNoz, they are useful implementation details.

For anyone migrating dashboards, they are valuable lessons learned ahead of time.

The point

Parity you can measure beats conversion you have to trust.

Our compiler moves a Grafana dashboard into SigNoz and then proves every translated panel.

For every chart we can demonstrate one of two outcomes:

SigNoz returns the same values as Prometheus.
Or we show exactly where the disagreement occurs and which layer caused it.

That transforms dashboard migration from a leap of faith into a repeatable engineering process.

And that is what gives teams the confidence to actually migrate.

The repository includes:

the PromQL compiler
the bounded translation subset
the three-way parity verifier
the reproducible demo environment

If you're migrating dashboards from Grafana to SigNoz, or building tooling around observability, we'd love your feedback and contributions.

Building MCP Sentinel: An Open-Source Security Scanner for Model Context Protocol (MCP) Servers

Mukund B — Sun, 26 Jul 2026 17:06:41 +0000

Building a Firewall and a Flight Recorder for AI Agent Tool Calls with SigNoz

How we built an OpenTelemetry-native policy engine that blocks malicious AI agent actions while creating a complete forensic trail.

Our on-call AI agent picked up a production incident, read the investigation notes, pulled a production database password, and then attempted to send it to an external URL.

Every individual tool call returned a perfectly normal success response.

Nothing on the dashboard looked suspicious.

The only reason we caught it was because the same OpenTelemetry span that SigNoz was visualizing was also the exact record our policy engine used to make its enforcement decision.

The enforcement decision and the forensic evidence were the same object.

That became the central idea behind MCP Sentinel.

What We Built

We built MCP Sentinel for the Agents of SigNoz Hackathon.

It's a lightweight proxy that sits between an AI agent and every tool it can invoke using the Model Context Protocol (MCP).

Every tool call passes through a single interception point.

At that point we:

Start an OpenTelemetry span
Evaluate security policies
Either allow, redact, or block the request
Record everything as telemetry

The architecture is intentionally simple.

AI Agent
    │
    ▼
MCP Sentinel
    │
 ┌──┴────────────┐
 │ Policy Engine │
 │ OpenTelemetry │
 └──┬────────────┘
    │
    ▼
 SigNoz
    │
    ▼
Actual Tool

Why We Built It

AI agents no longer wait for humans to tell them exactly which API to call.

They decide for themselves.

Give an agent production credentials and a single poisoned prompt can turn it into a data exfiltration engine—even if every API involved returns HTTP 200.

Traditional monitoring tells you that:

the database query succeeded
the HTTP request succeeded
the filesystem read succeeded

It does not tell you:

"This agent just read a secret and immediately attempted to send it outside your network."

We wanted one place that both:

enforced security policies
recorded exactly what happened

SigNoz became the center of that design because it already provides:

traces
metrics
logs
dashboards
alerts

from a single self-hosted OpenTelemetry backend.

One Span Per Tool Call

MCP Sentinel is built on FastMCP.

Every tool invocation passes through a single middleware hook:

on_call_tool

For every request we:

Evaluate policies
Start a span
Attach metadata
Record the decision

The core logic looks like this (trimmed):

decision = self._engine.evaluate(session_id, tool_name, arguments)

with self._tracer.start_as_current_span(
    f"execute_tool {tool_name}",
    kind=SpanKind.CLIENT
) as span:

    span.set_attribute("mcp.tool.name", tool_name)
    span.set_attribute("gen_ai.tool.name", tool_name)
    span.set_attribute(
        "gen_ai.tool.call.arguments",
        _as_json(arguments)
    )

    span.set_attribute(
        "sentinel.verdict",
        decision.verdict
    )

    span.set_attribute(
        "sentinel.risk_score",
        decision.risk_score
    )

    if decision.rules:
        span.set_attribute(
            "sentinel.rules",
            ",".join(decision.rules)
        )

    if decision.reason:
        span.set_attribute(
            "sentinel.reason",
            decision.reason
        )

We reused the standard OpenTelemetry semantic conventions wherever possible:

mcp.*
gen_ai.*

Everything specific to Sentinel lives under:

sentinel.verdict
sentinel.rules
sentinel.reason
sentinel.risk_score

One FastMCP detail turned out to be important:

To block a tool call you must return an error result from the middleware hook.

Raising an exception simply becomes a generic tool failure, meaning the agent cannot distinguish a policy decision from a runtime error.

The Policies

Every session is evaluated against five security policies.

The most interesting is risky_sequence.

It fires whenever an agent:

reads sensitive information
then attempts to send data outside the trusted environment

Another policy, pii_egress, scans outbound arguments for secrets and personally identifiable information.

In our attack scenario both policies triggered simultaneously on the same http_post request.

The request never reached the destination.

Caption

The live control plane. The agent reads incident notes, retrieves a production secret, then attempts an http_post. Both risky_sequence and pii_egress trigger, and the request is blocked before leaving the system.

What SigNoz Gave Us

Once every tool call became a span, most of the remaining functionality came almost for free.

Every policy decision is also written as a structured log correlated by trace ID.

Opening the blocked request in SigNoz immediately answers:

which policy fired
why it fired
which span recorded it

No custom audit UI required.

IMAGE 2

Caption

The blocked request appears as a correlated log record. The trace ID links directly back to the OpenTelemetry span that captured the decision.

Dashboards Directly from Traces

For the overview dashboard we deliberately skipped the metrics query builder.

Instead, we queried SigNoz's ClickHouse trace store directly using SQL.

The questions we cared about were already encoded as span attributes.

Counting calls by verdict became:

SELECT
    attributes_string['sentinel.verdict'] AS verdict,
    count() AS calls
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'mcp-sentinel'
  AND name LIKE 'execute_tool%'
GROUP BY verdict
ORDER BY calls DESC

One dashboard answers the security question we cared about most:

Did any blocked request actually leave the system?

Because blocked requests are never forwarded, every blocked span is proof that nothing escaped.

IMAGE 3

Caption

MCP Sentinel Overview dashboard showing tool call activity, policy decisions, per-tool latency, and security insights, all generated directly from OpenTelemetry spans.

No additional data pipeline was required.

Everything came directly from telemetry.

For teams that prefer metrics, Sentinel also exports:

sentinel.tool_calls

tagged with:

verdict
tool
rule

making it easy to build alerts and dashboards using standard metric workflows.

SigNoz Provisioned Itself (Almost)

One feature we're particularly happy with is that Sentinel provisions its own observability environment.

A setup script authenticates against the SigNoz REST API and automatically creates:

dashboards
notification channels
alerts

The system that generates security events also installs the dashboards that monitor them.

What We Learned

Two observations stayed with us.

1. Agents Don't Stop After the First Block

In another experiment we blocked a destructive command.

Instead of giving up, the agent began searching for alternative ways to achieve the same goal.

It requested:

an admin token
a bypass key

It found neither.

Even if it had, any attempt to send them externally would have triggered the same sequence policy.

This behavior wasn't deterministic—we only observed it once—but it changed how we think about agent security.

Guardrails need to cover the entire sequence, not just individual API calls.

Every step needs to be recorded.

2. Security Decisions Are Just Telemetry

The second insight surprised us.

Treating policy decisions as telemetry worked because SigNoz is already built to process telemetry.

We never built:

a separate audit database
a custom event store
a reconciliation service

The OpenTelemetry span itself became:

the enforcement record
the audit log
the dashboard source
the alert trigger

For a use case SigNoz wasn't explicitly designed for—AI agent security—it fit remarkably well.

Where We Want to Go Next

The next improvement is end-to-end trace context propagation.

Today the agent, Sentinel, and downstream tools generate related traces.

The next version will connect them into a single distributed trace so the complete execution path appears in one timeline.

The core idea, however, is already working.

One proxy.

One span per tool call.

One record that both stops an attack and documents exactly why it was stopped.

If you're giving AI agents access to real production systems, building that control plane early is worth the effort.

The repository includes:

MCP proxy
Policy engine
Five built-in security policies
Attack scenarios
SigNoz provisioning scripts
OpenTelemetry instrumentation

If you're experimenting with production AI agents, we'd love to hear your feedback and contributions.