We Asked SigNoz How to Flag a Hallucinating Agent. They Said "We Haven't Figured That Out." So We Did.

Emmanuel Joshua — Sat, 25 Jul 2026 15:49:16 +0000

By Team ThunderBoltz · Agents of SigNoz Hackathon · Track 1: AI & Agent Observability

In the kickoff Q&A for the Agents of SigNoz hackathon, we asked the SigNoz team a question we'd been chewing on: how do you flag an AI agent that hallucinates? Their answer, on the record: "We haven't figured that out yet."

So we spent the next five days figuring it out. This is the story of SentinelX — a predictive observability platform that catches AI hallucinations in real time, predicts outages before they happen, and lets agents write their own dashboards — all on SigNoz, deployed via Foundry, with agents that read and write through the MCP server.
Why this matters (it's not theoretical)

AI agents are shipping answers to real users right now. A legal copilot cites a court case that doesn't exist in a real brief. A triage bot invents a dosage. A support agent promises a refund the company never authorized. And the systems they run on fail silently: a connection pool exhausts at 3 a.m. while every dashboard still reads green and the on‑call engineer is still asleep.

Traditional observability catches failures after they happen. We wanted to catch lies the instant they're born, and predict failures before the alert fires. SentinelX does both, using three AI agents that talk to SigNoz through its MCP server — reading telemetry the way a senior SRE would, and writing dashboards no human ever clicked into existence.

Deploying SigNoz via Foundry (and the boot bug that nearly killed us)
We deployed SigNoz the way the hackathon rules require: through Foundry, with a casting.yaml and a committed casting.yaml.lock so anyone can reproduce the stack from scratch.

# casting.yaml (simplified)
apiVersion: v1alpha1
kind: Installation
metadata:
  name: sentinelx-signoz
spec:
  deployment:
    mode: docker
    flavor: compose
  mcp:
    spec:
      enabled: true

And then, at 2 a.m. on the first night, the collector came up with an empty pipeline. No OTLP receivers. No traces. Just a health check saying "Everything is ready" while port 4318 accepted TCP connections and returned nothing.
The root cause was a chicken‑and‑egg at startup: the backend's OpAMP config channel refuses to serve a collector until an organization exists in the database, but the organization doesn't exist until you create the first admin user through the UI — which you can't do until the backend is fully up. The collector, waiting on a channel that wasn't serving, fell back to a nop (no‑operation) pipeline with zero receivers.
The fix was two‑fold: provision a root admin user through environment variables so the org exists at boot, and add a compose.override.yaml that drops the --manager-config flag from the ingester so it uses its own static receiver config instead of waiting on OpAMP:

# pours/deployment/compose.override.yaml
services:
  sentinel-signoz-signoz-0:
    environment:
      - SIGNOZ_USER_ROOT_ENABLED=true
      - SIGNOZ_USER_ROOT_EMAIL=admin@sentinel.dev
      - SIGNOZ_USER_ROOT_PASSWORD=<secure-password>
      - SIGNOZ_USER_ROOT_ORG_NAME=default
  ingester:
    command:
      - -c
      - |
        /signoz-otel-collector migrate sync check &&
        /signoz-otel-collector --config=/etc/otel-collector-config.yaml --copy-path=/var/tmp/collector-config.yaml

That override is committed and documented. If you're deploying SigNoz via Foundry and your collector comes up with no receivers, this is the boot race to watch for. Provision the root user before you cast.

The Hallucination Radar — turning "is the model lying?" into a metric
The core insight: a hallucination isn't a vague vibes problem. It's a grounding problem. If an AI's response contains information not supported by the context it was given, that's measurable.
So we built a two‑model design. A RAG agent answers a question using retrieved context. Then a separate LLM — a "critic" — re‑reads that answer against the sources and scores how much of it is ungrounded, from 0.0 (fully grounded) to 1.0 (completely fabricated). The score is emitted as a real OpenTelemetry span attribute:

span.set_attribute("gen_ai.evaluation.hallucination_score", float(score))
span.set_attribute("gen_ai.evaluation.hallucination_reason", reason)
span.set_attribute("gen_ai.evaluation.method", "llm_critic")

That single attribute is the bridge between "an LLM said something sketchy" and "a queryable, graphable, alertable metric in SigNoz." In the Traces explorer, you can click any hallucination_radar.evaluate span and read the score and the critic's one‑line reason. From there we built it up the stack the way an on‑call engineer would:
A three‑panel dashboard in the Query Builder: a high‑risk count (Value panel), a risk‑over‑time series (Time Series), and a table of the exact fabrications caught (Table), all scoped to name = 'hallucination_radar.evaluate' AND gen_ai.evaluation.hallucination_score > 0.7.
A traces‑based alert that fires the moment a score crosses 0.8.
A hallucination went from being an invisible failure — the kind that ships to a customer — to being something you can graph, threshold, and get paged about.

Oracle — predicting the outage before the alert
For the predictive agent, we simulated one of the most common production failures: connection‑pool exhaustion. A pool is a fixed set of reusable database connections; a "leak" is a bug where connections are checked out but never returned, so the occupied count climbs toward the hard ceiling. When it hits the max, new requests block and the app appears to die — usually at 3 a.m., usually while every dashboard still reads green because no threshold has been crossed yet.
We added a db_pool_active gauge to our target app (exported over OTLP to SigNoz every 10 seconds) and a /chaos/leak endpoint that simulates the leak. Then Oracle reads that time‑series straight out of SigNoz through the MCP tool signoz_query_metrics:

result = await mcp.query_metrics(
    metricName="db_pool_active",
    timeRange="30m",
    requestType="time_series",
)

It runs a linear regression on the data points (numpy polyfit), computes the slope and an R² confidence, and solves (threshold − current) / slope for the minutes until breach. The prediction is emitted as its own span (oracle.predict) with attributes like sentinel.prediction.minutes_until_breach and sentinel.prediction.confidence.

The key distinction: a normal threshold alert is reactive — it fires when you're already in the danger zone. Oracle is predictive — it warns while the pool is still at 50, totally "fine," no alert anywhere, and says "you have 18 minutes." That gap is the window where a human can fix the leak calmly instead of getting paged mid‑outage.

Scribe — agents that write observability
Read is half the loop. Write is the other half. Scribe builds a full dashboard payload — title, widgets, queries, layout — and posts it to SigNoz through the MCP tool signoz_create_dashboard:

result = await mcp.call_tool("signoz_create_dashboard", {
    "title": "🛡️ SentinelX — Auto-Generated",
    "layout": [],   # server auto-places panels
    "widgets": [ ... ],  # 3 widgets: value, time-series, table
})

The dashboard appears in SigNoz. No human opened the UI. No human clicked a panel. An agent built the observability itself. Read and write over the same MCP bridge.

The debugging journey (the details only someone who did it would know)
No hackathon story is honest without the walls. Here are ours:
The IPv6 trap. On Windows, localhost resolved to ::1 (IPv6), but Docker mapped the OTLP port on IPv4 only. Our Python exporter silently dropped every trace until we pinned 127.0.0.1 in every endpoint. If your OTLP exports vanish on Windows Docker Desktop, check this first.
The Gemini quota death. Our first LLM backend (Gemini 2.0 Flash free tier) hit a hard‑zero quota mid‑build — limit: 0, not "try tomorrow." We abstracted the LLM layer (sentinel_core/llm.py) so the project could swap to OpenRouter without rewriting the agents. That abstraction also gave us an offline demo mode for recording.
The MCP 403. Our service account authenticated fine (tools/list returned 41 tools) but every data query returned 403: only viewers/editors/admins can access this resource. The key was valid; the role was missing.

Creating an Admin service account through the SigNoz UI fixed it in 60 seconds. If your MCP reads 403, check the role, not the key.

The 200‑field schema. signoz_create_dashboard has a deeply nested schema where every widget must carry query, selectedLogFields, selectedTracesFields, thresholds, contextLinks — and query must include queryType, promql, clickhouse_sql, and builder all present, even the empty ones. We read the full inputSchema from tools/list, built a _widget() factory that hardcodes every required empty field, and Scribe's dashboard landed on the first attempt.

What we learned (and what we'd tell our past selves)
What worked: Foundry for reproducible deployment (the casting.yaml.lock is a gift for judges). The MCP server as the agents' eyes and hands — read and write over one bridge. The hallucination_score attribute as the bridge between LLM evaluation and observability. And the community: the SigNoz Slack team (Nagesh, Chethan, Bishal, Vibhu) pointed us at the root‑user fix that unblocked the entire stack. That single nudge was the turning point.

What didn't: Trusting localhost on Windows. Assuming the free LLM tier would hold. Assuming MCP auth "just works" (it needs a role). And trying to hand‑craft a 200‑field dashboard payload from memory instead of reading the schema first.

What we'd tell our past selves: "The collector's OpAMP boot race will eat your first night. Provision the root user before you cast. Pin 127.0.0.1 in every OTLP endpoint. And read the MCP schema before you write the payload."

Conclusion

We set out to answer one question from the kickoff — how do you flag an agent that hallucinates? — and to give autonomous systems the instruments they shipped without. SentinelX catches the lie, predicts the fire, and writes the dashboard. All on SigNoz, all via Foundry, all with agents that read and write.

The question was "how do you flag a hallucinating agent?" Consider it answered.

Links:

GitHub:

Emman-code / sentinelx-signog

🛡️ Sentinel — Predictive Observability for the Agentic Era

Agents of SigNoz · Track 01 (AI & Agent Observability) · by <YOUR NAME / TEAM>

"How do you flag an agent that hallucinates?" — asked in the kickoff Q&A The SigNoz team's answer: "We haven't figured that out yet." Sentinel is the answer. It detects AI hallucinations in real-time and predicts production failures before alerts fire — and its agents read and write SigNoz through the MCP server.

What Sentinel does

Agent	Role	SigNoz surface
🎯 Hallucination Radar	A live LLM critic scores every AI reply 0–1 for grounding; the score ships as the OTel attribute `gen_ai.evaluation.hallucination_score` → dashboard + alert.	traces · dashboard · alert
🔮 Oracle	Pulls a metric trend via MCP, runs linear regression, and predicts the threshold breach minutes before any alert (with an R² confidence).	metrics (MCP read) · prediction spans
📝 Scribe	Auto-generates a

…

View on GitHub

Demo video: https://youtu.be/WFIE3YkhWT0

SigNoz: signoz.io · Foundry docs · MCP server

Built for the Agents of SigNoz hackathon by Team SentinelX. AI coding assistants were used to accelerate implementation; all architecture decisions, debugging, and verification were performed by the team. Disclosed per hackathon rules.

DEV Community: Emmanuel Joshua

We Asked SigNoz How to Flag a Hallucinating Agent. They Said "We Haven't Figured That Out." So We Did.

Emman-code / sentinelx-signog

🛡️ Sentinel — Predictive Observability for the Agentic Era

What Sentinel does