SIGNOZ

KOMARI Subheeksh — Fri, 24 Jul 2026 08:17:21 +0000

I built a deterministic verdict engine for AI agents on SigNoz — no LLM in the loop

Your AI agent hallucinates. It loops tool calls. It burns tokens. And you won't know until a customer complains.

That's the problem Gaze solves. It watches your agent's traces through SigNoz, runs nine deterministic rules, and issues a recomputable verdict. No LLM in the verdict path. Same input always produces the same sha256 hash.

I built this for the Agents of SigNoz Hackathon 2026. Here's how it works and what I learned.

The problem

Every observability tool tells you what your agent COST. LangSmith, Burnrate, every LLM gateway — they all show dollars spent and tokens consumed. But none of them tell you if the agent was RIGHT.

An agent can repeat the same wrong answer for hours. It can cite documents that don't exist. It can get stuck in a tool loop burning API credits. Existing SRE tools catch when your service is down. They don't catch when your agent is wrong.

Gaze sits between your agent and SigNoz. It reads the traces, evaluates behavior, and issues a verdict score from 0 to 100. When the score drops, SigNoz fires an alert.

How it works

Your agent emits OpenTelemetry spans with GenAI semantic conventions — prompt, completion, tokens, model name. SigNoz stores them in ClickHouse. Gaze queries ClickHouse directly, runs nine deterministic rules against every span, computes a score, writes the verdict back to SigNoz via OTLP.

All of this happens inside your deployment. No external service. No API key. You run foundryctl cast --file casting.yaml and everything comes up: SigNoz, ClickHouse, the ingester, the UI.

The nine rules

Every rule is pure Python. No ML models, no embeddings APIs, no external dependencies.

repetition loop — n-gram similarity > 80% across 5+ consecutive spans. Catches agents stuck repeating the same output.
embedding drift — character bigram hashing. Tokenizes output into bigrams, hashes them to a fixed-length vector, compares cosine distance from the agent's baseline. If the distance exceeds 0.40, the agent's output quality is degrading.
tool loop — detects circular tool calls. Same (tool, args) pair repeated 3+ times means the agent is stuck.
unauthorized tool — cross-references tool calls against the agent's registered manifest. Catches jailbreak attempts that try to access unregistered tools.
prompt injection — 47 regex patterns from the verazuo/jailbreak_llms dataset. Matches known injection vectors like "ignore all instructions", "DAN", "developer mode", "system override".
cost explosion — token usage > 3x the 7-day rolling average for the same agent + model.
latency degradation — P95 span duration > 2x the 7-day rolling baseline.
empty response — agent returned null or empty output.
hallucinated source — agent cited a document not found in the retrieval spans.

The verdict

Each verdict carries a sha256 hash of (trace snapshot + rule set version + agent ID). Same input always produces the same hash. Anyone can recompute and verify.

$ curl -X POST https://gaze-4fy2.onrender.com/verdict \
  -d '{"agent_id": "support-bot-01"}'

{
  "verdict_hash": "sha256:a1b2c3d4...",
  "score": 94,
  "status": "HEALTHY",
  "rules_evaluated": 9
}

The verdict spans are written back to SigNoz via OTLP so they appear alongside your agent's traces.

How I used SigNoz

This project wouldn't exist without SigNoz. Here's exactly what I used:

MCP server

SigNoz exposes an MCP (Model Context Protocol) server that allows external tools to query traces programmatically. Gaze uses this to pull agent spans for evaluation.

# mcp_client.py
async def query_spans(agent_id: str, window_minutes: int = 5):
    async with signoz_mcp_client() as client:
        return await client.execute_tool(
            "signoz_list_spans",
            {"service": agent_id, "minutes": window_minutes}
        )

ClickHouse direct queries

For deeper analysis, Gaze queries ClickHouse directly. SigNoz stores all trace data in signoz_traces.distributed_signoz_index_v3. Gaze extracts GenAI-specific attributes:

SELECT
    span_id, trace_id,
    JSONExtractString(tagMap, 'gen_ai.prompt') as prompt,
    JSONExtractString(tagMap, 'gen_ai.completion') as completion,
    JSONExtractInt(tagMap, 'gen_ai.usage.input_tokens') as input_tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'support-bot-01'
  AND timestamp > now() - INTERVAL 5 MINUTE

OTLP verdict export

After evaluation, Gaze creates verdict spans and exports them to SigNoz via OTLP. This means verdicts appear as regular spans in the SigNoz UI, right next to your agent's traces.

# otel_exporter.py
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(exporter)

Foundry deployment

Everything deploys with one command:

foundryctl cast --file casting.yaml

The casting.yaml defines the full SigNoz stack — ClickHouse, ingester, query service, UI, alert manager. Judges can re-run Foundry against your casting.yaml.lock to reproduce the exact deployment.

SigNoz dashboard

I built a 6-panel dashboard in dashboards/gaze-verdict.json that shows:

Agent score cards (0-100 for each registered agent)
Verdict timeline with rule breakdowns
Evidence explorer linking each violation to the exact span
Token usage over time vs rolling average
Latency P95 vs baseline
Alert rules for score drops

Import it directly into your SigNoz UI.

What I learned

Deterministic beats probabilistic for observability

Every verdict tool out there uses an LLM to judge agent quality. This is fundamentally wrong for observability. If your monitoring tool can give different answers for the same input, it's not monitoring — it's guessing.

Gaze uses character bigram hashing for embedding drift detection instead of calling an embedding API. No model variance. No API latency. No cost. Same text always produces the same vector. This was the hardest lesson: resist the urge to use AI for everything. Sometimes a hash function is better.

ClickHouse is underrated for trace analysis

Most observability tools abstract ClickHouse behind APIs. But direct queries give you power you can't get otherwise. Gaze's rules use window functions, JSON extraction, and statistical aggregates that would be painful through a REST API. ClickHouse handles them natively.

SigNoz MCP is real infrastructure

The MCP server isn't a demo feature — it's a production-grade trace query interface. I built the entire Gaze integration on it. Zero issues with connection stability or query performance.

Foundry makes self-hosting trivial

I've set up SigNoz manually before. It's 4+ containers with networking, volumes, and config. Foundry reduces it to one YAML file and one command. For a hackathon deadline, this is the difference between finishing and not.

The code

Everything is open source: github.com/subheeksh5599/Gaze

Backend: Python 3.11+, FastAPI, Pydantic v2. 8 endpoints. 35/35 tests passing.
Frontend: Vite + React 19 + TypeScript + Tailwind — gaze-omega.vercel.app
Rules engine: 9 deterministic rules. Zero ML dependencies. Pure Python.
Deployment: Foundry for SigNoz, Render for backend, Vercel for frontend.

Try it yourself:

git clone https://github.com/subheeksh5599/Gaze
cd Gaze
pip install -r backend/requirements.txt
cd backend && python3 gaze/server.py

Then visit http://localhost:8000/docs for the OpenAPI docs.

What's next

Historical replay — recompute verdicts for past trace windows
Custom rule builder — YAML-based rules, shareable rule packs
Agent pause integration — auto-pause on CRITICAL verdict
Slack/Discord bot for querying verdicts from chat

Built for the Agents of SigNoz Hackathon 2026. MIT licensed.

DEV Community: KOMARI Subheeksh