Making AI agent quality a first-class observability signal with SigNoz

Making AI agent quality a first-class observability signal with SigNoz

An AI support agent is asked whether the store price-matches a competitor. It
thinks for two seconds, calls a knowledge-base tool, and answers confidently:
"We don't offer a price-match guarantee."

That policy does not exist. Nobody wrote it. The agent invented it.

Now look at what your monitoring saw: HTTP 200. Latency 1.8s, well inside the
p95. Zero exceptions. Zero error-rate movement. Every dashboard green.

This is the failure mode that makes AI systems different from the software we
already know how to operate. A hallucination is a successful request. It
passes every health check we have, because every health check we have measures
whether the machine responded — not whether the answer was true.

I built AgentLens for the Agents of SigNoz hackathon to close that gap: to
make "was this answer actually grounded in something real?" a signal you can
graph, alert on, and block a deploy on, using the same tooling you already use
for latency and error rate.

---

The core idea: read the traces back out

Most eval tooling wraps your agent. You import a library, wrap your calls, and
it grades things inline. That couples evaluation to the agent's runtime, its
language, and its deploy cycle.

AgentLens does the opposite, and this is the one design decision worth stealing:

1. The agent is instrumented with plain OpenTelemetry — no proprietary SDK. Every LLM call and tool call is a span, using OTel's GenAI semantic conventions.
2. Those spans go to SigNoz over OTLP. Normal telemetry, nothing special.
3. The evaluator then queries finished traces back out of SigNoz through its query API, rebuilds each agent run from its spans, and grades it with an LLM judge.
4. The scores are written back into SigNoz as OpenTelemetry metrics.

The evaluator never imports the agent. It never calls it. It never shares a
process with it. It only ever reads spans.

That decoupling buys three things that a wrapper cannot:

• It can grade an agent written in any language. The interface is OpenTelemetry, not a Python import.
• It can grade runs from yesterday, because the traces are already stored. Evaluation stops being something that must happen at request time.
• Quality becomes telemetry. Once a score is an OTel metric in the same backend as your latency, you can put it on the same dashboard, alert on it with the same rules, and query it with the same language. No second system, no separate quality portal to remember to check.

That last point is why AgentLens deliberately ships no custom UI. The
interface is SigNoz. Building a bespoke dashboard would have argued against
the whole premise.

---

How SigNoz is used

SigNoz is not a log sink here. It is load-bearing in three distinct roles:

1. As the trace store the agent writes to. The agent emits spans named
invoke_agent support-triage, with child spans for each LLM call and tool call,
carrying GenAI conventions plus a few custom attributes — most importantly
agentlens.variant, which tags which version of the agent produced the run.

2. As the evaluator's data source. This is the unusual part. The evaluator
issues queries against SigNoz's query API to find agent runs in a time window,
pulls their spans, and reconstructs what the agent actually did — which tools it
called, in what order, what it finally said. SigNoz is the input to evaluation,
not just the output.

3. As the destination for the scores. Grades come back as metrics —
agentlens.eval.score.groundedness, an overall score, token counts — plus eval
spans named evaluate_run support-triage carrying
agentlens.eval.failure_mode and the judge's reasoning. So when a score drops,
you can click into the exact evaluation that produced it and read why, in the
judge's own words, next to the trace of the run being judged.

Two panels tell the whole story:

• Groundedness by variant — avg of agentlens.eval.score.groundedness, grouped by agentlens.variant
• Agent latency p95 by variant — p95 span duration, same grouping

Deployed side by side, the top panel falls off a cliff while the bottom panel
stays perfectly flat. That contrast is the entire argument for the project.

---

The experiment

I ran the same golden task suite against two variants of the same agent.

baseline — full grounding rules in the system prompt: cite what you looked
up, say you don't know when the knowledge base doesn't cover it.

regressed — one plausible-sounding prompt change, the kind a real team
ships on a Tuesday to improve customer satisfaction scores: "be confident,
always give the customer a concrete answer."

No syntax error. No failing test. No new dependency. No latency change. It just
quietly makes the agent invent policies rather than admit ignorance.

The suite includes deliberate traps — an order ID that doesn't exist, a policy
question the knowledge base genuinely doesn't answer, an international shipping
question that requires combining two lookups. The correct behaviour on a trap is
often to decline to answer.

Baseline results

Found 6 agent run(s) in SigNoz.
  PASS  T07-international-shipping baseline   overall=0.92 ground=0.90 tokens=2185
  PASS  T07-international-shipping baseline   overall=0.90 ground=1.00 tokens=2426
  PASS  T06-undocumented-policy    baseline   overall=0.86 ground=1.00 tokens=2398 [incomplete_answer]
  PASS  T06-undocumented-policy    baseline   overall=0.93 ground=0.95 tokens=2350

Six runs, zero errors, groundedness between 0.90 and 1.00. Note the
incomplete_answer flag on a run that still passed — the judge records how
an answer was weak even when it clears the bar, which is exactly the granularity
you want when a score moves and you need to know what changed.

Regressed results

The same suite, the same model, the same tools — only the system prompt changed.
The judge flags the failing runs with a hallucinated_policy failure mode: the
agent stated a company policy that appears nowhere in the retrieved evidence.
Groundedness drops below the gate's floor while latency and error rate do not
move at all, which is precisely the blind spot this project exists to close.

---

Blocking CI on quality

Because the scores are just telemetry, a CI job can query them and fail the
build:

npm run gate -- --variant regressed

The gate reads evaluation results out of SigNoz for a variant and time window,
and enforces thresholds: minimum overall score, minimum groundedness, maximum
tokens per run. If groundedness has fallen below the floor, it exits non-zero
and the deploy stops.

That's the payoff. A hallucination is a 200 OK — latency and error rate can
never catch it. This can.

---

Tech stack

• Agent + evaluator: Node.js / TypeScript, run with tsx
• Telemetry: OpenTelemetry SDK — OTLP HTTP exporters for both traces and metrics, GenAI semantic conventions
• LLM: Groq (openai/gpt-oss-20b) over the OpenAI-compatible protocol, for both the agent under test and the judge. Provider is one env var; Gemini, OpenAI, Anthropic, OpenRouter and Ollama all work unchanged.
• Observability: self-hosted SigNoz, ClickHouse-backed, deployed with Foundry
• CI: GitHub Actions job running the quality gate

The judge and the agent being the same model is deliberate for reproducibility
on a free tier, and the judge is given the retrieved evidence rather than asked
to recall facts — it grades whether the answer is supported, not whether it is
true in general.

---

What I learned

SigNoz's install path changed under me mid-hackathon. The docker-compose
manifests and deploy/install.sh are deprecated; self-hosting is now
declarative through foundryctl cast against a casting.yaml. My setup script
was hunting for compose files that no longer exist. Pinning your assumptions
about someone else's deploy tooling is a bug waiting to happen — and generated
lock files belong in the repo, so a reviewer can reproduce your exact stack.

"Settings → API Keys" no longer exists in current builds; service accounts
replaced it. A read-only viewer role is enough for an evaluator that only
queries.

Ingestion lag is real and you must design for it. The evaluator reads what
the agent just wrote, and ClickHouse needs a beat. Every read path takes a
lookback window, and the fix for an empty result is patience, not a retry storm.
It's an honest constraint of the read-it-back-out architecture, and worth
stating plainly rather than hiding behind a sleep.

Instrument the judge too. The evaluation itself is a traced span with its
own token counts. The cost of measuring quality is visible in the same place as
the quality.

---

Try it

Repo: https://github.com/eiza763/agentlens

casting.yaml and casting.yaml.lock are committed, so foundryctl cast
reproduces the exact SigNoz deployment I used. npm run selftest runs 47 checks
with no credentials and no network if you just want to confirm the logic. There
is a Codespaces devcontainer that brings up SigNoz for you, because not everyone
has Docker and admin rights on the machine in front of them.

Quality is not a vibe you check manually before a release. It's a signal. Put it
on the dashboard next to latency, and let CI block on it.