Testing AI Agents by Reading Their Traces: Building TraceCheck on Self-Hosted SigNoz
"I have successfully issued a refund for order 123." Confident, polite, completely correct-sounding. Also, in this particular run, the agent had just called lookup_order six times in a row before giving up and doing it anyway — a hard loop I'd seeded on purpose to prove a point. If I'd only been grading that final sentence, I would never have known.
That's the exact problem I built TraceCheck to solve for the Agents of SigNoz hackathon: a testing platform where OpenTelemetry traces stored in self-hosted SigNoz are the ground truth for whether an AI agent actually behaved correctly, not just whether it said something plausible.
The gap I kept running into
Most ways of testing an LLM agent boil down to: send it an input, check the output. That's black-box testing, and it has a real blind spot. An agent can call the same tool in a loop, skip a step it was supposed to do, or quietly blow through a token budget — and still land on a reply that reads fine. The failure happens in the process, not the final text, and if you never look at the process, you never catch it.
I'd been reading about OpenTelemetry and figured: if every LLM call and every tool call gets recorded as a span, and SigNoz stores that trace, then a test runner could pull the trace back out and check the actual span tree — which tools ran, in what order, how many times, how many tokens, how long it took — right alongside the usual output check. White-box and black-box assertions, from the same real trace. That's TraceCheck.
Getting SigNoz running was its own lesson
Before any of the actual testing logic, I had to get a real agent sending real spans into a real SigNoz instance, self-hosted via Foundry per the hackathon rules. This alone taught me more than I expected.
First attempt: foundryctl cast -f casting.yaml died at almost exactly 300 seconds, twice, pulling different amounts of Docker images each time. That consistency (same timeout, different progress) told me it wasn't a flaky network — it was a hardcoded timeout on the pull step. Fix: pre-pull the five images (signoz/signoz, signoz/signoz-otel-collector, clickhouse/clickhouse-server, clickhouse/clickhouse-keeper, postgres:16) manually with plain docker pull, which has no such limit. Once cached, cast finished in about 20 seconds.
Second lesson, this time on the agent side: Gemini's free tier gave me Quota exceeded ... limit: 0 even with a freshly generated key. Turns out "free tier" in the current Gemini API means something specific — you still need billing linked to that exact Google Cloud project to get real (if free) quota, not just a verified account. Once I linked billing at aistudio.google.com/spend, the zero-quota errors disappeared instantly.
What the agent actually looks like
The demo agent is a small customer-support bot with three tools — lookup_order, search_faq, issue_refund — using a plain manual tool-calling loop against Gemini (gemini-3.1-flash-lite, after two other model names failed for two unrelated reasons — always check a provider's live model list, not a six-month-old blog post). Every LLM call and tool call is wrapped in an OpenTelemetry span:
with _tracer.start_as_current_span("llm.call") as llm_span:
response = client.models.generate_content(model=MODEL, contents=contents, config=config)
llm_span.set_attribute("gen_ai.request.model", MODEL)
llm_span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_token_count or 0)
llm_span.set_attribute("gen_ai.usage.output_tokens", usage.candidates_token_count or 0)
Using OpenTelemetry's standard gen_ai.* attribute names (instead of inventing my own) means SigNoz recognizes these as LLM-call spans out of the box, no custom mapping needed.
Writing tests as YAML, grading against the real trace
A test scenario is just plain YAML:
scenario: "refund for order #123"
input: "I want a refund for order 123, it arrived broken"
assertions:
trace:
- tool_called: lookup_order
before: issue_refund
- max_tool_repeats: 2
- max_total_tokens: 8000
- max_duration_ms: 30000
output:
- judge: "Confirms the refund and apologizes for the broken item"
The runner sends the input to the real agent, gets back a trace ID, and polls SigNoz's /api/v5/query_range API until the trace actually lands (spans are batch-exported, so there's a short delay). Then it checks the real span tree against every rule — did lookup_order happen before issue_refund, did any tool repeat past the limit, what were the total tokens, how long did it take — plus one black-box check, an LLM-as-judge call grading the final reply against a plain-English rubric.
To prove the assertions actually catch real failures and not just pass trivially, I kept a correct agent on main and created three branches, each with one deliberately seeded bug:
-
bug/tool-loop: a prompt change that made the agent re-verify the same order repeatedly. Real result:lookup_ordercalled 6 times against a limit of 2 — caught, while the final reply still read completely normally. -
bug/skips-lookup: told the agent the lookup step was "optional." Result: it skipped straight toissue_refund, caught by the tool-ordering assertion. -
bug/token-blowup: padded the system prompt with ~4,400 tokens of filler. Result: total usage jumped to 10,059 tokens against an 8,000 limit — caught, with every other assertion still passing, isolating exactly the one thing that broke.
The CI pipeline taught me the most
Wiring this into GitHub Actions — so a real regression blocks a real pull request — turned out to be the hardest part, and where I learned the most about how SigNoz's auth actually works.
My first approach created a service account and API key via SigNoz's API for the CI run. It kept hitting 403 Forbidden on the trace-query endpoint. Root cause: a freshly created service account has zero permissions until a role is assigned, normally done by hand in the SigNoz UI — something I couldn't replicate over the API on a brand-new, scripted instance. The fix was simpler than I expected: use the logged-in admin's JWT directly as a bearer token, since that's the exact same credential the SigNoz UI itself uses when you browse the Traces page. No role assignment needed.
Once auth was solved, I hit a second, sneakier issue: the OTel collector's port would accept a plain TCP connection before the actual gRPC service behind it was really wired up to handle traffic — so a simple "is the port open" check passed, while real span exports still got reset. The real fix was pushing an actual dummy span through the whole pipeline — SDK → collector → ClickHouse → query API — and retrying that full round trip until it genuinely came back out, before trusting the pipeline was ready. That's a very specific lesson: a listening socket is not the same thing as a working service, and health checks should prove the thing you actually depend on, not a proxy for it.
What I'd tell my past self
Pre-pull your Docker images before running an installer with a pull step baked in. Link billing to your specific Gemini project even for the free tier. Test both halves of agent testing — the trace and the output — because they catch genuinely different failure classes; the tool-loop bug was invisible to output grading entirely. And when you're proving a service is "ready" in CI, prove it by using the service the way your real code will, not by checking a port.
Where it landed
TraceCheck ended up as: a real agent instrumented with OpenTelemetry, a self-hosted SigNoz instance as the actual source of truth (not a bolt-on dashboard), an assertion engine covering four white-box checks plus an LLM-judge black-box check, three proven seeded-bug branches, a GitHub Action that provisions a fresh SigNoz and blocks a bad merge on every push, and a small live dashboard to watch it all happen. Open source, self-hosted, and framework-agnostic — any agent that emits OpenTelemetry spans is testable this way.
Top comments (0)