DEV Community

Cover image for I built TraceGate because my AI agent demo passed, but the traces told a different story
Rohit Purkait
Rohit Purkait

Posted on

I built TraceGate because my AI agent demo passed, but the traces told a different story

#ai #observability #typescript #opentelemetry

Most AI agent demos stop at the final answer.

Mine did too at first. I had a support agent that could answer a refund question, call a policy tool, and avoid a prompt-injection path. From the outside, it looked fine.

Then I asked a less comfortable question: if this agent broke in production, would I actually know what happened?

That question became TraceGate.

TraceGate is a release gate for AI agents built with OpenTelemetry and SigNoz. It runs agent scenarios, sends telemetry to SigNoz, then checks whether the run produced enough evidence to safely ship.

tracegate flow

The problem

AI agents can pass a demo while still being hard to debug.

A tool can retry three times and still return a successful answer. An LLM call can happen without cost metadata. A prompt-injection test can pass, but leave no trace of which safety path was used. Those are not product failures in the usual sense. They are observability failures.

That is what I wanted TraceGate to catch.

The idea is simple: before an agent ships, it should satisfy an observability contract. If the contract fails, the release is blocked.

What TraceGate checks

TraceGate uses a YAML contract to describe what a safe release run should include.

Here is part of the contract I used:

name: TraceGate AI Agent Release Contract
serviceName: tracegate-demo-agent

budgets:
  maxRunCostUsd: 0.005
  maxP95LatencyMs: 2000
  maxToolRetries: 1

checks:
  - id: span-agent-run
    type: required-span
    spanName: agent.run
    severity: critical

  - id: span-llm-call
    type: required-span
    spanName: llm.call
    severity: critical

  - id: attr-llm-model
    type: required-attribute
    spanName: llm.call
    attribute: gen_ai.request.model
    severity: critical

  - id: retry-budget-trace-lookup
    type: max-tool-retries
    toolName: trace.lookup
    maxRetries: 1
    severity: critical
Enter fullscreen mode Exit fullscreen mode

The default demo intentionally fails one check. The trace.lookup tool retries three times, but the contract only allows one retry.

That makes the demo useful because TraceGate is not pretending everything is healthy. It blocks the release and explains why.

TraceGate FAIL
Checks: 7/8 passed
Critical failures: 1
FAIL retry-budget-trace-lookup - Worst retry count for 'trace.lookup' was 3; limit 1.
Enter fullscreen mode Exit fullscreen mode

tracegate overview

The architecture

The stack is:

Vite
React
TypeScript
Node.js
OpenTelemetry
SigNoz
OpenAI
YAML contracts
Enter fullscreen mode Exit fullscreen mode

The flow looks like this:

Scenario -> Agent runner -> OpenTelemetry -> SigNoz -> TraceGate contract evaluator -> Pass or block
Enter fullscreen mode Exit fullscreen mode

The Node runner executes the scenario. OpenTelemetry records spans, metrics, and logs. SigNoz receives the telemetry through OTLP. TraceGate reads the run result and evaluates it against the contract.

For the OpenTelemetry setup, I used the OTLP HTTP exporters:

const endpoint =
  process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: `${endpoint}/v1/traces`
  }),
  logRecordProcessor: new BatchLogRecordProcessor(
    new OTLPLogExporter({
      url: `${endpoint}/v1/logs`
    })
  ),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: `${endpoint}/v1/metrics`
    }),
    exportIntervalMillis: 1000
  })
});
Enter fullscreen mode Exit fullscreen mode

In my local setup, SigNoz received spans for:

agent.run
llm.call
tool.ticket.lookup
tool.policy.search
tool.trace.lookup
Enter fullscreen mode Exit fullscreen mode

signoz overview

A build issue I hit

The part that took the most debugging was the local SigNoz setup.

I used Foundry to run SigNoz locally. The first time I started everything, the UI was up, but the telemetry path was not useful. The collector looked alive, but the pipeline was effectively not sending the data I expected.

The issue was in the generated local configuration around the OpAMP and MCP service wiring. I patched the generated setup so the ingester pointed to the actual SigNoz service and the MCP server pointed to the SigNoz UI service. After that, the OTLP export worked and I could verify spans in ClickHouse.

That was the most “real” part of the project for me. The code for the gate was straightforward compared to proving that telemetry actually reached the backend.

The workbench

TraceGate has a landing page and a workbench.

The landing page explains the idea: ship agents only when the traces agree.

The workbench shows:

latest release verdict
checks passed
critical failures
cost
latency
gate matrix
failed evidence
SigNoz investigation query
Enter fullscreen mode Exit fullscreen mode

The gate matrix is the part I care about most. It turns vague release confidence into specific checks.

For example:

Root agent run span: pass
LLM calls traced: pass
Model attribute present: pass
Cost attribute present: pass
Cost budget: pass
Trace lookup retry budget: fail
Refund support scenario: pass
Prompt injection scenario: pass
Enter fullscreen mode Exit fullscreen mode

That gives a reviewer a much better starting point than “the demo worked on my machine.”

Judge test mode

I also added a judge-operated test mode because I did not want the hosted app to feel like a static mockup.

A judge can enter:

service name
scenario
tool name
retry budget
observed retries
cost
latency
Enter fullscreen mode Exit fullscreen mode

Then TraceGate generates a new verdict.

For example:

service: checkout-agent
tool: inventory.lookup
allowed retries: 1
observed retries: 3
Enter fullscreen mode Exit fullscreen mode

That blocks the release.

If the observed retries are changed to 0, the release becomes ready.

signoz workflow

This makes the product easier to test without asking someone to set up my whole local environment first.

What I learned

The main thing I learned is that observability is more useful when it is tied to a decision.

Before this project, I mostly treated observability as something I would open after a bug. With AI agents, that feels too late. If an agent takes a risky path, skips metadata, hides cost, or retries a flaky tool several times, I want to know during release validation.

The second thing I learned is that “the agent worked” is too broad. I now split it into two questions:

Did the agent produce the expected outcome?
Did the agent produce enough evidence to debug that outcome?
Enter fullscreen mode Exit fullscreen mode

TraceGate focuses on the second question.

What I would build next

The next version should have a visual contract editor. Writing YAML is fine for a hackathon, but a team should be able to create a release gate by choosing required spans, attributes, budgets, and scenarios in the UI.

I would also add GitHub Actions support. The natural place for TraceGate is in CI, where it can block an agent release before merge.

The final improvement would be deeper SigNoz artifacts. Each contract should generate a matching dashboard, alert, and investigation prompt so the release gate and the debugging workflow stay connected.

Final thought

TraceGate started from one uncomfortable question: if my AI agent fails later, will I have the evidence to understand it?

SigNoz already gives teams a strong place to inspect telemetry. TraceGate adds a release workflow on top of that telemetry.

The project is small, but the idea feels useful: agents should not ship only because the final answer looked good. They should ship when their behavior is traceable enough to trust.

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

That gap between demo success and trace reality is exactly where agent systems get risky. The user-visible output can look right while the trace shows retries, skipped checks, fabricated intermediate claims, or a tool call that only worked by accident.

I like the idea of making traces part of the acceptance criteria. For agent workflows, "it produced the answer" is weaker than "it produced the answer through an allowed path."