DEV Community

Victory Lucky
Victory Lucky

Posted on

Detecting AI Agent Behavioral Drift with OpenTelemetry and SigNoz

#ai #agents #otel #signoz

Recently, I have been working on AI agents and MCPs (I just completed one for a Fintech)
A screenshot showing a financial expense graph in claude
and some of the challenges that kept showing up was,

  • how do I know what this agent is doing?
  • how do I know how much token it consumed?
  • what was the cost in USD?
  • how do I know when this agent drifts from the intended action?
  • and finally, how do I tell what the latency was, and when it spiked?

While still thinking through on how to solve this problem, I came across this tweet on X

A screenshot showing a post on X
In the noon of that same day, I got an email from WeMakeDevs about the SigNoz hackathon, I went through the details and saw that it was about telemetry, and that was all the cue I needed.

How the tool was born

The frustration I encountered trying to understand the processes of these agents,such as wondering what changed, what model was used? what was the thought process of the model and the skill before it arrived at the previous result? was what led me to build DriftWatch

The solution

Introducing DriftWatch: An AI agent that observes itself.

So what is DriftWatch?
DriftWatch is an AI agent SDK that observes itself. Every skill (tool)
call and LLM step is traced via OpenTelemetry into
SigNoz, and an AI layer on top of those traces flags
behavioral drift — shifts in tool-call mix, error rate, latency, or
token spend between two time windows.

DriftWatch does this by observing every process, tool calls, duration, drifts, as well as the model and/or skill used, and provides you with an Open telemetry records passed and analysed in SigNoz, which provides deep insights into the whats, whys and whens of every process.

How DriftWatch works

DriftWatch wraps tool calls, and use open telemetry elements to track every process from initiation to response. Concretely, that means every agent run gets one parent agent.run span, every tool call gets its own child span, and three custom metrics ride alongside them: agent.tool.calls (counter, labelled by tool + outcome), agent.tool.duration (histogram), and agent.tokens (counter, labelled by model/provider/type). None of this is SigNoz-specific yet — it's just the OpenTelemetry Node SDK, instrumented once at process startup:

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: telemetryConfig.serviceName,
    'agent.kind': 'driftwatch',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${telemetryConfig.otlpEndpoint}/v1/traces`,
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: `${telemetryConfig.otlpEndpoint}/v1/metrics`,
    }),
    exportIntervalMillis: 10_000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
Enter fullscreen mode Exit fullscreen mode

Point OTEL_EXPORTER_OTLP_ENDPOINT at a self-hosted SigNoz collector (http://localhost:4318 if you're running SigNoz's own docker compose up locally) and every span and metric above lands in SigNoz over OTLP/HTTP with zero extra glue code — this is the part that's genuinely "free": SigNoz's collector supports OTLP natively, so there's no translation layer to write.

That gets you visibility.
But here's the part I actually wanted, noticing when an agent's behavior quietly changes, I needed something SigNoz doesn't do out of the box: comparing two time windows and asking whether the difference means anything. So DriftWatch queries SigNoz's /api/v4/query_range builder API directly for two one-hour windows (baseline vs. current) and diffs them:

const requestBody = {
  start: startTimeMs,
  end: endTimeMs,
  step: 60,
  compositeQuery: {
    queryType: 'builder',
    panelType: 'table',
    builderQueries: {
      A: { dataSource: 'metrics', aggregateAttribute: { key: 'agent.tool.calls', dataType: 'float64' },
           aggregateOperator: 'sum', groupBy: [{ key: 'tool' }, { key: 'outcome' }], expression: 'A' },
      B: { dataSource: 'metrics', aggregateAttribute: { key: 'agent.tool.duration', dataType: 'float64' },
           aggregateOperator: 'p95', expression: 'B' },
      C: { dataSource: 'metrics', aggregateAttribute: { key: 'agent.tokens', dataType: 'float64' },
           aggregateOperator: 'sum', expression: 'C' },
    },
  },
};

await fetch(`${signozBaseUrl}/api/v4/query_range`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'SIGNOZ-API-KEY': signozApiKey },
  body: JSON.stringify(requestBody),
});
Enter fullscreen mode Exit fullscreen mode

Three builder queries in one request — tool-call counts (for tool mix and error rate), p95 tool latency, and total token spend — fanned out for both windows in parallel. The SIGNOZ-API-KEY header comes from SigNoz's own UI (Settings → API Keys), and the builder query shape took a few failed requests to get right, since the v4 API's aggregateAttribute/dataType fields aren't obvious from the dashboard alone — I ended up reverse-engineering the exact payload by watching the network tab while building a panel manually in the SigNoz UI, then copying that shape into code.

[Screenshot: SigNoz dashboard showing the agent.tool.calls, agent.tool.duration, and agent.tokens panels over a live run]

Once both windows come back as plain numbers (tool mix percentages, error rate, p95 latency, token spend), DriftWatch hands them to whatever model you've configured and asks it to judge the drift — with a strict JSON-only system prompt, retried up to a few times if the model wraps its answer in markdown:

You are an SRE copilot that classifies whether an AI agent's behavior
has drifted enough to warrant a human alert. Reply with a SINGLE raw
JSON object: {"drift": bool, "severity": "none"|"low"|"medium"|"high",
"reasons": string[], "recommended_action": string}.
Enter fullscreen mode Exit fullscreen mode

That verdict, plus the raw baseline/current numbers is what shows up as the /drift endpoint response, and it's the same JSON the DriftWatch console renders as a human-readable card: severity, the specific metrics that moved, and a one-sentence recommendation.

[Screenshot: a drift verdict — e.g. "search_docs share went from 40% to 75%, error rate 2% → 9%", next to the SigNoz trace that shows the actual slow calls]

Seeing it in SigNoz

The whole loop is testable end-to-end without touching production traffic:

# bring up SigNoz locally
git clone https://github.com/SigNoz/signoz && cd signoz/deploy/docker && docker compose up -d

# run DriftWatch pointed at it, then generate some load
git clone https://github.com/codewithveek/driftwatch &&
cd drift-watch && pnpm dev
BASE_URL=http://localhost:3000 pnpm seed 40

# pull a live drift verdict computed from SigNoz data
curl localhost:3000/drift
Enter fullscreen mode Exit fullscreen mode

pnpm seed 40 fires 40 mixed requests through the agent, which is enough to populate both the baseline and current windows with real spans — open SigNoz at localhost:8080 right after and you can watch the agent.run traces stack up, click into any one, and see every tool call as a child span with its own duration and status. That trace view is honestly the most useful part for debugging: when the drift judge flags "search_docs share 40% → 75%," you don't have to take its word for it, you click through to the actual traces from that window and see which tool calls dominated.

[Screenshot: a single agent.run trace expanded, showing child spans for each tool call with duration and gen_ai.* attributes]

Autopilot: closing the loop

Once drift detection works, the obvious next question is "okay, now what"? so DriftWatch has a second, optional loop that turns a drift verdict into an action: pause the agent, roll it back to a known-good state, or just notify (Slack/Telegram/webhook), gated by policy rules and, for anything destructive, a human approval step. It's a thin layer on top of the same SigNoz-derived numbers, not a separate data source — I'm not going to detail it fully here since this post is about the SigNoz integration specifically, but it's what turns "we noticed the agent drifted" into something you can leave running unattended.

Takeaways

What worked: OpenTelemetry's auto-instrumentation for Fastify/HTTP meant I got baseline request tracing for free the moment sdk.start() ran, before I'd written a single custom span. Layering agent.run/tool spans and the three custom metrics on top of that was straightforward once I had the exporter pointed at SigNoz's collector correctly.

What was hard: there's no existing semantic convention for "AI agent behavior" the way there is for HTTP or DB spans, so I had to invent my own attribute names (agent.task_id, agent.skills_used, gen_ai.usage.*) and just be consistent about them. Getting SigNoz's v4 query builder payload shape right from code (rather than the UI) took real trial and error — the error messages from a malformed compositeQuery aren't always specific about which field is wrong. And getting a model to reliably return bare JSON for the drift verdict was less reliable than I expected; the retry-with-correction-prompt approach was a hack that turned out to work well enough for a weekend build.

Conclusion

DriftWatch turns SigNoz from "a place traces go to die" into an actual feedback loop for AI agents — instrument once with OpenTelemetry, query SigNoz's builder API for two time windows, and let a model tell you when something changed. Code: https://github.com/codewithveek/drift-watch — live instance: https://driftwatch.veek.me/console/.

Top comments (0)