DEV Community

Cover image for AI Agent observability with DriftWatch and SigNoz
Victory Lucky
Victory Lucky

Posted on

AI Agent observability with DriftWatch and SigNoz

#signoz #ai #opentelemetry #observability

In the agentic AI era, one of the major challenges is token and agent optimization, but then how can you truly optimize this without deep insights into the activities of your AI agent?
Those were my thoughts while exploring agentic AI and MCP servers (I recently built one for a Fintech),
I knew the problem exists but I never truly understood the best way to approach it nor the tools required to solve that problem, until recently when I came across this tweet on X
a screenshot showing a post on X about agent telemetryCoincidentally, I got an email from WeMakeDevs about the Signoz hackathon on the same day I saw the above tweet, then it clicked that this was the right time to solve this problem, so I built DriftWatch.

The Problem

Most agentic AI applications are built blindly without deep insight into,

  • What tools were called most
  • How long did it take (latency, throughput)
  • How much tokens were consumed for input and output
  • How much did it cost in USD?
  • Which prompts consumed the most tokens
  • Which model was used, and
  • When did the agent start drifting from the intended action.

This and more are the problems DriftWatch is solving.

So what is DriftWatch and how does it solve the problems

DriftWatch is a self-observing AI agent SDK that integrates with the Vercel AI SDK.
It traces every tool call and model step an agent makes as OpenTelemetry into SigNoz,
Signoz dashboard showing driftwatch tool call traces
then let's a drift-judge watch that telemetry for behavioral drift by querying the SigNoz API, and detects shifts in tool-call mix, error rate, latency, or token spend.

import { detectBehavioralDrift } from '@driftwatch/sdk';

const report = await detectBehavioralDrift({
  modelClient: openai('gpt-4o-mini'),
  driftDetectionConfig: config.driftDetection, // SigNoz URL + API key
});

console.log(report.verdict);
// {
//   drift: true,
//   severity: 'medium',
//   reasons: ['error rate 1% → 8%', 'p95 latency 210ms → 680ms'],
//   recommended_action: 'Investigate the rising error rate before it spreads.'
// }
Enter fullscreen mode Exit fullscreen mode

Along with this, DriftWatch also provides per-task guardrails, which lets you control the max token and cost of each task and what to do when that limit is met,

const result = await runAgentTask({
  prompt,
  modelClient: openai('gpt-4o-mini'),
  tools,
  maxSteps: config.agent.maxSteps,
  guardrails: {
    maxTokensPerTask: 40_000,   // 0 disables this check
    maxCostUsd: 0.5,            // 0 disables; derived from the prices below
    pricePer1kInput: 0.005,
    pricePer1kOutput: 0.015,
    onExceed: 'stop',           // 'stop' halts mid-loop; 'flag' finishes and marks it
  },
});

if (result.guardrailTriggered) {
  console.warn('guardrail hit:', result.guardrailReason);
}
Enter fullscreen mode Exit fullscreen mode

How to use DriftWatch and connect to Signoz

Using DriftWatch and connecting it to SigNoz is straightforward, and can be done in a few ways.

Step 1

Install the SDK from npm, along with any ai provider of your choice

npm install @driftwatch/sdk ai zod
npm install @ai-sdk/openai   # or any AI SDK provider you already use
Enter fullscreen mode Exit fullscreen mode

Step 2: Provide env variables and set up the telemetry

Connecting to SigNoz Cloud

# Ingestion: For pushing telemetry to SigNoz
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.<region>.signoz.cloud:443
OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<your-ingestion-key>

# Query API: For pulling data for drift behaviour (create a key with at least the Viewer role)
SIGNOZ_URL=https://<your-team>.<region>.signoz.cloud
SIGNOZ_API_KEY=<your-api-key>

# Read by your provider package (@ai-sdk/openai), not by the SDK itself.
OPENAI_API_KEY=sk-...
Enter fullscreen mode Exit fullscreen mode

Connecting to self-hosted SigNoz
Self-hosted SigNoz bundles the collector and query service, and needs no ingestion header:

git clone https://github.com/SigNoz/signoz && cd signoz/deploy/docker
docker compose up -d          # UI + query service on :8080, OTLP collector on :4318
Enter fullscreen mode Exit fullscreen mode 
# Ingestion: For pushing telemetry to SigNoz
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318     # collector (HTTP/proto)
OTEL_EXPORTER_OTLP_HEADERS=                            # none for self-hosted


# Query API: For pulling data for drift behaviour (create a key with at least the Viewer role)
SIGNOZ_URL=http://localhost:8080                       # query service / UI
SIGNOZ_API_KEY=<key from Settings → API Keys>

# Read by your provider package (@ai-sdk/openai), not by the SDK itself.
OPENAI_API_KEY=sk-...
Enter fullscreen mode Exit fullscreen mode

Start telemetry, create a telemetry.ts file (you can name the file anything)

// telemetry.ts — preload; must run before anything else imports
import 'dotenv/config'; // load .env into process.env before anything reads it
import { bootstrapTelemetry, loadDriftWatchConfigFromEnv } from '@driftwatch/sdk';

bootstrapTelemetry(loadDriftWatchConfigFromEnv().telemetry);
Enter fullscreen mode Exit fullscreen mode

bootstrapTelemetry starts OpenTelemetry and sends traces, metrics, and logs to SigNoz. It must run before your application code, in a
separate file, loaded with --import. This is not a style choice: OpenTelemetry instruments modules (fetch, http, …) by patching them the moment they load. If your application file has already imported those modules by the time bootstrapTelemetry runs, the patch is too late, and nothing is captured.

Step 3. Run an agent task

The runAgentTask function runs a traced tool-use loop. Give it a model and tools. It returns the result and a usage summary, and because telemetry.ts already started OpenTelemetry, it emits the agent.run trace and agent.tool.* metrics as it goes.

The example below is runnable. The tool calls Open-meteo's public API.

import 'dotenv/config'
import { runAgentTask, loadDriftWatchConfigFromEnv } from '@driftwatch/sdk';
import { openai } from '@ai-sdk/openai';
import { tool } from 'ai';
import { z } from 'zod';

const config = loadDriftWatchConfigFromEnv();
const modelClient= openai('gpt-5');
// Tools the agent can call. Both hit Open-Meteo's free, keyless API — https://open-meteo.com.
const tools = {
  get_weather: tool({
    description: 'Get current weather for a city',
    inputSchema: z.object({ city: z.string() }),
    execute: async ({ city }) => {
      const geoRes = await fetch(
        `https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(city)}&count=1`,
      );
      const geoData = await geoRes.json();
      const location = geoData.results?.[0];
      if (!location) {
        return { error: `No location found for "${city}"` };
      }

      const weatherRes = await fetch(
        `https://api.open-meteo.com/v1/forecast?latitude=${location.latitude}&longitude=${location.longitude}&current_weather=true`,
      );
      const weatherData = await weatherRes.json();
      return {
        city: location.name,
        country: location.country,
        tempC: weatherData.current_weather?.temperature,
        windSpeedKmh: weatherData.current_weather?.windspeed,
      };
    },
  }),

  search_docs: tool({
    description: 'Search the Open-Meteo API documentation for a query term',
    inputSchema: z.object({ query: z.string() }),
    execute: async ({ query }) => {
      const res = await fetch('https://open-meteo.com/en/docs');
      const html = await res.text();
      const text = html.replace(/<[^>]+>/g, ' ');
      const hits = text.split(new RegExp(query.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'), 'i')).length - 1;
      return { query, hits };
    },
  }),
};

const result = await runAgentTask({
  prompt: 'What is the weather in Lagos, and how many times does the Open-Meteo docs page mention "temperature"?',
  modelClient, // any AI SDK provider; needs its API key
  tools,
  maxSteps: config.agent.maxSteps,
});

console.log(result.responseText);
console.log(result.skillsUsed, result.tokenUsage);
Enter fullscreen mode Exit fullscreen mode

Then run agent.ts with telemetry.ts preloaded ahead of it.

npx tsx --import ./telemetry.ts ./agent.ts
Enter fullscreen mode Exit fullscreen mode

And that's it, You get the model's answer and the exact usage for that call:

The current weather in Lagos, Nigeria is **27°C** with a wind speed of **15 km/h**.

The Open-Meteo documentation mentions the word "temperature" **59 times**.
[ 'get_weather', 'search_docs' ] { inputTokens: 877, outputTokens: 254, totalTokens: 1131 }
Enter fullscreen mode Exit fullscreen mode

Give it a few minutes, search service_name=driftwatch from the SigNoz dashboard, and watch the traces, logs and metrics show up,
if you need an already configured dashboard, import observability/signoz-dashboard.json
via SigNoz Dashboards → New dashboard → Import JSON — it contains all three panels below, pre-built.

A signoz dashboard showing driftwatch behaviour data

Another way to use DriftWatch is through the withSkillExecutionSpan function,
Wrap a tool's execute in withSkillExecutionSpan. Every call then produces a
tool.<name> span and increments the agent.tool.calls and
agent.tool.duration metrics. These are the signals the drift detector reads.
Use this for any tool, whether it's a database lookup, an HTTP call, or a vector search:

import { withSkillExecutionSpan } from '@driftwatch/sdk';
import { tool } from 'ai';
import { z } from 'zod';

const get_repo = tool({
  description: 'Get public information about a GitHub repository',
  inputSchema: z.object({ owner: z.string(), repo: z.string() }),
  execute: (input) =>
    withSkillExecutionSpan({
      skillName: 'get_repo',
      skillInput: input,
      executeSkill: async () => {
        const res = await fetch(
          `https://api.github.com/repos/${input.owner}/${input.repo}`,
          { headers: { 'user-agent': 'driftwatch-demo' } },
        );
        const data = await res.json();
        return { stars: data.stargazers_count, language: data.language };
      },
    }),
});
Enter fullscreen mode Exit fullscreen mode

What I learnt and challenges faced

While building DriftWatch, I encounter series of challenges and bugs,

  1. Telemetry silently going nowhere While testing out DriftWatch, I had issues with my telemetry not showing up on the dashboard, and had to reach out to SigNoz's support through email, and Shivams response was truly helpful. After which I realized that part of the problem was that I loaded my .env file using dotenv/config inside the application's entry point, which runs after the preload. So when the preload tried to read OTEL_EXPORTER_OTLP_ENDPOINT, it got undefined.

2.Traces worked, but my metrics disappeared
After fixing the exporter, traces appeared almost immediately.
but the custom metrics did not.

Metrics such as:
agent.tokens,agent.tool.calls,agent.tool.duration were nowhere to be found, while runtime metrics like:
http.server.duration and nodejs.eventloop showed up perfectly.

This turned out to be a subtle OpenTelemetry JavaScript issue.

Originally, I created the metric instruments at module load time.

const meter = metrics.getMeter('driftwatch');

const tokenCounter = meter.createCounter('agent.tokens');
Enter fullscreen mode Exit fullscreen mode

Unfortunately, that module was imported before my telemetry bootstrap registered the global MeterProvider.
When no provider exists, the Metrics API returns a no-op meter, so
every instrument created from that meter silently drops measurements forever.

The confusing part was that traces did not behave this way.

The solution was to create instruments lazily.

let tokenCounter: Counter | undefined;

function getTokenCounter() {
  return (
    tokenCounter ??=
      metrics
        .getMeter('driftwatch')
        .createCounter('agent.tokens')
  );
}
Enter fullscreen mode Exit fullscreen mode

The first request now creates the counter after the provider has already been registered.

  1. Calling the wrong Query Builder After fixing the telemetry flow, it was finally time to build the drift detector. The detector queries SigNoz through /api/v4/query_range but I was sending the older v3 query format: 
aggregateOperator: "sum"
Enter fullscreen mode Exit fullscreen mode

Meanwhile, the v4 expects aggregation to be split across separate fields:
temporality, timeAggregation, and spaceAggregation.

Once I updated my requests, counters started returning data.
But there was still one final mystery.

agent.tool.duration always returned an empty result.

The answer turned out to be hidden inside how SigNoz stores histograms.

Instead of storing a histogram under one metric name, SigNoz expands it into several related series:
.bucket .count, .sum, .min,.max

The base metric name itself is not directly queryable.

To calculate percentiles, you query the .bucket series instead.

{
  "dataSource": "metrics",
  "aggregateAttribute": {
    "key": "agent.tool.duration.bucket",
    "type": "Histogram"
  },
  "temporality": "Delta",
  "spaceAggregation": "p95"
}
Enter fullscreen mode Exit fullscreen mode

Lessons learnt
but at the end I was able to learn a number of new things, which includes:

  • Deeper understanding of telemetry (this is my first time working with it), I now know what histograms and counters are, differences between span and meter.
  • How to create a dashboard using the UI or by importing a JSON.
  • A better understanding of Vercel AI SDK

Conclusion

DriftWatch SDK does more than what I talked about in this article, but I had to stop so as not to make it too long, Check out the documentation at drift-watch-docs.vercel.app/docs/sdk for the full details about DriftWatch capabilities, and other tools which I didn't mention in this post, including @driftwatch/autopilot which lets you setup schedulers and notifiers to get notify on Slack, Telegram or Webhook, whenever your agent drifts.

Driftwatch is also open-source and contributions are welcomed on the Github repo.

Top comments (0)