I Couldn't See What My AI App Was Doing — So I Added OpenTelemetry and SigNoz in One Afternoon

Tarun Prajapati — Sat, 18 Jul 2026 16:42:25 +0000

Nebva is an AI content-generation SaaS I'm building on Node.js, TypeScript, MongoDB, and Redis, with Gemini doing the actual generation work. Until recently, the only telemetry it had was a homemade buffer that wrote events into MongoDB and hoped I'd go query them later. When a request was slow, or a generation got blocked, I had no real way to find out why — just logs and guesses. Here's what changed in one afternoon.

Before You Start

Node.js/Express backend (swap for your own stack)
MongoDB + Redis running
A Gemini API key (or whichever LLM you're calling)
SigNoz running locally (Docker Compose or Foundry)
No prior OpenTelemetry experience required

What I Was Flying Blind On

Nebva's only telemetry used to be TelemetryProxyProvider.ts — a manual buffer that wrote events straight into MongoDB. It worked, in the sense that events got written. What it couldn't do was connect anything: an LLM call, the storage write that followed it, and the notification that went out after — three separate entries with no shared ID between them. If a request felt slow, I had no way to tell whether the delay was Gemini, MongoDB, or my own code. There was no dashboard and no alerting, just documents sitting in a collection I'd query by hand when something broke.

A single content-generation request in Nebva actually moves through several distinct stages — pulling signal, generating an embedding, calling Gemini for the actual content, writing the result to storage. None of that structure was visible anywhere. It was just "the request," one opaque block of time.

The One File That Changed Everything

Auto-instrumentation is the fastest win in the whole migration, and it lives entirely in one file: instrumentation.ts.

// instrumentation.ts — REPLACE the body below with your actual file
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'nebva-backend',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

⚠️ This file has to load before anything else in your app — before your routes, before your business logic. OpenTelemetry's own docs are explicit about this: import it after your other modules and the instrumentations silently do nothing. [FILL IN: if you actually hit this, describe what you saw instead of this line]

That's it for infrastructure. Every Express route, every Mongoose query, every outgoing HTTP call, and every Redis operation now shows up as a span, without touching a single line of business code.

Wrapping My AI Provider Without Touching It

Auto-instrumentation covers infrastructure. It has no idea what "generating a blog post" means, or that a guardrail just blocked an output — those are Nebva-specific concepts, and there's no way for a generic instrumentation library to invent attributes for things it doesn't know about.

That's what TelemetryProxyProvider is for. It implements the same IAIProvider interface every other part of the codebase already calls, so nothing upstream had to change. Underneath, every call gets wrapped in a span tagged with OpenTelemetry's GenAI attributes — gen_ai.usage.total_tokens, gen_ai.usage.output_tokens, model name, and so on.

// TelemetryProxyProvider.ts — REPLACE with your actual implementation
class TelemetryProxyProvider implements IAIProvider {
  constructor(private readonly inner: IAIProvider) {}

  async generate(input: GenerationInput) {
    return tracer.startActiveSpan('gen_ai.generate', async (span) => {
      span.setAttribute('gen_ai.request.model', input.model);
      try {
        const result = await this.inner.generate(input);
        span.setAttribute('gen_ai.usage.total_tokens', result.usage.totalTokens);
        span.setAttribute('gen_ai.usage.output_tokens', result.usage.outputTokens);
        return result;
      } finally {
        span.end();
      }
    });
  }
}

The part I actually think is worth stealing: I used the same pattern for Nebva's content-safety checks. Every time a guardrail blocks a generation — a soft refusal, a hate-speech match, whatever the reason — that's now also a span attribute: gen_ai.guardrail.triggered and gen_ai.guardrail.reason, tagged against the user who triggered it. It didn't need a separate system. Same Decorator, same spans, one more attribute.

What It Actually Looks Like in SigNoz

Once real traffic runs through it, the Services page breaks the whole pipeline into stages, each with its own P50/P95/P99:

I'd assumed the pipeline was roughly evenly slow across the board. It isn't — image generation alone is the outlier, and errors cluster entirely on the write path.

The guardrail attributes turned into a dashboard with no extra plumbing — same span data, a different query:

Two Things I'd Tell My Past Self

Auto-instrumentation gets you infrastructure for free. It will never know your business. Express, Mongoose, HTTP, Redis — all free. "This generation got blocked for a safety reason" is not a concept OpenTelemetry has ever heard of, and it's worth tagging that from day one instead of bolting it on later.
Watch your async queues. Nebva uses BullMQ for background jobs. By default, BullMQ jobs show up as completely disconnected traces in SigNoz. Auto-instrumentation doesn't automatically propagate the OTel context across the Redis queue boundary unless you explicitly inject the trace context into the job payload and extract it on the worker side. If you don't do this, your trace waterfall breaks the moment a job hits the queue.

Wrapping Up

One file for infrastructure, one Decorator for business logic, and a MongoDB buffer I could finally delete. If you're running any LLM behind an API and flying blind on what it's actually doing, this is the smallest version of the fix.

If I had more time, the next things I'd add: alerting on guardrail-trigger spikes per user, a cost dashboard broken out by generation type instead of just totals, and tracing through the queue side of the pipeline to see whether async jobs stay connected to the request that started them.

DEV Community: Tarun Prajapati