End-to-End Observability for AI Applications Using OpenTelemetry and SigNoz
How I implemented distributed tracing, monitored an AI application end-to-end, and transformed debugging with OpenTelemetry and SigNoz.
Introduction
Building an AI application is exciting. Debugging it when requests travel through multiple services is an entirely different challenge.
When I started building Yaatra, my AI-powered travel assistant, the architecture looked simple.
A user would:
- Enter travel preferences.
- The backend would process the request.
- The AI engine would generate a personalized itinerary.
- The response would be displayed on the frontend.
At least, that's how it looked on paper.
Behind the scenes, a single request was traveling through multiple independent services. When something became slow—or worse, failed—I had no easy way to identify the root cause.
Questions started appearing almost immediately:
- Was the backend slow?
- Was the AI engine taking longer than expected?
- Was the external LLM causing delays?
- Did the request even reach the AI service?
Traditional logging only provided fragments of the story. I needed a complete picture.
That's when I decided to implement end-to-end observability using OpenTelemetry and SigNoz.
In this blog, I'll share my journey of instrumenting an AI-native application, the challenges I encountered, the lessons I learned, and how observability completely changed the way I debug distributed systems.
The Problem
Yaatra isn't a monolithic application.
Although users interact with a single interface, every request passes through multiple services before a response is generated.
Initially, my debugging strategy looked something like this:
console.log("Request received");
console.log("Calling AI Engine...");
console.log("Response generated");
This worked during early development.
As the project grew, the logs quickly became insufficient.
I couldn't answer questions like:
- Which service is responsible for high latency?
- How long does the AI engine spend processing?
- Are requests successfully reaching the AI service?
- Which request generated this log?
- How can I follow one request across multiple services?
I needed more than logs.
I needed visibility into the complete lifecycle of every request.
Why OpenTelemetry?
I wanted an observability solution that wasn't tied to a single vendor.
OpenTelemetry stood out because it has become the industry standard for collecting telemetry data.
Instead of manually adding monitoring logic throughout my codebase, OpenTelemetry automatically instruments supported libraries and exports telemetry to any compatible backend.
For visualization, I chose SigNoz, an open-source observability platform that supports:
- Distributed Traces
- Metrics
- Logs
- Dashboards
Together, they provided everything I needed:
- Distributed tracing
- Automatic instrumentation
- Performance monitoring
- Request latency analysis
- Error tracking
- Custom dashboards
My Tech Stack
Gateway Service
- Node.js
- Express.js
- OpenTelemetry SDK
- OTLP Exporter
- HTTP Instrumentation
AI Engine
- FastAPI
- Groq LLM
- Python OpenTelemetry SDK
Observability Stack
- OpenTelemetry
- SigNoz
- OTLP Protocol
- Docker
Understanding Distributed Tracing
Before implementing observability, I had heard about distributed tracing, but never fully understood its value.
Once I started using it, everything clicked.
Every incoming request creates a Trace.
Every operation performed while processing that request becomes a Span.
Instead of viewing every service independently, I could now follow a request throughout its entire journey.
Frontend
│
▼
Gateway Service
│
▼
AI Engine
│
▼
LLM (Groq)
│
▼
Response
Each step became visible.
This was probably the biggest mindset shift during the project.
Instrumenting the Gateway
The gateway receives requests from the frontend and forwards them to the AI engine.
To instrument it, I added the OpenTelemetry SDK to my Node.js application.
The SDK included:
- Node SDK
- OTLP Trace Exporter
- HTTP Instrumentation
- Resource Configuration
One issue confused me for quite some time.
No traces were appearing inside SigNoz.
After debugging, I discovered that OpenTelemetry must be initialized before importing Express.
Changing the initialization order immediately solved the issue.
A tiny implementation detail—but one that consumed several hours.
Instrumenting the AI Engine
The AI engine is built using FastAPI.
Unlike Node.js, it required the Python OpenTelemetry SDK.
I instrumented:
- FastAPI
- Requests library
- OTLP Exporter
Once configured, every request reaching the AI engine automatically became a child span of the gateway request.
This was the moment distributed tracing finally made sense.
Instead of seeing unrelated requests, SigNoz displayed a complete waterfall showing the entire request journey.
Context Propagation Was the Missing Piece
One concept that initially confused me was context propagation.
Generating spans alone isn't enough.
The trace context must travel alongside every request.
Without context propagation:
- Gateway creates one trace
- AI Engine creates another trace
There is no relationship between them.
With propagation enabled:
Single Trace
Gateway
│
├── AI Engine
│ │
│ └── LLM Request
│
└── Response
Everything belongs to one trace.
This became one of the most valuable lessons from the entire implementation.
Sending Telemetry to SigNoz
Both services exported telemetry using the OTLP protocol.
Once telemetry reached SigNoz, I could immediately visualize:
- Active services
- Incoming requests
- Distributed traces
- Span duration
- Service dependencies
- Errors
Instead of scrolling through thousands of log lines, I could simply click a trace and inspect the complete execution path.
Building Meaningful Dashboards
Collecting traces wasn't my end goal.
I wanted dashboards that answered real operational questions.
Request Count
Tracks incoming requests over time.
Useful for identifying traffic spikes.
Average Request Latency
Measures response times.
Latency increases become immediately visible.
AI Processing Time
A custom metric showing how much time the AI engine spends generating responses.
Request Volume
A bar chart comparing request activity across different time periods.
Error Monitoring
Displays failed requests and helps identify problems before users report them.
Challenges I Faced
Not everything worked on the first attempt.
Missing Traces
Initially, no traces appeared.
The problem turned out to be the SDK initialization order.
Broken Context Propagation
The gateway and AI engine were generating completely independent traces.
Proper propagation fixed the issue.
Force Flush Issues
During development, telemetry wasn't always exported immediately.
Understanding how and when telemetry is flushed made debugging much easier.
Dashboard Design
Creating dashboards is easy.
Creating useful dashboards is much harder.
I realized dashboards should answer questions—not simply display numbers.
What I Learned
Logs Are Not Enough
Logs tell you what happened.
Traces tell you:
- Where it happened
- How long it took
- What happened before
- What happened after
Observability Should Be Added Early
Adding instrumentation later is possible.
Adding it early makes development and debugging significantly easier.
Distributed Systems Need Distributed Tracing
As applications become more service-oriented, console logs become increasingly difficult to manage.
Distributed tracing provides context that logs alone cannot.
Dashboards Save Time
Instead of manually inspecting logs, dashboards immediately reveal:
- Increased latency
- Higher request volume
- Failed requests
- Slow downstream services
Architecture Overview
Frontend
│
▼
Gateway Service (Node.js)
│
OpenTelemetry SDK
│
OTLP Trace Exporter
│
▼
AI Engine (FastAPI)
│
OpenTelemetry SDK
│
▼
Groq LLM
│
▼
Response to Client
────────────────────────────────
Telemetry Flow
Gateway
─────────►
OTLP
◄─────────
AI Engine
─────────►
▼
SigNoz
(Traces • Metrics • Logs)
Screenshots
Dashboard Overview
Docker Containers
Yaatra System Architecture
SigNoz Deployment
OpenTelemetry Implementation
What I'd Improve Next
Now that end-to-end observability is working, there are several improvements I'd like to make.
- Instrument the React frontend for browser tracing.
- Add structured logging alongside traces.
- Configure alerts for high latency and increased error rates.
- Track business metrics such as itinerary generation time.
- Monitor AI token usage.
- Measure user experience from browser interaction to AI response.
Final Thoughts
Before this project, I believed observability was primarily a production concern.
After implementing OpenTelemetry and SigNoz, I realized it's equally valuable during development.
Instead of relying on scattered console logs, I can now:
- Follow every request across the application.
- Identify bottlenecks within seconds.
- Understand how each service contributes to response time.
- Debug distributed systems with confidence.
The biggest takeaway wasn't learning another tool.
It was learning a completely new way to think about debugging.
Modern applications are distributed.
Our debugging approach should be distributed too.
If you're building AI applications, microservices, or any system where requests travel across multiple components, I highly recommend investing in observability early.
It will save countless hours of debugging and provide confidence that every request tells a complete story.
Thanks for reading!
References
- OpenTelemetry Documentation
- SigNoz Documentation
- YouTube Tutorials
- ChatGPT (for brainstorming, explanations, and documentation assistance)
Top comments (0)