I Built an AI Observability Platform to Understand What Happens Inside LLM Applications

DHEVIKA M — Sat, 18 Jul 2026 10:56:23 +0000

Introduction

Your AI application is working.

The response is generated.
The API returns 200.
The user is happy.

But inside the system, you have no idea what actually happened.

Was the LLM slow?
Did the agent waste tokens?
Where did latency come from?

This was the exact problem I faced while building an AI application during the Agents of SigNoz hackathon.

The AI response was generated successfully, but I had no visibility into the internal execution flow.

A user only sees:

"The AI generated a response."

But as an AI engineer, I need deeper answers:

Why was the response slow?
Was the delay caused by my backend or the LLM?
How many tokens were consumed?
Which step failed?
What was the estimated cost of the execution?

This experience made me realize something important:

Building an AI application is only half the challenge. Understanding its behavior in production is equally important.

This motivated me to build ObservEx Lite, an AI Agent Observability Platform using OpenTelemetry and SigNoz.

The goal was simple:

Transform AI debugging from guesswork into measurable engineering.

The Problem: AI Applications Are Black Boxes

Traditional applications usually have a predictable flow:

Monitoring these systems is relatively straightforward.

However, modern AI applications are different.

An AI workflow can look like:

                  User Request

                     |
                     |

                   AI Agent

                     |
                     |

                 Reasoning Process

                     |
                     |

                  LLM Call

                     |
                     |

                Generated Response

Inside this workflow, many things can go wrong:

LLM latency can increase
Token usage can become expensive
A particular AI step can fail
The model response can become unpredictable

Traditional monitoring can tell us:

"The API request took 5 seconds."

But AI developers need answers like:

 "The LLM call consumed most of the latency and used 205 tokens."

This gap is where AI observability becomes necessary.

Introducing ObservEx Lite

ObservEx Lite is an observability layer designed specifically for AI workflows.

It captures the complete lifecycle of an AI request:

AI request tracking
Agent execution flow
LLM execution traces
Token consumption
Cost estimation
Error tracking
Performance analysis

Instead of treating AI as a single API call, ObservEx Lite makes every important execution step visible.

System Architecture

ObservEx Lite is built around an OpenTelemetry-based observability pipeline that connects an AI application with a monitoring platform.

The architecture has four major layers:

1. AI Application Layer

A FastAPI-based AI agent receives user requests and executes the AI workflow.

2. Telemetry Collection Layer

OpenTelemetry SDK captures:

Distributed traces
Application logs
AI-specific metadata

such as model name, token usage, and execution cost.

3. Telemetry Transport Layer

The collected telemetry is exported using the OpenTelemetry Protocol (OTLP).

4. Observability Layer

SigNoz receives and visualizes the telemetry data through:

Logs Explorer
Trace Explorer

Dashboards

         User Request
               |
               |
               v

      FastAPI AI Agent

               |
  ----------------------------
  |                          |
  v                          v

OpenTelemetry Tracing OpenTelemetry Logging

      |                          |
      ----------------------------
                   |
                   v

                OTLP Exporter

                   |
                   v

                SigNoz

    -----------------------------------
    |                |                |
    v                v                v

  Logs            Traces         Dashboard

The key design decision was separating the AI application from the observability backend.

OpenTelemetry acts as the standard telemetry layer, allowing ObservEx Lite to generate portable telemetry data without being tightly coupled to a single monitoring platform.

This makes the system easier to extend for future AI applications.

Technology Stack

Backend

Python
FastAPI

Observability

OpenTelemetry SDK
OTLP Exporter
SigNoz

AI Monitoring

LLM execution tracing
Token usage tracking
Latency monitoring
Cost estimation

Implementation:
Step 1:FastAPI Instrumentation


from opentelemetry import trace
tracer = trace.get_tracer("observex-agent")
@app.get("/")
async def ai_agent():
    with tracer.start_as_current_span(
        "ai-agent-request"
    ) as request_span:
        request_span.set_attribute(
            "request.type",
            "AI inference"
        )
        with tracer.start_as_current_span(
            "llm-call"
        ) as llm_span:
            llm_span.set_attribute(
                "llm.model",
                "gemini-1.5-pro"
            )
            llm_span.set_attribute(
                "llm.provider",
                "Google"
            )
            response = "AI agent response generated"
        return {
            "message": response
        }

Step 2: Adding AI-Specific Metadata

One of the biggest learnings from this project was:

AI applications require more than traditional application metrics.
Traditional monitoring focuses on:

CPU usage
Memory
Request count
Response time

AI systems require additional information:

Model name
Provider
Input tokens
Output tokens
Cost estimation
Task type

I added custom OpenTelemetry span attributes:


python

llm_span.set_attribute(
    "llm.model",
    "gemini-1.5-pro"
)
llm_span.set_attribute(
    "llm.provider",
    "Google"
)
llm_span.set_attribute(
    "llm.input_tokens",
    120
)
llm_span.set_attribute(
    "llm.output_tokens",
    85
)

llm_span.set_attribute(
        "llm.total_tokens",
        205
    )
    llm_span.set_attribute(
        "llm.cost_estimate",
        0.002
    )

Now every AI execution contains meaningful intelligence:

llm.model = gemini-1.5-pro
llm.provider = Google
llm.input_tokens = 120
llm.output_tokens = 85
llm.total_tokens = 205
llm.cost_estimate = 0.002

This converts an AI call from a black box into measurable data.

Step 3: Implementing Structured AI Logs


python

import logging
logger = logging.getLogger("observex")
logger.info(
    "Calling Gemini model",
    extra={
        "model": "gemini-1.5-pro",
        "task": "text-generation"
    }
)

Along with traces, I added structured application logs.

The system captures events like:

AI request received
Processing AI agent request
Calling Gemini model
LLM response generated
Request completed successfully

Logs provide the timeline of what happened during execution.

Challenges I Faced: Making AI Telemetry Actually Useful

Building ObservEx Lite was not only about adding monitoring code. The real challenge was making AI execution understandable.

1. Connecting FastAPI with SigNoz

The first challenge was creating a complete telemetry pipeline.
My FastAPI application was running successfully, but initially the traces were not appearing in SigNoz.

I had to debug multiple components:

OpenTelemetry SDK configuration
OTLP exporter endpoint
SigNoz collector
Docker services
Telemetry ports

After validating the complete pipeline, my first AI execution trace appeared inside SigNoz.

That was an important milestone because the AI workflow was no longer invisible.

2. Understanding AI-Specific Observability

Traditional application monitoring focuses on metrics like:

Request count
CPU usage
Memory usage
Response latency

However, AI applications require additional context.

A slow AI response could happen because of:

Large token generation
Slow LLM response
Inefficient agent workflow
External API delays

I learned that AI observability requires tracking:

Model information
Token consumption
Execution steps
Latency contribution
Estimated cost

Without this information, debugging AI systems becomes guesswork.

3. Designing Meaningful Telemetry

Another challenge was deciding what information was actually useful.

Collecting every possible metric creates noise.

The important question was:

"What information would help an AI engineer fix a production issue?"

This led me to focus on meaningful telemetry:

AI request lifecycle
LLM execution span
Token usage
Cost estimation
Error events

This made the observability data more actionable.

Why I Chose SigNoz

While building ObservEx Lite, I needed an observability platform that could understand the complete AI execution flow.

I chose SigNoz because it provides:

1. OpenTelemetry Native Architecture

OpenTelemetry allows applications to generate standard telemetry data without being locked into a specific monitoring platform.

This allowed ObservEx Lite to use a clean architecture:

                  FastAPI Application
                          |
                    OpenTelemetry
                          |
                         OTLP
                          |
                        SigNoz

2. Complete Visibility in One Platform

AI debugging requires connecting multiple signals together.

SigNoz provides:

Traces to understand execution flow
Logs to understand events
Dashboards to analyze performance

Instead of checking different tools, developers can investigate the complete AI lifecycle in one place.

3. Developer-Friendly Debugging

The most valuable feature during this project was trace visualization.

Instead of seeing:

   Request failed

I could see:

                  AI Request
                      |
                      |
                   LLM Call
                      |
                      |
                 Response Generated

This made identifying latency and failures much easier.

The Debugging Journey

The first challenge was not writing telemetry code.

The challenge was making telemetry actually flow.

My application was generating spans, but SigNoz showed no data.

I verified:

OTLP endpoint configuration
Port 4317 availability
SigNoz collector status
Docker services

After fixing the exporter configuration, my first trace appeared in SigNoz.

That moment confirmed that my AI workflow was finally observable.

Initially:

The FastAPI application was running
Requests were working
But traces were not visible inside SigNoz

I investigated:

OpenTelemetry exporter configuration

OTLP endpoint
SigNoz services
Docker containers
Telemetry ports

After debugging the telemetry pipeline, traces started appearing in SigNoz.

Seeing my first AI execution trace inside SigNoz was the moment the project became real.

It showed that my AI workflow was no longer invisible.

Building the SigNoz AI Monitoring Dashboard

After connecting OpenTelemetry with SigNoz, I created an AI monitoring dashboard.

The dashboard shows AI request volume, latency, and execution performance.

Understanding AI Execution Through Traces
The SigNoz Trace Explorer shows the complete AI execution path.

Example:

           ai-agent-request

                  |

                  |

                  └── llm-call

                  |

                  |

           Response Generated

Instead of asking:

"Why is my application slow?"

I can now answer:

"The LLM execution consumed most of the request time."

Inspecting LLM Intelligence

The span details contain AI-specific information:

Model:
gemini-1.5-pro

Provider:
Google

Input Tokens:
120

Output Tokens:
85

Total Tokens:
205

Task:
text-generation

This information helps with:

Cost optimization
Performance tuning
Debugging failures
Understanding AI behavior

Monitoring AI Logs

The Logs Explorer provides complete visibility into application events.

Example events:

AI request received
Calling Gemini model
LLM response generated
Request completed successfully

Performance Analysis Using Flamegraphs

Flamegraphs help identify where execution time is spent.

For AI applications, this answers:

Is the agent workflow slow?
Is the LLM response delayed?
Which component requires optimization?

** Why ObservEx Lite Is Different**

Traditional monitoring tells developers:

"Your API took 3 seconds."

ObservEx Lite provides AI-specific answers:

"The LLM call took 2 seconds, consumed 205 tokens, and contributed most of the latency."

The difference is visibility.

AI systems are not just APIs anymore.

They are intelligent workflows that require intelligent monitoring.

What I Learned

This project changed my perspective on AI engineering.

Before building ObservEx Lite, I focused mainly on:

Model integration
Prompt engineering
Application functionality

After this project, I learned that production AI systems also require:

Observability
Reliability
Performance analysis
Cost awareness

An AI system is not production-ready until developers understand what happens inside it.

Future Improvements

Real LLM Provider Integration
Multi-Agent Workflow Visualization
AI Cost Analytics Dashboard
Automated Anomaly Detection
Cloud-Native Deployment

Conclusion

ObservEx Lite demonstrates how OpenTelemetry and SigNoz can bring production-grade observability to AI applications.

Instead of asking:

"Why is my AI application slow?"

Developers can answer:

"The LLM call took 2 seconds, used 205 tokens, and caused most of the latency."

By combining logs, traces, and AI metadata, ObservEx Lite transforms AI debugging from guesswork into measurable engineering.

The future of AI engineering is not only building smarter models.

It is building AI systems that we can understand.

Project Repository

GitHub: https://github.com/DHEVIKA/Hackathon

DEV Community: DHEVIKA M

I Built an AI Observability Platform to Understand What Happens Inside LLM Applications