I Built an AI Observability Platform to Understand What Happens Inside LLM Applications

Introduction

Your AI application is working.

1. The response is generated.
2. The API returns 200.
3. The user is happy.

But inside the system, you have no idea what actually happened.

1. Was the LLM slow?
2. Did the agent waste tokens?
3. Where did latency come from?

This was the exact problem I faced while building an AI application during the Agents of SigNoz hackathon.

The AI response was generated successfully, but I had no visibility into the internal execution flow.

A user only sees:

"The AI generated a response."

But as an AI engineer, I need deeper answers:

• Why was the response slow?
• Was the delay caused by my backend or the LLM?
• How many tokens were consumed?
• Which step failed?
• What was the estimated cost of the execution?

This experience made me realize something important:

Building an AI application is only half the challenge. Understanding its behavior in production is equally important.

This motivated me to build ObservEx Lite, an AI Agent Observability Platform using OpenTelemetry and SigNoz.

The goal was simple:

Transform AI debugging from guesswork into measurable engineering.

The Problem: AI Applications Are Black Boxes

Traditional applications usually have a predictable flow:

                    User
                     |
                    API
                     |
                  Database
                     |
                  Response

Monitoring these systems is relatively straightforward.

However, modern AI applications are different.

An AI workflow can look like:

                  User Request

                     |
                     |

                   AI Agent

                     |
                     |

                 Reasoning Process

                     |
                     |

                  LLM Call

                     |
                     |

                Generated Response

Inside this workflow, many things can go wrong:

1. LLM latency can increase
2. Token usage can become expensive
3. A particular AI step can fail
4. The model response can become unpredictable

Traditional monitoring can tell us:

"The API request took 5 seconds."

But AI developers need answers like:

 "The LLM call consumed most of the latency and used 205 tokens."

This gap is where AI observability becomes necessary.

Introducing ObservEx Lite

ObservEx Lite is an observability layer designed specifically for AI workflows.

It captures the complete lifecycle of an AI request:

• AI request tracking
• Agent execution flow
• LLM execution traces
• Token consumption
• Cost estimation
• Error tracking
• Performance analysis

Instead of treating AI as a single API call, ObservEx Lite makes every important execution step visible.

System Architecture

ObservEx Lite is built around an OpenTelemetry-based observability pipeline that connects an AI application with a monitoring platform.

The architecture has four major layers:

1. AI Application Layer

A FastAPI-based AI agent receives user requests and executes the AI workflow.

2. Telemetry Collection Layer

OpenTelemetry SDK captures:

• Distributed traces
• Application logs
• AI-specific metadata

such as model name, token usage, and execution cost.

3. Telemetry Transport Layer

The collected telemetry is exported using the OpenTelemetry Protocol (OTLP).

4. Observability Layer

SigNoz receives and visualizes the telemetry data through:

• Logs Explorer
• Trace Explorer

• Dashboards

           User Request
                 |
                 |
                 v
  
        FastAPI AI Agent
  
                 |
    ----------------------------
    |                          |
    v                          v
  

OpenTelemetry Tracing OpenTelemetry Logging

      |                          |
      ----------------------------
                   |
                   v

                OTLP Exporter

                   |
                   v

                SigNoz

    -----------------------------------
    |                |                |
    v                v                v

  Logs            Traces         Dashboard

The key design decision was separating the AI application from the observability backend.

OpenTelemetry acts as the standard telemetry layer, allowing ObservEx Lite to generate portable telemetry data without being tightly coupled to a single monitoring platform.

This makes the system easier to extend for future AI applications.

Technology Stack

Backend

1. Python
2. FastAPI

Observability

1. OpenTelemetry SDK
2. OTLP Exporter
3. SigNoz

AI Monitoring

1. LLM execution tracing
2. Token usage tracking
3. Latency monitoring
4. Cost estimation

Implementation:
Step 1:FastAPI Instrumentation

from opentelemetry import trace
tracer = trace.get_tracer("observex-agent")
@app.get("/")
async def ai_agent():
    with tracer.start_as_current_span(
        "ai-agent-request"
    ) as request_span:
        request_span.set_attribute(
            "request.type",
            "AI inference"
        )
        with tracer.start_as_current_span(
            "llm-call"
        ) as llm_span:
            llm_span.set_attribute(
                "llm.model",
                "gemini-1.5-pro"
            )
            llm_span.set_attribute(
                "llm.provider",
                "Google"
            )
            response = "AI agent response generated"
        return {
            "message": response
        }

Step 2: Adding AI-Specific Metadata

One of the biggest learnings from this project was:

AI applications require more than traditional application metrics.
Traditional monitoring focuses on:

1. CPU usage
2. Memory
3. Request count
4. Response time

AI systems require additional information:

1. Model name
2. Provider
3. Input tokens
4. Output tokens
5. Cost estimation
6. Task type

I added custom OpenTelemetry span attributes:

python

llm_span.set_attribute(
    "llm.model",
    "gemini-1.5-pro"
)
llm_span.set_attribute(
    "llm.provider",
    "Google"
)
llm_span.set_attribute(
    "llm.input_tokens",
    120
)
llm_span.set_attribute(
    "llm.output_tokens",
    85
)

llm_span.set_attribute(
        "llm.total_tokens",
        205
    )
    llm_span.set_attribute(
        "llm.cost_estimate",
        0.002
    )

Now every AI execution contains meaningful intelligence:

llm.model = gemini-1.5-pro
llm.provider = Google
llm.input_tokens = 120
llm.output_tokens = 85
llm.total_tokens = 205
llm.cost_estimate = 0.002

This converts an AI call from a black box into measurable data.

Step 3: Implementing Structured AI Logs

python

import logging
logger = logging.getLogger("observex")
logger.info(
    "Calling Gemini model",
    extra={
        "model": "gemini-1.5-pro",
        "task": "text-generation"
    }
)

Along with traces, I added structured application logs.

The system captures events like:

1. AI request received
2. Processing AI agent request
3. Calling Gemini model
4. LLM response generated
5. Request completed successfully

Logs provide the timeline of what happened during execution.

Challenges I Faced: Making AI Telemetry Actually Useful

Building ObservEx Lite was not only about adding monitoring code. The real challenge was making AI execution understandable.

1. Connecting FastAPI with SigNoz

The first challenge was creating a complete telemetry pipeline.
My FastAPI application was running successfully, but initially the traces were not appearing in SigNoz.

I had to debug multiple components:

1. OpenTelemetry SDK configuration
2. OTLP exporter endpoint
3. SigNoz collector
4. Docker services
5. Telemetry ports

After validating the complete pipeline, my first AI execution trace appeared inside SigNoz.

That was an important milestone because the AI workflow was no longer invisible.

2. Understanding AI-Specific Observability

Traditional application monitoring focuses on metrics like:

1. Request count
2. CPU usage
3. Memory usage
4. Response latency

However, AI applications require additional context.

A slow AI response could happen because of:

1. Large token generation
2. Slow LLM response
3. Inefficient agent workflow
4. External API delays

I learned that AI observability requires tracking:

1. Model information
2. Token consumption
3. Execution steps
4. Latency contribution
5. Estimated cost

Without this information, debugging AI systems becomes guesswork.

3. Designing Meaningful Telemetry

Another challenge was deciding what information was actually useful.

Collecting every possible metric creates noise.

The important question was:

"What information would help an AI engineer fix a production issue?"

This led me to focus on meaningful telemetry:

1. AI request lifecycle
2. LLM execution span
3. Token usage
4. Cost estimation
5. Error events

This made the observability data more actionable.

Why I Chose SigNoz

While building ObservEx Lite, I needed an observability platform that could understand the complete AI execution flow.

I chose SigNoz because it provides:

1. OpenTelemetry Native Architecture

OpenTelemetry allows applications to generate standard telemetry data without being locked into a specific monitoring platform.

This allowed ObservEx Lite to use a clean architecture:

                  FastAPI Application
                          |
                    OpenTelemetry
                          |
                         OTLP
                          |
                        SigNoz

2. Complete Visibility in One Platform

AI debugging requires connecting multiple signals together.

SigNoz provides:

1. Traces to understand execution flow
2. Logs to understand events
3. Dashboards to analyze performance

Instead of checking different tools, developers can investigate the complete AI lifecycle in one place.

3. Developer-Friendly Debugging

The most valuable feature during this project was trace visualization.

Instead of seeing:

   Request failed

I could see:

                  AI Request
                      |
                      |
                   LLM Call
                      |
                      |
                 Response Generated

This made identifying latency and failures much easier.

The Debugging Journey

The first challenge was not writing telemetry code.

The challenge was making telemetry actually flow.

My application was generating spans, but SigNoz showed no data.

I verified:

• OTLP endpoint configuration
• Port 4317 availability
• SigNoz collector status
• Docker services

After fixing the exporter configuration, my first trace appeared in SigNoz.

That moment confirmed that my AI workflow was finally observable.

Initially:

1. The FastAPI application was running
2. Requests were working
3. But traces were not visible inside SigNoz

I investigated:

OpenTelemetry exporter configuration

1. OTLP endpoint
2. SigNoz services
3. Docker containers
4. Telemetry ports

After debugging the telemetry pipeline, traces started appearing in SigNoz.

Seeing my first AI execution trace inside SigNoz was the moment the project became real.

It showed that my AI workflow was no longer invisible.

Building the SigNoz AI Monitoring Dashboard

After connecting OpenTelemetry with SigNoz, I created an AI monitoring dashboard.

The dashboard shows AI request volume, latency, and execution performance.

Understanding AI Execution Through Traces
The SigNoz Trace Explorer shows the complete AI execution path.

Example:

           ai-agent-request

                  |

                  |

                  └── llm-call

                  |

                  |

           Response Generated

Instead of asking:

"Why is my application slow?"

I can now answer:

"The LLM execution consumed most of the request time."

Inspecting LLM Intelligence

The span details contain AI-specific information:

Model:
gemini-1.5-pro

Provider:
Google

Input Tokens:
120

Output Tokens:
85

Total Tokens:
205

Task:
text-generation

This information helps with:

1. Cost optimization
2. Performance tuning
3. Debugging failures
4. Understanding AI behavior

Monitoring AI Logs

The Logs Explorer provides complete visibility into application events.

Example events:

1. AI request received
2. Calling Gemini model
3. LLM response generated
4. Request completed successfully

Performance Analysis Using Flamegraphs

Flamegraphs help identify where execution time is spent.

For AI applications, this answers:

1. Is the agent workflow slow?
2. Is the LLM response delayed?
3. Which component requires optimization?

** Why ObservEx Lite Is Different**

Traditional monitoring tells developers:

"Your API took 3 seconds."

ObservEx Lite provides AI-specific answers:

"The LLM call took 2 seconds, consumed 205 tokens, and contributed most of the latency."

The difference is visibility.

AI systems are not just APIs anymore.

They are intelligent workflows that require intelligent monitoring.

What I Learned

This project changed my perspective on AI engineering.

Before building ObservEx Lite, I focused mainly on:

1. Model integration
2. Prompt engineering
3. Application functionality

After this project, I learned that production AI systems also require:

1. Observability
2. Reliability
3. Performance analysis
4. Cost awareness

An AI system is not production-ready until developers understand what happens inside it.

Future Improvements

1. Real LLM Provider Integration
2. Multi-Agent Workflow Visualization
3. AI Cost Analytics Dashboard
4. Automated Anomaly Detection
5. Cloud-Native Deployment

Conclusion

ObservEx Lite demonstrates how OpenTelemetry and SigNoz can bring production-grade observability to AI applications.

Instead of asking:

"Why is my AI application slow?"

Developers can answer:

"The LLM call took 2 seconds, used 205 tokens, and caused most of the latency."

By combining logs, traces, and AI metadata, ObservEx Lite transforms AI debugging from guesswork into measurable engineering.

The future of AI engineering is not only building smarter models.

It is building AI systems that we can understand.

Project Repository

GitHub: https://github.com/DHEVIKA/Hackathon