DEV Community

Akash Goyal
Akash Goyal

Posted on

Custom Metrics Dashboard for GenAI apps with SigNoz and OpenTelemetry

SigNoz supports lot of features and has a lot of documentations around those features. It has a full section dedicated to LLM observability across different service providers.

But, this blog is a bit different. Instead of relying on out-of-the-box vendor presets, it shows you how to create a completely custom dashboard tailored for your specific GenAI/agentic application as per your needs.

Just tracking generic infrastructure metrics like CPU and memory utilization isn't enough. To understand how your AI features are truly performing, you need specialize application-level telemetry.

All the code discussed in this post is available in the GitHub Repository.

Signoz supports both Trace Data and Metrics Data - for setting metric widget queries. This feature is really helpful in scaling the existing instrumented codebases & has uniqueness.

Part 1: Setting Up SigNoz Locally (Self-Hosted Docker)

Referred Signoz Doc Section for setting this up.

### Step 1: Install Foundry
curl -fsSL https://signoz.io/foundry.sh | bash

### Step 2: Create `casting.yaml`
vi casting.yaml  #add below
"""
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
"""

### Step 3: Deploy SigNoz - spin up your local SigNoz infrastructure:
foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

Part 2: Application Architecture & Quick Start

I designed a minimal llm-inference based workflow application, with two language models, to work on this topic.

1. Application Architecture Block

Demo Application Design

Explanation of the diagram:

  1. Traffic Flow: The LLM Client actively invokes endpoints on the LLM Server to run inference tasks.
  2. Telemetry Collection: Both the client and the server generate performance data during these interactions.
  3. Data Export: The gathered telemetry is continuously pushed out via a Metrics periodic export routine utilizing OpenTelemetry standard components (OTLP Span Exporter and OTLP Metric Exporter).
  4. Storage and Visualization: The exported data flows directly into SigNoz, where it is permanently stored in a ClickHouse DB.
  5. Observability: Finally, developers can monitor the application using the integrated Trace Viewer and Dashboards.

2. Getting Started with App

Clone the repo from github. Then, follow these steps to initialize the environment and launch your components:

# 1. Setup local virtual environment
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode 
# 2. Ensure metric computation logic is added.
Enter fullscreen mode Exit fullscreen mode
  1. CODE - custom metrics relevant to the application.
  2. CODE - Setting span attrs & record metrics Recording metrics & setting Span attributes 
# 3. Spin up the LLM Server container 
# Create your docker-compose.yaml file first, then run:
docker compose up -d llm-server
Enter fullscreen mode Exit fullscreen mode

llm-server is up

# 4. Configure local environment and run client loops sequentially
pip install httpx opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp 
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_RESOURCE_ATTRIBUTES="service.name=llm-client-service"
python llm_client.py
Enter fullscreen mode Exit fullscreen mode

Part 3: Visualising the Traces

A look at the traces :

  1. Success Client requests - Trace View:
    Success Client Requests Trace

  2. If the infra capacity is limited, after some requests, time-out errors might start surfacing. (Nothing to worry, just restart the llm-server from docker app). Related traces examples below:

Requests getting Timed Out one after other. (Server was hung)
Requests getting Timed Out

Errored Request Trace (server was not up by that time)
Trace of Errored Request

Part 4: Custom Metrics Dashboard in SigNoz

Creating the Custom Dashboard - Below are the steps I followed in order:

  1. Implemented the custom metrics relevant to the application.
  2. Added relevant attributes in spans & record metrics.
  3. Referred one of the template dashboards in Signoz Dashboard Page. Took the Agno Dashboard as Base to build upon.
  4. Took the code for 'Error rate' metric widget. Created new metric widgets, in sync with the metric_record.py logic.
  5. Modified the queries of existing widgets in Base Json.

For any further edits (metric-name, widget-type, etc.), I used the SigNoz's native Query Builder.

Update Metric Widget in DB Builder

The dashboard JSON used for this experiment is shared in the repo file.

How it appears in SigNoz:

Signoz Default Metrics Section :
Signoz Default metrics section view

Your Custom Metrics dashboard - data rendered using both span params & metrics :

Custom Metrics dashboard

Conclusion & Impact: An insight into custom dashboards with Signoz

These are my findings on how Signoz supports custom metrics dashboards, and what changes are needed or good to have while working on such requirements.

Signoz supports both Trace Data and Metrics Data - for setting metric widget queries. This feature is really helpful in scaling the existing instrumented codebases & has uniqueness.

I hope this content would help readers enhance their knowledge.

Ready to get started? Clone the experimental code from GitHub Repository and spin it up in your local developer environment!

Top comments (0)