Tracing Voice AI is Hard: How I Instrumented Streaming LLMs with OpenTelemetry and SigNoz

Voice AI agents are magical until they start lagging. Suddenly my Groq-powered voice assistant started taking 3 seconds to respond and standard APM tools were useless. They could tell me that an HTTP request was open, but they couldn’t tell me if the bottleneck was the Speech-to-Text (STT) transcription, the LLM’s Time to First Token (TTFT), or the Text-to-Speech (TTS) synthesis.

Here’s how I did this by building Zooid, a custom OpenTelemetry instrumentation layer that traces streaming Voice AI, tracks FinOps token costs, and visualizes everything natively in SigNoz.

The Challenge: Web Tracing or Voice Tracing

A typical CRUD web application is easy to trace. A request comes in, hits a database, and returns a JSON response. You drop in the opentelemetry-instrumentation-flask package, and you are done.

Voice AI is a whole other beast. A "Voice Turn" consists of:

1. Obtaining raw audio streams (STT).
2. Streaming text tokens from an LLM as they are produced.
3. On the fly conversion of those text chunks back to audio streams (TTS)
4. Managing sudden interruption from the user (barge-in).

This is where the normal auto-instrumentation fails, since the LLM returns an async generator and not a static response. If you just trace the function execution, the span ends when the generator is created, not when the stream actually finishes.

I needed a way to track the lifecycle of a streaming conversation manually, measure Voice specific metrics (like Time To First Audio) and get this data into SigNoz.# Step 1. Instrumenting the async stream

I created a custom Python decorator (@instrument_voice_app) that manually manages the OpenTelemetry tracer to address the streaming problem.

Instead of wrapping a function, it injects a VoiceTurnContext object into the agent. This enables the inner AI logic to tell you exactly when specific milestones are achieved, such as when the initial audio byte is streamed back to the user.

Here's the basic logic I used to follow the stream:

from opentelemetry import trace
import time

tracer = trace.get_tracer("zooid.voice")

def instrument_voice_app(func):
    async def wrapper(*args, **kwargs):
        with tracer.start_as_current_span("voice_turn") as span:
            turn_start = time.time()
            context = VoiceTurnContext()

            try:
                # Execute the Voice Agent
                result = await func(context, *args, **kwargs)

                # Record Voice-Specific Metrics
                if context.first_audio_time:
                    ttfa = context.first_audio_time - turn_start
                    span.set_attribute("voice.ttfa_ms", ttfa * 1000)

                if context.stt_confidence:
                    span.set_attribute("stt.confidence", context.stt_confidence)

                if context.cost_usd:
                    span.set_attribute("voice.turn_cost_usd", context.cost_usd)

                span.set_status(trace.StatusCode.OK)
                return result

            except Exception as e:
                span.set_status(trace.StatusCode.ERROR, str(e))
                span.record_exception(e)
                raise
    return wrapper

This gave me a beautiful, continuous trace spanning the entire user interaction, but it introduced a new problem: How do I visualize these custom attributes in SigNoz?

Step 2: The ClickHouse “Gotcha” (and how I fix it)

Sending data to the OpenTelemetry Collector (localhost:4317) was quite easy. I could immediately see the traces in the SigNoz "Traces" tab.

But when I went to the "Dashboards" tab and tried to create a Time-Series panel for voice.turn_cost_usd and voice.ttfa_ms, the dashboard showed "No Data".## The Problem with It
This is where I found a very important piece of information on how SigNoz stores OpenTelemetry attributes in ClickHouse.

In Python, when you do span.set_attribute("voice.ttfa_ms", 450.5) it is sent as a Number by OpenTelemetry. This is cleanly stored in the new SigNoz trace schema (signoz_index_v3). However, SigNoz custom dashboard query builder depends on materialized views to map these attributes correctly to aggregate these metrics effectively.

-- The crucial missing lines in the Materialized View
attributes_number as numberTagMap,
attributes_bool as boolTagMap

Once I applied this fix and backfilled the index, the query builder instantly recognized my custom metrics. I was able to query numberTagMap.voice.turn_cost_usd as a Number data type directly in the SigNoz UI.

Step 3: Building the Staff SRE Dashboard

With the data flowing correctly, I built a custom JSON Dashboard in SigNoz tailored entirely for Voice AI.

(If you are following along, you can import this directly into your SigNoz instance by clicking "Import JSON" in the Dashboards tab).

Instead of looking at generic CPU usage, my dashboard tracks:

• P99 TTFA (Time To First Audio): The most critical metric for conversational AI. If this goes over 800ms, the user thinks the bot is broken.
• FinOps / Cost Per Turn: Aggregating the voice.turn_cost_usd attribute to track Groq/OpenAI token spend in real-time.
• Barge-in Rate: Tracking the voice.interrupted boolean attribute to see how often users cut the AI off (a great indicator of whether the bot is talking too much).

(Insert Screenshot of the SigNoz Dashboard showing the TTFA Time Series and FinOps Cost panels here)

Alerting on AI Hallucinations and Latency

Finally, I utilized SigNoz's Alert Rules to act like a Staff SRE. I set up an alert that triggers if the P99 TTFA > 1000ms for more than 5 minutes.

Because we injected the LLM_PROVIDER as a span attribute (e.g., Groq vs OpenAI), the alert payload directly tells me which provider is lagging, allowing my team to instantly route traffic to a fallback provider.

What I Learned

1. OpenTelemetry is incredibly extensible. You aren't limited to HTTP spans. By managing the context manually, you can trace practically any asynchronous, streaming workflow.
2. Know your ClickHouse schema. If your custom attributes aren't showing up in SigNoz dashboards, check how your Number and Boolean tags are being mapped in the underlying ClickHouse tables.
3. Voice AI requires FinOps tracing. Tracking latency isn't enough anymore. Injecting token costs (voice.turn_cost_usd) directly into the trace span means you can correlate slow performance directly with high inference costs.

Conclusion

Standard APM isn't built for the generative AI era, but OpenTelemetry and SigNoz give you the primitives to build exactly what you need. By manually instrumenting our async generators and mapping custom number attributes in ClickHouse, we turned a black-box Voice AI script into a fully observable, enterprise-grade system.

If you want to try it out for your own Voice AI projects, you can grab the Python SDK from PyPI, generate a full boilerplate app via npx create-zooid (npm), or check out the full code on GitHub.

---

Built for the WeMakeDevs & SigNoz Observability Hackathon.