# Tracking the Ghosts in the Machine: Instrumenting Asynchronous AI Agent Workflows with OpenTelemetry and SigNoz

When an LLM-based agent takes 12 seconds to respond, where exactly does the time go?

Was it a slow vector database retrieval? Did your orchestration framework get stuck in an agent loop? Or was your LLM provider throttling your API requests?

If your architecture relies on asynchronous background workers to process agent steps, finding the culprit is notoriously difficult. Standard HTTP auto-instrumentation falls flat the moment a job gets serialized and handed off to a background queue. The trace context is dropped, the execution path goes dark, and you're left digging through disconnected log lines trying to manually stitch timestamps together.

This is a common wall people hit while building asynchronous pipelines for things like real-time risk estimation or hallucination detection on LLM outputs. Getting past it means moving beyond basic auto-instrumentation into manual OpenTelemetry propagation, paired with SigNoz for visualization.

Here's how to bypass the generic setup guides, bridge asynchronous boundaries, and map out exactly what your AI agents are doing in production.

The Architecture: Why Async Breaks Tracing

To understand why tracing gets complicated, look at a typical production-ready agent flow:

[FastAPI Client]
       │
       ▼ (Submit Task)
[Redis / Celery Queue]  ◄─── Trace Context usually dies here!
       │
       ▼ (Worker Picks Up Task)
[AI Agent Worker]
  ├── Step 1: Query Vector DB (Embedding)
  ├── Step 2: Prompt LLM (Cerebras / OpenRouter)
  └── Step 3: Run Post-Processing Evaluator

When an API request hits FastAPI, OpenTelemetry middleware automatically starts a trace. But when the FastAPI app pushes a background task to a Redis queue, that trace context isn't automatically passed along.

When the background worker picks up the job, it starts execution with a completely blank trace context. The initial API call and the actual agent execution show up in SigNoz as two disconnected events. To see the whole journey — from the user's click to the final token generated — that context has to be propagated manually.

Step 1: Injecting Context into the Queue

To bridge the gap, the current span's context needs to be serialized into a carrier dictionary, passed through the task queue, and then extracted on the worker side to reconstruct the parent-child span relationship.

In OpenTelemetry terms, this is called injecting and extracting trace context.

Here's how to configure the task producer (the FastAPI endpoint) to inject context before dispatching the background job:

import json
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from celery_app import run_agent_workflow

app = FastAPI()
tracer = trace.get_tracer("agent.api")

@app.post("/v1/agent/run")
async def trigger_agent(prompt: str):
    with tracer.start_as_current_span("http_trigger_agent") as span:
        span.set_attribute("user.prompt_length", len(prompt))

        # 1. Create a carrier dictionary to hold trace headers
        carrier = {}

        # 2. Inject the active trace context into the carrier
        TraceContextTextMapPropagator().inject(carrier)

        # 3. Pass the carrier (containing traceparent headers) to our async task
        run_agent_workflow.delay(prompt=prompt, otel_carrier=carrier)

        return {"status": "accepted", "trace_id": format(span.get_span_context().trace_id, "032x")}

The dictionary passed to the worker will contain keys similar to:

{"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"}

Step 2: Extracting Context on the Worker Side

When the asynchronous worker picks up the job, it needs to extract those headers and explicitly start its span as a child of the original HTTP request trace.

from celery import Celery
from opentelemetry import trace
from opentelemetry.context import attach, detach
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

celery_app = Celery("tasks", broker="redis://localhost:6379/0")
tracer = trace.get_tracer("agent.worker")

@celery_app.task
def run_agent_workflow(prompt: str, otel_carrier: dict):
    # 1. Extract the parent context from the incoming carrier dict
    parent_context = TraceContextTextMapPropagator().extract(carrier=otel_carrier)

    # 2. Attach the extracted parent context to the current thread
    token = attach(parent_context)

    try:
        # 3. Start a new span. It will automatically detect parent_context
        with tracer.start_as_current_span("execute_agent_workflow") as span:
            span.set_attribute("agent.task_name", "hallucination_evaluation")

            # Run the actual agent steps
            result = run_evaluation_loop(prompt, span)
            return result

    finally:
        # 4. Clean up the thread context to avoid context leaks
        detach(token)

By explicitly attaching and detaching the context, every database call, HTTP fetch, and inference request executed inside run_evaluation_loop is guaranteed to appear nested visually under the initial user interaction in SigNoz.

Step 3: Tracking the AI Black Box (Custom Span Attributes)

Once context propagation is working, the next challenge is getting visibility into the LLM step itself.

Standard HTTP auto-instrumentation will only report that a POST request to your LLM provider took 3.2 seconds — it won't tell you why. Making that telemetry useful means injecting custom attributes into spans that reflect actual LLM realities: token counts, model variations, and custom status flags.

Here's a clean, manual instrumentation pattern for an LLM invocation step:

import time
import requests
from opentelemetry import trace

tracer = trace.get_tracer("agent.steps")

def call_llm_provider(prompt: str, parent_span) -> str:
    # Creating a sub-span for the LLM execution step
    with tracer.start_as_current_span("llm_generation") as span:
        span.set_attribute("llm.model", "llama-3.1-70b-instruct")
        span.set_attribute("llm.temperature", 0.2)
        span.set_attribute("llm.provider", "OpenRouter")

        start_time = time.time()

        # Simulate or execute your actual LLM post request
        response = requests.post(
            url="https://openrouter.ai/api/v1/chat/completions",
            headers={"Authorization": "Bearer YOUR_KEY"},
            json={
                "model": "meta-llama/llama-3.1-70b-instruct",
                "messages": [{"role": "user", "content": prompt}]
            }
        )

        latency = time.time() - start_time
        span.set_attribute("llm.latency_seconds", latency)

        if response.status_code == 200:
            data = response.json()
            # Extract billing and performance metrics
            usage = data.get("usage", {})
            prompt_tokens = usage.get("prompt_tokens", 0)
            completion_tokens = usage.get("completion_tokens", 0)

            # Injecting valuable business and system metrics
            span.set_attribute("llm.tokens.prompt", prompt_tokens)
            span.set_attribute("llm.tokens.completion", completion_tokens)
            span.set_attribute("llm.tokens.total", prompt_tokens + completion_tokens)

            # Tag system success status
            span.set_status(trace.StatusCode.OK)
            return data["choices"][0]["message"]["content"]
        else:
            span.set_status(trace.StatusCode.ERROR, f"API Error: {response.text}")
            span.record_exception(Exception(f"LLM Provider returned code {response.status_code}"))
            raise ValueError("LLM Call Failed")

Visualizing the System in SigNoz

With this manual instrumentation in place, this translates directly into a much richer picture inside the SigNoz UI.

Instead of isolated execution points, the Traces tab reveals a nested, Gantt-style parent-child tree:

• http_trigger_agent (FastAPI span — user request metadata)
  • execute_agent_workflow (Celery background worker span)
  • vector_db_retrieval (querying database/embeddings)
  • llm_generation (external request to the LLM API)

Digging into Custom Tags

Clicking on the llm_generation span in SigNoz surfaces custom metadata beyond basic HTTP headers:

Attribute	Value
llm.model	llama-3.1-70b-instruct
llm.provider	OpenRouter
llm.tokens.prompt	1024
llm.tokens.completion	256
llm.latency_seconds	2.84

These attributes can drive targeted dashboards. Filtering on llm.model = llama-3.1-70b-instruct, for example, makes it possible to plot average latency for that engine against cheaper alternatives, like smaller 8B-parameter models.

Technical Lessons Learned (Gotchas to Avoid)

A few roadblocks are almost guaranteed the first time you set this up — things that don't show up in the basic quickstarts:

1. Beware of thread pools and coroutines. If you're using Python's asyncio inside your worker loop, standard tracing can lose track of context when tasks yield execution. Run agent runs inside safe context managers, or use asyncio.gather while manually wrapping tasks with context propagation helper utilities.
2. Do not over-instrument. It's tempting to wrap every helper function, array manipulation, and validation step in its own span. Don't. Too many spans create excessive tracing overhead, clutter your database, and make SigNoz dashboards hard to read. Stick to boundaries where I/O happens: API calls, database reads, file access, and LLM inferences.
3. Handle rate limits elegantly. If your external LLM provider returns a 429 Too Many Requests error, capture the Retry-After header and write it directly into your span properties. It will save hours of debugging when a background worker queue suddenly stalls.

Conclusion & Next Steps

Tracing complex, multi-agent AI applications doesn't have to be a guessing game. By propagating trace contexts across queue boundaries and enriching spans with LLM-specific metrics, a black-box system becomes a transparent, debuggable execution path — the kind of precision that turns a prototype into production-grade software.

To go further:

• Dive into the OpenTelemetry Specification on Context Propagation to understand the standard behind this pattern.
• Spin up the open-source observability platform on SigNoz's GitHub repository to start visualizing your own trace architectures.