DEV Community

Cover image for Observing a Multi-Agent LLM System: What I Learned Instrumenting 8 Agents with OpenTelemetry
AVISH MANIAR
AVISH MANIAR

Posted on

Observing a Multi-Agent LLM System: What I Learned Instrumenting 8 Agents with OpenTelemetry

#opentelemetry #signoz

I spent three days trying to figure out why my multi-agent LLM workflow was taking 15 minutes when it should take 5. Without observability, I was stuck adding print statements and rerunning the entire pipeline. That's when I decided to properly instrument it with OpenTelemetry and SigNoz.

This post walks through how I added traces, logs, and metrics to a Flask app that orchestrates 8 LLM agents using Ollama and Mistral. More importantly, it covers what I learned about monitoring agent-based systems that you won't find in the standard OpenTelemetry docs.

The Problem: A Black Box LLM Pipeline

My user story automation app processes requirements documents through 8 specialized agents:

  • RequirementsAgent extracts functionalities
  • EpicExtractorAgent creates epics in batches
  • EpicRefinerAgent polishes the output
  • StoryGeneratorAgent writes user stories
  • TestCaseAgent generates test scenarios

The workflow runs for 10-15 minutes per document, making debug cycles painful. When something went wrong, I had no idea which agent was slow or failing. I needed to see inside the pipeline.

Setting Up OpenTelemetry: The Basics Plus Agent Tracing

The standard Flask instrumentation gives you HTTP endpoints and database queries, but it doesn't know anything about your agents. Here's what I added to src/backend/otel_config.py:

def init_telemetry(app):
    SIGNOZ_BASE = os.getenv('SIGNOZ_OTLP_ENDPOINT', 'http://localhost:4318')

    resource = Resource.create({
        "service.name": "user-story-automation",
        "service.version": "1.0.0",
        "deployment.environment": "development",
    })

    # Traces: Track agent operations and workflow execution
    trace_provider = TracerProvider(resource=resource)
    trace_exporter = OTLPSpanExporter(endpoint=f"{SIGNOZ_BASE}/v1/traces")
    trace_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
    trace.set_tracer_provider(trace_provider)

    # Logs: Centralized logging with context
    logger_provider = LoggerProvider(resource=resource)
    log_exporter = OTLPLogExporter(endpoint=f"{SIGNOZ_BASE}/v1/logs")
    logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
    set_logger_provider(logger_provider)

    # Metrics: Custom histograms for agent performance
    metric_exporter = OTLPMetricExporter(endpoint=f"{SIGNOZ_BASE}/v1/metrics")
    meter_provider = MeterProvider(
        resource=resource,
        metric_readers=[PeriodicExportingMetricReader(metric_exporter)]
    )
    metrics.set_meter_provider(meter_provider)

    # Auto-instrumentation
    FlaskInstrumentor().instrument_app(app)
    SQLAlchemyInstrumentor().instrument()
    RequestsInstrumentor().instrument()
Enter fullscreen mode Exit fullscreen mode

The auto-instrumentation got me basic traces, but here's the first gotcha: Flask traces showed me /process-document taking 674 seconds, but that single span told me nothing about which agent was slow. I needed manual spans.

Manual Spans: The Only Way to See Agent Operations

I wrapped each agent call with a manual span in src/backend/routes/api.py:

# Requirements extraction
with create_manual_span("agent.requirements_extraction", {
    "agent.name": "RequirementsAgent",
    "document.length": len(extracted_text)
}):
    requirements = rat(refine_doc, extract_functionarity, extracted_text, chat, mode)

# Epic extraction with batch tracking
for batch_idx, batch in enumerate(batches):
    with create_manual_span("agent.epic_extraction_batch", {
        "agent.name": "EpicExtractorAgent",
        "batch.index": batch_idx + 1,
        "batch.total": len(batches),
        "batch.size": len(batch)
    }):
        batch_result = extract_epics_func(batch_text, chat, mode)
Enter fullscreen mode Exit fullscreen mode

This is where I learned something important: batch operations need special handling. Without tracking batch.index and batch.total, I couldn't tell if batch 2 was consistently slower than batches 1 and 3 (spoiler: it was).

What The Data Revealed

Once the instrumentation was running, I built dashboards using SigNoz's MCP integration. Here's what I actually learned from the data.

Finding the Bottleneck

The main dashboard showed average workflow time at 13.28 minutes across 3 test runs. Breaking it down by agent revealed the issue:
SigNoz dashboard showing 3 total workflow requests, 13.28 minute average workflow time, 23 total agent operations, and 0 errors, with line charts for Agent Execution Time P95 and Workflow Requests Over Time<br>

  • Epic Extraction Batch: 11 operations, 2.06 minutes average
  • Requirements Extraction: 3 operations, 2.68 minutes average
  • Epic Refinement: 2 operations, 5.77 minutes average

Epic Refinement was taking 3x longer per operation than anything else. That's where I needed to optimize.

Table and donut chart showing Agent Performance Breakdown: epic_extraction_batch (11 ops, 2.06 min avg), test_case_generation_batch (7 ops, 1.21 min avg), requirements_extraction (3 ops, 2.68 min avg), and epic_refinement (2 ops, 5.77 min avg, highest P95 at 7.49 min)

The pie chart showed epic extraction batch consuming 48% of total operations, but the table showed refinement had the worst P95 latency at 7.49 minutes.

P99 vs P50: Why Percentiles Matter

I added a Performance Bottlenecks dashboard to track percentile latency. This is where things got interesting:

P99 vs P50 latency chart showing P99 spiking to 16.67 minutes while P50 stays near 3.33 minutes, alongside a Time Distribution by Agent donut chart showing epic_extraction_batch consuming 22.63 minutes total

The graph shows P99 latency spiking to 16.67 minutes while P50 stayed around 3.33 minutes. That's a 5x difference. This told me the LLM (Ollama running Mistral locally) had massive variance in response time, probably due to resource contention.

The time distribution pie chart revealed another insight: epic extraction batch consumes 45% (22.83 mins) of total workflow time across all agents. Even though individual operations are fast (2.06 min average), the sheer volume makes it the biggest time sink.

Error Tracking by Agent

I built an Advanced Analytics dashboard with per-agent error rates:

Advanced Analytics dashboard with four panels: Latency Breakdown showing P50/P90/P95/P99 with P99 spiking to 13.33 minutes, Agent Success Rate Over Time near 100%, Error Rate by Agent Type showing a 4% spike for epic_refinement, and Workflow Completion Rate above the 90% critical threshold

The error rate chart showed epic_refinement had a 4% error spike on July 17th, then recovered. Without this view, I would never have known there was a transient issue that resolved itself.

The success rate panel showed 100% overall, but the multi-percentile latency breakdown (P50/P90/P95/P99 on one chart) made it obvious where the variance was coming from.

Using SigNoz MCP for Dashboard Creation

Instead of clicking through the UI, I used SigNoz's Model Context Protocol (MCP) server with Claude. I configured it in ~/.claude.json:

{
  "mcp": {
    "signoz": {
      "url": "http://localhost:8000/mcp",
      "headers": {
        "SIGNOZ-API-KEY": "your-api-key-here"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Then I could create dashboards by describing what I wanted:

Me: "Create a dashboard showing agent performance with P95 latency"
Assistant: *Uses signoz_create_dashboard MCP tool*
Result: Complete dashboard with 10 widgets created in seconds
Enter fullscreen mode Exit fullscreen mode

This was 10x faster than clicking through the UI. I created all 5 dashboards this way, including complex multi-query widgets for the Advanced Analytics dashboard.

Alerts: Catching Issues Before They Matter

I set up 7 alerts total, but two were particularly useful:

1. Agent Performance Anomaly Detection

This uses z-score anomaly detection (2 standard deviations) on the agent_duration_seconds metric with hourly seasonality. It caught a case where epic refinement suddenly took 2x normal time due to LLM resource contention. A static threshold would have missed this gradual degradation.

2. Workflow Completion Rate

I track agent.requirements_extraction (the first operation in every workflow) and calculate success rate with a formula query:

(successful_operations / total_operations) * 100
Enter fullscreen mode Exit fullscreen mode

The alert fires if completion rate drops below 90%. This was tricky to set up because I initially filtered for /process-document span name, which doesn't exist. I had to search traces to find the actual span names being generated.

What I Learned (The Stuff Not In The Docs)

1. Manual Spans Are Non-Negotiable for Agent Systems

Auto-instrumentation is great for standard web apps, but it can't see your domain logic. Every agent operation needs an explicit span with relevant attributes like agent.name, batch.index, and batch.size.

2. Batch Operations Need Tracking

If you're processing items in batches, track:

  • batch.index: Which batch is this? (1, 2, 3)
  • batch.total: How many batches total?
  • batch.size: How many items in this batch?

This let me discover that batch 2 was consistently 20% slower than batches 1 and 3, which pointed to a data distribution problem.

3. P95/P99 Reveals What Average Hides

Average latency was 3.33 minutes, but P99 was 13.33 minutes. If I'd only looked at averages, I would have thought performance was fine. The high percentiles exposed the LLM variance issue.

4. Resource Attributes Speed Up Queries

I tagged all data with service.name as a resource attribute (not a span attribute). SigNoz queries filter on resource attributes first, making every query significantly faster. This is in the docs but easy to miss.

5. MCP Integration Changed My Workflow

Being able to describe dashboards in natural language ("show me P50/P90/P95/P99 on one chart") instead of configuring query builders saved hours. The MCP server turned dashboard creation from a chore into a conversation.

The Results

Before instrumentation:

  • No visibility into which agent was slow
  • Had to add print statements and rerun 15-minute workflows to debug
  • Unknown if errors were happening in production

After instrumentation:

  • Identified Epic Refinement as the bottleneck (5.77 min average)
  • Discovered batch 2 consistently slower than batches 1 and 3
  • Caught transient 4% error spike that self-resolved
  • P99 latency tracking revealed 5x variance in LLM performance
  • Alerts catch failures within 1 minute

Most importantly, I now have actionable optimization targets:

  • Epic Refinement could be parallelized to reduce the 5.77 min bottleneck
  • Investigate why batch 2 is slower (probably contains more complex epics)
  • Consider LLM response caching to reduce P99 variance

Conclusion

Instrumenting a multi-agent LLM system taught me that observability isn't just about adding traces. It's about choosing the right attributes to track (like batch index), measuring the right percentiles (P95/P99, not just average), and building dashboards that answer specific questions ("which agent is slow?" not "here's all the data").

The combination of OpenTelemetry's flexibility and SigNoz's MCP integration made this project possible in a weekend. If you're building with LLM agents, start with manual spans around each agent operation and work from there.

Tech Stack: Flask, LangChain, Ollama (Mistral), OpenTelemetry, SigNoz Foundry
Dashboards: 4 dashboards, 7 alerts, all created via MCP

Top comments (0)