DEV Community

Cover image for One Trace Per Task: Tracing Multi-Agent Systems with OpenTelemetry
vivek
vivek

Posted on

One Trace Per Task: Tracing Multi-Agent Systems with OpenTelemetry

In Part 1 we saw that multi-agent systems fail in the gaps between agents, and we built a handoff record to capture what each agent sent, passed and understood.

But a handoff record floating on its own isn't much use. You need it sitting inside one trace that holds the entire task: every agent, every handoff, every tool call, every MCP server, in order. Otherwise you're back to four green traces and one wrong answer.

This post walks through five steps that turn a team of agents into one readable story:

  1. One trace per task
  2. A span per agent turn
  3. A span per handoff
  4. A clean hierarchy
  5. Outcomes tagged per agent

Then we'll write a small root-cause walker that reads the trace and points at the agent that broke the task.


Step 1: One trace per task

Create a root span the moment the task starts, and pass its trace context to every agent, sub-agent, tool and MCP server the task touches. OpenTelemetry's context propagation handles this across process and service boundaries using the W3C traceparent header.

# orchestrator.py
from opentelemetry import trace
from opentelemetry.propagate import inject

tracer = trace.get_tracer("multi-agent")

def run_task(user_request: str):
    with tracer.start_as_current_span("task") as root:
        root.set_attributes({
            "app.task.type": "churn_report",
            "app.task.goal": user_request[:500],
        })
        plan = call_agent("planner", {"goal": user_request})
        for step in plan["steps"]:
            call_agent(step["agent"], step)
Enter fullscreen mode Exit fullscreen mode

Agents over HTTP:

import httpx

def call_agent(name: str, payload: dict) -> dict:
    headers = {}
    inject(headers)                         # adds traceparent / tracestate
    r = httpx.post(AGENT_URLS[name], json=payload, headers=headers, timeout=120)
    return r.json()
Enter fullscreen mode Exit fullscreen mode
# researcher_service.py (receiving side)
from opentelemetry.propagate import extract

@app.post("/run")
def run(request: Request, payload: dict):
    ctx = extract(dict(request.headers))
    with tracer.start_as_current_span("invoke_agent researcher", context=ctx):
        return researcher.run(payload)
Enter fullscreen mode Exit fullscreen mode

Agents over a queue (very common for long-running agent work): put the carrier in the message itself.

# producer
carrier = {}
inject(carrier)
queue.publish({"task": step, "otel": carrier})

# consumer
msg = queue.consume()
ctx = extract(msg["otel"])
with tracer.start_as_current_span("invoke_agent writer", context=ctx):
    writer.run(msg["task"])
Enter fullscreen mode Exit fullscreen mode

Do the same for MCP servers and any downstream services. A single missed hop splits the trace again, and the split always seems to happen at exactly the hop you need to see.


Step 2: A span per agent turn

Each agent's work becomes a child span carrying its name, role and version, so you can filter and compare by role later.

AGENT_META = {
    "researcher": {"role": "research", "version": "researcher-v7", "model": "your-model-id"},
    "writer":     {"role": "drafting", "version": "writer-v12",    "model": "your-model-id"},
    "reviewer":   {"role": "verify",   "version": "reviewer-v4",   "model": "your-model-id"},
}

def agent_span(name: str):
    meta = AGENT_META[name]
    span = tracer.start_as_current_span(f"invoke_agent {name}")
    return span, {
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": name,
        "gen_ai.request.model": meta["model"],
        "app.agent.role": meta["role"],
        "app.agent.version": meta["version"],
    }
Enter fullscreen mode Exit fullscreen mode

The version tag matters more here than in single-agent systems. You might ship a new writer prompt on Monday and a new reviewer on Wednesday. When quality dips Thursday, per-agent versions tell you which change to suspect.


Step 3: A span per handoff

Handoffs get their own spans, not just log lines. That puts them on the timeline, between the agents they connect.

def handoff(sender: str, receiver: str, task: str, context: dict,
            permissions: list[str], original_goal: str):
    with tracer.start_as_current_span(f"handoff {sender}->{receiver}") as span:
        span.set_attributes({
            "app.handoff.sender": sender,
            "app.handoff.receiver": receiver,
            "app.handoff.task_sent": task[:1000],
            "app.handoff.context_keys": list(context.keys()),
            "app.handoff.context_tokens": count_tokens(context),
            "app.handoff.permissions": permissions,
            "app.task.original_goal": original_goal[:500],
        })
        result = call_agent(receiver, {
            "task": task, "context": context,
            "permissions": permissions, "original_goal": original_goal,
        })
        # receiver returns its one-sentence restatement (see Part 1)
        span.set_attribute("app.handoff.task_understood",
                           result.get("task_understood", "")[:1000])
        return result
Enter fullscreen mode Exit fullscreen mode

Now the comparison from Part 1 (sent vs. understood vs. original goal) can run directly on spans in your trace backend.


Step 4: Model the hierarchy

Multi-agent traces get unreadable fast. A clean, consistent hierarchy keeps them navigable:

task                                   ← root: one per user request
└── invoke_agent planner               ← agent turn
    ├── chat plan                      ← model call
    ├── handoff planner->researcher    ← handoff
    │   └── invoke_agent researcher
    │       ├── chat search_strategy
    │       ├── execute_tool sql_query ← tool call
    │       └── execute_tool web_fetch
    ├── handoff planner->writer
    │   └── invoke_agent writer
    │       └── chat draft
    └── handoff writer->reviewer
        └── invoke_agent reviewer
            └── chat review
Enter fullscreen mode Exit fullscreen mode

The pattern: task → agent → (handoff → agent)* → model call → tool call. Pick a shape and stick to it across every agent and team. Consistent span names (invoke_agent X, handoff A->B, execute_tool Y) make queries like "all handoffs into the writer this week" trivial.


Step 5: Tag outcomes per agent

Mark every agent span with its own result, not just the task's. Otherwise every failure gets blamed on whoever acted last, which is usually the writer or the reviewer, and usually wrong.

from opentelemetry.trace import Status, StatusCode

def finish_agent(span, outcome: str, reason: str | None = None):
    # outcome: ok | degraded | failed | rejected_input
    span.set_attribute("app.agent.outcome", outcome)
    if reason:
        span.set_attribute("app.agent.outcome_reason", reason)
    if outcome in ("failed", "rejected_input"):
        span.set_status(Status(StatusCode.ERROR, reason or outcome))
Enter fullscreen mode Exit fullscreen mode

Where do per-agent outcomes come from? Cheap checks you already have: schema validation on each agent's output, the handoff drift score, a tool error, the reviewer's verdict, an eval run on a sample.


Finding the agent that broke the task

When a task fails, read the trace from the root and find the first handoff where the task changed shape. That's usually where the problem started, even if the damage showed up three agents later.

Here's that idea as code, working on spans exported from your backend:

from dataclasses import dataclass

@dataclass
class Suspect:
    span_name: str
    agent: str
    reason: str

def first_bad_hop(spans: list[dict], drift, threshold=0.75) -> Suspect | None:
    """spans: one trace, sorted by start time."""
    for s in spans:
        a = s["attributes"]

        # 1. A handoff where the received task drifted from the goal
        if s["name"].startswith("handoff "):
            score = drift(a.get("app.handoff.task_understood", ""),
                          a.get("app.task.original_goal", ""))
            if score < threshold:
                return Suspect(s["name"], a["app.handoff.receiver"],
                               f"task changed shape (similarity {score:.2f})")

        # 2. An agent that marked its own output as failed or degraded
        if a.get("app.agent.outcome") in ("failed", "degraded"):
            return Suspect(s["name"], a.get("gen_ai.agent.name", "?"),
                           a.get("app.agent.outcome_reason", "agent reported failure"))

        # 3. A tool error that the agent carried on past
        if a.get("gen_ai.operation.name") == "execute_tool" and s["status"] == "ERROR":
            return Suspect(s["name"], a.get("gen_ai.tool.name", "?"),
                           "tool error before task completed")
    return None
Enter fullscreen mode Exit fullscreen mode

Run it on the EU-churn example from Part 1:

>>> first_bad_hop(spans, drift)
Suspect(span_name='handoff planner->researcher',
        agent='researcher',
        reason='task changed shape (similarity 0.71)')
Enter fullscreen mode Exit fullscreen mode

Not the writer. Not the reviewer. The first hop. Run the same function across every failed trace in a week and group by span_name, and you get a ranked list of your weakest handoffs, which is a much better backlog than "the agents are flaky."


Practical notes

  • Sample whole tasks, not spans. If you sample, decide at the root and keep or drop the entire task. Half a multi-agent trace is worse than none. Tail-based sampling in the OpenTelemetry Collector can keep 100% of failed or high-risk tasks.
  • Truncate, don't drop, large attributes. Store the first N characters of task text and context on the span; put the full payload in logs keyed by span ID.
  • Redact at capture. Handoff context often carries customer data. Scrub it before it leaves the process.
  • Expect convention churn. OpenTelemetry's GenAI conventions are still in Development. Keep attribute names in one module.

For a broader look at what to collect across single and multi-agent setups, see this complete guide to AI agent observability.


TL;DR

  • Propagate context so one task = one trace, across HTTP, queues and MCP servers.
  • Give every agent turn and every handoff its own span, tagged with name, role and version.
  • Keep a consistent hierarchy: task → agent → handoff → agent → model call → tool call.
  • Tag outcomes per agent, then walk the trace for the first hop where the task changed shape.

In Part 3, we'll move from single traces to dashboards: system, role, handoff and verification metrics, the alert that catches a rubber-stamp reviewer, and the security signals for when one compromised agent starts instructing the rest. It all builds on the checklist in this breakdown of multi-agent monitoring metrics and security signals.

How are you propagating context between agents today: HTTP headers, queue messages, or hoping for the best? 🧵

Top comments (0)