An agent run that exports more than one trace root did not record a run. It recorded fragments. Those fragments cannot be ordered, cannot be diffed against yesterday’s export, and cannot tell you which tool returned after the parent step had already closed. The process may still have printed a final answer. The observation is already broken.
HTTP services live with forests. A load balancer, a client that never set a propagator, a one-percent sampler — all of them mint extra roots, and on-call still reconstructs a request from logs. Agent loops do not get that luxury. One model step fans into tools. Tools fork. Forks talk to shells, sandboxes, and stdio servers. If traceparent dies at the fork, the collector stores a tree plus a pile of stumps. You will debug the stumps.
This article treats a split forest as a failed run, not as a telemetry warning. The artifact is a small Python runner, a W3C header injector, and a checker you can keep next to the loop. The snippets are labeled examples. They are not production measurements.
Why the forest appears
Imagine a parent span named run that starts a tool named checkout_repo. The tool is a subprocess. Subprocesses do not inherit in-process context objects. They inherit environment variables and file descriptors. If you forget to write TRACEPARENT into that environment, the tool starts a new trace_id. Your backend now holds two traces that share a run_id attribute if you remembered to copy it, or share nothing if you did not.
Logs will still interleave on stdout. They always do. That interleaving is why a log stream cannot be the source of truth for tool order. The forest can, but only when it is a tree.
A second, quieter failure is the late child. The parent step hits a deadline and closes. The subprocess later emits a span that still names the old parent. Some backends attach it. Some drop it. Some create a new root. From the agent’s point of view the tool failed with timeout. From the collector’s point of view a span arrived from a closed world. If you do not name that case, the retry looks like a second successful tool call.
Copied HTTP sampling makes this worse. Edge traffic is high volume and low diagnostic value per span. Agent traffic is the opposite. Dropping the only tool span that mutated state is not “saving cardinality.” It is deleting the receipt.
A contract small enough to test
Three rules fit in one paragraph. One root per run_id. Every tool span must carry a parent that exists in the same trace. A span that starts after its parent ended is not a child; it is either an event named late_tool on that parent, or it is a failure of the run.
That is stricter than typical APM. It should be. You can execute the contract locally with JSONL and no vendor. OpenTelemetry can emit the same shape later. The point is the assertion, not the exporter logo.
Inject context before the tool forks
The runner below mints one trace_id for the run, formats a W3C traceparent, and puts it in the child environment. The child is responsible for starting its body span under that parent. If it ignores the header, it becomes a second root, and the checker fails.
# agent_trace_contract.py — example runner, not a library
from __future__ import annotations
import json, os, secrets, subprocess, sys, time
from dataclasses import asdict, dataclass, field
from typing import Optional
def _hex(n: int) -> str:
return secrets.token_hex(n)
def format_traceparent(trace_id: str, span_id: str, sampled: bool = True) -> str:
flags = "01" if sampled else "00"
return f"00-{trace_id}-{span_id}-{flags}"
def parse_traceparent(value: str) -> tuple[str, str]:
parts = value.split("-")
if len(parts) != 4 or parts[0] != "00" or len(parts[1]) != 32 or len(parts[2]) != 16:
raise ValueError(f"invalid traceparent: {value!r}")
return parts[1], parts[2]
@dataclass
class Span:
trace_id: str
span_id: str
parent_span_id: Optional[str]
name: str
start_ns: int
end_ns: Optional[int] = None
status: str = "UNSET"
attrs: dict = field(default_factory=dict)
def close(self, status: str = "OK") -> None:
self.end_ns = time.time_ns()
self.status = status
class JsonlExporter:
def __init__(self, path: str) -> None:
self.path = path
open(self.path, "w", encoding="utf-8").close()
def emit(self, span: Span) -> None:
with open(self.path, "a", encoding="utf-8") as fh:
fh.write(json.dumps(asdict(span)) + "\n")
class Tracer:
def __init__(self, exporter: JsonlExporter, run_id: str) -> None:
self.exporter = exporter
self.run_id = run_id
self.trace_id = _hex(16)
self._stack: list[Span] = []
def start(self, name: str, attrs: Optional[dict] = None) -> Span:
parent = self._stack[-1].span_id if self._stack else None
span = Span(
trace_id=self.trace_id,
span_id=_hex(8),
parent_span_id=parent,
name=name,
start_ns=time.time_ns(),
attrs={"run_id": self.run_id, **(attrs or {})},
)
self._stack.append(span)
return span
def end(self, span: Span, status: str = "OK") -> None:
span.close(status)
if self._stack and self._stack[-1].span_id == span.span_id:
self._stack.pop()
self.exporter.emit(span)
def run_tool(tracer: Tracer, argv: list[str], timeout_s: float) -> dict:
parent = tracer.start("tool:" + os.path.basename(argv[1] if argv[0] == sys.executable else argv[0]))
env = os.environ.copy()
env["TRACEPARENT"] = format_traceparent(parent.trace_id, parent.span_id)
env["RUN_ID"] = tracer.run_id
env["SPAN_OUT"] = tracer.exporter.path
try:
proc = subprocess.run(argv, env=env, capture_output=True, text=True, timeout=timeout_s)
except subprocess.TimeoutExpired:
tracer.end(parent, "ERROR")
return {"ok": False, "error": "timeout"}
tracer.end(parent, "OK" if proc.returncode == 0 else "ERROR")
return {"ok": proc.returncode == 0, "stdout": proc.stdout, "stderr": proc.stderr}
The child process is deliberately tiny. It either joins the parent trace or it lies.
# tool_worker.py — example child; pass --orphan to mint a new root
import json, os, sys, time, secrets
def main() -> None:
out = os.environ["SPAN_OUT"]
run_id = os.environ.get("RUN_ID", "")
header = os.environ.get("TRACEPARENT", "")
orphan = "--orphan" in sys.argv
if orphan or not header:
trace_id, parent = secrets.token_hex(16), None
else:
parts = header.split("-")
trace_id, parent = parts[1], parts[2]
start = time.time_ns()
time.sleep(0.05)
record = {
"trace_id": trace_id,
"span_id": secrets.token_hex(8),
"parent_span_id": parent,
"name": "tool_body:checkout_repo",
"start_ns": start,
"end_ns": time.time_ns(),
"status": "OK",
"attrs": {"run_id": run_id},
}
with open(out, "a", encoding="utf-8") as fh:
fh.write(json.dumps(record) + "\n")
print(json.dumps({"files": 12}))
if __name__ == "__main__":
main()
Wire them with a one-shot command. The first invocation should pass. The second should fail the forest check on purpose.
python - <<'PY'
import sys
from agent_trace_contract import JsonlExporter, Tracer, run_tool, load_and_assert
exp = JsonlExporter("/tmp/run.jsonl")
tr = Tracer(exp, run_id="run-demo-1")
root = tr.start("run")
run_tool(tr, [sys.executable, "tool_worker.py"], timeout_s=2.0)
tr.end(root)
print(load_and_assert("/tmp/run.jsonl", "run-demo-1"))
PY
Assert one tree, then diff two trees
The checker is the part that belongs in CI. It does not parse stdout. It does not trust timestamps to rebuild order. It counts roots, distinct trace_ids, parents that point nowhere, and children that start after the parent end_ns.
# same module — example assertion
def load_and_assert(path: str, run_id: str) -> dict:
with open(path, encoding="utf-8") as fh:
spans = [json.loads(line) for line in fh if line.strip()]
mine = [s for s in spans if s.get("attrs", {}).get("run_id") == run_id]
if not mine:
raise AssertionError("no spans for run_id")
ids = {s["span_id"] for s in mine}
roots = [s for s in mine if not s.get("parent_span_id")]
missing = [s for s in mine if s.get("parent_span_id") and s["parent_span_id"] not in ids]
by_id = {s["span_id"]: s for s in mine}
late = []
for s in mine:
parent = by_id.get(s.get("parent_span_id") or "")
if parent and parent.get("end_ns") and s["start_ns"] > parent["end_ns"]:
late.append(s["span_id"])
report = {
"roots": len(roots),
"trace_ids": len({s["trace_id"] for s in mine}),
"missing_parents": len(missing),
"late_children": len(late),
"span_count": len(mine),
}
if report["trace_ids"] != 1 or report["roots"] != 1 or missing or late:
raise AssertionError(f"split forest: {report}")
return report
def graph_key(spans: list[dict], run_id: str) -> list[tuple]:
mine = [s for s in spans if s.get("attrs", {}).get("run_id") == run_id]
by_id = {s["span_id"]: s for s in mine}
rows = []
for s in mine:
parent = by_id.get(s.get("parent_span_id") or "")
rows.append((s["name"], parent["name"] if parent else None, s["status"]))
return sorted(rows)
A reusable debug loop follows from those two functions. Export run A. Assert a single root. Change one thing — drop TRACEPARENT, add --orphan, or shrink the tool timeout. Export run B. Diff graph_key(A) against graph_key(B). If the graphs diverge in roots or parent names, stop. Do not open the prompt. Do not grep stdout for “success.” The forest already named the fault.
That loop is cheap enough to run on every fixture. Agent volume is not web traffic. Recording every tool span is the conservative default until a span name explodes into unbounded cardinality, which is a different contract and a different failure.
When the question is whether a split forest came from a missing injector or from a flaky model step, you need a second execution environment that does not change the tool graph. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode here only as that second environment: its free model access and free server option are enough to rerun the same loop and compare two JSONL exports, without turning the debug session into an infra ticket. The checker does not depend on that environment. If you already have a scratch box, keep using it.
What this does not cover
The contract is wrong for ordinary HTTP services. Those systems need sampling, tail-based decisions, and the right to mint extra roots at trust boundaries. Copying “one root or fail” onto a public API will page you for client bugs you do not own.
It is also wrong when you cannot wrap the tool. A third-party SaaS that never returns traceparent, and never accepts one, will always look like a stump unless you create a local client span and treat the remote call as an event. Do not pretend the remote side joined your tree.
JSONL is not a collector. It will not backpressure, deduplicate, or survive a crashed runner. Use it to prove the injector. Then export the same fields through OTLP if you already have a backend. Clock skew still exists; this checker uses parent ids and end_ns on spans you control, not wall clocks from two hosts.
Skip the approach if the job is a batch of independent items with no coordinator. Those runs are supposed to be many roots. Inventing a fake parent so the checker stays green is worse than a forest you understand.
If the forest splits, the run is not green. Repair the injector before you tune the prompt. The model cannot cite a source the tracer never joined.
Top comments (0)