An assistant transcript is a press release. A child span is a receipt. If the model says it ran the test suite and the trace has no child span for that tool under the generation that made the claim, the run is not done. The claim is still open, the same way a merge request stays open when CI never started.
Teams keep scoring agent runs from the last paragraph of the chat. That paragraph is generated text. It can describe a successful pytest invocation, a patched module, and a green exit code while the tool layer never executed. Walk the span tree instead of grading the prose, and the mismatch becomes mechanical.
Treat the generation span as a function call. Tool spans are the callees. A named tool inside that generation is an unresolved symbol until a child with a matching parent_id and tool name exists, reaches a terminal status, and, when the protocol provides it, carries the same tool_call_id. This is not line coverage of your application; it is claim coverage of the agent’s own statements.
The rest of this piece is a local checker and a fixture. The checker reads JSON, builds a parent map, extracts claims from generation spans, and prints every claim that never closed. It does not need a vendor UI, and it does not need you to trust the model’s summary.
A usable span is boring on purpose. Give it an id, an optional parent_id, a name, start and end timestamps in milliseconds, a status, and a small attribute bag. Put the assistant’s claimed tools on the generation span as structured objects, not as a sentence to be regexed later. If your exporter already follows OpenTelemetry-style GenAI fields, map those names in; the invariant does not care which SDK wrote the file.
#!/usr/bin/env python3
"""claim_span_coverage.py
Close an agent claim only when a child tool span exists.
A claim is a (generation_span_id, tool_name, tool_call_id|None) tuple
taken from the generation span's `claimed_tools` attribute.
A child closes the claim when:
- child.parent_id == generation_span_id
- child.name == tool_name
- child.status in {"ok", "error"} # error still counts as observed
- if tool_call_id is present on both, they match
This file is meant to be run locally against the fixture below.
It performs no network I/O.
"""
from __future__ import annotations
import json
import sys
from collections import defaultdict
from typing import Any
TERMINAL = {"ok", "error"}
def load_trace(path: str) -> dict[str, Any]:
with open(path, encoding="utf-8") as handle:
payload = json.load(handle)
if not isinstance(payload.get("spans"), list):
raise ValueError("trace must contain a spans array")
return payload
def index_spans(spans: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
by_id: dict[str, dict[str, Any]] = {}
for span in spans:
span_id = span.get("id")
if not span_id or span_id in by_id:
raise ValueError(f"missing or duplicate span id: {span_id!r}")
by_id[span_id] = span
return by_id
def children_of(spans: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
tree: dict[str, list[dict[str, Any]]] = defaultdict(list)
for span in spans:
parent = span.get("parent_id")
if parent:
tree[parent].append(span)
return tree
def claims_from(span: dict[str, Any]) -> list[dict[str, Any]]:
raw = span.get("attributes", {}).get("claimed_tools", [])
claims = []
for item in raw:
if isinstance(item, str):
claims.append({"name": item, "tool_call_id": None})
elif isinstance(item, dict) and item.get("name"):
claims.append(
{
"name": item["name"],
"tool_call_id": item.get("tool_call_id"),
}
)
return claims
def coverage_report(trace: dict[str, Any]) -> dict[str, Any]:
spans = trace["spans"]
by_id = index_spans(spans)
tree = children_of(spans)
open_claims = []
closed_claims = []
orphans = []
for span in spans:
parent_id = span.get("parent_id")
if parent_id and parent_id not in by_id:
orphans.append(span["id"])
for span in spans:
for claim in claims_from(span):
matched = None
for child in tree.get(span["id"], []):
if child.get("name") != claim["name"]:
continue
if child.get("status") not in TERMINAL:
continue
child_cid = child.get("attributes", {}).get("tool_call_id")
if (
claim["tool_call_id"]
and child_cid
and claim["tool_call_id"] != child_cid
):
continue
matched = child
break
record = {
"generation_id": span["id"],
"tool": claim["name"],
"tool_call_id": claim["tool_call_id"],
}
if matched is None:
open_claims.append(record)
else:
record["child_id"] = matched["id"]
record["child_status"] = matched.get("status")
closed_claims.append(record)
return {
"closed": closed_claims,
"open": open_claims,
"orphans": orphans,
"ok": not open_claims and not orphans,
}
def main() -> int:
if len(sys.argv) != 2:
print(
"usage: python3 claim_span_coverage.py fixture.json",
file=sys.stderr,
)
return 2
report = coverage_report(load_trace(sys.argv[1]))
json.dump(report, sys.stdout, indent=2)
sys.stdout.write("\n")
return 0 if report["ok"] else 1
if __name__ == "__main__":
raise SystemExit(main())
Save a fixture that contains one honest closed claim and one dangling generation. The dangling row is the interesting case: the model named run_tests, and nothing under that parent ever ran.
{
"trace_id": "run-2026-09-21-claim-coverage",
"spans": [
{
"id": "gen-1",
"parent_id": null,
"name": "generation",
"start_ms": 0,
"end_ms": 40,
"status": "ok",
"attributes": {
"claimed_tools": [
{"name": "read_file", "tool_call_id": "c1"}
]
}
},
{
"id": "tool-1",
"parent_id": "gen-1",
"name": "read_file",
"start_ms": 12,
"end_ms": 18,
"status": "ok",
"attributes": {"tool_call_id": "c1", "path": "src/app.py"}
},
{
"id": "gen-2",
"parent_id": null,
"name": "generation",
"start_ms": 41,
"end_ms": 90,
"status": "ok",
"attributes": {
"claimed_tools": [
{"name": "run_tests", "tool_call_id": "c2"}
]
}
}
]
}
Wire the two files and run the checker as a command, not as a dashboard screenshot.
python3 claim_span_coverage.py fixture.json
echo "exit=$?"
Expected stdout on this fixture is a JSON object with read_file under closed, run_tests under open, an empty orphans list, and ok set to false. The process exit code is 1. That exit code is the gate: a CI step can refuse to label the agent run green when a claim never grew a child.
Error status still closes the claim. An error child means the tool was observed, not that the user task succeeded. If you need task success, inspect the tool payload in a later gate. Mixing the two checks is how teams treat a missing pytest span and a failing pytest span as the same incident. They are not.
Orphan spans are the other refusal. A tool row whose parent_id points at a span that is not in the file is a side effect that escaped the DAG. Subprocess helpers do this when the agent forks a script and the script logs to a different tracer. The work may have happened on disk. It did not happen inside this trace, so this trace cannot close the claim.
Clock order is a weak substitute for parent_id. Two spans that overlap in wall time might be concurrent callees, or they might be a generation talking about work that already finished under a previous parent. The checker ignores start_ms except as documentation. If you later add a causality assertion, require child.start_ms >= parent.start_ms and child.end_ms <= parent.end_ms only after you know the exporter uses inclusive envelopes. Many runtimes close the generation before the last tool callback is flushed. That pattern produces false open claims unless you buffer tool exits before you emit the parent end event.
The export wrapper is the other half of the workflow. Record the tool name and id on the generation span as soon as the model returns a structured tool-call object, start the child in that same context, and flush children before ending the parent. The following is a labeled sketch, not a drop-in SDK.
# Proposal: attach claims at dispatch time so the checker is not parsing prose.
# Adapt field names to your runtime. This block is unexecuted.
def on_model_tool_calls(generation, tool_calls):
generation.attributes.setdefault("claimed_tools", [])
for call in tool_calls:
generation.attributes["claimed_tools"].append(
{"name": call.name, "tool_call_id": call.id}
)
def run_tool(generation, call, dispatch):
child = generation.start_child(name=call.name)
child.attributes["tool_call_id"] = call.id
try:
payload = dispatch(call)
child.status = "ok"
return payload
except Exception:
child.status = "error"
raise
finally:
child.end()
A conservative parser over free text is a last resort. Prose such as “you could run pytest” will over-claim. Prefer the model API’s tool-call objects when they exist. Structured objects plus child spans are the smallest honest picture of an agent run.
Token counts, file diffs, and summary embeddings do not substitute for this gate. A cheap completion can emit “tests passed” in a handful of tokens. Two summaries can be close in embedding space while only one run actually executed run_tests. Presence of a terminal child span is a cheaper and stricter signal than either of those proxies.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The coverage file is host-agnostic. If you need a machine to wrap an agent, export this JSON, and iterate without first provisioning your own box, MonkeyCode is an open-source project that currently offers free model access and a free server option. Use that path only as a collector. Keep the checker and the fixture in your own repo so the invariant survives any host.
The approach has sharp edges. Sampling that drops tool spans will invent open claims. Truncated exports that omit children will do the same. Encrypted or redacted tool names will fail the name match even when the child exists. Multi-agent runs that do not share a trace_id will look like two empty forests. None of those failures mean the agent was idle. They mean the evidence was incomplete, and incomplete evidence cannot close a claim.
Do not use this gate as a proof of semantic correctness. A write_file child with status ok does not prove the bytes on disk are the bytes you wanted. Do not use it as an authorization log, a billing meter, or a safety interlock for anything that moves money, rotates credentials, or actuates hardware. Those domains need signed tool results and an allowlist, not a parent pointer.
Skip the whole pattern if your runtime cannot emit parent links at all. A flat log of tool names without ids is not a DAG, and forcing it into this checker will only print noise. Fix context propagation first. Then turn the missing-child case into a failing exit code.
The reusable debug loop is short. Export the trace. Run the checker. For every open claim, either emit the missing child, delete the claim from the generation, or fail the run. For every orphan, repair propagation before you debate the model. Keep the failing fixture next to the prompt that produced it. The next regression will be another missing child, not a new narrative in the chat window.
Top comments (0)