Treat Span Cardinality as a Hard Error

Agent traces usually fail as landfills, not as black boxes. The run emitted spans. Too many of them. Each tool call copied the prompt, the raw JSON, and a pile of metadata that no later diff will read. The debug loop then drowns in its own evidence.

Cardinality is the failure mode that looks like success. Status is OK. Parent ids exist. Timestamps are monotonic enough to sort. The trace is still useless, because the tree grew faster than anyone can compare. A free endpoint does not change that arithmetic. It only makes it cheaper to grow the landfill.

Think of a suitcase and a stack of store receipts. The itinerary tells you which cities you visited and in which order. The receipts prove a purchase happened, but stuffing every receipt into the bag does not make the itinerary clearer. Agent traces need the itinerary. They rarely need the full receipt. Hash the receipt. Keep the route.

The practical rule is simple. Treat span cardinality as a hard error, the same way you treat an unclosed file handle. If a run exceeds a byte budget, a child-fan-out cap, or a payload-shape contract, fail the run before you archive it. A huge green trace is not a green run. It is an unbounded write.

This is adjacent to logging culture, not a replacement for it. Logs can be verbose on a laptop. A shared debug loop cannot. Once two runs must be compared, extra attributes stop being context and start being noise. The comparison key has to be small, stable, and reconstructible from the span tree rather than from prose buried in a blob.

A usable span for a tool call is narrow. Name, parent id, status, duration, a canonical payload hash, and a schema fingerprint of sorted keys. That is enough to ask whether this run called the same tools, with the same shapes, in the same tree. Full prompts belong behind a redact-then-store gate, not in the default export.

Suppose eight child tools each carry a 4KB body. That is 32KB before attributes. Ten turns become 320KB. Retries double it. The model then echoes prior tool results into the next prompt, and the next span copies that echo again. Nothing in that sequence looks like a 429. The server still fills. Diffs still lie, because two runs that did the same work hash differently once a timestamp or a request id leaks into the body.

Canonicalization is the fix, not sampling. Sampling drops the failure path you came for. Canonical JSON with sorted keys, then SHA-256, keeps the comparison and throws away the noise. If a field is allowed to change between runs, strip it before the hash. If it is not allowed to change, leave it in and let the hash break. That break is the signal.

The harness below is a local, unexecuted example. It does not report production timings or vendor quotas. Run it with Python 3.11+. It encodes three gates: a total attribute-byte budget, a maximum number of children per parent, and a requirement that every tool.call span has a matching tool.result child. Fail any gate and the trace is not archived.

# span_budget.py
from __future__ import annotations

import hashlib
import json
import unittest
from dataclasses import dataclass, field
from typing import Any, Iterable

DROP_KEYS = frozenset({"ts", "timestamp", "request_id", "trace_id", "span_id"})
MAX_ATTR_BYTES = 32_768
MAX_CHILDREN = 8


def canonical(value: Any) -> Any:
    if isinstance(value, dict):
        return {
            str(k): canonical(v)
            for k, v in sorted(value.items(), key=lambda kv: str(kv[0]))
            if str(k) not in DROP_KEYS
        }
    if isinstance(value, list):
        return [canonical(v) for v in value]
    return value


def payload_hash(payload: Any) -> str:
    blob = json.dumps(canonical(payload), separators=(",", ":"), ensure_ascii=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()


def schema_fingerprint(payload: Any) -> str:
    if not isinstance(payload, dict):
        return type(payload).__name__
    return ",".join(sorted(str(k) for k in payload.keys()))


@dataclass
class Span:
    span_id: str
    parent_id: str | None
    name: str
    kind: str
    status: str
    payload: dict[str, Any] = field(default_factory=dict)

    def compact(self) -> dict[str, Any]:
        return {
            "span_id": self.span_id,
            "parent_id": self.parent_id,
            "name": self.name,
            "kind": self.kind,
            "status": self.status,
            "schema": schema_fingerprint(self.payload),
            "payload_sha256": payload_hash(self.payload),
        }


class SpanBudgetError(RuntimeError):
    pass


def attr_bytes(spans: Iterable[Span]) -> int:
    return sum(len(json.dumps(s.compact(), separators=(",", ":")).encode()) for s in spans)


def enforce_budget(spans: list[Span]) -> list[dict[str, Any]]:
    children: dict[str, int] = {}
    calls = {s.span_id: s for s in spans if s.kind == "tool.call"}
    results_by_parent = {s.parent_id for s in spans if s.kind == "tool.result"}

    for span in spans:
        if span.parent_id:
            children[span.parent_id] = children.get(span.parent_id, 0) + 1
            if children[span.parent_id] > MAX_CHILDREN:
                raise SpanBudgetError(
                    f"fan-out {children[span.parent_id]} at parent {span.parent_id}"
                )

    missing = [sid for sid in calls if sid not in results_by_parent]
    if missing:
        raise SpanBudgetError(f"tool.call missing tool.result: {missing}")

    size = attr_bytes(spans)
    if size > MAX_ATTR_BYTES:
        raise SpanBudgetError(f"compact trace is {size} bytes; cap is {MAX_ATTR_BYTES}")

    return [s.compact() for s in spans]


def tree_hash(compact_spans: list[dict[str, Any]]) -> str:
    ordered = sorted(compact_spans, key=lambda s: (s["parent_id"] or "", s["name"], s["span_id"]))
    return payload_hash(ordered)


class SpanBudgetTests(unittest.TestCase):
    def test_hash_ignores_volatile_fields(self):
        a = {"query": "orders", "ts": "2026-09-09T00:00:00Z", "request_id": "r1"}
        b = {"request_id": "r2", "query": "orders", "ts": "2026-09-09T00:00:01Z"}
        self.assertEqual(payload_hash(a), payload_hash(b))

    def test_orphan_tool_call_fails(self):
        spans = [
            Span("root", None, "agent.run", "run", "ok"),
            Span("c1", "root", "sql.query", "tool.call", "ok", {"sql": "select 1"}),
        ]
        with self.assertRaises(SpanBudgetError):
            enforce_budget(spans)

    def test_matching_result_compacts(self):
        spans = [
            Span("root", None, "agent.run", "run", "ok"),
            Span("c1", "root", "sql.query", "tool.call", "ok", {"sql": "select 1"}),
            Span("r1", "c1", "sql.query", "tool.result", "ok", {"rows": 1}),
        ]
        compact = enforce_budget(spans)
        self.assertEqual(len(compact), 3)
        self.assertEqual(len(tree_hash(compact)), 64)


if __name__ == "__main__":
    unittest.main()

Run it as a gate, not as a viewer.

python span_budget.py

Wire the same function at the end of an agent turn. Compact first. Hash the tree. Persist only the compact list. If you need the raw payload later, write it to a side channel that is off by default and keyed by payload_sha256. The debug loop then diffs tree hashes. A changed hash with an unchanged schema fingerprint points at data. A changed fingerprint points at contract drift. A budget error points at the agent growing a bush instead of a path.

That loop is where a free model endpoint earns its keep. Re-running the agent to see whether the tree hash is stable should not require a paid inference budget on day one. A free server is relevant for the same reason: the compact export is small enough to keep next to the process, and the raw landfill is not. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit this workflow only as an iteration surface. They do not remove the budget. They do not define span semantics. If the compact trace cannot be explained without the product name, the trace is still too large.

Do not confuse this with sampling ratios copied from request-centric APM. An agent run is one business action with a fan-out of tools. Dropping 90% of spans drops the tool that mattered. A hard cap on children is stricter and more honest. If the agent needed nine tools, the budget should fail in CI, not silently omit the ninth in production.

Limitations are real. Hashing destroys ad-hoc grep. If a legal process requires the literal prompt, this default export will not satisfy it; keep a locked raw channel with a retention clock, and do not pretend the compact trace is the record. Schema fingerprints also lie when two payloads share keys but not types. Extend schema_fingerprint to walk types if your tools return unions. The byte cap is a constant, not a discovered optimum. Raise it only when a failing test names the span that would not fit.

Skip this approach if the agent is a single function call with no tools. Skip it if every span is already under a few hundred bytes and two runs already diff cleanly. Skip it if you cannot strip volatile keys, because the hash will flap and the team will disable the gate. A disabled cardinality gate is worse than no gate. It trains people to ignore the red bar.

The core conclusion does not depend on a vendor. Archive the itinerary. Hash the receipts. Fail the run when the suitcase will not close. If a free server is already idle, run python span_budget.py before adding another raw log line to a tool span.