Your Agent Trace Is Too Big: Budget Bytes per Span

The scarce resource in agent observability is not spans. It is bytes.

A single read_file against a lockfile, one HTTP call to a chatty API, or one git diff over a generated directory can outweigh the other forty spans of a run combined. Picture a run whose span metadata totals 30 KB while one tool result carries 1.2 MB of JSON — those figures are illustrative, the shape is not. Store that verbatim and three things break at once: the trace viewer crawls, the disk on whatever box runs the agent fills up, and the signal you actually needed (which arguments went in, what shape came out) drowns in payload.

So budget bytes per span, and make the budget a test. This is the loop I keep coming back to when a trace gets fat.

Decide what earns verbatim storage

Two questions per payload, answered before the record is written rather than after: will you need exact bytes to reproduce the bug, and how large is it? The second question is cheap. The first one requires you to know what kind of tool you are instrumenting.

Payload kind	Keep inline	Spill to blob store	Why
Small tool arg / result (< 2 KB)	Full	No	Cheap, and exact bytes matter most here
File read	Head + tail + sha256	Yes, if over budget	You rarely debug the 900th line of a lockfile
HTTP response body	Head + tail + digest + status	Yes, if over budget	Status and headers explain more than the body
Model prompt / completion	Full, redacted upstream	No	Mid-payload truncation destroys reasoning context

Note the digest column. Every record carries a sha256 of the full payload even when the payload itself is truncated. That buys two things: deduplication, since identical file reads resolve to one blob, and a cheap equality check when you suspect two runs saw different bytes.

Capture inside a head/tail budget

import base64, hashlib, json, os, pathlib

STORE = pathlib.Path(os.environ.get("TRACE_BLOB_DIR", "./trace-blobs"))


def capture(raw: bytes, budget: int = 2048) -> dict:
    """Shrink one tool payload to a fixed-size record. Never lossless."""
    half = max(budget // 2, 0)
    digest = hashlib.sha256(raw).hexdigest()
    rec = {
        "sha256": digest,
        "bytes": len(raw),
        "truncated": len(raw) > budget,
        "head_b64": base64.b64encode(raw[:half]).decode(),
        "tail_b64": base64.b64encode(raw[-half:] if half else b"").decode(),
    }
    if rec["truncated"]:
        put_blob(digest, raw)
    return rec


def put_blob(digest: str, raw: bytes) -> None:
    STORE.mkdir(parents=True, exist_ok=True)
    path = STORE / digest
    if path.exists():
        return
    tmp = path.with_suffix(".tmp")
    tmp.write_bytes(raw)
    tmp.replace(path)


def rehydrate(rec: dict) -> bytes:
    path = STORE / rec["sha256"]
    if path.exists():
        return path.read_bytes()
    head = base64.b64decode(rec["head_b64"])
    tail = base64.b64decode(rec["tail_b64"])
    gap = rec["bytes"] - len(head) - len(tail)
    return head + f"<{gap} bytes omitted>".encode() + tail

Head answers "what is this object". Tail answers "how did it end", which is where stack traces, exit codes, and truncation markers usually live. The middle is what you sacrifice, and rehydrate states the size of that sacrifice instead of hiding it.

Two properties worth testing immediately. The blob store is content-addressed, so re-reading the same file ten times costs one file on disk. And the roundtrip must be exact when the payload fits:

def test_rehydrate_roundtrip():
    raw = b"x" * 10_000 + b"tail"
    rec = capture(raw, budget=512)
    assert rec["truncated"] is True
    assert rec["sha256"] == hashlib.sha256(raw).hexdigest()
    assert rehydrate(rec) == raw

When the blob is present, rehydrate returns the original bytes. When it has been garbage-collected, it returns a lossy but honest reconstruction. Both are debuggable; a 1.2 MB span in a web UI is not.

Make the budget a test, not a habit

A budget written in a document decays. A budget that fails a build survives. The test below measures the serialized size of each span record and names the largest offender.

import json, pathlib

LIMIT = 20_000  # bytes per span record; illustrative, tune per project


def span_bytes(span: dict) -> int:
    return len(json.dumps(span, separators=(",", ":"), sort_keys=True).encode())


def test_trace_byte_budget(trace: dict):
    sizes = sorted(
        ((span_bytes(s), s["name"]) for s in trace["spans"]), reverse=True
    )
    total = sum(n for n, _ in sizes)
    assert sizes[0][0] <= LIMIT, f"largest span {sizes[0][1]} is {sizes[0][0]} bytes"
    assert total <= LIMIT * 8, f"run total {total} bytes across {len(sizes)} spans"
    print(sizes[:3])

Run it the same way you run any other regression check:

python -m pytest tests/test_trace_bytes.py -q
du -sh trace-blobs | sort -h | tail -3

The first command tells you which tool is writing fat spans. The second tells you whether the blob store is growing faster than your retention policy can delete. When the assert fires, you get a name and a number, which is a decision: widen the budget for that tool, or fix the tool to return less.

Where this helps, and where it hurts

Truncation is lossy by design, and the loss lands in the middle of the payload — sometimes exactly where the malformed record sits. Digests prove that two runs differed; they never explain why. And a blob store without garbage collection is just a slower disk leak, so retention has to key off span status: keep blobs referenced by failed spans, drop the rest on a schedule you can defend.

Do not apply this to traces that are themselves a compliance or audit record. Those need exact bytes, immutable storage, and encryption at rest; a truncated record with a digest is not evidence. Likewise, if your tracing backend already offloads large payloads and gives you a stable handle back, you are duplicating work — use its offload instead. And redaction stays upstream of everything here: a blob is still storage, so secrets must be stripped before capture is called, not after.

One practical environment where the budget stops being theoretical: running the agent on a free server option, where the disk is yours to manage and nobody else will notice it filling. That is the case where per-span byte accounting pays for itself in the first week, and where a cheap summarization pass inside the debug loop is worth wiring up rather than skipping. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are what make that loop reachable without a budget conversation first; the byte ledger above is the part you keep either way.

Start by measuring one real trace you already have. Sum the serialized size of its spans, sort descending, and look at the top three. If one payload owns most of the bytes, you have found where to spend your instrumentation effort — and a budget test that will keep it honest.