Trace Retention Triage: Keep the Agent Runs You Will Actually Debug

Storage breaks first in agent observability. Not the parser, not the dashboard — the disk. A single agent run can emit thousands of spans, most of them near-identical re-reads of the same file, and the one run that failed in a genuinely new way is the one that got rotated out before you looked. Retention is a triage decision, and it belongs at write time.

I am not going to hang this on a headline. Trends rotate weekly; span counts do not lie. So the artifact below is a policy you can measure against your own runs, plus four checks that tell you whether the policy is behaving.

Decide in three tiers, and make the third tier tiny

The failure mode I see most often is a binary choice: keep everything, or sample 1% uniformly. Both are wrong for debugging, because the value of a run is not uniform — it is concentrated in the first sighting of a new failure signature.

Tier	Written to disk	Survives eviction	Purpose
full	entire run, all spans	yes, until byte cap	the runs you will actually replay
digest	one line: fingerprint, run id, one-sentence summary	yes, cheap	keeps dropped runs searchable
drop	nothing beyond the hot ring	no	repetition, not information

The rule that does the work is blunt: the first time a fingerprint appears, keep the whole run. Every later appearance of the same fingerprint is compressed to a digest line. A green run repeating itself is dropped outright, because a passing run that looks like the last passing run teaches you nothing about the bug.

Fingerprint the failure, not the run id

A run id identifies an event. A fingerprint identifies a problem, which is the unit you debug. Build it from an order-stable projection of the run: error class, number of spans, and the ordered list of tool calls with their statuses and exit codes.

# triage.py — Python 3.11, stdlib only
import hashlib, json
from pathlib import Path

DROP_CLASSES = {"RateLimit", "Timeout", "UpstreamUnavailable"}

def canonical(run: dict) -> bytes:
    """Order-stable view of what makes two runs the same problem."""
    sig = {
        "error": run.get("error_class", "ok"),
        "shape": len(run["spans"]),
        "tools": [
            [s["tool"], s.get("status", "ok"), s.get("exit_code")]
            for s in run["spans"] if s.get("kind") == "tool"
        ],
    }
    return json.dumps(sig, sort_keys=True, separators=(",", ":")).encode()

def fingerprint(run: dict) -> str:
    return hashlib.sha256(canonical(run)).hexdigest()[:16]

def decide(run: dict, seen: dict[str, int]) -> tuple[str, str]:
    fp = fingerprint(run)
    prior = seen.get(fp, 0)
    seen[fp] = prior + 1
    if prior == 0 and run.get("error_class") not in DROP_CLASSES:
        return fp, "full"     # first sighting: keep every span
    if run.get("status") == "ok":
        return fp, "drop"     # a green run repeating itself is noise
    return fp, "digest"

Note what the fingerprint deliberately ignores: message text, timestamps, and token counts. Those change between identical failures. Note what it also ignores, and should not forever: tool arguments. A parameter-sensitive bug will collapse into one fingerprint under this projection, which is the main reason the collision audit below is not optional.

A hot ring keeps promotion possible

Digest lines are useless if the raw body is already gone, so write every run into a bounded hot ring first. The ring is dumb FIFO by mtime, capped by count, and it is the only place a digest can be promoted from.

class Retainer:
    def __init__(self, root: Path, cap_bytes: int = 512 * 1024 * 1024, hot_runs: int = 200):
        self.root, self.cap, self.hot_runs = root, cap_bytes, hot_runs
        for sub in ("full", "hot", "digest"):
            (root / sub).mkdir(parents=True, exist_ok=True)

    def full_bytes(self) -> int:
        return sum(p.stat().st_size for p in (self.root / "full").glob("*.jsonl"))

    def admit(self, run: dict, mode: str) -> str:
        body = json.dumps(run, sort_keys=True)
        self._push_hot(run)
        if mode == "drop":
            return "drop"
        if mode == "digest":
            line = {"fp": fingerprint(run), "run_id": run["run_id"],
                    "digest": run.get("digest", "")}
            with (self.root / "digest" / "d.jsonl").open("a") as f:
                f.write(json.dumps(line) + "\n")
            return "digest"
        while self.full_bytes() + len(body) > self.cap and self._evict_oldest_full():
            pass
        (self.root / "full" / f'{run["run_id"]}.jsonl').write_text(body)
        return "full"

The two caps answer different questions. cap_bytes is how much debugging history you can afford to keep raw. hot_runs is how far back you can still rescue a run you already decided to compress. Set the second one to roughly a week of your run rate and forget it.

The debug loop is three commands

Once a fingerprint is recorded on every span and log line, the loop stops being archaeology. You grep the digest, promote the signature you care about, then read only the tool spans.

python -m triage stats                       # counts per tier, dedupe ratio, bytes in full/
python -m triage promote --fp 9c1f3a7d2b40e881
python -m triage show --fp 9c1f3a7d2b40e881 \
  | jq '.spans[] | select(.kind == "tool") | {tool, status, exit_code, args_hash}'

The stats line is the honest part of the exercise. If 400 runs produced 11 distinct fingerprints, your agent is retrying one bug and your retention policy just told you that for free. If 400 runs produced 380 fingerprints, your signature is too fine — drop exit_code or normalize argument hashes until repeats start collapsing.

Where a free model and a free server actually earn their place

The collector above is one stdlib file with no database, so the boring hosting question has a boring answer: run it on a small always-on box. The free server option is enough for this because the writer is append-only and the caps are yours to choose. The free model access covers a narrower job — writing the one-sentence digest for digest-tier runs, so a compressed run is still findable in plain language instead of only by hash.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep that model call on a short leash, because a digest step inside a retry loop is how free quotas evaporate. Cap the completion at roughly 120 output tokens, hard-stop after N digests per hour, and log a counter next to stats. The operator describes the free tier as 10 million tokens plus a free server option; I have not independently verified either the quota mechanics or how long the offer lasts, so instrument your own spend rather than trusting a number in a blog post.

Four checks before you trust the policy

1. Dedupe ratio. Feed a fixture file of 500 recorded runs. Expect stats to show a full/keep ratio well below 1; if it is above 0.5 your fingerprint is not collapsing repeats.
2. Cap discipline. Feed 10,000 synthetic runs, then du -sb store/full. It must not exceed cap_bytes plus one run, or your eviction loop is not running.
3. Promotion durability. Promote a fingerprint, feed 5,000 more runs, confirm the promoted file still exists. This catches the classic bug where promotion writes into the same eviction pool.
4. Collision audit. Collect distinct stack traces that share a fingerprint and read ten of them. Any two that a human would debug differently mean your projection is too coarse.

The reference code above is a sketch I have not benchmarked on your workload, and the numbers you get are the only numbers that matter. Run check 3 first; it is the one that silently corrupts a debugging session.

Limits, and who should not do this

The fingerprint is lossy by design, so this approach hides low-frequency variants of a common error. The hot ring is FIFO by mtime, which means a burst of noisy runs can evict a quiet but important one. And digest lines carry no ordering, so reconstruction requires the raw body while it is still hot.

Skip this if you need a complete audit trail for compliance, if you are sustaining thousands of runs per minute where per-file writes will thrash, or if your team already queries a real trace backend with full-text search over everything. In those cases retention is a solved, paid problem and a ring buffer is a step backwards.

If you do try it, start with a 512 MB cap and one digest per unique failing fingerprint. Watch the ratio for a week, then decide whether the free server and the free model were doing real work or just sitting there.