Sample Runs, Not Spans: Shrink Agent Traces Without Losing Failures

At 09:12 the on-call page fired. A checkout agent returned HTTP 500 for forty minutes.

We opened the trace store and found two weeks of spans. The failing run was not in it.

Our sampler had dropped the interesting spans. It kept the boring ones. That is the classic sampling bug, and it is a retention problem, not a tracing problem.

Why span-level sampling fails

Most teams sample at the span level. A coin flip decides each span independently.

That breaks causality. A tool call survives; its parent LLM call does not. You cannot replay a run with holes in it.

Worse, failures are rare and correlated. A 5% span sample catches roughly 5% of a broken run, and the interesting leaf spans are exactly the ones your status filter already flags.

So you pay 5% of the storage and lose 95% of the debuggability. The incident you care about is the one you stored least.

Change the unit of sampling

Sample runs, not spans. A run is either fully recorded or fully summarized. No partial runs, no orphan spans.

Three storage modes are enough:

1. Full — every span, every payload, exact bytes, sequence numbers intact.
2. Summary — one lossy record per span: id, kind, status, duration, token counts, byte counts. No payloads.
3. Anomaly-pinned — the same as full, but exempt from sampling forever, because the run already proved it can break.

Then add one rule that overrides chance. Any run that touches a known anomaly keeps full detail, regardless of the sample rate.

A decision table you can argue about

Span or run class	Decision	Retention	Why
run has any error span	full	30 days	root-cause evidence
any span has status = retry	full	30 days	retry storms hide real bugs
span has a truncation flag	full	14 days	the payload was already cut
span exceeds the latency p99 budget	full	14 days	regression evidence
clean run, hash-selected	full	7 days	baseline for diffing
clean run, not selected	summary	90 days	trend lines and cost accounting
health-check span	neither	0	pure noise

Two things stand out. Clean runs are kept longest in summary form, because they are cheap and they answer trend questions. Broken runs are kept shortest in full form, because the fix arrives fast.

The sampler, in about forty lines

This is a proposal, not a benchmarked result in your environment. Run it against archived traces before you put it in front of production traffic.

# sampler.py — proposal; measure it on your own archive first
import hashlib
import os

SALT = os.environ.get("TRACE_SAMPLE_SALT", "change-me")
ANOMALY_STATUSES = {"error", "retry", "truncated", "budget_exceeded"}

def is_anomalous_span(span: dict) -> bool:
    if span.get("status") in ANOMALY_STATUSES:
        return True
    return bool(span.get("truncated"))

def run_is_pinned(run: dict) -> bool:
    return run.get("anomaly_count", 0) > 0

def selected_for_full(run_id: str, success_sample_rate: float = 0.05) -> bool:
    """Deterministic, stable per run_id. Sample whole runs, never spans."""
    digest = hashlib.sha256((SALT + run_id).encode()).hexdigest()
    bucket = int(digest[:8], 16) % 10_000
    return (bucket / 10_000.0) < success_sample_rate

def store_mode(run: dict) -> str:
    if run_is_pinned(run):
        return "full"
    if selected_for_full(run["run_id"]):
        return "full"
    return "summary"

Two design choices matter here.

• Hashing on run_id, not a random draw. The same run gets the same verdict in every process, on every retry, and after a restart. Random draws make your retention policy non-reproducible.
• A SALT from the environment. Without it, anyone can compute which runs you kept. With it, selection is stable but not guessable.

The one test that matters

A single invariant protects the whole policy: an anomalous run is never summarized away.

# test_retention.py — proposal; wire into CI
def test_anomalous_runs_are_never_summarized():
    for status in ("error", "retry", "truncated", "budget_exceeded"):
        run = {"run_id": "r-1", "anomaly_count": 1, "status": status}
        assert store_mode(run) == "full"

def test_selection_is_stable_across_calls():
    a = [selected_for_full("r-42") for _ in range(100)]
    assert len(set(a)) == 1

def test_selection_is_not_all_or_nothing():
    kept = sum(selected_for_full(f"r-{i}") for i in range(10_000))
    assert 300 < kept < 700  # ~5% with sane variance

The third test is the subtle one. A salt or modulus bug can silently push your sample rate to 0% or 100%, and nobody notices until an incident review.

Where the pipeline runs

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The collector and summarizer are small, periodic, and CPU-bound. That is a good fit for a free server option, because you are not paying for idle capacity between jobs.

MonkeyCode is an open-source project with free model access, and the operator lists a 10M-token free allowance alongside the free server option. Treat both as claims to verify against the current terms before you depend on them.

The model access has a genuine role here. Full payloads are expensive to keep; one-line run digests are not. So generate a digest for every run, not just the sampled ones, and store it permanently next to the summary records. Something like:

Input: 40 summary records for run r-8891
Output: one sentence — what the run attempted, what changed, what failed

That digest is what an on-call engineer reads first at 09:12. It costs a few hundred tokens per run, which is why a free allowance changes what you can afford to keep.

Two cautions. Never put payload contents you cannot share into a third-party model call, and never treat a free tier as a system of record. Check the current limits and retention terms yourself.

Measure before you trust the policy

Do not adopt this on vibes. Compute the ratio on one day of archived traces.

# total bytes and runs
wc -c < traces.jsonl
wc -l < traces.jsonl

# bytes owned by anomalous runs (approximation: runs with an error line)
jq -c 'select(.status == "error") | .run_id' traces.jsonl \
  | sort -u > pinned.txt
rg -F -f pinned.txt traces.jsonl | wc -c

Then report two numbers: pinned_runs / total_runs and kept_bytes / total_bytes.

Acceptance criteria I would use, and you should replace with your own:

• pinned_runs / total_runs under 2% in steady state.
• kept_bytes / total_bytes under 15% after summarization.
• Zero runs missing a stored digest.

If pinned_runs is much higher than your error rate, your anomaly rules are too broad. Fix the rules before you lower the storage bill.

Limitations

• Summaries cannot answer unanticipated questions. If you did not record a field, you cannot query it later. Add fields to the summary schema only when you can justify the bytes.
• Sampling breaks exact replay for unselected runs. Keep the sample rate high enough that a baseline exists for diffing.
• Hash sampling can correlate with your id scheme. If run_id starts with a tenant prefix, some tenants get sampled more than others. Salt alone does not fix that; hash the full id or re-bucket.
• Latency budgets need a warm baseline. On a sleeping free server, the first request after idle inflates duration. Exclude cold-start spans or your p99 rule pins everything.
• Not for audit or legal hold. A sampling policy is an engineering choice, not a compliance record.

Who should skip this

If you run fewer than a few thousand agent runs per day, storage is cheap and full-fidelity tracing is simpler. Skip the sampler.

If you operate under retention obligations, keep everything and solve cost elsewhere. Skip the sampler.

If your failures are exactly the class you would sample away, for example silent wrong answers with clean statuses, pin on an outcome signal instead of a status code.

Start with one day of traffic

Sample runs, not spans. Pin anomalies unconditionally. Store a model-written digest for every run.

Take one archived day of traces. Run the sampler over it. Report the two ratios and the three invariants.

If the numbers hold, you have shrunk the store without losing the one run you will need at 09:12.