Failure-Biased Trace Sampling: Keep the Incidents, Drop the Noise

A collector has been running for nine days. Disk is at 91%. Someone adds a cleanup cron job at 2 a.m. By morning, the failing run you needed is gone. The passing runs survived.

That is not a storage problem. It is a policy problem. Random sampling treats the incident and the boring run as equals.

The failure mode: traces are append-only, disks are not

Agent traces grow faster than normal logs. Each step carries a prompt, a tool payload, and a result. A 30-step run can be larger than a day of application logs.

So teams do one of two things. They keep everything and hit a wall. Or they sample uniformly and lose the failures.

Both outcomes cost you the same thing: the ability to explain a bad run after the fact.

Policy first: what must never be sampled away

Write the keep rules before you write the sampler. Four classes should always survive:

1. Any run with a terminal error status.
2. Any run that retried a tool call two or more times.
3. Any run that exceeded its token or wall-clock budget.
4. Any run flagged by a human or a downstream check.

Everything else is a candidate for sampling. That split is the whole design.

Deterministic sampling by trace ID

Random sampling per collector instance breaks down with multiple services. Two services disagree about the same trace.

Hash the trace ID instead. The decision becomes a pure function of the ID, with no shared state.

import hashlib

BUCKETS = 10_000
BASE_RATE = 0.05

def bucket(trace_id: str) -> int:
    digest = hashlib.sha256(trace_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % BUCKETS

def should_keep(trace: dict, rate: float = BASE_RATE) -> bool:
    if trace["status"] == "error":
        return True
    if trace.get("retries", 0) >= 2:
        return True
    if trace.get("tokens", 0) > trace.get("token_budget", 10**9):
        return True
    return bucket(trace["trace_id"]) < int(rate * BUCKETS)

Same trace ID, same answer, on every node. Propagate IDs with W3C Trace Context so the key is stable across services (spec).

Test the rate, not the vibe

A sampler without a distribution test will drift silently. Pin it with one assertion.

def test_rate_is_stable():
    ids = [f"trace-{i}" for i in range(20_000)]
    kept = sum(bucket(t) < 500 for t in ids)
    assert 900 < kept < 1100

Run it in CI. If someone changes the hash, this fails before your disk does.

Keep the aggregate when you drop the payload

Dropping spans should not drop the numbers. Emit a rollup record for every run, sampled or not.

def rollup(trace: dict) -> dict:
    spans = trace["spans"]
    return {
        "trace_id": trace["trace_id"],
        "status": trace["status"],
        "steps": len(spans),
        "tool_calls": sum(1 for s in spans if s["kind"] == "tool"),
        "duration_ms": trace["duration_ms"],
        "tokens": trace["tokens"],
    }

Now p95 tool latency and token spend stay queryable. You lose the arguments, not the shape.

A three-tier retention table

Decide tiers before the disk decides for you.

Tier	What stays	Retention	Typical query
Hot	full spans and payloads	72 h	debugging a live incident
Warm	rollups plus failing spans	30 d	regression hunting
Cold	counters only	12 mo	capacity planning

Promotion is one-way. Failing runs jump from hot to warm. Passing runs expire quietly.

Backpressure beats deletion

When free space drops, lower the sampling rate. Never delete the newest failures.

import shutil

def effective_rate(base: float = 0.05, path: str = "/var/lib/traces") -> float:
    gb = shutil.disk_usage(path).free / 1e9
    if gb > 50:
        return base
    if gb > 20:
        return base / 2
    if gb > 5:
        return base / 10
    return 0.0  # hard-keep classes only

Log the active rate with every write. When someone asks why a trace is missing, you have an answer instead of a shrug.

Generating the corpus on free capacity

You cannot tune a sampler without a realistic trace corpus. That is the expensive part.

This is where MonkeyCode fits into the loop. Its free model access is enough to run a batch harness and produce a few hundred varied agent traces at no token cost. The free server option can host the collector endpoint if you do not want a laptop awake for days.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest constraints. Free tiers have limits, so read the current docs before sizing a batch. And a hosted collector is a convenience, not a durability guarantee — export to your own storage.

A 30-minute test plan

1. Record 200 runs with sampling disabled. That is your ground truth.
2. Inject 20 known failures into the corpus.
3. Run the sampler at rate=0.05 over the same IDs.
4. Assert all 20 failures survive. Assert the keep rate lands in range.
5. Diff the rollups against ground truth. Counts must match exactly.
6. Set the disk threshold above current usage and confirm the rate degrades, not the failures.

uv run pytest -q tests/test_sampler.py

If step 5 mismatches, your rollup runs after sampling. Move it upstream.

Limitations and who should skip this

This design is wrong for several cases:

• Teams under ~100 runs a day. Keep everything; the complexity is not worth it.
• Audited or regulated workloads needing complete retention. Sampling is non-negotiable there.
• Workloads where failures are rare and cheap to store. Sample the passes, not the errors.
• Anyone who cannot re-run the corpus. Without ground truth, you cannot verify the sampler.

Sampling also hides low-frequency patterns. A 5% keep rate needs roughly 20x the traffic to surface a rare tool bug.

The loop to reuse

Write keep rules. Hash the trace ID. Roll up before you sample. Degrade rate under disk pressure. Verify against a labeled corpus.

That loop survives model changes and framework upgrades. It also makes the 2 a.m. cleanup cron job unnecessary.

If you want to try the corpus step first, the free model access is the cheapest entry point.