DEV Community

Alessio
Alessio

Posted on Edited on Originally published at py-forge.com

Find the slow span in your traces with a 600-line Python script

You have a pile of exported spans, a p99 that jumped, and no time to click through a tracing UI one trace at a time. What you want is a quick answer to two questions: which operations are slower than their own normal, and which span in each bad trace is to blame?

TraceSift is a single Python file (standard library only) that answers that from a JSON file of spans.

Try it without your own data

# Generate 50 mock traces (about 200 spans) and analyze them
python3 tracesift.py --generate-mock --mock-count 50 -o spans.json
python3 tracesift.py -i spans.json
Enter fullscreen mode Exit fullscreen mode

Real output from that run:

Total Ingested Spans: 206 across 50 traces
Total Anomalies Detected: 13
Root Causes Isolated: 9
------------------------------------------------------------------------------
TOP ROOT CAUSES IDENTIFIED:
  [ROOT_ERROR] Trace: trc-228bb000f -> Culprit: database-cluster::UPDATE inventory_qty (20.1ms, depth=2)
  [LATENCY_BOTTLENECK] Trace: trc-266da0018 -> Culprit: edge-gateway::GET /api/v1/checkout (808.3ms, depth=0)
------------------------------------------------------------------------------
TOP COST OPTIMIZATION OPPORTUNITIES:
  edge-gateway -> GET /healthz: vol=5.34% noise=95% rec_sample=1% | Saving: $0.79/mo
Enter fullscreen mode Exit fullscreen mode

(--demo runs a built-in self-test and demonstration instead.)

What the input looks like

A flat JSON list of spans (or an object with a spans list). Field names are matched loosely, so trace_id/traceId, span_id/spanId, parent_span_id/parentId, service/serviceName, operation/name and duration_ms (or duration_us, duration_ns) all work:

[
  {"trace_id": "t1", "span_id": "a", "parent_span_id": null,
   "service": "edge-gateway", "operation": "GET /api/v1/checkout",
   "duration_ms": 127.4, "error": false},
  {"trace_id": "t1", "span_id": "b", "parent_span_id": "a",
   "service": "database-cluster", "operation": "UPDATE inventory_qty",
   "duration_ms": 20.1, "error": true}
]
Enter fullscreen mode Exit fullscreen mode

It does not read Jaeger, Zipkin or OTLP export files natively. If your backend exports nested formats, flatten them to one object per span first (a few lines of jq or Python).

How it decides

  1. Anomalies: spans are grouped by (service, operation). For each group it computes a robust mean and standard deviation, and a span is flagged when it is above that group's p95 and its z-score is at or above the threshold (default 2.5, change it with -z). Comparing each operation with itself avoids "the DB call is slower than the cache call" noise.
  2. Root cause per bad trace: if the trace has failed spans, the culprit is the deepest failed span in the call tree (where the error started, not the gateway that reported it). Otherwise it is the longest span in the trace.
  3. Sampling suggestions: health checks are suggested at 1%, stable error-free operations at 10%, and operations with more than 5% errors are never downsampled.

Reports for CI or a PR

python3 tracesift.py -i spans.json -z 3 --format markdown -o trace-report.md
python3 tracesift.py -i spans.json --format json -o trace-report.json
Enter fullscreen mode Exit fullscreen mode

Limitations (read these)

  • The dollar "savings" are a rough estimate, not a bill: it assumes about 600 bytes per span, scales your sample to a monthly volume with a fixed multiplier, and uses --cost-per-gb (default $0.65). Use it to rank operations, not to promise numbers.
  • It works on a file you give it; it is not a live collector or a tracing UI.
  • Small samples make z-scores unreliable: with only a handful of spans per operation, expect few or no anomalies.

TraceSift is on PyForge: get TraceSift for $49, or get it together with every other PyForge tool, including future ones, with the All-Access Pass.

Top comments (0)