How to Sift Distributed Traces and Detect Spans Outliers in Python

Modern microservice architectures generate massive volumes of telemetry. When an API endpoint slows down or returns intermittent server errors, engineering teams often face thousands of raw span events dumped into log aggregators or telemetry backends. Manually filtering through individual trace IDs to spot whether a latency spike originated from a database query, a downstream HTTP client, or an edge proxy takes significant time.

TraceSift is a self-contained Python script designed to normalize heterogeneous trace formats, calculate statistical latency baselines, and surface anomalous spans or execution failures without requiring an external telemetry cluster.

The Problem: Unfiltered Telemetry and Schema Mismatch

Distributed tracing instrumentation across different frameworks often uses disparate schemas. OpenTelemetry, Jaeger, Zipkin, and custom internal loggers output slightly different keys:

• Timestamps might be reported in nanoseconds, microseconds, or milliseconds.
• Duration keys alternate between duration, duration_ms, duration_ns, or duration_us.
• Identifiers might be labeled trace_id, traceId, spanId, or id.
• Error states might be flagged by a boolean error field, an HTTP status code integer, or string tags.

Beyond schema variance, identifying which spans actually degraded an operation requires statistical context. Looking at a single span duration in isolation does not tell you if it exceeds the baseline of expected performance. Without computing percentiles and distribution deviations across service operations, finding the root culprit across distributed spans becomes manual guesswork.

Install & Usage

TraceSift runs as a single Python script using standard library modules. It does not require installing third-party libraries via pip.

Setup

Save the script source to a local file named tracesift.py:

# Save the source code into tracesift.py
chmod +x tracesift.py
python tracesift.py --help

Running the Analyzer

The script accepts trace data formatted as JSON or can execute using its built-in mock topology generator.

To run the script directly with simulated microservice traffic:

python tracesift.py

To adjust the sensitivity threshold for latency outlier detection when analyzing traces:

python tracesift.py --z-threshold 3.0

To evaluate telemetry ingest cost estimates with a custom ingestion price per gigabyte:

python tracesift.py --cost-per-gb 0.85

How It Works

The engine is built around three primary classes: TraceNormalizer, MockTraceGenerator, and SiftingEngine.

1. Ingesting and Normalizing Heterogeneous Spans

The TraceNormalizer.normalize_dict method handles schema variations and converts differing units into standard millisecond representations stored inside a strongly-typed Span dataclass:

            # Timestamp normalization (normalize to milliseconds)
            raw_start = raw.get("start_time_ms") or raw.get("startTime") or raw.get("timestamp") or 0.0
            raw_start = float(raw_start)
            if raw_start > 1e14:  # Nanoseconds
                start_ms = raw_start / 1e6
            elif raw_start > 1e11:  # Microseconds
                start_ms = raw_start / 1e3
            else:
                start_ms = raw_start

            # Duration normalization (normalize to milliseconds)
            raw_dur = raw.get("duration_ms") or raw.get("duration") or raw.get("duration_ns") or raw.get("duration_us") or 0.0
            raw_dur = float(raw_dur)

            if raw.get("duration_ms") is not None:
                duration_ms = float(raw.get("duration_ms"))
            elif raw.get("duration_ns") is not None:
                duration_ms = float(raw.get("duration_ns")) / 1e6
            elif raw.get("duration_us") is not None:
                duration_ms = float(raw.get("duration_us")) / 1e3
            elif raw_start > 1e14:  # Correlating timestamp in nanoseconds
                duration_ms = raw_dur / 1e6
            elif raw_start > 1e11:  # Correlating timestamp in microseconds
                duration_ms = raw_dur / 1e3
            elif raw_dur > 1e7:  # Nanoseconds heuristic
                duration_ms = raw_dur / 1e6
            elif raw_dur > 1e5:  # Microseconds heuristic
                duration_ms = raw_dur / 1e3
            else:
                duration_ms = raw_dur

The method inspects key variants like traceId, spanId, and status codes such as 500 or 503, outputting uniform Span objects for downstream analysis.

2. Statistical Outlier Detection

The SiftingEngine groups non-error spans by (service, operation) pairs to compute baselines. When evaluating anomalies, it calculates the interquartile range (IQR) to prune heavy outliers before computing the mean and standard deviation:

        for key, values in self.baselines.items():
            # Robust baseline calculation: prevent severe outliers from distorting mean & stdev
            if len(values) >= 4:
                q25 = self.calculate_percentile(values, 25.0)
                q75 = self.calculate_percentile(values, 75.0)
                iqr = q75 - q25
                if iqr > 0:
                    upper_bound = q75 + 1.5 * iqr
                    clean_values = [v for v in values if v <= upper_bound]
                    base_values = clean_values if len(clean_values) >= 2 else values
                else:
                    base_values = values
            else:
                base_values = values

Each span is evaluated using two criteria:

1. Explicit Errors: Spans marked with error flags or failure status codes are flagged immediately as an AnomalyRecord.
2. Statistical Latency Outliers: Non-error spans are measured against the baseline using a Z-score calculation. If the duration exceeds both the calculated p95 threshold and the configured Z-score cutoff (defaulting to 2.5), the span is logged as a latency anomaly.

Limitations of the Implementation

When using TraceSift, consider the following implementation details:

• In-Memory Storage: The script stores all traces and baseline duration arrays inside in-memory dictionaries (defaultdict(list)). Analyzing millions of spans concurrently will scale memory consumption linearly with span counts.
• Timestamp Unit Heuristics: The normalization logic uses numeric thresholds (such as checking if the raw timestamp is greater than 1e14 or 1e11) to guess whether timestamps are nanoseconds, microseconds, or milliseconds. Unusual custom timestamps may misalign with these heuristics.
• Isolated Linear Baselines: Spans are grouped strictly by the tuple (service, operation). The baseline does not account for multidimensional factors such as request payload size, client location, or high-cardinality tags.
• Outlier Sample Size: For operations with fewer than two recorded spans, the standard deviation defaults to 0.0, which skips latency Z-score anomaly detection for that specific operation until more samples are observed.

Frequently Asked Questions

What dependencies do I need to install?

None. TraceSift relies exclusively on Python standard library modules: sys, os, json, math, argparse, random, statistics, dataclasses, typing, collections, and datetime. No pip install commands are required.

How does TraceSift distinguish errors from latency spikes?

The script checks error states during the normalization step by evaluating error, is_error, or HTTP status codes (500, 502, 503, 504, CRITICAL). Spans with explicit errors are recorded directly as execution failure anomalies. Spans without error flags are evaluated through Z-score deviations and the 95th percentile calculations.

Can TraceSift normalize OpenTelemetry and Jaeger formats simultaneously?

Yes. The TraceNormalizer.normalize_dict method extracts fields using fallback key checks. It attempts to read standard keys such as service_name, serviceName, process.serviceName, span_id, and id, enabling uniform parsing across differing JSON trace exports.

---

Full version and the rest of the toolkit: TraceSift on PyForge. Product page: https://py-forge.com/products/tracesift