You don't need a six-figure observability contract to get causal root cause analysis. This is a step-by-step build using Python, Docker Compose, and OpenTelemetry runnable on a laptop, with a mock incident you can trigger yourself to watch the pipeline correctly isolate the failing service in real time.
By the end of this tutorial you'll have a working local pipeline that ingests traces, builds a service dependency graph, and prints a structured "here's what broke and why" report.
Prerequisites
Docker + Docker Compose
Python 3.10+
15 minutes
Step 1: Spin up the collection stack
yaml
docker-compose.yml
version: "3.8"
services:
otel-collector:
image: otel/opentelemetry-collector:latest
volumes:
- ./otel-config.yaml:/etc/otel/config.yaml
command: ["--config=/etc/otel/config.yaml"]
ports:
- "4317:4317"
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686"
- "14250:14250"
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
bash
docker compose up -d
Jaeger UI is now live at localhost:16686 that's where you'll visually confirm the trace data once the demo app starts sending it.
Step 2: A 3-service demo app with an intentional dependency
Three tiny Flask services: frontend → orders → inventory. Each is instrumented with the OpenTelemetry SDK.
python
orders_service.py
from flask import Flask
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
import requests, time, random
app = Flask(name)
FlaskInstrumentor().instrument_app(app)
tracer = trace.get_tracer(name)
@app.route("/order")
def order():
with tracer.start_as_current_span("call-inventory"):
resp = requests.get("http://localhost:5002/inventory")
return resp.json()
if name == "main":
app.run(port=5001)
Repeat the same pattern for frontend (calls orders) and inventory (the leaf service). Full code for all three is in the repo linked at the end copy them in as frontend_service.py, orders_service.py, inventory_service.py and run each with python .py.
Step 3: Extracting trace metadata and building the DAG
python
dag_builder.py
import requests
import networkx as nx
def fetch_traces(service="frontend", lookback="1h"):
resp = requests.get(
f"http://localhost:16686/api/traces?service={service}&lookback={lookback}"
)
return resp.json()["data"]
def build_dag(traces):
graph = nx.DiGraph()
for trace in traces:
spans = {s["spanID"]: s for s in trace["spans"]}
for span in trace["spans"]:
for ref in span.get("references", []):
parent = spans.get(ref["spanID"])
if parent:
graph.add_edge(
parent["operationName"],
span["operationName"],
duration=span["duration"],
)
return graph
This walks every trace's span references to reconstruct who-calls-whom, weighting each edge by observed latency the raw material the causal ranking step needs.
Step 4: Scoring anomalous hops
python
rank_causes.py
import statistics
def rank_by_latency_anomaly(graph):
scored = []
for u, v, data in graph.edges(data=True):
durations = [d["duration"] for _, _, d in graph.edges(data=True) if _ == u]
if len(durations) > 1:
z = (data["duration"] - statistics.mean(durations)) / (statistics.stdev(durations) or 1)
scored.append((v, z))
return sorted(scored, key=lambda x: x[1], reverse=True)
The service with the highest z-score on its incoming edge duration is your root-cause candidate. This is deliberately simple a z-score threshold, not a full ML model because the point of this tutorial is to show the shape of the pipeline. Swap in a more sophisticated anomaly detector once the plumbing works.
Step 5: Live demo inject latency and watch it get caught
python
inject_latency.py
import time
from flask import Flask
app = Flask(name)
@app.route("/inventory")
def inventory():
time.sleep(2.5) # simulate a regression
return {"status": "ok", "items": 42}
if name == "main":
app.run(port=5002)
Restart the inventory service with this injected version, hit /order on the frontend a few times to generate traces, then run:
bash
python dag_builder.py && python rank_causes.py
Expected output:
Root cause candidate: inventory (z-score: 3.8)
Suspect edge: orders → inventory
The pipeline correctly isolates inventory as the regression source exactly the kind of answer that normally takes a human ten minutes of dashboard-hopping to reach.
What's next
This local version proves the algorithm. The companion AWS Builders article wires the same logic into a real cloud pipeline with EventBridge and Lambda, so triage runs automatically instead of on a manual script invocation. Full code for this tutorial, including all three demo services, is in the repo below.
Repo: github.com//telemetry-causal-triage
If this saved you a debugging session, a star on the repo helps more people find it. Next up: how this generalizes to agentic/LLM pipeline failures specifically, where execution time itself is non-deterministic.
Top comments (0)