Modern Python services rarely fail inside a single synchronous HTTP handler. Most production latency spikes happen between a FastAPI endpoint, a PostgreSQL query, and an asynchronous background worker.
In this guide, we wire OpenTelemetry traces, structured logs, and custom latency histograms together in Python so every request carries a single trace_id end-to-end.
1. Install the OpenTelemetry SDK and Instrumentation
pip install fastapi uvicorn opentelemetry-api opentelemetry-sdk \
opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi
2. Configure TracerProvider, MeterProvider, and OTLP Export
Create telemetry.py to export spans and metrics over gRPC to any OTLP collector (such as SigNoz or an OpenTelemetry Collector):
import os
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
def setup_telemetry(service_name: str = "billing-api"):
endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
resource = Resource.create({"service.name": service_name, "deployment.environment": "prod"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True))
)
trace.set_tracer_provider(tracer_provider)
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint=endpoint, insecure=True),
export_interval_millis=10000,
)
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))
3. Propagating Trace Context into Background Tasks
When a FastAPI route enqueues work onto a queue or thread pool, copy the W3C traceparent carrier so worker spans attach to the parent HTTP trace instead of starting disconnected root traces:
from fastapi import FastAPI, BackgroundTasks
from opentelemetry import trace, metrics
from opentelemetry.propagate import inject, extract
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
app = FastAPI()
setup_telemetry()
FastAPIInstrumentor.instrument_app(app)
tracer = trace.get_tracer("billing.worker")
meter = metrics.get_meter("billing.metrics")
job_latency = meter.create_histogram("worker.job.duration_ms", unit="ms")
def run_invoice_job(order_id: str, carrier: dict):
ctx = extract(carrier)
with tracer.start_as_current_span("worker.generate_invoice", context=ctx) as span:
span.set_attribute("order.id", order_id)
# Simulated work + metric recording
job_latency.record(42.5, {"job.type": "invoice"})
@app.post("/orders/{order_id}/invoice")
def enqueue_invoice(order_id: str, bg: BackgroundTasks):
carrier = {}
inject(carrier)
bg.add_task(run_invoice_job, order_id, carrier)
return {"queued": True, "order_id": order_id}
4. Key Takeaways
- Always bind
BatchSpanProcessorrather thanSimpleSpanProcessorin production to avoid blocking the FastAPI event loop on collector I/O. - Propagate
traceparentacross queues and background workers viainject()/extract(). - Pair custom histograms with trace exemplars so p99 alerts jump straight to the slow span.
Top comments (0)