DEV Community

nmorestd
nmorestd

Posted on

End-to-End OpenTelemetry Tracing & Metrics in FastAPI Background Workers

Modern Python services rarely fail inside a single synchronous HTTP handler. Most production latency spikes happen between a FastAPI endpoint, a PostgreSQL query, and an asynchronous background worker.

In this guide, we wire OpenTelemetry traces, structured logs, and custom latency histograms together in Python so every request carries a single trace_id end-to-end.

1. Install the OpenTelemetry SDK and Instrumentation

pip install fastapi uvicorn opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi
Enter fullscreen mode Exit fullscreen mode

2. Configure TracerProvider, MeterProvider, and OTLP Export

Create telemetry.py to export spans and metrics over gRPC to any OTLP collector (such as SigNoz or an OpenTelemetry Collector):

import os
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

def setup_telemetry(service_name: str = "billing-api"):
    endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
    resource = Resource.create({"service.name": service_name, "deployment.environment": "prod"})

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True))
    )
    trace.set_tracer_provider(tracer_provider)

    metric_reader = PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint=endpoint, insecure=True),
        export_interval_millis=10000,
    )
    metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))
Enter fullscreen mode Exit fullscreen mode

3. Propagating Trace Context into Background Tasks

When a FastAPI route enqueues work onto a queue or thread pool, copy the W3C traceparent carrier so worker spans attach to the parent HTTP trace instead of starting disconnected root traces:

from fastapi import FastAPI, BackgroundTasks
from opentelemetry import trace, metrics
from opentelemetry.propagate import inject, extract
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
setup_telemetry()
FastAPIInstrumentor.instrument_app(app)

tracer = trace.get_tracer("billing.worker")
meter = metrics.get_meter("billing.metrics")
job_latency = meter.create_histogram("worker.job.duration_ms", unit="ms")

def run_invoice_job(order_id: str, carrier: dict):
    ctx = extract(carrier)
    with tracer.start_as_current_span("worker.generate_invoice", context=ctx) as span:
        span.set_attribute("order.id", order_id)
        # Simulated work + metric recording
        job_latency.record(42.5, {"job.type": "invoice"})

@app.post("/orders/{order_id}/invoice")
def enqueue_invoice(order_id: str, bg: BackgroundTasks):
    carrier = {}
    inject(carrier)
    bg.add_task(run_invoice_job, order_id, carrier)
    return {"queued": True, "order_id": order_id}
Enter fullscreen mode Exit fullscreen mode

4. Key Takeaways

  1. Always bind BatchSpanProcessor rather than SimpleSpanProcessor in production to avoid blocking the FastAPI event loop on collector I/O.
  2. Propagate traceparent across queues and background workers via inject() / extract().
  3. Pair custom histograms with trace exemplars so p99 alerts jump straight to the slow span.

Top comments (0)