Structured Logging and Distributed Tracing: Moving Beyond console.log

We have all been there during an outage: staring at terminal logs searching for an error:

[2026-09-25 02:14:12] Error processing request: object is None
[2026-09-25 02:14:12] Failed to charge credit card
[2026-09-25 02:14:13] User checked out

Which user failed? What was the order ID? Did the database drop or did the payment gateway time out?

Unstructured, free-text string logging (print(), console.log()) is completely inadequate for distributed systems. When fifty microservices process thousands of interleaved concurrent requests, tracking the lifecycle of an error requires two foundational capabilities:

1. Machine-Readable Structured JSON Logging.
2. Distributed Tracing with Correlation IDs (trace_id / span_id).

Structured JSON vs Free-Text

Unstructured:

Order 9812 failed for customer cust_44 due to timeout after 4500ms

Structured JSON:

{
  "timestamp": "2026-09-25T02:14:12.842Z",
  "level": "ERROR",
  "service": "payment-api",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "customer_id": "cust_44",
  "order_id": "9812",
  "duration_ms": 4500,
  "error": "Stripe API did not respond within timeout window"
}

With structured logs, you can run queries in Grafana Loki or Datadog in seconds:

service="payment-api" level="ERROR" duration_ms > 4000 | stats count() by customer_id

Implementing Correlation Middleware in FastAPI

import uuid
import time
import logging
import json
from contextvars import ContextVar
from fastapi import FastAPI, Request, Response
from starlette.middleware.base import BaseHTTPMiddleware

correlation_id_ctx: ContextVar[str] = ContextVar("correlation_id", default="")

class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        log_data = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "correlation_id": correlation_id_ctx.get(),
        }
        if hasattr(record, "extra_data"):
            log_data.update(record.extra_data)
        return json.dumps(log_data)

logger = logging.getLogger("api")
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

app = FastAPI()

class TelemetryMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        corr_id = request.headers.get("X-Correlation-ID", str(uuid.uuid4()))
        token = correlation_id_ctx.set(corr_id)
        start_time = time.perf_counter()

        try:
            response: Response = await call_next(request)
            duration_ms = round((time.perf_counter() - start_time) * 1000, 2)

            logger.info(
                f"HTTP {request.method} {request.url.path} {response.status_code}",
                extra={"extra_data": {
                    "path": request.url.path,
                    "status_code": response.status_code,
                    "duration_ms": duration_ms
                }}
            )
            response.headers["X-Correlation-ID"] = corr_id
            return response
        finally:
            correlation_id_ctx.reset(token)

app.add_middleware(TelemetryMiddleware)

Observability Rules

1. Log Context, Not Just Messages: Never write logger.error("Something went wrong"). Include user_id, request_id, and exact error details.
2. Never Log Sensitive PII: Strip passwords, credit card numbers, and bearer tokens in your logging serializer.
3. Propagate Trace Headers: When calling downstream services, always pass headers={"X-Correlation-ID": correlation_id_ctx.get()}.