Trace Any API Request Across Your Stack: A Practical Guide to Request IDs

It was 2 AM, and a customer's payment had failed. Not in our payment service — the request entered our API, then vanished. Three services later, someone had dropped the ball, and I had no idea who.

The logs told a sad story. Our gateway logged the incoming request. Our billing service logged a failure. Our email service logged a retry. But nothing connected them. I spent four hours stitching log lines together by timestamp and user ID, hoping nothing else happened at the same second.

The fix was embarrassingly simple: request IDs. One header, a few lines of middleware, and every log line in every service suddenly belongs to the same story.

Here's how to do it properly, with code you can actually run.

What a request ID is

A request ID (also called correlation ID or trace ID) is a unique identifier generated when a request first enters your system. It travels with the request through every service, every queue, every retry. Every log line, error report, and response carries it.

When something breaks, you grab the ID from the error message and grep once. Done. No archaeology.

Step 1: Generate it at the edge (and only at the edge)

The rule that matters most: only the entry point generates a new ID. Every downstream service must forward the ID it received, never generate its own.

In FastAPI, that looks like this:

import uuid
from fastapi import FastAPI, Request
from starlette.middleware.base import BaseHTTPMiddleware

REQUEST_ID_HEADER = "X-Request-ID"

class RequestIDMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        request_id = request.headers.get(REQUEST_ID_HEADER)
        # Accept an external ID only if it looks sane.
        # This lets your clients correlate on their side too.
        if not request_id or len(request_id) > 128:
            request_id = str(uuid.uuid4())
        request.state.request_id = request_id
        response = await call_next(request)
        response.headers[REQUEST_ID_HEADER] = request_id
        return response

app = FastAPI()
app.add_middleware(RequestIDMiddleware)

Two details people get wrong:

1. Always return the ID in the response. Your support team's first question to a user becomes "what error code and request ID did you see?" instead of "tell me exactly what you clicked at 2 AM."
2. Cap the length of incoming IDs. If you blindly echo a header back, you've created a log injection vector — someone sends a 10 KB header full of newlines and your logs get forged entries. Validate or regenerate.

Step 2: Forward it on every outbound call

Generating the ID is useless if it dies at the service boundary. Every HTTP client needs to pass it along.

With httpx, the clean way is a transport hook or a small helper:

import httpx

def client_with_request_id(request_id: str) -> httpx.AsyncClient:
    return httpx.AsyncClient(
        headers={"X-Request-ID": request_id},
        timeout=10.0,
    )

# In your route handler:
async def charge_customer(request: Request):
    client = client_with_request_id(request.state.request_id)
    async with client:
        resp = await client.post(
            "http://billing-service:8080/v1/charges",
            json={"amount_cents": 4999},
        )

The alternative — writing a manual headers= dict at every call site — fails the moment a developer forgets. One shared helper means you forget once, fix it once.

Step 3: Put the ID in every log line

A request ID that isn't in your logs is a request ID that doesn't exist.

Using structlog (or stdlib logging with a filter), bind the ID once per request and stop thinking about it:

import structlog

logger = structlog.get_logger()

@app.middleware("http")
async def bind_request_id(request: Request, call_next):
    structlog.contextvars.bind_contextvars(
        request_id=request.state.request_id,
        path=request.url.path,
        method=request.method,
    )
    try:
        return await call_next(request)
    finally:
        structlog.contextvars.unbind_contextvars(
            "request_id", "path", "method"
        )

Now every log line in that request's lifecycle — including in functions that never touch the request object — carries the ID:

2026-09-06T14:22:31Z [info] charge initiated request_id=9f2c... path=/v1/charges
2026-09-06T14:22:31Z [error] upstream timeout request_id=9f2c... service=billing

If you're on stdlib logging, a logging.Filter that reads a thread-local or contextvars value works the same way. The mechanism matters less than the invariant: every handler has access to the ID without threading it through every function signature.

Step 4: Carry it through async work and retries

This is where most hand-rolled systems break. If your request enqueues a background job and the job runs five minutes later, the ID must travel with the job payload:

job = {
    "type": "send_receipt",
    "request_id": request.state.request_id,  # not optional
    "charge_id": charge.id,
}

Same for retries. When your retry library fires attempt number three, it should reuse the original ID — a new ID per attempt would scatter one logical operation across three unrelated trace lines. Keep attempt counters in a separate header (X-Retry-Count) if you need them.

Message queues (RabbitMQ, SQS, Kafka) have standard header fields for this. When you consume a message, read the ID from the message headers, bind it to your logging context, and forward it on anything you produce downstream.

What this actually buys you

After we shipped this, a real incident went like this:

1. Customer reports a failed order with the request ID from their error screen.
2. I run one grep across our log aggregator.
3. Twenty lines, in order: gateway → billing → retry → billing again → email → success on the third attempt.

The bug (a flaky connection to our payment provider, masked by a silent timeout) was visible in under a minute. The old process would have taken an hour of timestamp-matching.

The honest numbers from our team: average time-to-understanding for a production error dropped from roughly 45 minutes to under 10. Most of that wasn't clever tooling — it was one header, consistently propagated.

Five rules to remember

1. Only the edge generates IDs. Downstream services forward, never invent.
2. Return the ID in the response. Your users become your first-line debuggers.
3. Validate incoming IDs (length, charset) before echoing them anywhere.
4. Bind the ID to logging context once — don't thread it through signatures.
5. Persist it across async boundaries: queues, retries, and scheduled jobs.

If your API doesn't have request IDs yet, this is a two-hour change that pays for itself the first time you debug a distributed failure. Start with the middleware and the response header; add propagation as you touch each service.

The alternative is another 2 AM session, grepping logs by timestamp, hoping nobody else's request interleaved with yours. I've been there. It's not worth it.

---

Building APIs and writing about the parts nobody warns you about. This article is part of my series on API resilience patterns — feedback and war stories welcome in the comments.