Building Distributed Tracing for a Data-Intensive Platform: The X-Ray Journey on AWS

The Problem We Started With

Our donation platform runs on 50+ AWS Lambda functions, handling everything from Stripe/PayPal payments to DB writes to SQS-based background processing (queue consolidation, tax receipts, user events, and more). When something went wrong in production, we had no easy way to answer basic questions:

• Which specific function failed, and why?
• Did a donation actually reach MongoDB, or did it fail silently?
• When a webhook queues a background job, does that job actually run, and how long does the whole chain take?

We needed distributed tracing: a way to see, end-to-end, what happens when a request flows through our system. AWS X-Ray is the natural tool for this on Lambda. This is the story of getting it working, including the two hardest problems we hit: "Unknown Host" nodes polluting every trace, and SQS producer→consumer visibility.

Attempt #1: ADOT (OpenTelemetry) - The Modern Path That Wasn't Ready

AWS's recommended modern approach for Lambda tracing is the ADOT (AWS Distro for OpenTelemetry) Lambda layer. It auto-instruments HTTP, AWS SDK calls, and more, with zero code changes, attach a layer and set an environment variable.

We attached it. Every function immediately started crashing on cold start with:

Runtime.CallbackHandlerDeprecated: AWS Lambda has removed support for
callback-based function handlers starting with Node.js 24.

Investigation confirmed this wasn't a config mistake, it's a real, unresolved AWS limitation. Node.js 24 dropped support for callback-style Lambda handlers, and ADOT's Node.js wrapper hasn't been updated to reflect this change. I found open GitHub issues on aws-observability/aws-otel-lambda from other teams hitting the same wall, with no fix timeline from AWS. Their own docs list Node 22 as the latest officially supported runtime for ADOT.

Decision: stick with Node.js 24 (rather than downgrade) and use the classic AWS X-Ray SDK (via AWS Lambda Powertools' Tracer) instead. Less automatic, but stable.

Attempt #2: Powertools Tracer, Getting the Basics Right

We rebuilt tracing using @aws-lambda-powertools/tracer:

export const tracer = new Tracer({ serviceName: "<XYZ>_Backend" });

export const withMetrics = (handlerName: string, handlerLogic) => {
  return async (event, context) => {
    const segment = tracer.getSegment();
    const subsegment = segment?.addNewSubsegment(handlerName);
    try {
      return await handlerLogic(event, context);
    } catch (error) {
      subsegment?.addError(error);
      throw error;
    } finally {
      subsegment?.close();
    }
  };
};

Every one of our ~50+ Lambda handlers got wrapped with withMetrics, giving each function its own named trace with duration and error status. Combined with enabling tracingConfig: { mode: "Active" } on every function in our CDK backend definition, this gave us basic, working, per-function tracing.

Problem #1: "Unknown Host"

Once external calls (MongoDB, Stripe, PayPal) started happening inside these traced functions, X-Ray started showing a strange red node labeled "Unknown host" for outbound network activity with no way to tell what it actually was.

Root Cause

Digging into the Powertools Tracer source code revealed the culprit:

// Tracer.js, inside the constructor
if (this.isTracingEnabled() && this.captureHTTPsRequests) {
    this.provider.captureHTTPsGlobal();
    this.provider.instrumentFetch();
}

captureHTTPsRequests defaults to true. This means Powertools Tracer was silently patching Node's core http/https module on every cold start, the exact same kind of monkey-patching that's known to misbehave on Node.js 24's updated internals. It wasn't crashing outright, but it was producing broken, unlabeled subsegments instead of properly-tagged ones (or in some cases, nothing at all).

The Fix

Two changes, both verified against the actual library source before applying:

1. Turn off the broken auto-instrumentation:

export const tracer = new Tracer({
  serviceName: "XYZ_Backend",
  captureHTTPsRequests: false
});

2. Manually wrap external calls, and mark them correctly for the trace map:

export const traceExternalCall = async (name, operation) => {
  const segment = tracer.getSegment();
  if (!segment) return operation();

  const subsegment = segment.addNewSubsegment(name);
  subsegment.namespace = "remote"; // <- tells X-Ray to render this as its own node
  try {
    return await operation();
  } catch (error) {
    subsegment.addError(error);
    throw error;
  } finally {
    subsegment.close();
  }
};

The namespace = "remote" line matters more than it looks; it's literally the same flag the AWS X-Ray SDK's own HTTP patcher sets internally when it recognizes an external call. Without it, a subsegment just sits in the timeline list; with it, X-Ray draws it as a proper node on the trace map.

Making It Automatic, Not Manual-Per-Function

Rather than wrapping every Mongo/Stripe/PayPal call across 50+ files by hand, we centralized the fix into the few shared places all those calls already flow through:

• MongoDB: wrapped once, inside the shared connectToDb() helper. Every function that connects to Mongo gets a MongoDB-Connect node automatically.
• Stripe: the Stripe SDK exposes native request/response events. We hooked into those once, inside our shared initializeStripe() factory:

  stripe.on("request", (event) => {
    const subsegment = tracer.getSegment()?.addNewSubsegment("api.stripe.com");
    subsegment.namespace = "remote";
    openSubsegments.set(event.request_start_time, subsegment);
  });
  stripe.on("response", (event) => {
    const subsegment = openSubsegments.get(event.request_start_time);
    if (event.status >= 400) subsegment.addErrorFlag();
    subsegment.close();
  });

• PayPal / Zoho / Coda / any HTTP call via axios a single global interceptor on the shared axios instance, since every one of these services happens to call out via axios:

  axios.interceptors.request.use((config) => {
    const host = new URL(config.url, config.baseURL).hostname;
    const subsegment = tracer.getSegment()?.addNewSubsegment(host);
    subsegment.namespace = "remote";
    config.xraySubsegment = subsegment;
    return config;
  });
  axios.interceptors.response.use(
    (response) => { response.config.xraySubsegment?.close(); return response; },
    (error) => { error.config?.xraySubsegment?.close(); return Promise.reject(error); }
  );

Result: zero code changes needed per new function, as long as it reuses these shared utilities. "Unknown host" nodes were replaced with correctly-labeled MongoDB-Connect, api.stripe.com, api-m.paypal.com, etc.

Problem #2: Seeing SQS Producer → Consumer Flows

Our architecture is heavily queue-based: a webhook function does its work, then drops a message on an SQS queue; a separate Lambda, triggered by that queue, picks it up and continues (e.g. writing to MongoDB). We wanted to see that whole chain as one connected story in X-Ray.

First (Wrong) Conclusion

Our first attempt used the classic pattern of reading the SQS message's AWSTraceHeader attribute and manually overwriting the consumer's segment trace_id/parent_id to match the producer's. We tested it directly, injected a known fake trace ID into a synthetic SQS event, invoked the consumer, and checked what trace ID it reported.

It didn't work. The consumer always reported its own, fresh trace ID, regardless of what we injected. We confirmed this at the API level too, using aws xray get-trace-graph on the producer's trace ID; the consumer never appeared in the graph.

Why: on Lambda, the top-level trace ID is assigned by the AWS Lambda platform before your code runs. Application code can't override it after the fact, that door is locked from the platform side. This is different from services like ECS/EC2, where the X-Ray SDK itself creates the root segment and can be told what trace ID to use.

We initially documented this as "SQS producer→consumer tracing isn't possible on Node 24 without ADOT" and moved on.

The Correction

I later discovered we'd tested the wrong thing. AWS has a separate mechanism "linked traces" that connects two distinct trace IDs together for console viewing, without merging them into one trace ID. Opening the consumer's trace in the X-Ray console showed a banner: "This trace is part of a linked set of traces", with the producer trace displayed alongside it, connected.

This worked because the AWSTraceHeader was already being correctly propagated onto the SQS message (this part worked from the start) X-Ray's console uses that header to link the two traces together for display, even though they remain separate IDs under the hood.

Making the Queue Itself Visible

One more piece was missing: our SQS client wasn't wrapped with any tracing at all, so the actual "send to queue" step was invisible, and the queue never appeared as its own node on the trace map. The fix was a single line, in the one shared place all our queue sends flow through:

// common/services/messageQueue.service.ts
this.messageQueue = tracer.captureAWSv3Client(new SQSClient({}));

captureAWSv3Client hooks into the AWS SDK v3's own middleware stack (not Node's core modules), so it's unaffected by the Node 24 monkey-patching issues entirely. After this change, the Trace Map showed the SQS queue as an explicit node https://sqs.../XYZSaveUsersQueue.fifo sitting visually between the producer and consumer functions, with the queue's real URL.

End Result

For a real signup flow (postConfirmationFunction → SQS → saveUsersFunction), the Trace Map now shows:

Client → postConfirmation (Lambda) → SQS Queue → saveUsers (Lambda) → MongoDB-Connect
                                                                     → MongoDB-FindUser

With the linked-traces banner connecting the two Lambda traces, and the MongoDB node showing "Ok 100%" with zero faults a fully connected, correctly labeled, verified-working picture of the whole flow.

What I Learned

1. Verify library defaults, don't assume them. The "Unknown host" bug existed because a true default was silently active the whole time. Reading the actual source code (Tracer.js) settled it in minutes, versus hours of guessing.
2. Test claims empirically, especially "automatic" ones. AWS's own documentation implied SQS linking "just works" for Lambda consumers. It doesn't not the way we first tested it. The real feature (linked traces) exists, but it's a different mechanism than the docs' wording suggested.
3. Centralize instrumentation, don't scatter it. Wrapping ~5 shared utility files (Mongo connect, Stripe factory, axios instance, SQS service) gave us tracing coverage across 50+ functions, with zero risk of new functions being "forgotten." The only manual step left is wrapping each new handler's export in withMetrics a single line, documented in our dev guide.
4. Node.js version currency has real, unglamorous costs. Being on the newest Node runtime meant losing access to AWS's newest tracing tooling (ADOT) for the time being. That's a real trade-off, not a bug to route around.

Production Considerations Going Forward

• Sampling: X-Ray doesn't trace every request by default (1/sec + 5% of the rest). What we tested manually (1 request at a time) always gets sampled; production traffic won't be 100% visible without tuning sampling rules.
• Alarms over eyeballs: the Trace Map is a "glance and check" tool, not something to watch live. Real usage means CloudWatch Alarms on fault rate/latency, with traces used for investigation after an alarm fires.
• Annotations for searchability: to find one specific customer's failed transaction later, we plan to add searchable annotations (e.g. order ID) to segments something to build next.