DEV Community: M. Abdullah Bin Aftab

Building Distributed Tracing for a Data-Intensive Platform: The X-Ray Journey on AWS

M. Abdullah Bin Aftab — Fri, 10 Jul 2026 06:27:21 +0000

The Problem We Started With

Our donation platform runs on 50+ AWS Lambda functions, handling everything from Stripe/PayPal payments to DB writes to SQS-based background processing (queue consolidation, tax receipts, user events, and more). When something went wrong in production, we had no easy way to answer basic questions:

Which specific function failed, and why?
Did a donation actually reach MongoDB, or did it fail silently?
When a webhook queues a background job, does that job actually run, and how long does the whole chain take?

We needed distributed tracing: a way to see, end-to-end, what happens when a request flows through our system. AWS X-Ray is the natural tool for this on Lambda. This is the story of getting it working, including the two hardest problems we hit: "Unknown Host" nodes polluting every trace, and SQS producer→consumer visibility.

Attempt #1: ADOT (OpenTelemetry) - The Modern Path That Wasn't Ready

AWS's recommended modern approach for Lambda tracing is the ADOT (AWS Distro for OpenTelemetry) Lambda layer. It auto-instruments HTTP, AWS SDK calls, and more, with zero code changes, attach a layer and set an environment variable.

We attached it. Every function immediately started crashing on cold start with:

Runtime.CallbackHandlerDeprecated: AWS Lambda has removed support for
callback-based function handlers starting with Node.js 24.

Investigation confirmed this wasn't a config mistake, it's a real, unresolved AWS limitation. Node.js 24 dropped support for callback-style Lambda handlers, and ADOT's Node.js wrapper hasn't been updated to reflect this change. I found open GitHub issues on aws-observability/aws-otel-lambda from other teams hitting the same wall, with no fix timeline from AWS. Their own docs list Node 22 as the latest officially supported runtime for ADOT.

Decision: stick with Node.js 24 (rather than downgrade) and use the classic AWS X-Ray SDK (via AWS Lambda Powertools' Tracer) instead. Less automatic, but stable.

Attempt #2: Powertools Tracer, Getting the Basics Right

We rebuilt tracing using @aws-lambda-powertools/tracer:

export const tracer = new Tracer({ serviceName: "<XYZ>_Backend" });

export const withMetrics = (handlerName: string, handlerLogic) => {
  return async (event, context) => {
    const segment = tracer.getSegment();
    const subsegment = segment?.addNewSubsegment(handlerName);
    try {
      return await handlerLogic(event, context);
    } catch (error) {
      subsegment?.addError(error);
      throw error;
    } finally {
      subsegment?.close();
    }
  };
};

Every one of our ~50+ Lambda handlers got wrapped with withMetrics, giving each function its own named trace with duration and error status. Combined with enabling tracingConfig: { mode: "Active" } on every function in our CDK backend definition, this gave us basic, working, per-function tracing.

Problem #1: "Unknown Host"

Once external calls (MongoDB, Stripe, PayPal) started happening inside these traced functions, X-Ray started showing a strange red node labeled "Unknown host" for outbound network activity with no way to tell what it actually was.

Root Cause

Digging into the Powertools Tracer source code revealed the culprit:

// Tracer.js, inside the constructor
if (this.isTracingEnabled() && this.captureHTTPsRequests) {
    this.provider.captureHTTPsGlobal();
    this.provider.instrumentFetch();
}

captureHTTPsRequests defaults to true. This means Powertools Tracer was silently patching Node's core http/https module on every cold start, the exact same kind of monkey-patching that's known to misbehave on Node.js 24's updated internals. It wasn't crashing outright, but it was producing broken, unlabeled subsegments instead of properly-tagged ones (or in some cases, nothing at all).

The Fix

Two changes, both verified against the actual library source before applying:

1. Turn off the broken auto-instrumentation:

export const tracer = new Tracer({
  serviceName: "XYZ_Backend",
  captureHTTPsRequests: false
});

2. Manually wrap external calls, and mark them correctly for the trace map:

export const traceExternalCall = async (name, operation) => {
  const segment = tracer.getSegment();
  if (!segment) return operation();

  const subsegment = segment.addNewSubsegment(name);
  subsegment.namespace = "remote"; // <- tells X-Ray to render this as its own node
  try {
    return await operation();
  } catch (error) {
    subsegment.addError(error);
    throw error;
  } finally {
    subsegment.close();
  }
};

The namespace = "remote" line matters more than it looks; it's literally the same flag the AWS X-Ray SDK's own HTTP patcher sets internally when it recognizes an external call. Without it, a subsegment just sits in the timeline list; with it, X-Ray draws it as a proper node on the trace map.

Making It Automatic, Not Manual-Per-Function

Rather than wrapping every Mongo/Stripe/PayPal call across 50+ files by hand, we centralized the fix into the few shared places all those calls already flow through:

MongoDB: wrapped once, inside the shared connectToDb() helper. Every function that connects to Mongo gets a MongoDB-Connect node automatically.
Stripe: the Stripe SDK exposes native request/response events. We hooked into those once, inside our shared initializeStripe() factory:

  stripe.on("request", (event) => {
    const subsegment = tracer.getSegment()?.addNewSubsegment("api.stripe.com");
    subsegment.namespace = "remote";
    openSubsegments.set(event.request_start_time, subsegment);
  });
  stripe.on("response", (event) => {
    const subsegment = openSubsegments.get(event.request_start_time);
    if (event.status >= 400) subsegment.addErrorFlag();
    subsegment.close();
  });

PayPal / Zoho / Coda / any HTTP call via axios a single global interceptor on the shared axios instance, since every one of these services happens to call out via axios:

  axios.interceptors.request.use((config) => {
    const host = new URL(config.url, config.baseURL).hostname;
    const subsegment = tracer.getSegment()?.addNewSubsegment(host);
    subsegment.namespace = "remote";
    config.xraySubsegment = subsegment;
    return config;
  });
  axios.interceptors.response.use(
    (response) => { response.config.xraySubsegment?.close(); return response; },
    (error) => { error.config?.xraySubsegment?.close(); return Promise.reject(error); }
  );

Result: zero code changes needed per new function, as long as it reuses these shared utilities. "Unknown host" nodes were replaced with correctly-labeled MongoDB-Connect, api.stripe.com, api-m.paypal.com, etc.

Problem #2: Seeing SQS Producer → Consumer Flows

Our architecture is heavily queue-based: a webhook function does its work, then drops a message on an SQS queue; a separate Lambda, triggered by that queue, picks it up and continues (e.g. writing to MongoDB). We wanted to see that whole chain as one connected story in X-Ray.

First (Wrong) Conclusion

Our first attempt used the classic pattern of reading the SQS message's AWSTraceHeader attribute and manually overwriting the consumer's segment trace_id/parent_id to match the producer's. We tested it directly, injected a known fake trace ID into a synthetic SQS event, invoked the consumer, and checked what trace ID it reported.

It didn't work. The consumer always reported its own, fresh trace ID, regardless of what we injected. We confirmed this at the API level too, using aws xray get-trace-graph on the producer's trace ID; the consumer never appeared in the graph.

Why: on Lambda, the top-level trace ID is assigned by the AWS Lambda platform before your code runs. Application code can't override it after the fact, that door is locked from the platform side. This is different from services like ECS/EC2, where the X-Ray SDK itself creates the root segment and can be told what trace ID to use.

We initially documented this as "SQS producer→consumer tracing isn't possible on Node 24 without ADOT" and moved on.

The Correction

I later discovered we'd tested the wrong thing. AWS has a separate mechanism "linked traces" that connects two distinct trace IDs together for console viewing, without merging them into one trace ID. Opening the consumer's trace in the X-Ray console showed a banner: "This trace is part of a linked set of traces", with the producer trace displayed alongside it, connected.

This worked because the AWSTraceHeader was already being correctly propagated onto the SQS message (this part worked from the start) X-Ray's console uses that header to link the two traces together for display, even though they remain separate IDs under the hood.

Making the Queue Itself Visible

One more piece was missing: our SQS client wasn't wrapped with any tracing at all, so the actual "send to queue" step was invisible, and the queue never appeared as its own node on the trace map. The fix was a single line, in the one shared place all our queue sends flow through:

// common/services/messageQueue.service.ts
this.messageQueue = tracer.captureAWSv3Client(new SQSClient({}));

captureAWSv3Client hooks into the AWS SDK v3's own middleware stack (not Node's core modules), so it's unaffected by the Node 24 monkey-patching issues entirely. After this change, the Trace Map showed the SQS queue as an explicit node https://sqs.../XYZSaveUsersQueue.fifo sitting visually between the producer and consumer functions, with the queue's real URL.

End Result

For a real signup flow (postConfirmationFunction → SQS → saveUsersFunction), the Trace Map now shows:

Client → postConfirmation (Lambda) → SQS Queue → saveUsers (Lambda) → MongoDB-Connect
                                                                     → MongoDB-FindUser

With the linked-traces banner connecting the two Lambda traces, and the MongoDB node showing "Ok 100%" with zero faults a fully connected, correctly labeled, verified-working picture of the whole flow.

What I Learned

Verify library defaults, don't assume them. The "Unknown host" bug existed because a true default was silently active the whole time. Reading the actual source code (Tracer.js) settled it in minutes, versus hours of guessing.
Test claims empirically, especially "automatic" ones. AWS's own documentation implied SQS linking "just works" for Lambda consumers. It doesn't not the way we first tested it. The real feature (linked traces) exists, but it's a different mechanism than the docs' wording suggested.
Centralize instrumentation, don't scatter it. Wrapping ~5 shared utility files (Mongo connect, Stripe factory, axios instance, SQS service) gave us tracing coverage across 50+ functions, with zero risk of new functions being "forgotten." The only manual step left is wrapping each new handler's export in withMetrics a single line, documented in our dev guide.
Node.js version currency has real, unglamorous costs. Being on the newest Node runtime meant losing access to AWS's newest tracing tooling (ADOT) for the time being. That's a real trade-off, not a bug to route around.

Production Considerations Going Forward

Sampling: X-Ray doesn't trace every request by default (1/sec + 5% of the rest). What we tested manually (1 request at a time) always gets sampled; production traffic won't be 100% visible without tuning sampling rules.
Alarms over eyeballs: the Trace Map is a "glance and check" tool, not something to watch live. Real usage means CloudWatch Alarms on fault rate/latency, with traces used for investigation after an alarm fires.
Annotations for searchability: to find one specific customer's failed transaction later, we plan to add searchable annotations (e.g. order ID) to segments something to build next.

Amazon Q CLI: Rock Paper Scissor with Python 🕹️

M. Abdullah Bin Aftab — Wed, 04 Jun 2025 04:35:40 +0000

In this blog post, I'll walk you through my journey of creating a fun Rock Paper Scissors game using Python and Pygame, with the assistance of Amazon Q CLI. I'll cover everything from setting up the development environment to implementing game mechanics and generating custom assets.

🔧 Installing Amazon Q CLI on macOS

Amazon Q CLI is a powerful tool that helped me throughout the development process. Here's how I installed it on my macOS:

First install the AWS CLI using Homebrew:

brew install awscli

Then I configured my AWS credentials:

aws configure

Next, I installed the Amazon Q CLI:

pip install amazon-q-cli

After installation, I verified it was working:

q --version

I started using Amazon Q by running:

q chat

In the chat, I asked:

Can you help me structure a Rock Paper Scissors game using Pygame?

Amazon Q CLI became my coding companion, helping me generate code snippets, debug issues, and even create project documentation.

🎮 Designing the Game Structure

Based on Amazon Q's suggestions, I designed the game with three main states:

Menu State: The main menu where players can start the game.
Playing State: Where players select their move (Rock, Paper, or Scissors).
Result State: Displaying the outcome and allowing players to play again.

I created the main game file structure with Amazon Q's help:

q chat

I asked:

Can you help me create a main.py file for my Rock Paper Scissors game with state management?

🎯 Implementing Game Mechanics

For the core game mechanics, I needed to implement:

Player choice selection
Computer random choice generation
Winner determination logic

🎁 Final Thoughts

Building this Rock Paper Scissors game was a fun and educational experience.

Amazon Q CLI significantly accelerated my development process by:

Providing code snippets and structure
Helping debug issues quickly
Generating documentation
Offering best practices and optimization tips

The final game includes:

An intuitive user interface
Smooth animations and transitions
Visual feedback with particle effects
Custom-generated game assets
Multiple game states for a complete experience
If you're developing games or any software project, I highly recommend giving Amazon Q CLI a try. It's like having an expert developer by your side, ready to help whenever you need it.

🕹️ To run the game yourself:

git clone https://https://github.com/heighterses/rock-paper-scissor-Q-CLI
cd rock-paper-scissor-Q-CLI
pip install pygame
python main.py

Enjoy playing Rock Paper Scissors! 🎮

Case Study: Creating an ETL Data Pipeline using AWS Services - Real-World Problem

M. Abdullah Bin Aftab — Fri, 06 Dec 2024 18:04:47 +0000

Architecture

Overview

This ETL pipeline leverages several AWS services to fetch, process, and store YouTube videos with translated subtitles. The core components include:

AWS Lambda for processing video and audio
Step Functions for Orchestrating Workflows
S3 Buckets for storing raw and processed data
AWS Transcribe for converting audio to text
AWS Translate for translating the text into desired languages

Pipeline Steps

Fetching Video from YouTube

Video is fetched from YouTube and sent to an initial Lambda Function.
This function splits the video into separate audio and video files.

Step Functions Orchestration

AWS Step Functions orchestrate all the following processes.
The workflow controls each step, passing data between services and managing the pipeline flow.

Storing Raw Video and Audio

Audio and video files are stored separately here.
The bucket sends an email notification to confirm the files were successfully stored.

Merging Video and Audio Files

This Lambda function combines the audio and video files into a single file.
Error Handling: If merging fails, the pipeline stores the failed files in an Error Bucket.

Transcribing Audio to Text

The merged video’s audio is sent to AWS Transcribe to convert audio into text.
The resulting text is passed to the next Lambda function.

Proofreading Transcribed Text

This Lambda function checks the transcription for accuracy and readability.
If the text is poor quality, it can be flagged for manual review or re-transcription.

Translating Text into Target Language

The proofread text is sent to AWS Translate to convert it into a chosen language (e.g., Arabic, Italian, or Spanish).
The translated text is then passed to the next Lambda function.

Generating Subtitles and Merging Paragraphs

This function formats the translated text as subtitles and merges paragraphs if needed.
The final file is prepared for storage.

Storing Processed Video with Subtitles

The fully processed video with subtitles in the desired language is stored here.
This is the final output location where the processed video is accessible.

Error Handling

Error Bucket for Failed Merges: If audio and video files fail to merge, the Lambda Merge Function sends them to a designated Error Bucket.
Transcription and Proofreading Quality Check: Poor transcription quality detected by the Proofread Function can trigger a flag for manual review or re-transcription.

Notifications

**Email Notifications **for File Arrival in S3:After the raw video and audio files are stored in the S3 Raw Video and Audio Bucket, an email notification is sent to confirm successful storage.

Benefits of This Pipeline

Automation: Streamlines the entire video processing workflow without manual intervention.
Scalability: AWS services like Lambda and Step Functions allow the pipeline to handle many videos.
Language Support: AWS Translate enables easy translation to multiple languages, broadening the video’s reach.
Error Management: Dedicated error-handling buckets and flags ensure issues are logged and handled efficiently.

How to Import Pandas(library) in AWS Lambda Functions - AWS Lambda Layers

M. Abdullah Bin Aftab — Wed, 16 Oct 2024 19:13:43 +0000

Imagine you need to run a Python script on the AWS Lambda function and you get this error 🔻

{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'pandas',
  "errorType": "Runtime.ImportModuleError"
  ...
}

Don't worry this is a common error and I am not going to make this long

How do you import Pandas in AWS Lambda Functions?

There are several ways but I am going to give you the easiest way to import pandas in AWS Lambda Function is to add Lambda Layer 🥪

What is AWS Lambda Layer?

It is a 🧀 cheese layer in Lambda Function containing additional code like libraries, dependencies, etc.

In Simple Words

AWS Lambda Layers are like building blocks for your functions. Imagine you need extra tools (like the Pandas library) to complete a project.

Instead of packing all those tools inside every single project (which wastes space and time), AWS allows you to create layers of tools (libraries, dependencies, or shared code).

These layers sit outside your main function but are always available when your function needs them.

In short, Lambda Layers help you:

Separate the main logic from the extra libraries to save space in your code. You can reuse libraries and code across multiple Lambda functions.

Easily update or manage your dependencies without changing your core function code.

Think of layers as an extra storage box attached to your Lambda function, holding everything your function needs to work smoothly. You can stack multiple layers on your function without cluttering your main code.

Steps to add Lambda Function Layer and Import Pandas

It takes only 3 steps to run Pandas in your Lambda Function successfully

Step 1 - Open Lambda Function through your AWS Management Console

As you can see we have an option Layers under the name of our Lambda Function, in my case, it's "import-pandas-function" and the Layers count is 0

Step 2- Add Script in your AWS Lambda Function

This step is further divided into two steps because we need to add a Python script that contains some Pandas code and write a test event in JSON to verify whether the code is running correctly.

2.1 - Add Python script - you can copy this code 🡻

import json
import pandas as pd

def lambda_handler(event, context):
    data = event.get('data', [])

    df = pd.DataFrame(data)

    if not df.empty:
        mean_value = df['column_name'].mean()

        result = {
            "mean_value": mean_value,
            "data_shape": df.shape,
            "summary": df.describe().to_dict()
        }
    else:
        result = {
            "message": "Empty DataFrame"
        }

    # Return the response
    return {
        'statusCode': 200,
        'body': json.dumps(result)
    }

2.2 - Add Test script in Json in the test tab - you can copy this code 🡻

{
  "data": [
    {"column_name": 10, "other_column": "A"},
    {"column_name": 20, "other_column": "B"},
    {"column_name": 30, "other_column": "C"},
    {"column_name": 40, "other_column": "D"}
  ]
}

Press the test button you probably got the 🔴error:-
"errorMessage": "Unable to import module 'lambda_function': No module named 'pandas', "errorType": "Runtime.ImportModuleError" ...

Step 3 - Add AWS Lambda Layer to Successfully run the Pandas in your Code

Scroll down to your Lambda Function, you probably can see the "Layers" separate section at the end of the page

3.1 - Click "Add a Layer"

After Clicking the "Add a Layer" you can see the page which has a couple of sections "Function runtime settings" and "Choose a layer"

3.2 - Click "AWS layers"

You can see three options in the "Choose a layer" section click the "AWS layers".

3.3 - Choose "AWS layers"

After selecting the AWS layers you can see the dropdown under "AWS layers".

3.4 - Choose "AWS layers" and "Version"

In a dropdown of "AWS layers" select -> AWSSDKPandas-Python312

In a dropdown of "Version" select -> 13(select the most one)

click the "Add" button

3.5 - Make sure the "Function Overview"

When your page is directed to the function overview you can see the layer is added below the function name "import-pandas-function"

Step 4 - Test the Function

You've successfully got the Response "statusCode": 200

{
  "statusCode": 200,
  "body": "{\"mean_value\": 25.0, \"data_shape\": [4, 2], \"summary\": {\"column_name\": {\"count\": 4.0, \"mean\": 25.0, \"std\": 12.909944487358056, \"min\": 10.0, \"25%\": 17.5, \"50%\": 25.0, \"75%\": 32.5, \"max\": 40.0}}}"
}

Keep Coding 😎