DEV Community: Anish Ummenthala

Building a Scalable Fan Out Architecture on AWS with SNS and SQS

Anish Ummenthala — Sun, 29 Jun 2025 09:57:52 +0000

In distributed systems, decoupling your components is the first step to achieving scalability and resilience.

If you have ever built a system where one thing needs to trigger a bunch of other things, you must have already stumbled upon the idea of fan-out architecture.

In this post, I’ll explain what fan-out architecture is, why it’s powerful, and how we can build it using AWS SNS and SQS. I’ll also share a simple CDK project I made that implements this architecture (a notification system that sends email, SMS, and push notifications all in parallel).

TL;DR

Fan-out Architecture is a design pattern where a single message is sent to multiple downstream systems in parallel. Using AWS SNS + SQS, we can build a fully decoupled, scalable, and fault-tolerant system.

In this blog:

We break down what fan-out architecture is.
Show how to implement it with AWS SNS (publisher)+ SQS (subscribers).
Walk through a real example: a notification system that sends email, SMS, and push notifications all in parallel.

What Is Fan Out Architecture?

Fan-out architecture is just a fancy way of saying:

“Hey, everyone who cares about this message — go do your thing!”

We aren’t doing everything in one place. We take one big task, break it into smaller tasks, and run them at the same time (in parallel).

In tech terms, it’s a messaging pattern where a single message is sent to multiple destinations at the same time. Each destination can process it in parallel and in total isolation from other destinations.

This architecture is super useful when we want things to be fast, reliable, and decoupled (services that don't need to know about each other to get the job done).

Pub/Sub Model?

The easiest way to implement this pattern is with the good old Publish-Subscribe (aka Pub/Sub) model.

We have a Topic (acts like a group chat).
A Publisher sends a message to that topic.
All the Subscribers get that message instantly and go do their work.

Why is this awesome?

The publisher doesn't need to know who is listening.
The subscriber doesn't care who sent the message.
Everyone does their job independently, and our system becomes more flexible, maintainable, and resilient.

Why Bother with Fan Out?

So, what do we actually get out of this architecture?

Parallel Processing — Multiple tasks run at the same time (= faster).
Failure Isolation — One service crashing won’t take the others down.
Decoupling — Easier to maintain, test, and deploy our services.

My Example: Notification System with CDK

I built a simple notification system using AWS CDK (in TypeScript). Here’s what it includes:

An SNS topic (NotificationTopic)
Three SQS queues are subscribed to this topic (EmailQueue, PushQueue, SMSQueue)
Three Lambda functions to handle each queue
Infra to wire them all together

A real-world example would be when a user signs up for your platform, your backend sends a single message to the NotificationTopic.

That message fans out to:

EmailQueue → triggers Email Lambda → sends welcome email
PushQueue → triggers Push Lambda → sends push notification to the device
SMSQueue → triggers SMS Lambda → sends text message to user

Each Lambda gets messages in batches from its queue and does its thing, completely unaware of the others.

Time to get into the good stuff: the code

1. Defining the SNS Topic

export const createNotificationTopic = (scope: Construct) => {
  return new Topic(scope, 'NotificationTopic', {
    displayName: 'FanoutNotificationTopic',
  });
};

This is the central piece of our fan-out system. Any message published here will be broadcast to all the subscribers.

2. Creating the SQS Queues

Here we have defined a reusable function that creates named SQS queues:

export const createNotificationQueue = (scope: Construct, name: string) => {
  return new Queue(scope, `${name}Queue`, {
    queueName: `${name.toLowerCase()}-notification-queue`,
    visibilityTimeout: Duration.seconds(30),
  });
};

We use this function to spin up EmailQueue, PushQueue, and SMSQueue in our stack.

3. Creating Lambda Consumers

Here we have defined a function that creates Lambda consumers for each SQS queue:

export const createNotificationConsumer = (
  scope: Construct,
  handlerPath: string,
  queue: Queue,
  name: string
) => {
  const lambdaFn = new NodejsFunction(scope, `${name}ConsumerFn`, {
    entry: handlerPath,
    runtime: Runtime.NODEJS_18_X,
    handler: 'handler',
    timeout: Duration.seconds(10),
  });

  queue.grantConsumeMessages(lambdaFn);
  lambdaFn.addEventSource(new SqsEventSource(queue, { batchSize: 5 }));

  return lambdaFn;
};

Each Lambda uses a separate handler file (emailHandler.ts, pushHandler.ts, smsHandler.ts) and consumes messages in batches from its corresponding queue.

4. Hooking Everything Together

Now, in the main stack, we wire all the resources together:

const topic = createNotificationTopic(this);

const emailQueue = createNotificationQueue(this, 'Email');
const pushQueue = createNotificationQueue(this, 'Push');
const smsQueue = createNotificationQueue(this, 'SMS');

// Subscribe queues to the topic
topic.addSubscription(new SqsSubscription(emailQueue));
topic.addSubscription(new SqsSubscription(pushQueue));
topic.addSubscription(new SqsSubscription(smsQueue));

// Hook up Lambdas
createNotificationConsumer(this, './src/lambda/emailHandler.ts', emailQueue, 'Email');
createNotificationConsumer(this, './src/lambda/pushHandler.ts', pushQueue, 'Push');
createNotificationConsumer(this, './src/lambda/smsHandler.ts', smsQueue, 'SMS');

Now, publishing a message to the topic will send it to all three queues, and each queue will invoke its Lambda to handle the message.

Full Project on GitHub!

If you want to see the full code, clone it, tweak it, or just poke around, I’ve uploaded everything to GitHub:

Github Repo

The repo includes:

Full CDK setup (SNS + SQS + Lambda)
Queue + consumer definitions
Sample handler code
Clean folder structure

Feel free to fork it, star it, or open issues if you run into anything or want to suggest improvements.

Wrapping Up

Fan-out architecture is a solid pattern for building event-driven, decoupled systems that scale. With AWS SNS and SQS, it’s super easy to implement, and with CDK, it’s even easier to define in code.

This system is a great base for building things like:

Notification systems
Audit/event logs
Data processing pipelines
Multi-tenant service triggers

One last thing, don’t forget to destroy your AWS resources when you’re done testing.

“Always clean up unless your hobby is funding Jeff Bezos’ next rocket launch.”

References

Redacting Sensitive Data On-the-Fly with S3 Object Lambda

Anish Ummenthala — Sun, 29 Jun 2025 09:36:06 +0000

“Mask what you must, without touching the source.”

Most modern applications often store user data in Amazon S3, including potentially sensitive details like names, emails, or contact information. But what if you could automatically hide the private data only when it's accessed, without ever touching the original file?

That's where Amazon S3 Object Lambda comes in.

In this post, I'll show you how to build a simple PII (Personal Identifiable Information) masker using the AWS CDK, S3 Access Points & Lambda Function that redacts email addresses from .txt files in real time.

Why On-the-Fly Transformation?

Traditionally, redacting sensitive information in S3 means:

Modifying the original file to sanitize information
Storing the sanitized copies
Managing multiple versions of the same data

But this approach can be clunky and error-prone sometimes.

With S3 Object Lambda, you can intercept the S3 GET requests and transform data on the fly without duplicating the source files. It is a serverless middleware for S3 objects.

What is an S3 Access Point?

An S3 Access Point is like a custom entry point to your S3 bucket. Instead of relying on one big, complicated bucket policy, you can create several of these entry points, each with its own name and set of rules. This lets you control who gets in and what they can access more easily.

For example, you might have:

One access point for your internal team,
Another one that allows the public only to read files,
And a third one that's mainly for use only by a Lambda function.

What is an S3 Object Lambda?

An S3 Object Lambda lets you customize or transform data on the fly as retrieved from S3. It sits between the user/client and the original object, invoking a Lambda function to tweak the data (masking sensitive info, reformatting content, or filtering fields) before it's returned.

It's an intelligent filter that processes the file just before it reaches the user. Some common ways it's used include:

Removing sensitive information like credit card numbers, email addresses, or personal information.
Creating content on the fly, such as resizing images or adding watermarks right before download.
Changing file formats as needed, for example, converting a CSV file into JSON without storing both versions.

Hands-On: On-the-Fly PII Redactor

Let's get our hands dirty and build something real: a PII redactor that automatically masks email addresses from text files stored in S3 without changing the original files.

You can follow along or clone the complete working code from my GitHub repo:
PII Redaction with Object Lambda

We'll use AWS CDK to provision the entire setup:

S3 bucket to store raw sensitive data
Standard S3 Access Point
Lambda function that performs redaction
Object Lambda Access Point that triggers the Lambda on reads

1. Creating the Raw Data Bucket

We create an S3 bucket configured with autoDeleteObjects: true and removalPolicy: DESTROY, meaning the bucket and its contents will be automatically removed when the stack is destroyed, ideal for dev or test environments.

const bucket = new Bucket(this, "RawDataBucket", {
  bucketName: "raw-bucket-with-sensitive-data",
  removalPolicy: RemovalPolicy.DESTROY,
  autoDeleteObjects: true,
});

2.Creating a Standard S3 Access Point

We create an access point that provides a secure and controlled interface for Lambda to read files from the raw data bucket. The ARN created here will be used later to indirectly connect the Object Lambda Access Point to the bucket.

const accessPoint = new CfnS3AccessPoint(this, "S3AccessPoint", {
  bucket: "raw-bucket-with-sensitive-data",
  name: "bucket-access-point",
});

const s3AccessPointArn = `arn:aws:s3:${this.region}:${this.account}:accesspoint/bucket-access-point`;

3. Creating the Lambda Function

We create a function invoked by Object Lambda to mask/redact data. The lambda code will be written in Python and handler is index.lambda_handler

const lambdaFn = new Function(this, "PiiMaskerLambda", {
  runtime: Runtime.PYTHON_3_9,
  functionName: "maskerLambdaFunction",
  code: Code.fromAsset("lambda"),
  handler: "index.lambda_handler",
  environment: {
    RAW_BUCKET_NAME: "raw-bucket-with-sensitive-data",
  },
  logRetention: RetentionDays.ONE_WEEK,
});

4. Granting Access Permissions

Lambda permission to:

Grant s3:GetObject access only via the Access Point, not the entire bucket
Ensures Lambda can retrieve data without exposing raw bucket permissions

lambdaFn.addToRolePolicy(
  new PolicyStatement({
    actions: ["s3:GetObject"],
    resources: [`${s3AccessPointArn}/object/*`],
  })
);

bucket.grantRead(lambdaFn);

5. Creating the Object Lambda Access Point

We then configure an Object Lambda Access Point, which invokes the Lambda during object retrieval, transforming the output. This is the gateway that clients will call instead of the bucket. It's linked to:

The regular S3 Access Point
The Lambda function (for transformation)

const objectLambdaAccessPoint = new CfnS3ObjectLambdaAccessPoint(
  this,
  "S3ObjectLambdaAccessPoint",
  {
    name: "object-lambda-access-point",
    objectLambdaConfiguration: {
      supportingAccessPoint: s3AccessPointArn,
      transformationConfigurations: [
        {
          actions: ["GetObject"],
          contentTransformation: {
            AwsLambda: {
              FunctionArn: lambdaFn.functionArn,
            },
          },
        },
      ],
    },
  }

6. Allowing Object Lambda to Use Lambda

We explicitly allow the S3 Object Lambda service to:

Invoke our redaction Lambda
Write transformed responses using s3-object-lambda:WriteGetObjectResponse

/**
 * Allow the S3 Object Lambda service to invoke the Lambda function
 */
lambdaFn.addPermission("AllowObjectLambdaInvoke", {
  principal: new ServicePrincipal("s3-object-lambda.amazonaws.com"),
  sourceArn: objectLambdaAccessPointArn,
});

/**
 * Allow Lambda to return a transformed version of the object
 */
lambdaFn.addToRolePolicy(
  new PolicyStatement({
    actions: ["s3-object-lambda:WriteGetObjectResponse"],
    resources: [objectLambdaAccessPointArn],
  })
);

7. Lambda Handler (Python)

Below is the Python code to redact the sensitive email addresses from the text files.

import os
import re
import boto3
from urllib.parse import urlparse

s3 = boto3.client("s3")

# Regex to match emails
EMAIL_PATTERN = re.compile(r"[a-zA-Z0-9.+_-]+@[a-zA-Z0-9._-]+\.[a-zA-Z]+")

def lambda_handler(event, context):
    # Determine the requested key
    requested_url = event["userRequest"]["url"]
    key = urlparse(requested_url).path.lstrip("/")

    # Fetch the raw object
    bucket = os.environ["RAW_BUCKET_NAME"]
    resp = s3.get_object(Bucket=bucket, Key=key)
    body = resp["Body"].read()

    # Only redact emails in .txt files
    if key.lower().endswith(".txt"):
        try:
            text = body.decode("utf-8")
            redacted = EMAIL_PATTERN.sub("[REDACTED_EMAIL]", text)
            output = redacted.encode("utf-8")
            print(f"Redacted emails in {key}")
        except Exception:
            output = body
    else:
        output = body

    # Return to the caller
    s3.write_get_object_response(
        Body=output,
        RequestRoute=event["getObjectContext"]["outputRoute"],
        RequestToken=event["getObjectContext"]["outputToken"],
    )
    return {"status_code": 200}

Testing via AWS CLI

Upload a .txt file containing emails to your S3 bucket.

aws s3 cp s3://raw-bucket-with-sensitive-data/test.txt <path_to_local_txt_file>

Get Object via Object Lambda Access Point

aws s3api get-object \
  --bucket arn:aws:s3-object-lambda:<region>:<account>:accesspoint/object-lambda-access-point \
  --key test.txt redacted-output.txt

Replace and with your AWS values.

Output: The emails will be replaced with [REDACTED_EMAIL].

Don't Forget to Clean Up! 🚨

Once you've finished testing the redaction flow, delete the resources you provisioned to avoid incurring charges. You can destroy everything with a single command:

cdk destroy

This removes:

The S3 bucket and objects (if autoDeleteObjects: true)
Lambda function and associated roles
S3 Access Point
S3 Object Lambda Access Point
IAM policies

⚠️ Note: Since we are using CDK and logRetention, the log events inside CloudWatch log groups will expire automatically after the retention period (7 days). However, the log groups themselves are not deleted by default. To clean them up completely, you can use AWS CLI / AWS Console to delete all log groups with a specific project tag.

References

Full source code available on GitHub: Link
Transforming objects with S3 Object Lambda
Transforming Data on the Fly with S3 Object Lambda Access Points
Transforming Your Data with Amazon S3 Object Lambda
151)What is S3 object lambda access point?
Object Lambda Access Point – Why you should use it