Anish Ummenthala

Posted on Jun 29 • Originally published at Medium

Building a Scalable Fan Out Architecture on AWS with SNS and SQS

#aws #sns #sqs #systemdesign

In distributed systems, decoupling your components is the first step to achieving scalability and resilience.

If you have ever built a system where one thing needs to trigger a bunch of other things, you must have already stumbled upon the idea of fan-out architecture.

In this post, I’ll explain what fan-out architecture is, why it’s powerful, and how we can build it using AWS SNS and SQS. I’ll also share a simple CDK project I made that implements this architecture (a notification system that sends email, SMS, and push notifications all in parallel).

TL;DR

Fan-out Architecture is a design pattern where a single message is sent to multiple downstream systems in parallel. Using AWS SNS + SQS, we can build a fully decoupled, scalable, and fault-tolerant system.

In this blog:

We break down what fan-out architecture is.
Show how to implement it with AWS SNS (publisher)+ SQS (subscribers).
Walk through a real example: a notification system that sends email, SMS, and push notifications all in parallel.

What Is Fan Out Architecture?

Fan-out architecture is just a fancy way of saying:

“Hey, everyone who cares about this message — go do your thing!”

We aren’t doing everything in one place. We take one big task, break it into smaller tasks, and run them at the same time (in parallel).

In tech terms, it’s a messaging pattern where a single message is sent to multiple destinations at the same time. Each destination can process it in parallel and in total isolation from other destinations.

This architecture is super useful when we want things to be fast, reliable, and decoupled (services that don't need to know about each other to get the job done).

Pub/Sub Model?

The easiest way to implement this pattern is with the good old Publish-Subscribe (aka Pub/Sub) model.

We have a Topic (acts like a group chat).
A Publisher sends a message to that topic.
All the Subscribers get that message instantly and go do their work.

Why is this awesome?

The publisher doesn't need to know who is listening.
The subscriber doesn't care who sent the message.
Everyone does their job independently, and our system becomes more flexible, maintainable, and resilient.

Why Bother with Fan Out?

So, what do we actually get out of this architecture?

Parallel Processing — Multiple tasks run at the same time (= faster).
Failure Isolation — One service crashing won’t take the others down.
Decoupling — Easier to maintain, test, and deploy our services.

My Example: Notification System with CDK

I built a simple notification system using AWS CDK (in TypeScript). Here’s what it includes:

An SNS topic (NotificationTopic)
Three SQS queues are subscribed to this topic (EmailQueue, PushQueue, SMSQueue)
Three Lambda functions to handle each queue
Infra to wire them all together

A real-world example would be when a user signs up for your platform, your backend sends a single message to the NotificationTopic.

That message fans out to:

EmailQueue → triggers Email Lambda → sends welcome email
PushQueue → triggers Push Lambda → sends push notification to the device
SMSQueue → triggers SMS Lambda → sends text message to user

Each Lambda gets messages in batches from its queue and does its thing, completely unaware of the others.

Time to get into the good stuff: the code

1. Defining the SNS Topic

export const createNotificationTopic = (scope: Construct) => {
  return new Topic(scope, 'NotificationTopic', {
    displayName: 'FanoutNotificationTopic',
  });
};

This is the central piece of our fan-out system. Any message published here will be broadcast to all the subscribers.

2. Creating the SQS Queues

Here we have defined a reusable function that creates named SQS queues:

export const createNotificationQueue = (scope: Construct, name: string) => {
  return new Queue(scope, `${name}Queue`, {
    queueName: `${name.toLowerCase()}-notification-queue`,
    visibilityTimeout: Duration.seconds(30),
  });
};

We use this function to spin up EmailQueue, PushQueue, and SMSQueue in our stack.

3. Creating Lambda Consumers

Here we have defined a function that creates Lambda consumers for each SQS queue:

export const createNotificationConsumer = (
  scope: Construct,
  handlerPath: string,
  queue: Queue,
  name: string
) => {
  const lambdaFn = new NodejsFunction(scope, `${name}ConsumerFn`, {
    entry: handlerPath,
    runtime: Runtime.NODEJS_18_X,
    handler: 'handler',
    timeout: Duration.seconds(10),
  });

  queue.grantConsumeMessages(lambdaFn);
  lambdaFn.addEventSource(new SqsEventSource(queue, { batchSize: 5 }));

  return lambdaFn;
};

Each Lambda uses a separate handler file (emailHandler.ts, pushHandler.ts, smsHandler.ts) and consumes messages in batches from its corresponding queue.

4. Hooking Everything Together

Now, in the main stack, we wire all the resources together:

const topic = createNotificationTopic(this);

const emailQueue = createNotificationQueue(this, 'Email');
const pushQueue = createNotificationQueue(this, 'Push');
const smsQueue = createNotificationQueue(this, 'SMS');

// Subscribe queues to the topic
topic.addSubscription(new SqsSubscription(emailQueue));
topic.addSubscription(new SqsSubscription(pushQueue));
topic.addSubscription(new SqsSubscription(smsQueue));

// Hook up Lambdas
createNotificationConsumer(this, './src/lambda/emailHandler.ts', emailQueue, 'Email');
createNotificationConsumer(this, './src/lambda/pushHandler.ts', pushQueue, 'Push');
createNotificationConsumer(this, './src/lambda/smsHandler.ts', smsQueue, 'SMS');

Now, publishing a message to the topic will send it to all three queues, and each queue will invoke its Lambda to handle the message.

Full Project on GitHub!

If you want to see the full code, clone it, tweak it, or just poke around, I’ve uploaded everything to GitHub:

Github Repo

The repo includes:

Full CDK setup (SNS + SQS + Lambda)
Queue + consumer definitions
Sample handler code
Clean folder structure

Feel free to fork it, star it, or open issues if you run into anything or want to suggest improvements.

Wrapping Up

Fan-out architecture is a solid pattern for building event-driven, decoupled systems that scale. With AWS SNS and SQS, it’s super easy to implement, and with CDK, it’s even easier to define in code.

This system is a great base for building things like:

Notification systems
Audit/event logs
Data processing pipelines
Multi-tenant service triggers

One last thing, don’t forget to destroy your AWS resources when you’re done testing.

“Always clean up unless your hobby is funding Jeff Bezos’ next rocket launch.”

DEV Community