DEV Community: Oleksandr Hanhaliuk

Scaling AWS FIFO SQS (queues) without blocking customers

Oleksandr Hanhaliuk — Mon, 16 Mar 2026 20:14:30 +0000

Let’s imagine you have two services (for example, on AWS Lambda). Service 1 communicates with Service 2 via commands: messages in FIFO SQS (commands VS events).

FIFO (First-In-First-Out) is necessary in this situation to ensure the order of command processing in Service 2 (regular AWS SQS does not guarantee the order of the message queue, and causes duplication).

To achieve parallelism, you can use message groups (message group ID), where each group is processed sequentially, but several groups are processed in parallel. This allows you to have separate “mini-queues” for different message sources, for example, for each customer.

Let’s imagine you have 2 customers, for each of which you transfer messages from Service 1 to Service 2:

Customer A: 1,000 commands per hour.*Customer *B: 500,000 commands per hour.

Each of them has its own message group ID (which ensures that they do not block each other). Service 2 can process approximately 100,000 commands per hour. Thus, Customer A’s messages should, in theory, be processed within that hour, as they are not blocked.

This would indeed be the case if it weren’t for the AWS FIFO SQS limitation: the queue only looks at the first 120,000 messages (until recently it was 20,000) to find available groups for processing.

Solution

In this case, it is necessary to create an architecture that will not allow one Customer to block another while maintaining the conditional order of messages. There are several solutions to this:

Store messages in additional storage if SQS has more than X messages in the queue
Make the processing speed on Service 2 faster
Dynamically create an additional FIFO SQS for a group of messages with high traffic

I will describe the last scenario below, as I have experience creating this type of architecture.

Architecture description

Tracking SQS load:

Each message adds a counter in DynamoDB for the corresponding userId.
After processing the message, Service 2 sends an SNS message about the completion of processing, to which Service 1 is subscribed — the counter in DynamoDB decreases.

Separate queue allocation:

If a Customer exceeds X (e.g., > 100,000) active messages, a separate FIFO SQS is created for it (using AWS SDK SQS Client).
All new messages from this Customer are redirected to this queue.
A separate trigger for Lambda is added to Service 2, which processes this queue (using AWS SDK Lambda client). PS: this fact violates the SoC principle, since the trigger for Lambda belongs to Service 2. However, you can optimise the architecture and create a separate service for monitoring SQS and creating triggers for Lambda.

Cleaning

All records in DynamoDB have a TTL of 1 day.
After it expires, the record is deleted.
The deletion event triggers Lambda
The newly created FIFO SQS queue and its Lambda trigger are deleted.

Pros

Real-time SQS load monitoring
Guaranteed non-blocking for customers with moderate traffic

Cons

An additional DynamoDB database and business logic on Lambda requires implementation, maintenance, and costs
Simpler implementations based on CloudWatch Alarms are possible, but they will have a delay in creating additional AWS FIFO SQS

Violation of the SoC (Separation of Concerns) principle: because Service 1 creates Lambda triggers in Service 2.

Conclusion

If you have a system where different customers can have very different loads, but you need to maintain message order, it is important to have a mechanism for dynamic traffic separation. The FIFO limit of 120,000 messages is not obvious but critical.

This approach ensures stability and independence between Customers even during peak periods.

I Built a “AWS CDK NodejsFunction”… but for AWS Fargate

Oleksandr Hanhaliuk — Sat, 14 Feb 2026 17:36:41 +0000

Deploying a NodeJS application on a Fargate shouldn’t be harder than deploying it on Lambda. But it usually does.

I worked with ECS Fargate for years and I know the pain: Dockerfiles, ECR, task definition, container definition, IAM roles, networking, logging, autoscaling. None of it is wrong, but its just repeating boilerplate. You can find full code example of Fargate Service on NodeJS here.

At the same time for Lambda its pretty simple — you just use package AWS provided for us https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs.NodejsFunction.html

This frustration led me to build fargate-nodejs — open source abstraction that helps developers create Fargate NodeJS service with just one Construct.

Idea: Lambda Style, Fargate Runtime

The core idea behind fargate-nodejs is simple:

You provide a Node.js entry file
The construct creates a production-ready Fargate service

No Dockerfile, no manual writing of ECS resources.
The goal is to hide most repetitive decision.

High-Level Architecture

Here’s how the construct works internally (simplified):

Your Node.js code is bundled using esbuild

A minimal container image is created automatically

An ECS task definition and Fargate service are generated

IAM roles and CloudWatch logs are configured

Optional integrations: Application Load Balancer (for HTTP services), Autoscaling, SQS-based workers

The result is standard AWS infrastructure, just created through a higher-level abstraction.

Example: HTTP Service on Fargate

A minimal Express app:

import express from 'express';

const app = express();

app.get('/', (_, res) => {
  res.send('Hello from Fargate');
});

import { FargateNodejsService } from 'fargate-nodejs'

new FargateNodejsService(this, 'MyService', {
  entry: './src/index.ts',
  containerPort: 3000,
});

When This Abstraction Makes Sense

This approach works best when:

Lambda feels too restrictive (timeouts, execution model)
You want long-running Node.js processes
Docker feels like unnecessary overhead
You’re already using AWS CDK
You value consistency and developer experience

It’s not intended for:

very custom container setups
complex, hand-tuned ECS clusters
Kubernetes-style workflows The abstraction is intentionally opinionated.

Full Walkthrough Video

I recorded a full video walkthrough where I:

walk through the code
show the deployment
and inspect what AWS actually creates in the console
test few use cases

Watch the full video:

Open Source & Feedback

The project is open source, and feedback is welcome:

GitHub repo: https://github.com/alexsanteenodev/fargate-nodejs

NPM package: https://www.npmjs.com/package/fargate-nodejs

AWS Bedrock Magic Summarizes Any Video Fast!

Oleksandr Hanhaliuk — Sun, 03 Aug 2025 16:45:06 +0000

AWS MCP and Cursor IDE Together Make AWS Simple!

Oleksandr Hanhaliuk — Sun, 08 Jun 2025 14:28:27 +0000

I show how to connect AWS’s new Model Context Protocol (MCP) to Cursor IDE and cut a 65-line CDK stack down to 12 lines.
In 8 minutes you’ll see:
• What MCP is in plain terms
• How to hook AWS MCP into Cursor IDE in 2 steps
• Side-by-side code before VS after MCP
• Bonus: AWS Cost Estimation using AI and AWS MCP

If you use AWS, this is worth a quick watch.

How to Reduce Your AWS Bill: 3 Hidden Costs You Might Be Ignoring

Oleksandr Hanhaliuk — Fri, 28 Mar 2025 12:06:50 +0000

I just published a quick YouTube video showing 3 easy fixes to lower your AWS costs on production workloads:

✅ Disable AWS X-Ray globally using Service Control Policies (SCP)
✅ Prevent unwanted Lambda logging to CloudWatch
✅ Reduce NAT Gateway costs with free VPC Endpoints for AWS services

Beat The Burst: Optimizing AWS ALB and ECS Fargate for Sudden Traffic Spikes

Oleksandr Hanhaliuk — Tue, 18 Mar 2025 21:25:37 +0000

AWS ECS Fargate is a serverless tool for managing your containers without the need to manage infrastructure. It is a very nice combination of configurable infrastructure that allows you to choose various CPU and memory numbers, provides robust autoscaling, and requires almost zero knowledge of DevOps.

However, in high-traffic systems with high bursts of traffic, autoscaling of Fargate can be slow, and this might lead to overloading of existing containers.

Problem

Imagine the following scenario:

You have an Application Load Balancer with a target group which targets to ECS service
ECS service has Fargate, which serves customers with a specific response
Fargate tasks contain an application that consumes a lot of resources (CPU and memory).
You have autoscaling based on CPU or connection per target or Task memory usage
When traffic suddenly spikes, the CloudWatch Alarm triggers Autoscaling.

Seems like a very common scenario for web applications, right? Nothing wrong here.

Hoverer when CloudWatch alarm triggers scaling to time Fargate task scaled and become healthy can be few minutes.

During this time, your existing tasks might be overloaded, which can lead to 503 responses to customers.

We don’t want that, right?

Solution

There are multiple solutions to this problem. Some of them we will cover in the next articles. In this article, we will cover the solution that allows traffic to be offloaded to Lambda during traffic spikes.

Diagram

Steps:

Create Lambda, which will serve traffic and have the same business as your Fargate container
Create an additional target group and point it to the Lambda
Add a target group to your Application Load Balancer listener:

EC2 -> Load balancers -> YourALB -> HTTPS:443 listener -> Edit listener -> Add target group
Create CloudWatch metrics which you want to rely on for traffic offload. It can be the following:
- Number of connections per target
- Average target response time
- ECS metrics, like CPU, Memory usage, etc.
- Try to configure metrics that predict traffic spikes as early as possible. For example, if the number of connections per target is increasing, we can suspect that the traffic spike is close.
Trigger Lambda on Cloud watch alarm

Previously, you would need to send an SNS message from an Alarm and trigger Lambda from SNS. However, recently, AWS introduced a new feature that allows you to trigger Lambda from alarms directly.
In your Lambda, write an application that updates the ALB listener weight using the AWS SDK or API (see docs). Add some traffic weight to the target group, which is configured in Step 1.

This will offload traffic from Fargate and give existing tasks some space until new tasks are Autoscaled.
Additionally, you can control Fargate autoscaling from this same Lambda in order to avoid conflicts with default scaling metrics. To do this, you can use SDK:

import {
  ECSClient,
  UpdateServiceCommand,
  DescribeServicesCommand,
  DescribeServicesCommandInput,
} from '@aws-sdk/client-ecs'


const describeServiceCommand = new DescribeServicesCommand({
    services: [ecsService],
    cluster: ecsCluster,
})
const serviceResponse = await escClient.send(describeServiceCommand)  

const desiredCount = serviceResponse.services[0].desiredCount

cosnt scalingUpDesiredCount = desiredCount + ${scalingSize}
cosnt scalingDownDesiredCount = desiredCount + ${scalingSize}

const updateParams = {
     service: ecsService,
     cluster: ecsCluster,
     desiredCount: scalingUpDesiredCount // or scalingDownDesiredCount,
}

const updateServiceCommand = new UpdateServiceCommand(updateParams)

await escClient.send(updateServiceCommand)

Summary

By setting up an extra target group for Lambda and adjusting traffic weights, we can keep things running smoothly until Fargate catches up. However, there remains a possibility of encountering 503 errors if our CloudWatch metrics fail to detect the traffic surge promptly.

In the next article, we will investigate how to use Lambda as a fallback for ECS Fargate

AWS Fargate and SQS for Long-Running Jobs with TypeScript

Oleksandr Hanhaliuk — Wed, 12 Mar 2025 22:02:26 +0000

This article will explain my experience in setting up the AWS ECS Fargate container which handles messages from SQS. The reason for having Fargate handle messages instead of lambda is that the message represents a long-running job which is usually longer than 15 minutes.

Setting up ECS Fargate infrastructure

Key Components Creation:

Cluster: A Fargate cluster is initialized to house the message processing tasks.
Task Definition: This definition specifies the computing resources required and points to the Dockerfile that contains the application code.
Fargate Service: Represents the service running on Fargate, facilitating public access and defining service scalability.
SQS Queue: An SQS queue is set up to hold the messages to be processed.

I. Import all needed libraries

import {
  Cluster,
  ContainerImage,
  FargateService,
  FargateTaskDefinition,
  LogDrivers,
  PropagatedTagSource,
} from 'aws-cdk-lib/aws-ecs'

import * as sqs from 'aws-cdk-lib/aws-sqs'

import { Duration, Stack } from 'aws-cdk-lib'
import {
  AdjustmentType,
  MetricAggregationType,
  ScalableTarget,
  StepScalingAction,
} from 'aws-cdk-lib/aws-applicationautoscaling'
import { Alarm, ComparisonOperator, MathExpression } from 'aws-cdk-lib/aws-cloudwatch'
import { ApplicationScalingAction } from 'aws-cdk-lib/aws-cloudwatch-actions'

II. Create cluster

// Create fargate cluster
  const cluster = new Cluster(stack, 'SQS_PROCESSING_CLUSTER', {
    // vpc: define your VPC here if needed,
    clusterName: 'SQS_PROCESSING_CLUSTER',
  })

III. Create task definition

  // Create a Task Definition for the container to start
  const taskDefinition = new FargateTaskDefinition(stack, 'SQS_PROCESSING_CONTAINER', {
    memoryLimitMiB: 4096,
    cpu: 1024,
  })
  taskDefinition.addContainer('SQS_PROCESSING_CONTAINER', {
    // Path to docker file in your local repository
    image: ContainerImage.fromAsset(`./src/`, { 
      file: 'Dockerfile',
    }),
    environment: {
       // put your environment
    },
    logging: LogDrivers.awsLogs({
      streamPrefix: '/fargate/sqs-processing/log-group',
      logRetention: aws_logs.RetentionDays.THREE_DAYS,
    }),
  })

IV. Create a Fargate service

  // Create a load-balanced Fargate service and make it public
  const service = new FargateService(stack, 'sqs-processing-service', {
    cluster: cluster, // Cluster from step 1
    taskDefinition: taskDefinition, // Task definition from step 2 
    serviceName: 'sqs-processing-service',
    assignPublicIp: true,
    minHealthyPercent: 100,
    maxHealthyPercent: 400,
    propagateTags: PropagatedTagSource.SERVICE,
    enableExecuteCommand: false,
  })

V. Create SQS

  const queue = new sqs.Queue(stack, 'sqs-id', {
    queueName: 'sqs-name',
    visibilityTimeout: Duration.seconds(300),
  })

VI. Allow Fargate to consume SQS message

queue.grantConsumeMessages(service.taskDefinition.taskRole)

Fargate autoscaling

To optimize resource utilization and costs, the setup includes autoscaling configurations. These ensure that the Fargate tasks scale up when messages are pending in the SQS queue and scale down once all messages have been processed or when no messages are present.

Scaling up

  const scalingTarget = service.autoScaleTaskCount({
    maxCapacity: 10,
    minCapacity: 0,
  })

  // Scale up when SQS has visible messages
  const numberOfMessagesVisibleMetric = queue.metricApproximateNumberOfMessagesVisible({
    period: Duration.seconds(30),
  })

  // Scale up if queue has messages
  scalingTarget.scaleOnMetric('QueueMessagesVisibleScaling', {
    metric: numberOfMessagesVisibleMetric,
    evaluationPeriods: 1,
    scalingSteps: [
      { upper: 0, change: 0 }, // DO not tear down task if there is no messages visible in SQS
      {
        lower: 1,
        upper: 5,
        change: +1,
      },
      {
        lower: 5,
        upper: 10,
        change: +5,
      },
      {
        lower: 10,
        change: +5,
      },
    ],
    cooldown: Duration.seconds(90),
  })

Scale down

Combine 3 metrics to avoid scaling down during message processing


  const messagesSentMetric = sqsQueue.metricNumberOfMessagesSent({
    period: Duration.seconds(30),
  })

  const messagesVisibleMetric = sqsQueue.metricApproximateNumberOfMessagesVisible({
    period: Duration.seconds(30),
  })

  const messagesNotVisibleMetric = sqsQueue.metricApproximateNumberOfMessagesNotVisible({
    period: Duration.seconds(30),
  })

  // Combine all metrics to create a custom metric using a math expression
  // The expression adds the 3 metrics, and we expect the sum to be zero to scale down.
  const customMetric = new MathExpression({
    expression: 'messagesSent + messagesVisible + messagesNotVisible',
    usingMetrics: {
      messagesSent: messagesSentMetric,
      messagesVisible: messagesVisibleMetric,
      messagesNotVisible: messagesNotVisibleMetric,
    },
    period: Duration.seconds(30),
  })

Create alarm and scalable target


  // Add an alarm to our Fargate service CPU usage
  const scaleInInit = new Alarm(stack, 'ScaleDownAlarm', {
    alarmDescription: 'When there is no messages in SQS, scale down to 0',
    metric: customMetric,
    evaluationPeriods: 10,
    datapointsToAlarm: 10,
    threshold: 0,
    comparisonOperator: ComparisonOperator.LESS_THAN_OR_EQUAL_TO_THRESHOLD,
  })

  // Define our auto scaling target for our Fargate service
  const scalableTarget = ScalableTarget.fromScalableTargetId(
    stack,
    'ZeroSQSMessage-ScalableTarget',
    `service/${service.cluster.clusterName}/${service.serviceName}|ecs:service:DesiredCount|ecs`
  )

Create step scaling action and alarm action


  // Define the action taken on our scaling target
  const scalingAction = new StepScalingAction(stack, 'scaleToZero', {
    scalingTarget: scalableTarget,
    adjustmentType: AdjustmentType.EXACT_CAPACITY,
    metricAggregationType: MetricAggregationType.MAXIMUM,
    cooldown: Duration.minutes(5),
  })

  // Create the adjustment made by our action
  scalingAction.addAdjustment({
    adjustment: 0,
    upperBound: 0,
  })

  // Finally add our alarm action to our Fargate alarm
  scaleInInit.addAlarmAction(new ApplicationScalingAction(scalingAction))

Bussines code implementation

Now that our infra is set, it is time to write business code to process messages

Docker file setup

The Dockerfile is carefully crafted to prepare the Node environment, execute the necessary installations, and set the application to run indefinitely, constantly listening for SQS messages to process.

FROM --platform=linux/amd64 node:18 AS appbuild
WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY ./ /app/

RUN npm run build

FROM node:18

WORKDIR /app
COPY --from=appbuild /app/ .
RUN npm prune production
ENV NODE_ENV=production


CMD ["node",  "/app/fargate/index.js" ]

Main business index.ts

A TypeScript file (index.ts) defines the core logic for message reception, processing, and deletion. This involves interacting with the SQS service through AWS SDK commands, specifically focusing on receiving and deleting messages.

import 'dotenv/config'
import { Message } from '@aws-sdk/client-sqs'

export async function runApp() {
  while (true) { // here we can also add delay
    await processMessages()
  }
}

export async function processMessages() {
  let messages: Message[] | undefined
  try {

    const command = new ReceiveMessageCommand({
      QueueUrl: 'sqs-url',
      WaitTimeSeconds: 20, // time to consume messages
      MaxNumberOfMessages: 1,
    })

    const response = await sqsClient.send(command)

    messages = response.Messages
    if (!messages || messages.length === 0) {
      console.warn('No messages received!')
      return
    }

    console.log('Received messages:', { messages })
    for (const msg of messages) {
      try {
        if (!msg.Body) {
          throw new Error('No body in message')
        }
        // here you need to handle you SQS message
      } catch (e) {
        console.error('Error processing message', { error: e, message: msg })
        // Optionally you can send message to Dead-letter SQS
      }

      // Now message is processed, we need to delete it
        const deleteCommand = new DeleteMessageCommand({
          QueueUrl: 'sqs-url',
          ReceiptHandle: message.ReceiptHandle,
        })

        await sqsClient.send(deleteCommand)
    }
  } catch (e) {
    // Uncaught error
    throw new Error('Error processing messages')
  }
}

void runApp()

Summary

This article outlines how to set up an AWS ECS Fargate container to process long-running jobs using messages from an SQS queue, an approach especially useful when tasks exceed the 15-minute execution limit of AWS Lambda. It covers the initial steps of setting up the ECS Fargate infrastructure, including creating a Fargate cluster, task definition, service, and SQS queue, followed by enabling autoscaling based on the queue’s workload. Additionally, it describes implementing business logic within a Docker container to continuously process and manage SQS messages. This setup leverages the scalability and flexibility of AWS services to efficiently handle long-running tasks while optimizing resource usage and costs.

Additional Considerations

When implementing this solution, consider monitoring and alerting mechanisms to keep track of the system’s health and performance. Employing AWS CloudWatch can provide insights into task execution metrics and trigger alarms in case of anomalies. Additionally, exploring the integration of dead-letter queues (DLQs) could further bolster the system’s resilience by handling messages that cannot be processed successfully.

Mastering CQRS: Command Query Responsibility Segregation in Modern Applications - Software Architecture Patterns

Oleksandr Hanhaliuk — Fri, 21 Feb 2025 15:07:27 +0000

Introduction

Modern applications demand scalability, performance, and flexibility. One common challenge in both monolithic and microservices architectures is handling reads and writes efficiently.

CQRS (Command Query Responsibility Segregation) is a software design pattern that separates read and write operations to optimize performance, scalability, and security.

In this article, we explore how CQRS works, its real-world microservice applications, and how it can be implemented on cloud platforms like AWS.

1. Why CQRS? The Problem with Traditional CRUD

In traditional CRUD-based architectures:

The same database schema is used for both reading and writing.
Complex joins and transactions slow down queries.
High-volume reads impact write performance, making scaling difficult.
Security concerns arise when sensitive data is exposed to read operations.

💡 CQRS solves these issues by splitting reads and writes into separate models!

2. Understanding CQRS

CQRS divides the system into two distinct models:

1. Command Model (Write Operations)

Handles data modifications: Create, Update, Delete.
Uses a normalized schema optimized for transactions.
Ensures strong consistency.
Commands don't return data, only success/failure.

2. Query Model (Read Operations)

Handles data retrieval: Get, List, Search.
Uses denormalized views optimized for fast reads.
Can be cached and scaled independently.
No side effects, ensuring read efficiency.

By separating writes (commands) from reads (queries), CQRS enables performance and security improvements.

3. How CQRS Works

User sends a command (e.g., Create Order).
Command service updates the database.
An event is published (e.g., OrderCreated).
A separate read model is updated asynchronously.
User queries the read model for updated data.

💡 Commands modify data, queries fetch data – both optimized separately!

4. CQRS with Event Sourcing

CQRS often works with Event Sourcing, where:

Instead of updating a row, each change is stored as an immutable event.
Events rebuild the state in real-time.
Provides auditability and rollback support.

📌 Example:

UserCreated → { "id": 1, "name": "John" }
UserUpdated → { "id": 1, "name": "John Doe" }

System state is reconstructed by replaying events!

5. CQRS on AWS

CQRS can be implemented using AWS services:

Write Model (Commands):

API Gateway + Lambda (or EC2/ECS)
Amazon RDS
EventBridge / SNS / SQS for event-driven processing

Read Model (Queries):

DynamoDB / ElastiCache / OpenSearch
API Gateway + Lambda for fast retrieval

🚀 Example CQRS Implementation on AWS:

Command → API Gateway → Lambda → RDS → EventBridge
Query   → API Gateway → Lambda → DynamoDB

6. CQRS with Materialized Views and a Single Database

CQRS does not always require separate databases. A single database can still benefit from CQRS using Materialized Views.

How it Works:

Commands (writes) modify normalized tables in the database.
A Materialized View (MV) is used to store precomputed read models.
Queries (reads) access the Materialized View instead of complex joins.
The Materialized View updates periodically or via triggers.

📌 Example:

Write Table: Orders Table
Materialized View: Orders Read Model

7. CQRS vs. Traditional CRUD

Feature	Traditional CRUD	CQRS
Single Schema	✅ Yes	❌ No
Performance Issues	✅ Yes	❌ No
Scalability	❌ Low	✅ High
Complexity	✅ Low	❌ Higher
Event-Driven	❌ No	✅ Yes

💡 CQRS trades complexity for scalability and performance!

Conclusion

The CQRS pattern is an essential tool for high-scale applications, ensuring efficient reads and writes while enabling performance, security, and flexibility.

🚀 Have you implemented CQRS in production? Share your thoughts below!

🔔 Follow for more insights on software architecture and cloud computing!

Ensuring Data Consistency with the Transactional Outbox Pattern on AWS

Oleksandr Hanhaliuk — Fri, 14 Feb 2025 21:53:02 +0000

Introduction

In distributed systems, ensuring data consistency across services is crucial. When database transactions and message queues are not atomic, failures can cause inconsistencies.

The Transactional Outbox Pattern ensures database writes and message publishing occur atomically, avoiding lost events.

1. The Problem: Database and Messaging Are Not Atomic

Imagine an Order Service that:
Writes order data to Amazon RDS (PostgreSQL/MySQL).
Publishes an event (OrderPlaced) to Amazon SQS for other services (e.g., Inventory, Shipping).

🔥 The Failure Scenario:
The service writes an order to RDS ✅
The service fails to publish OrderPlaced to SQS ❌ (network error, timeout, etc.)

💥 Problem: The order is stored in the database, but other services (Inventory, Shipping) never receive the event.
🚨 Data inconsistency can lead to:
Customers being charged without order fulfillment.
No inventory updates, leading to overselling.
No shipping triggered, leading to support escalations.

2. The Transactional Outbox Pattern Solution

🔹 Step 1: Define Outbox Table

CREATE TABLE order_outbox (
    id SERIAL PRIMARY KEY,
    order_id UUID NOT NULL,
    event_type TEXT NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    processed_at TIMESTAMP NULL
);

🔹 Step 2: Write Order & Event in One Transaction (TypeScript)

import { Pool } from 'pg';
const pool = new Pool({ connectionString: "your-db-connection" });

async function placeOrder(orderId: string, customerId: string, totalPrice: number) {
    const client = await pool.connect();
    try {
        await client.query('BEGIN');
        await client.query("INSERT INTO orders (order_id, customer_id, total_price) VALUES ($1, $2, $3)",
            [orderId, customerId, totalPrice]);
        await client.query("INSERT INTO order_outbox (order_id, event_type, payload) VALUES ($1, $2, $3)",
            [orderId, "OrderPlaced", JSON.stringify({ orderId, customerId, totalPrice })]);
        await client.query('COMMIT');
    } catch (error) {
        await client.query('ROLLBACK');
        throw error;
    } finally {
        client.release();
    }
}

🔹 Step 3: Poll & Publish Events (AWS Lambda / ECS Task)

import { Pool } from 'pg';
import { SNS } from 'aws-sdk';
const pool = new Pool({ connectionString: "your-db-connection" });
const sns = new SNS();
const topicArn = "arn:aws:sns:us-east-1:your-account-id:order-events";

async function processOutbox() {
    const client = await pool.connect();
    try {
        const { rows } = await client.query("SELECT id, payload FROM order_outbox WHERE processed_at IS NULL");
        for (const event of rows) {
            try {
                await sns.publish({ TopicArn: topicArn, Message: JSON.stringify(event.payload) }).promise();
                await client.query("UPDATE order_outbox SET processed_at = NOW() WHERE id = $1", [event.id]);
            } catch (error) {
                console.error(`Failed to send event ${event.id}:`, error);
            }
        }
    } catch (error) {
        console.error("Error processing outbox:", error);
    } finally {
        client.release();
    }
}

4. Why We Can't Just Rollback Transaction with SNS

🔹 SNS and Databases Don't Share Transactions

Databases (RDS, PostgreSQL, MySQL) support ACID transactions.
Amazon SNS is an external service and cannot be part of the same transaction.

🔹 Failure Scenarios Without an Outbox

DB Insert Succeeds, SNS Fails:
- Order is stored in RDS ✅
- SNS message fails ❌
- Other services never receive the event!
SNS Message Sent, DB Insert Fails:
- SNS message sent ✅
- DB insert fails ❌
- Other services see an event for an order that does not exist!

🔹 What Should Happen in Case of Failure?

If SNS fails: The event remains in the outbox and can be retried later.
If DB transaction fails: Nothing is committed, ensuring consistency.
Retries & Dead-letter Queue (DLQ): Failed SNS messages can be moved to a DLQ for further debugging.

5. Real-World Use Cases

E-commerce Order Processing: Ensuring orders are reliably propagated to inventory, billing, and shipping services.
Payment Processing: Making sure transactions trigger correct notifications and reports.
User Signups: Ensuring emails or notifications are reliably sent after new user registration.
IoT Device Events: Ensuring telemetry data from devices is consistently published.

6. Difference from Saga Pattern

Feature	Transactional Outbox	Saga Pattern
Approach	Ensures event publishing is atomic	Splits transactions into smaller steps
Use Case	Ensures reliable messaging	Handles multi-step business processes
Complexity	Simpler	More complex
Example	Reliable event publishing to SNS	Hotel & flight booking with rollback steps

Conclusion

The Transactional Outbox Pattern ensures event consistency by making database transactions and event publishing atomic. Implementing it with Amazon RDS, SNS, and Lambda/ECS provides a reliable, scalable, and decoupled solution.

🚀 Have you implemented this pattern? Share your experience in the comments!

🔔 Follow for more AWS architecture insights!

Serverless MapReduce for Excel: Scale Your Marketing Data with AWS

Oleksandr Hanhaliuk — Sat, 08 Feb 2025 18:27:08 +0000

Introduction

MapReduce is a programming model for processing large datasets in parallel. It splits the input data into chunks (Map), then combines or aggregates results (Reduce).

Map: Break down data into smaller parts.
Shuffle/Sort: Group related data.
Reduce: Aggregate or combine into final results.

Let's learn the Map-Reduce pattern with a real-world example: Event-Driven Serverless “MapReduce” AWS Architecture for Excel-Based Marketing Campaign Analytics.

1. Overview

If you’re dealing with Excel sheets full of marketing metrics (e.g., campaigns, CPC, revenue), this AWS serverless pipeline helps process and aggregate data automatically—no cluster management needed.

Key Steps:

Upload Excel: A marketing manager uploads a spreadsheet to an Amazon S3 bucket.
Map Lambda: Parses each row (date, campaign, source, cost, etc.) and saves intermediate results.
Reduce Lambda: Aggregates partial data into a final report for analytics or dashboards.

2. Architecture Flow

Excel File Upload: The marketing manager or an automated process places the Excel file into an S3 bucket.
Map Lambda: Triggered by an S3 event. It reads and parses each row, storing partial outputs in S3.
Reduce Lambda: Triggered by a subsequent event or schedule. Collects all partial results, aggregates them, and writes the final report to S3 or a database.

3. Step-by-Step

User uploads an Excel file with marketing data to an S3 bucket.
Map Lambda is triggered by an S3 event, processes each row, and stores intermediate data.
Reduce Lambda aggregates data across different marketing sources into a final report.
The processed report can be stored in S3 or used for visualization.

4. Key Benefits

✅ Serverless: No servers or clusters to maintain.

✅ Cost-Effective: Only pay for Lambda execution and minimal S3 usage.

✅ Automated Data Ingestion: Triggers when an Excel file is uploaded.

✅ Decoupled Architecture: Easily modify or extend each step.

5. Next Steps

Add validation/error handling in the “Map” phase for missing columns or invalid data.
Implement notifications (e.g., email or Slack) when final reports are generated.
Integrate with dashboard tools (e.g., QuickSight) to visualize aggregated marketing metrics.

6. Example Transformation Flow

Below is a simple example of how a single row from the Excel file is transformed during the Map step, and then combined in the Reduce step.

🔹 Input Excel Row

🔹 Map Output (Intermediate JSON)

{
  "date": "2025-01-07",
  "campaign": "WinterSales",
  "source": "Google AD",
  "impressions": 19394975,
  "clicks": 3878995,
  "cost": 8533789,
  "orders": 46935900,
  "revenue": 89216885,
  "cpc": 2.2
}

📂 (This might be stored in an S3 path /{date}/campaign/.)

🔹 Reduce Step
If there are multiple entries for the same date and campaign (e.g., different sources like SEO, Social, etc.), the Reduce Lambda will sum or aggregate values across all partial outputs.

🔹 Example Final Excel Report

Date,Campaign,TotlImpressions,TotalClicks,TotalCost,TotalOrders,TotalRevenue
2025-02-07,WinterSale,30000,500,340.0,16,1120.0

(Here, we combined data from AdWords, SEO, and other sources for WinterSale on 2025-02-07.)

7. Why Not Just Use Excel for Data Transformation?

For small, ad-hoc data tasks, Excel works. But a serverless, automated approach is ideal when you:

🚀 Need consistent transformations across multiple or frequently updated files.
📈 Require scalability (large datasets slow Excel and risk data limits).
📊 Want integration with dashboards, notifications, or further data pipelines.
🔍 Value version control and repeatable, automated processes.

8. What About AWS Glue?

🔹 AWS Glue
A fully managed ETL service, built on Apache Spark.
✅ Great for big data & complex transformations.
✅ Schema discovery & automatic scaling.
❌ More overhead than Lambda-based solutions.

🔹 Lambda-Based MapReduce
🚀 Lightweight and cost-effective for small-to-medium datasets.
✅ Easier to set up for frequent, structured Excel processing.
✅ You fully control the transformation logic.

Bottom Line:
If your datasets are huge or you need advanced ETL features, AWS Glue is better.
For lighter, serverless jobs, a Lambda-based MapReduce approach is simpler and cheaper.

💡 Would you like a tutorial on deploying this architecture step-by-step? Drop a comment!
🚀 Follow me for more AWS and data engineering content!

Ensuring Data Consistency in Microservices with the Saga Pattern (AWS DynamoDB & PostgreSQL)

Oleksandr Hanhaliuk — Fri, 07 Feb 2025 20:48:51 +0000

Introduction

The Saga pattern is a microservices architectural approach that coordinates a series of local transactions across different services through a sequence of events or commands. It aims to maintain data consistency without relying on a monolithic, long-running transaction. Instead, each microservice handles its own local database changes and publishes events to trigger subsequent steps.

Saga Theory

Why Avoid Two-Phase Commit (2PC)?

In a distributed environment, using 2PC can become overly complex, especially at scale. The Saga pattern counters this by breaking a single large transaction into a series of smaller local transactions. Each service:

Performs a local transaction.
Publishes an event (or sends a message).
If a subsequent step fails, the pattern invokes compensating transactions to roll back prior steps.

Two Main Approaches

Choreography: Each service produces and listens to domain events from other services.
Orchestration: A centralized controller (the Orchestrator) calls each service in turn and manages rollbacks if something fails.

Either approach aims to ensure data consistency across services without a global lock or a single transaction manager.

Why Use the Saga Pattern?

Data Consistency: Keep each service’s database in sync with the overall business process.
Scalability: Each microservice can be scaled independently.
Fault Tolerance: Isolate failure and roll back or compensate where necessary.
Loose Coupling: Services interact via events or commands, reducing direct dependencies.

Example Setup

Consider a typical e-commerce workflow:

Order Reservation Service (Service 1)
Payment Service (Service 2)
Shipment/Inventory Service (Service 3)

Each service uses:

AWS Lambda for compute.
Amazon SQS for inter-service communication.
Its own database (e.g., DynamoDB or PostgreSQL).

DynamoDB vs. PostgreSQL

For illustration:

Order Reservation Service uses DynamoDB.
Payment Service uses RDS (PostgreSQL).
Shipment/Inventory Service also uses RDS (PostgreSQL).

Flow Overview

1. Customer Request

The customer triggers an API call (via API Gateway) to place an order.

2. Order Reservation Service (DynamoDB)

Triggered by the API Gateway call (runs on AWS Lambda).
Persists the order record to its DynamoDB table.
Returns a “reservation pending” response to the customer.
Sends an SQS message to the Payment Service.

3. Payment Service (PostgreSQL)

Triggered by the SQS message.
Attempts to process payment (updates a PostgreSQL DB).
On success, sends an SQS message to Shipment/Inventory Service.
On failure, sends a rollback request to the Order Reservation Service.

4. Shipment/Inventory Service (PostgreSQL)

Triggered by the SQS message from Payment.
Confirms shipping and updates inventory (PostgreSQL).
On success, notifies Order Reservation Service of completion.
On failure, triggers compensating actions: Payment is reversed, Order Reservation is updated to “Canceled.”

Handling Transactions and Rollbacks

DynamoDB (Order Reservation)

Local Transactions: DynamoDB supports Transaction Write Items, allowing an atomic batch of writes/updates/deletes.
Rollback Strategy:
- Use a subsequent transactional write (or delete) to revert the item.
- Conditional checks to ensure you only rollback items in a pending state.
- Idempotency can be managed using item versioning or conditional expressions.

PostgreSQL (Payment, Shipment/Inventory)

Local Transactions: PostgreSQL provides standard SQL transactions with BEGIN, COMMIT, ROLLBACK.
Rollback Strategy:
- If failure occurs during an open transaction, ROLLBACK.
- If the transaction has already committed (e.g., after a credit card charge), a compensating transaction (like a refund) might be necessary.

Detailed Scenarios

1. All Services Succeed

Customer places an order (API Gateway → Lambda).
Order Reservation writes to DynamoDB.
Payment processes the payment with a PostgreSQL local transaction.
Shipment/Inventory updates inventory in its PostgreSQL transaction.
Order Reservation finalizes the order after receiving a success message.

2. Failure in Payment Service

Order Reservation completes successfully and sends a message to Payment.
Payment fails (e.g., insufficient funds).
Payment either rolls back the open transaction or issues a compensating refund.
Payment notifies Order Reservation.
Order Reservation updates DynamoDB to mark the order as “Canceled.”

3. Failure in Shipment/Inventory

Order Reservation and Payment succeed.
Shipment/Inventory fails (DB error, out of stock, etc.).
Shipment/Inventory rolls back the transaction if still open.
Notifies Payment (for refund if payment was already captured) and Order Reservation (to cancel the order).

4. Other Potential Failures

Timeouts: If Shipment/Inventory never responds, Order Reservation can define a timeout strategy.
Partial Failures: Payment might succeed but fail to write to its DB. Idempotency is critical to handle retries.
Dead Letter Queues (DLQs): Repeatedly failing messages can land in a DLQ for manual review.

Error Handling and Compensation

Local Rollbacks: Each service can undo its changes if notified of an upstream failure.
DynamoDB: Use a follow-up transactional write or delete.
PostgreSQL: Use ROLLBACK or a compensating UPDATE/DELETE if already committed.
Orchestration Logic: In the Orchestration approach, the orchestrator (e.g., the Order Reservation service) can track the overall saga state, sending compensating commands when needed.
Choreography: Each service publishes domain events for the others to consume.

Best Practices

Idempotency: Ensure repeated messages or API calls don’t create duplicate transactions.
Monitoring & Alerting: Use Amazon CloudWatch metrics (errors, queue size, latencies).
Security: Employ least-privilege IAM roles for each microservice.
Automated Tests: Integration, partial failure, and timeout tests are crucial for confidence.
DynamoDB Transactions: Use Transaction Write for atomic updates when you need them.
PostgreSQL ACID: Leverage strong consistency within each service boundary.

Conclusion

The Saga pattern is a powerful strategy for orchestrating and maintaining data consistency in distributed microservices. By structuring each step as a local transaction and employing events or an orchestrator, you avoid the pitfalls of large, complex global transactions. Whether you use DynamoDB transactions or PostgreSQL’s ACID properties, the key lies in designing robust compensating mechanisms for failures. With careful planning — including proper rollbacks, timeouts, and idempotent messages — the Saga pattern can reliably handle complex workflows such as e-commerce order processing.

Thanks for reading! If you enjoyed this guide, feel free to drop a comment or share how you implement Sagas in your own applications.

Single Responsibility Principle in Javascript

Oleksandr Hanhaliuk — Fri, 06 Dec 2024 16:31:03 +0000

Understanding the Single Responsibility Principle in JavaScript
When writing clean, maintainable code, one of the most important principles to follow is the Single Responsibility Principle (SRP). It’s one of the five SOLID principles in software development and ensures that your code is easier to read, test, and modify.

What is the Single Responsibility Principle?

The Single Responsibility Principle according to Robert C.Martin states that:

A class or function should have one and only one reason to change.

In simpler terms, each unit of your code (be it a function, class, or module) should be responsible for doing one thing and doing it well. When responsibilities are separated, changes in one area of your code won't unexpectedly affect others, reducing the risk of bugs and making your application easier to maintain and test.

Why is SRP Important?

Without SRP, you might face problems like:

Tangled Dependencies: When a function or class has multiple responsibilities, making a change to one can inadvertently break another.
Reduced Testability: Testing becomes harder when a unit of code does too much, as you’d need to mock unrelated dependencies. Poor Readability: Large, multi-purpose functions or classes are harder to understand, especially for new developers joining the project.
Difficult Maintenance: The more responsibilities a unit has, the more effort it takes to isolate and fix bugs or add new features.

Applying SRP in JavaScript

Let’s look at some practical examples of applying SRP in JavaScript.

Example 1: Refactoring a Function

Without SRP

function handleUserLogin(userData) {
    // Validate user data
    if (!userData.email || !userData.password) {
        logger.error("Invalid user data");
        return "Invalid input";
    }

    // Authenticate user
    const user = authenticate(userData.email, userData.password);
    if (!user) {
        console.error("Authentication failed");
        return "Authentication failed";
    }

    // Log success
    console.info("User logged in successfully");
    return user;
}

This function does too much: validation, authentication, and logging. Each of these is a distinct responsibility.

With SRP

We can refactor it by breaking it into smaller, single-purpose functions:

function validateUserData(userData) {
    if (!userData.email || !userData.password) {
        throw new Error("Invalid user data");
    }
}

function authenticateUser(email, password) {
    const user = authenticate(email, password); // Assume authenticate is defined elsewhere
    if (!user) {
        throw new Error("Authentication failed");
    }
    return user;
}

function handleUserLogin(userData, logger) {
    try {
        validateUserData(userData);
        const user = authenticateUser(userData.email, userData.password);
        logger.info("User logged in successfully");
        return user;
    } catch (error) {
        logger.error(error.message);
        return error.message;
    }
}

Now, each function has a single responsibility, making it easier to test and modify.

Example 2: Refactoring a Class

Without SRP

A class managing multiple concerns:

class UserManager {
    constructor(db, logger) {
        this.db = db;
        this.logger = logger;
    }

    createUser(user) {
        // Save user to DB
        this.db.save(user);
        this.logger.info("User created");
    }

    sendNotification(user) {
        // Send email
        emailService.send(`Welcome, ${user.name}!`);
        this.logger.info("Welcome email sent");
    } 
}

Here, UserManager handles user creation, logging, and sending emails—too many responsibilities.

With SRP

Refactor by delegating responsibilities to other classes or modules:

class UserService {
    constructor(db) {
        this.db = db;
    }

    createUser(user) {
        this.db.save(user);
    }
}

class NotificationService {
    sendWelcomeEmail(user) {
        emailService.send(`Welcome, ${user.name}!`);
    }
}

class UserManager {
    constructor(userService, notificationService, logger) {
        this.userService = userService;
        this.notificationService = notificationService;
        this.logger = logger;
    }

    createUser(user) {
        this.userService.createUser(user);
        this.notificationService.sendWelcomeEmail(user);
        this.logger.info("User created and welcome email sent");
    }
}

Each class now focuses on a single concern: persistence, notification, or logging.

Tips for Following SRP

Keep Functions Short: Aim for functions that are 5–20 lines long and serve one purpose.
Use Descriptive Names: A good function or class name reflects its responsibility.
Refactor Often: If a function feels too big or hard to test, split it into smaller functions.
Group Related Logic: Use modules or classes to group related responsibilities, but avoid mixing unrelated ones.

Conclusion

The Single Responsibility Principle is a cornerstone of clean code. By ensuring that every function, class, or module has only one reason to change, you make your JavaScript code more modular, easier to test, and simpler to maintain.

Start small—pick one messy function or class in your current project and refactor it using SRP. Over time, these small changes will lead to a significant improvement in your codebase.