DEV Community: Axel Dlv

How to Handle DynamoDB Reserved Keywords in Java (SDK v1 and v2)

Axel Dlv — Fri, 17 Oct 2025 08:32:59 +0000

TL;DR - Handling DynamoDB Reserved Keywords

The Problem: DynamoDB has reserved keywords (like order, name, status) that can't be used directly in queries, causing AmazonDynamoDBException errors.

The Solution: Use Expression Attribute Names as placeholders

Replace reserved words with placeholders starting with # (e.g., #ord for order)
Map the placeholder to the actual attribute name
Use in your filter expression: #ord = :val

Example:

// Instead of: "order = :val"
// Use: "#ord = :val"
Map<String, String> expressionAttributeNames = Map.of("#ord", "order");

Best Practices:

Always use placeholders defensively to avoid issues
Or design tables to avoid reserved words from the start (use prefixes like item_order or PascalCase)

Note: The article uses scan() which is expensive. Consider adding a Global Secondary Index (GSI) if you frequently query by a specific attribute.

When working with Amazon DynamoDB, it's easy to run into one of its many reserved keywords — words you can't use directly as attribute names in expressions.

Recently, I stumbled upon this issue while trying to query a table that had an attribute named order.

DynamoDB didn't like that, here's how I fixed it.

The Problem

Let's say you have a DynamoDB table mapped to a Template entity, and one of its attributes is named order.

You might try to scan it like this (AWS SDK v1):

DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
    .withFilterExpression("order = :val")
    .withExpressionAttributeValues(Map.of(":val", new AttributeValue().withN("1")))
    .withLimit(1);

But this will throw an AmazonDynamoDBException complaining that order is a reserved keyword.

DynamoDB reserves certain words (like size, name, order, status, etc.) for its own use — so you can't use them directly in filter or key expressions.

Performance Note: scan() is expensive and reads the entire table. If you frequently query by order, consider adding a Global Secondary Index (GSI) to use query() instead of scan().

The Solution: Expression Attribute Names

The trick is to use Expression Attribute Names, which let you create a "placeholder" for your reserved word.

Here's a working version of the same method:

AWS SDK v1 (DynamoDBMapper)

public Optional getByOrder(final Integer order) {
    if (order == null) {
        throw new IllegalArgumentException("Order must not be null");
    }

    // Map for replacing reserved attribute names
    final Map expressionAttributeNames = 
            Collections.singletonMap("#ord", "order");

    // Map for attribute values
    final Map expressionAttributeValues = 
            Collections.singletonMap(":val", new AttributeValue().withN(order.toString()));

    // Define the ScanExpression with limit
    final DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
            .withFilterExpression("#ord = :val")
            .withExpressionAttributeNames(expressionAttributeNames)
            .withExpressionAttributeValues(expressionAttributeValues)
            .withLimit(1);

    // Execute scan and return the first result if exists
    final PaginatedScanList scanResult = dynamoDBMapper.scan(Template.class, scanExpression);

    return scanResult.isEmpty() ? Optional.empty() : Optional.of(scanResult.get(0));
}

AWS SDK v2

public Optional getByOrder(Integer order) {
    if (order == null) {
        throw new IllegalArgumentException("Order must not be null");
    }

    // Get the DynamoDB table reference
    DynamoDbTable table = enhancedClient.table("Template", TableSchema.fromBean(Template.class));

    // Build expression with attribute name placeholder to handle reserved keyword "order"
    // #ord is a placeholder for the reserved word "order"
    // :val is a placeholder for the actual value to compare
    Expression expression = Expression.builder()
            .expression("#ord = :val")
            .expressionNames(Map.of("#ord", "order"))  // Map placeholder to actual attribute name
            .expressionValues(Map.of(":val", AttributeValue.builder().n(order.toString()).build()))
            .build();

    // try-with-resources ensures PageIterable is properly closed to avoid resource leaks
    try (var pages = table.scan(ScanEnhancedRequest.builder()
            .filterExpression(expression)
            .build())) {

        // Return the first matching item, if any
        return pages.items().stream().findFirst();
    }
}

What's Happening Here

#ord is a placeholder for the real attribute name order. You can name it anything as long as it starts with a #. Then, in the expressionAttributeNames map, you define what #ord actually represents.
:val is a placeholder for the value you're comparing against — this is standard in DynamoDB filter expressions.
Finally, the filter expression #ord = :val safely compares the order field without triggering a reserved word error.

Recommended Strategies

Option 1: Always use placeholders (defensive approach)

// Even if it's not a reserved word, use placeholders
.withExpressionAttributeNames(Map.of("#attr", "myAttribute"))

Option 2: Design - Avoid reserved words from the start

Use prefixes: item_order, user_status, product_name
PascalCase: OrderValue, StatusCode

Conclusion

Whenever DynamoDB throws an error like:

"Invalid KeyConditionExpression: Attribute name is a reserved keyword"

Just use Expression Attribute Names to "escape" the reserved word.

It's clean, simple, and works everywhere — filters, updates, and queries.

Photo by Alireza Akhlaghi on Unsplash

DynamoDB Deep Dive: Understanding WCU, RCU, LSI, and GSI

Axel Dlv — Tue, 29 Jul 2025 09:09:51 +0000

Introduction

Amazon DynamoDB is a powerful, serverless NoSQL database designed for high performance at scale. But with great power comes great complexity especially when it comes to indexes and capacity units.

A DynamoDB table has a primary key (partition key + optional sort key) that defines how data is stored and queried.

But what if you need alternative query patterns ?

In this blog, we’ll break down:

What are WCU (Write Capacity Units) and RCU (Read Capacity Units) ?
How to calculate them ?
Differences between LSI (Local Secondary Index) and GSI (Global Secondary Index).

What are WCU (Write Capacity Unit) and RCU (Read Capacity Unit) ?

DynamoDB operates under two capacity modes :

Provisioned capacity : You allocate RCUs and WCUs manually, use for Predictable, steady workloads.
On-demand : AWS handles scaling (costs more per request), use for unpredictable traffic/workloads.

To manage cost and performance effectively, you need to understand RCU (Read Capacity Unit) and WCU (Write Capacity Unit).

WCU - Write Capacity Unit

1 WCU = 1 write/sec for an item ≤1 KB.

WCU Formula:

WCUs = ceil(ItemSizeKB / 1KB) × writesPerSecond

Note: Item sizes are always rounded up.
Example: A 4.5 KB item counts as 5 KB → 5 WCUs for 1 write/sec.

RCU - Read Capacity Unit

Read cost depends on item size and read type:

Read Type	≤4 KB	4–8 KB	8–12 KB
Strongly Consistent Read	1 RCU	2 RCUs	3 RCUs
Eventually Consistent Read	0.5 RCU	1 RCU	1.5 RCUs
Transactional Read	2 RCUs	4 RCUs	6 RCUs

Note: Item sizes are rounded up to the next 4 KB.
Example: A 6 KB item → rounds to 8 KB → 2 RCUs for strong read → 1 RCU for eventual.

RCU Formula:

RCUs = ceil(ItemSizeKB / 4KB) × readsPerSecond

For strongly consistent reads, multiply by 1.
For eventually consistent reads, multiply by 0.5.
For transactional reads, multiply by 2.

Example: Scenario

You expect:

100 writes/sec of 2 KB items
200 reads/sec of 6 KB items
Read type = Eventually consistent

Write Capacity:

2 KB item = ceil(2/1) = 2 WCUs
100 writes/sec × 2 WCUs = 200 WCUs

Read Capacity:

6 KB item → round to 8 KB → ceil(8/4) = 2 RCUs (Strongly Consistent)
Eventual read = 2 × 0.5 = 1 RCU
200 reads/sec × 1 RCU = 200 RCUs

Provision at least 200 WCUs and 200 RCUs

Example: Scenario 2

You expect:

Provisioned RCUs = 10
Item size = 4 KB

Step 1: Determine RCUs per read

Strongly consistent read:
RCUs per read = (4/4) x 1 = 1

Eventually consistent read:
RCUs per read = (4/4) x 0.5 = 0.5

Step 2: Calculate Reads Per Second

Strongly consistent reads/sec:
Reads/sec = 10 / 1 = 10

Eventually consistent reads/sec:
Reads/sec = 10 / 0.5 = 20

Local Secondary Index (LSI) vs Global Secondary Index (GSI)

Secondary indexes let you query your DynamoDB table using alternative keys (not just the primary key).
There are two types LSI (Local Secondary Index) and GSI (Global Secondary Index).

Let’s say we have a table like this:

Partition Key (username)	Sort Key (createdAt)	Email	Role	Region
alice	2023-01-01	alice@example.com	admin	us-east-1
bob	2023-02-05	bob@example.com	user	eu-west-1

Note: A partition in DynamoDB is the physical storage where items are stored.
The partition key decides which partition an item goes to.
Items with the same partition key are grouped together for fast queries.

LSI - Local Secondary Index

An LSI lets you use the same partition key but query using a different sort key.

Same partition key as the base table, different sort key
Defined at table creation ONLY
Supports strongly consistent reads
Shares throughput and storage with the base table
Ways to sort or filter data within a partition
Max 5 LSIs per table

Example

Let’s say we want to sort users within the same username by their role.

Primary Key : username (PK) + createdAt (SK)
LSI : username (PK) + role (SK)

Note: LSIs must be defined at table creation and cannot be added later.

GSI - Global Secondary Index

A GSI lets you define a completely different partition key (and optionally a different sort key).

Different partition and sort keys allowed
Can be added after table creation
Eventually consistent reads only
Has its own RCU/WCU capacity. (GSI writes use the base table’s write capacity during replication. If the base table is saturated, throttling can propagate to GSI writes)
Max 20 GSIs per table

Example

Let’s say you want to query users by their role.

GSI : role (PK) + region (SK)

Note: GSIs can be added any time, but writes are duplicated → extra WCU cost.

LSI vs GSI Comparison

Feature	LSI	GSI
Partition Key	Same as table	Different allowed
Sort Key	Can differ	Can be different or Optional
Read Consistency	Strong + Eventual	Eventually only
Provisioning	Shared with base table	Separate RCUs/WCUs
Query Scope	Within a single partition	Across all partitions
Define at table creation	Yes	No (can add later)
Max per table	5	20
Use case	Sort/filter within a user	Query by different attribute

Conclusion

DynamoDB offers unmatched scalability, but designing tables efficiently is key:

Understand WCU & RCU to avoid throttling and control cost
Use LSI for alternative sort keys within a partition
Use GSI for cross-partition queries and new access patterns

Related Docs

DynamoDB read and write operations (AWS Docs)

Photo by Muhammadh Saamy on Unsplash

Traffic Shifting for AWS Lambda Deployments Using LocalStack and Terraform

Axel Dlv — Thu, 24 Jul 2025 10:10:31 +0000

"This project simulates a Traffic Shifting-style deployment using AWS Lambda aliases and manual traffic shifting via a script, suitable for testing in LocalStack. While not a true CodeDeploy-managed canary deployment, it mimics rollout behavior and can later be upgraded to full AWS CodeDeploy."

Technologies Used

Terraform for infrastructure-as-code
awslocal and tflocal for interaction with aws cli and terraform
LocalStack for simulating AWS locally
AWS Lambda with versioned deployments
API Gateway to expose Lambda function
Shell script to shift traffic between aliases

Infrastructure Modules and Files Overview

You can find the complete code and project structure on GitHub: HERE

Module / Directory	File	Purpose
Root Directory		Combines all modules and manages overall infrastructure
	`main.tf`	Calls all modules (Lambda, API Gateway, IAM) and wires them together
	`provider.tf`	Configures LocalStack AWS provider and endpoint overrides
	`variables.tf`	Declares global variables used in root module or passed to submodules
	`locals.tf`	Centralizes constants, derived values, or map-like logic
	`output.tf`	Outputs: API Gateway invoke URL
modules/lambda		Deploy and version Lambda functions, manage aliases
	`main.tf`	Creates the Lambda function, publishes versions, sets up aliases
	`variables.tf`	Declares inputs: function name, runtime, handler, IAM role ARN
	`output.tf`	Outputs: Lambda function name, ARN, alias, version
modules/iam		Set up IAM roles and attach necessary permissions
	`main.tf`	Creates Lambda execution role and policy documents
	`variables.tf`	common tags
	`output.tf`	Outputs: IAM role ARN (used by Lambda module)
modules/apigateway		Configure API Gateway to expose Lambda with alias support
	`main.tf`	Creates REST API, resources, methods, integrations with Lambda
	`variables.tf`	Declares inputs: Lambda alias ARN, stage name, HTTP methods
	`output.tf`	Outputs: invoke URL
script		Deployment logic and simulation scripts
	`traffic-shifting.sh`	Simulates alias-based routing by invoking different Lambda versions manually based on random chance

Traffic Shifting Script Breakdown

Configuration Section

FUNCTION_NAME="lambda_function"
ZIP_PATH=$1
REST_API_ID=$2
ALIAS_NAME="live"
REGION="eu-west-1"
WAIT_SECONDS=10
HEALTH_URL="http://$REST_API_ID.execute-api.localhost.localstack.cloud:4566/prod/test"
EXPECTED_STATUS=200

Step 1: Publish New Lambda Version

NEW_VERSION=$(awslocal lambda update-function-code \
  --function-name "$FUNCTION_NAME" \
  --zip-file "fileb://${ZIP_PATH}" \
  --publish \
  --query 'Version' \
  --output text)

Uploads new function code and publishes a version.
Stores the new version number for later use.

Step 2: Get Current Alias Version

CURRENT_VERSION=$(awslocal lambda get-alias \
  --function-name "$FUNCTION_NAME" \
  --name "$ALIAS_NAME" \
  --query 'FunctionVersion' \
  --output text)

Retrieves the currently live version the alias points to (used for rollback if needed).

Step 3: Shift 100% Traffic to New Version

awslocal lambda update-alias \
  --function-name "$FUNCTION_NAME" \
  --name "$ALIAS_NAME" \
  --function-version "$NEW_VERSION"

Updates the alias (e.g., live) to fully route traffic to the new version.

Step 4: Perform Health Check

STATUS_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$HEALTH_URL")

Waits a few seconds, then hits the API endpoint.
Checks if the response code matches the expected value (200).

Step 5: Promote or Rollback

if [ "$STATUS_CODE" -eq "$EXPECTED_STATUS" ]; then
  echo "Health check passed — keeping v$NEW_VERSION live"
  echo "Traffic shifting deployment finished."
else
  echo "Health check failed (got $STATUS_CODE) — rolling back to v$CURRENT_VERSION"
  awslocal lambda update-alias \
    --function-name "$FUNCTION_NAME" \
    --name "$ALIAS_NAME" \
    --function-version "$CURRENT_VERSION"
    echo "Traffic shifting deployment failed."
fi

If the health check passes: the new version stays live.
If it fails: rollback to the previous stable version.

This is a lightweight, local traffic shifting strategy for testing Lambda changes safely using LocalStack.

Running and Verifying Infrastructure Locally

1. Set up and run LocalStack using Docker

Make sure Docker is installed on your machine.

Run LocalStack container with AWS services enabled (Lambda, API Gateway, IAM, etc.):

docker run \
  -d \
  -p 127.0.0.1:4566:4566 \
  -p 127.0.0.1:4510-4559:4510-4559 \
  -p 127.0.0.1:443:443 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  localstack/localstack

2. Configure AWS CLI to point to LocalStack

Set AWS CLI environment variables for LocalStack endpoint (replace values as needed):

export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_REGION=eu-west-1
export AWS_ENDPOINT_URL=http://localhost:4566

3. Deploy your Terraform infrastructure to LocalStack

In your Terraform root directory, initialize and apply:

tflocal init
tflocal apply

Outputs:
api_gateway_invoke_url = "http://psjlmrppsc.execute-api.localhost.localstack.cloud:4566"

The invoke URL format is:

http://{rest_api_id}.execute-api.localhost.localstack.cloud:4566/{stage_name}/test

For example, if rest_api_id = psjlmrppsc and your stage_name is prod:

curl -X GET http://psjlmrppsc.execute-api.localhost.localstack.cloud:4566/prod/test

Expected output should be something like:

Hello from version 1 !

4. Verify resources in LocalStack

awslocal lambda list-functions --endpoint-url=http://localhost:4566
awslocal apigateway get-rest-apis --endpoint-url=http://localhost:4566

Or use LocalStack’s web UI if available on https://app.localstack.cloud.

5. Traffic Shifting script

To invoke traffic shiffting script:

./script/traffic-shifting.sh {rest_api_id} {path_lambda_zip}

For example, if rest_api_id = psjlmrppsc and your path_lambda_zip is ./lambda/v2/lambda-v2.zip:

./script/traffic-shifting.sh psjlmrppsc ./lambda/v2/lambda-v2.zip

Updating to version 2

Run your script to simulate traffic shifting deployment:

./script/traffic-shifting.sh psjlmrppsc ./lambda/v2/lambda-v2.zip

Publishing new Lambda version from ./lambda/v2/lambda-v2.zip
Current version: 1, New version: 2
Simulating traffic shifting: updating alias to v2 (100%)
{
    "AliasArn": "arn:aws:lambda:eu-west-1:000000000000:function:lambda_function:live",
    "Name": "live",
    "FunctionVersion": "2",
    "Description": "",
    "RevisionId": "3d562afa-4d56-4447-b7a0-f7b44e6d86cf"
}
Waiting 10 seconds for traffic to flow...
Checking health at http://psjlmrppsc.execute-api.localhost.localstack.cloud:4566/prod/test
Health check passed — keeping v2 live
Traffic shifting deployment finished.

The script should invoke the Lambda functions through the API Gateway alias routing locally via LocalStack.

Test the API endpoint to verify it's responding with the correct V2 response:

curl -X GET http://psjlmrppsc.execute-api.localhost.localstack.cloud:4566/prod/test

Expected output should be something like:

Hello from version 2 !

Updating to version 3

Run your script to simulate traffic shifting deployment:

./script/traffic-shifting.sh psjlmrppsc ./lambda/v3/lambda-v3.zip

Publishing new Lambda version from ./lambda/v3/lambda-v3.zip
Current version: 2, New version: 3
Simulating traffic shifting: updating alias to v3 (100%)
{
    "AliasArn": "arn:aws:lambda:eu-west-1:000000000000:function:lambda_function:live",
    "Name": "live",
    "FunctionVersion": "3",
    "Description": "",
    "RevisionId": "c913bf5e-caf2-4354-bdc3-57594adc5b5d"
}
Waiting 10 seconds for traffic to flow...
Checking health at http://psjlmrppsc.execute-api.localhost.localstack.cloud:4566/prod/test
Health check failed (got 502) — rolling back to v2
{
    "AliasArn": "arn:aws:lambda:eu-west-1:000000000000:function:lambda_function:live",
    "Name": "live",
    "FunctionVersion": "2",
    "Description": "",
    "RevisionId": "107f7f53-0c87-414f-8aa7-07732b47b4a9"
}
Traffic shifting deployment failed.

Try calling the same endpoint again:

curl -X GET http://psjlmrppsc.execute-api.localhost.localstack.cloud:4566/prod/test

If traffic shifting is properly configured or your script handles fallback logic, you should still receive the response from version 2, confirming resilience as the v3 application raise an error.

Hello from version 2 !

Destroy terraform environment

when you have finished, don't forget to destroy the environment :

tflocal destroy -auto-approve

Conclusion

Implementing traffic shifting deployments for AWS Lambda using Terraform both on LocalStack and real AWS brings the best of both development speed and production safety. By simulating real-world deployment flows locally, you can test new Lambda versions, validate functionality through health checks, and automate rollback when things go wrong all without risking your live environment.

Whether you're building locally or deploying to AWS, this approach provides a repeatable, reliable, and testable strategy for Lambda version management.

Picture of David S on Unsplash

How to Build an Event-Driven Outbox Pattern with AWS, Terraform and LocalStack

Axel Dlv — Tue, 24 Jun 2025 07:18:41 +0000

Introduction

When working with microservices, guaranteeing consistency between a database write and an event emission (like an SQS message) can be tricky. This is especially true in distributed systems, where network failures and retries can easily cause duplicates or loss.

The Outbox Pattern offers a robust solution. And in this article, we’ll build it from scratch using:

Terraform for infrastructure-as-code
LocalStack for running AWS services locally
AWS services: DynamoDB, Lambda, SQS, IAM

We’ll walk through setting up the architecture, defining the Terraform modules, and testing the event-driven system locally.

You can find the complete code and project structure on GitHub: HERE

What Is the Outbox Pattern ?

The Outbox Pattern avoids the dual-write problem by splitting responsibilities:

1. Application writes business data and an event into the same database transaction.
2. A separate process (e.g., a Lambda) reads the orders.outbox table and pushes the event to an external system (e.g., SQS).

This guarantees durability and avoids losing messages during partial failures.

Architecture Overview

Two DynamoDB tables: orders and orders.outbox
Stream enabled on orders.outbox table
Lambda triggered by stream, sends events to an SQS FIFO queue

Terraform Modules Breakdown

1. Provider & Root Configuration

We configure Terraform to connect AWS services to LocalStack:

provider "aws" {
  region = local.region

  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true

  endpoints {
    cloudformation = local.localstack_endpoint
    cloudwatch     = local.localstack_endpoint
    dynamodb       = local.localstack_endpoint
    iam            = local.localstack_endpoint
    lambda         = local.localstack_endpoint
    s3             = local.localstack_s3_endpoint
    sqs            = local.localstack_endpoint
  }
}

module "iam" {
  source      = "./modules/iam"
  common_tags = local.common_tags
}

module "dynamodb" {
  source      = "./modules/dynamodb"
  common_tags = local.common_tags
}

module "lambda" {
  source                     = "./modules/lambda"
  lambda_execution_role      = module.iam.lambda_execution_role
  localstack_endpoint        = local.localstack_endpoint
  dynamodb_outbox_stream_arn = module.dynamodb.dynamodb_outbox_stream_arn
  dynamodb_outbox_arn        = module.dynamodb.dynamodb_outbox_arn
  dynamodb_orders_arn        = module.dynamodb.dynamodb_orders_arn
  dynamodb_outbox_name       = module.dynamodb.dynamodb_outbox_name
  dynamodb_orders_name       = module.dynamodb.dynamodb_orders_name
  sync_queue_url             = module.sqs.sync_queue_fifo_url
  common_tags                = local.common_tags
  region                     = local.region
}

module "sqs" {
  source      = "./modules/sqs"
  common_tags = local.common_tags
}

This setup ensures every AWS service call is redirected to LocalStack.

2. IAM Module

Sets up the necessary IAM roles and permissions for Lambda to interact with other AWS services.

resource "aws_iam_role" "lambda_execution_role" {
  name = "lambda-execution-role"

  assume_role_policy = <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Principal": {
        "Service": "lambda.amazonaws.com"
        },
      "Effect": "Allow",
      "Sid": ""
      }]
    }
    EOF

  tags = merge(var.common_tags, {
    Name = "lambda-execution-role"
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "lambda-dynamodb-policy"
  role = aws_iam_role.lambda_execution_role.name

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      # DynamoDB permissions
      {
        Effect = "Allow",
        Action = [
          "dynamodb:PutItem",
          "dynamodb:GetItem",
          "dynamodb:UpdateItem",
          "dynamodb:DeleteItem",
        ],
        Resource = "*"
      },
      # CloudWatch Logs permissions (for logging)
      {
        Effect = "Allow",
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        Resource = "*"
      }
    ]
  })
}

3. DynamoDB Module

We define two DynamoDB tables:

orders — the main business table.
orders.outbox — stores events related to changes in the orders table, with DynamoDB Streams enabled to capture changes.

resource "aws_dynamodb_table" "orders" {
  name         = "orders"
  billing_mode = "PAY_PER_REQUEST"

  hash_key  = "userId"
  range_key = "orderId"

  attribute {
    name = "userId"
    type = "S"
  }

  attribute {
    name = "orderId"
    type = "S"
  }

  tags = merge(var.common_tags, {
    name = "dynamodb_orders_table"
  })
}

resource "aws_dynamodb_table" "outbox" {
  name         = "orders.outbox"
  billing_mode = "PAY_PER_REQUEST"

  stream_enabled   = true
  stream_view_type = "NEW_IMAGE"

  hash_key  = "orderId"
  range_key = "eventId"

  attribute {
    name = "orderId"
    type = "S"
  }

  attribute {
    name = "eventId"
    type = "S"
  }

  tags = merge(var.common_tags, {
    Name = "dynamodb_outbox_table"
  })
}

4. SQS Module

Creates an SQS FIFO queue used to dispatch events from the outbox to other services.

resource "aws_sqs_queue" "sync_queue_fifo" {
  name                        = "sync-queue.fifo"
  fifo_queue                  = true
  content_based_deduplication = true

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.sync_queue_dlq_fifo.arn,
    maxReceiveCount     = 3
  })

  tags = merge(var.common_tags, {
    Name = "sync_queue_fifo"
  })
}

resource "aws_sqs_queue" "sync_queue_dlq_fifo" {
  name                        = "sync-queue-dlq.fifo"
  fifo_queue                  = true
  content_based_deduplication = true

  tags = merge(var.common_tags, {
    Name = "sync_queue_dlq_fifo"
  })
}

5. Lambda Module

The Lambda function acts as the outbox processor:

Triggers on the DynamoDB stream from the outbox table.
Processes events and forwards them to the SQS queue.

resource "aws_lambda_function" "sync_event_lambda" {
  function_name = "syncEventLambda"
  handler       = "sync-event.lambda_handler"
  runtime       = "python3.12"
  filename      = "${path.root}/app/sync-event.zip"

  role = var.lambda_execution_role

  source_code_hash = var.force_lambda_update ? filebase64sha256("${path.root}/app/sync-event.zip") : "" # Forcer la mise à jour du code de la fonction Lambda

  environment {
    variables = {
      DYNAMODB_ENDPOINT     = var.localstack_endpoint
      OUTBOX_TABLE          = var.dynamodb_outbox_name
      SQS_QUEUE_URL         = var.sync_queue_url
      SQS_ENDPOINT          = var.localstack_endpoint
      REGION                = var.region
      AWS_ACCESS_KEY_ID     = "test" # For localstack
      AWS_SECRET_ACCESS_KEY = "test" # For localstack
    }
  }

  tags = merge(var.common_tags, {
    Name = "sync_event_lambda"
  })
}

resource "aws_lambda_function" "outbox_lambda" {
  function_name = "outboxProcessingLambda"
  handler       = "outbox-lambda.lambda_handler"
  runtime       = "python3.12"
  filename      = "${path.root}/app/outbox-lambda.zip"

  role = var.lambda_execution_role

  source_code_hash = var.force_lambda_update ? filebase64sha256("${path.root}/app/outbox-lambda.zip") : "" # Forcer la mise à jour du code de la fonction Lambda

  environment {
    variables = {
      ORDERS_TABLE          = var.dynamodb_orders_name
      OUTBOX_TABLE          = var.dynamodb_outbox_name
      DYNAMODB_ENDPOINT     = var.localstack_endpoint
      REGION                = var.region
      AWS_ACCESS_KEY_ID     = "test" # For localstack
      AWS_SECRET_ACCESS_KEY = "test" # For localstack      
    }
  }

  tags = merge(var.common_tags, {
    Name = "outbox_lambda"
  })
}

resource "aws_lambda_event_source_mapping" "trigger_outbox" {
  event_source_arn  = var.dynamodb_outbox_stream_arn
  function_name     = aws_lambda_function.sync_event_lambda.arn
  starting_position = "LATEST"
  batch_size        = 10

  filter_criteria {
    filter {
      pattern = jsonencode({
        dynamodb = {
          NewImage = {
            status = {
              S = ["PENDING"]
            }
          }
        }
      })
    }
  }

  tags = merge(var.common_tags, {
    Name = "trigger_outbox"
  })
}

Lambda Function Code Overview

Make sure to zip both Python scripts before uploading.

To complement the infrastructure setup, here’s a high-level look at the Lambda function implementations in Python.

outbox-lambda.py
This Lambda inserts business order data and an outbox event atomically into DynamoDB tables. It is using the same transaction.
It generates a unique event ID and marks the event status as "PENDING".

def insert_order_and_outbox(data, timestamp):
    response = dynamodb_client.transact_write_items(
        TransactItems=[
            {
                'Put': {
                    'TableName': ORDERS_TABLE,
                    'Item': {
                        'userId': {'S': data["userId"]},
                        'orderId': {'S': data["orderId"]},
                        'courierId': {'S': data.get("courierId")},
                        'notificationId': {'S': data.get("notificationId")},
                        'message': {'S': data.get("message")},
                        'createdAt': {'S': str(timestamp)}

                    }
                }
            },
            {
                'Put': {
                    'TableName': OUTBOX_TABLE,
                    'Item': {
                        'orderId': {'S': data["orderId"]},
                        'eventId': {'S': str(uuid4())},
                        'eventType': {'S': data["eventType"]},
                        'eventTimestamp': {'S': str(timestamp)},
                        'status': {'S': "PENDING"},
                        'payload': {'S': json.dumps(data)}
                    }
                }
            }
        ]
    )
    return response

This approach ensures the event is durably stored alongside the business data in the same transaction boundary.

sync-event.py
Triggered by DynamoDB Streams on the outbox table, this Lambda:

Deserializes new records with status "PENDING".
Sends the event payload to an SQS FIFO queue with retry and exponential backoff.
Upon successful send, updates the outbox event’s status to "SENT" to avoid duplicate processing.

def send_message_to_sqs_with_retry(queue_url: str, message: dict, attributes: dict, max_retries=3, base_delay=0.5) -> str:
    for attempt in range(max_retries + 1):
        try:
            response = sqs.send_message(
                QueueUrl=queue_url,
                MessageBody=json.dumps(message),
                MessageGroupId="outbox-event",
                MessageAttributes=attributes
            )
            status_code = response['ResponseMetadata']['HTTPStatusCode']
            message_id = response.get("MessageId")

            if status_code == 200 and message_id:
                logger.info(f"SQS message sent successfully. MessageId: {message_id}")
                return message_id
            else:
                raise Exception(f"Unexpected SQS response: {response}")
...

def lambda_handler(event, context):
...
# Send to SQS with validation and retry
message_id = send_message_to_sqs_with_retry(
    queue_url=SQS_QUEUE_URL,
    message=message,
    attributes={
        'eventType': {
            'DataType': 'String',
            'StringValue': event_type
        }
    }
)

# Update DynamoDB status only after successful send
table = dynamodb.Table(OUTBOX_TABLE)
table.update_item(
    Key={
        'orderId': order_id,
        'eventId': event_id
    },
    UpdateExpression='SET #s = :sent',
    ConditionExpression='#s = :pending',
    ExpressionAttributeNames={'#s': 'status'},
    ExpressionAttributeValues={
        ':sent': 'SENT',
        ':pending': 'PENDING'
    }
)
...

This design guarantees reliable event dispatch with strong delivery semantics.

For full code, visit the GitHub repository

Testing the Flow Locally

1. Start LocalStack:

localstack start

2. Deploy the infrastructure:

terraform init
terraform apply

3. Invoke the outboxProcessingLambda manually to simulate an event:
First, you need to install awscli-local.

awslocal lambda invoke \
  --function-name outboxProcessingLambda \
  --region eu-west-1 \
  --payload '{
    "userId": "user-456",
    "notificationId": "notif-9999",
    "orderId": "12345",
    "courierId": "courier-789",
    "eventType": "Delivered",
    "eventTimestamp": "2025-04-23T13:45:00Z",
    "message": "Your order #12345 has been delivered by courier-789."
  }' \
  --cli-binary-format raw-in-base64-out \
  output.json && cat output.json

5. Check DynamoDB (via LocalStack UI or CLI) to verify that the event has been inserted in the orders and orders.outbox database.

orders table

orders.outbox table

6. Check SQS messages (via LocalStack UI or CLI) to verify that the event was propagated.

Advantages of This Setup

LocalStack lets you test cloud-native apps without incurring AWS costs.
Using Terraform allows you to reuse and version infrastructure code easily.
The Outbox Pattern ensures data and event integrity in distributed systems.

Conclusion

Combining the Outbox Pattern with Terraform and LocalStack gives you a powerful, reliable, and testable approach for building event-driven microservices. Whether you're experimenting locally or preparing for production-scale architecture, this pattern helps ensure strong delivery guarantees and system resiliency.

You can find the complete code and project structure on GitHub: HERE

From Prompt to Play: Battleship Game Tutorial using Amazon Q, React and TypeScript

Axel Dlv — Fri, 13 Jun 2025 11:18:14 +0000

Introduction

I recently came across this AWS Community post, which inspired me to build a classic Battleship game using React, TypeScript, and the Amazon Q Developer CLI.
Battleship is a timeless strategy game I loved as a kid, and it felt like the perfect choice for showcasing how Amazon Q can assist with building interactive browser-based games using modern web technologies.

Why Battleship ?

I chose Battleship because it’s a nostalgic favorite from my childhood. I remember playing it on those plastic game boards with miniature ships. The rules are simple but strategic: you hide your ships, take turns firing shots, and try to sink the opponent’s fleet.

That made it perfect for a small web game, it’s self-contained, logical, and full of fun UI possibilities. I also liked the challenge of designing an opponent AI that could be both fair and challenging.

How to Use Amazon Q Like I Did

You’ll first need to install the Amazon Q Developer CLI. It’s a powerful tool that lets you chat with Amazon Q directly from your local environment or IDE.

Installation Guide (Windows) from Ricardo: The Essential Guide to Installing Amazon Q Developer CLI on Windows

Once installed, you can use q chat right from your terminal to generate, debug, and improve code just like I did.

Effective Prompting Techniques I Learned

I started with a "clear" and "detailed" prompt for Amazon Q (in terminal, q chat "prompt") :

"Create a browser-based Battleship game for one player against AI.
The game should follow standard Battleship rules (a beautiful 10x10 grid, ships logo with the size of 5, 4, 3, 3, and 2), with players taking turns.
The game should work in the browser without page reloads.
The player should have their own board to place ships and track hits/misses on the opponent’s board.
Add simple UI to place ships manually, and a start button to begin the game.
Handle game state for human player (ships placed, hits/misses, current turn).
Show a simple message or modal when a player wins.
Use TypeScript and a React front-end framework."

I found that prompt quality directly impacts code quality.

Here’s what worked best:

Be clear about the frontend tech (React + TypeScript)
Describe UI expectations and game flow in plain language

Later, I refined the prompt to add new logic:

"Can you add which ship is destroyed during the game ?"

This small change resulted in a useful UX improvement.

Now, the game tells you exactly which ship you’ve sunk, not just that you hit something.

How AI Handled Classic Programming Challenges

Amazon Q handled several programming challenges surprisingly well:

Ship placement validation: Making sure no ships overlap and all stay within grid bounds
Turn-based logic: Enforcing correct sequence of moves
AI logic: The bot fires intelligently and doesn’t reuse previous moves
Game state tracking: Managing separate boards, hit/miss states, and victory conditions

When I introduced a bug by modifying the ship destruction logic, I hit an error:

Module parse failed: Duplicate export 'default' (354:7)

I asked Q to fix it, and it correctly identified and removed the redundant export. That ability to both generate and debug code made development feel smoother than traditional trial-and-error debugging.

AI-Generated Code Examples (on GitHub)

Rather than flood this post with code, You can find the code below.

Github : Get the code here

Gameplay Preview

Here’s what you can expect:

Ship placement screen with click-to-place

Real-time feedback on hits and misses and "You sunk my Battleship!" messages.

End-of-game modal with reset option

Final Thoughts

This was a great hands-on project that combined frontend UI, game logic, and AI collaboration. Using Amazon Q Chat for code generation, debugging, and logic enhancement was a genuinely productive experience.

Whether you’re building games, dashboards, or apps. AI + React is a serious combo, and tools like Amazon Q make it even better.

Thanks for reading!

AWS Python Utility to Inspect and Purge SQS Queues on LocalStack

Axel Dlv — Wed, 21 May 2025 12:21:36 +0000

TL;DR: This Python script lets you easily debug AWS SQS queues locally via LocalStack. Features include listing messages, purging queues, and selecting from available queues interactively — all without touching the AWS cloud.

When working with Amazon SQS during development, setting up a fast feedback loop can be a challenge — especially if you want to avoid pushing messages to a real AWS account. That’s where LocalStack comes in, and this little utility script can make your life easier.

Here’s a breakdown of what it does and how to use it.

What This Script Does

Connects to a local SQS endpoint (e.g., LocalStack).
Lists all queues and lets you interactively choose one.
Displays message details with optional JSON body parsing.
Purges all messages in a queue (with confirmation).
Simple CLI interface for selecting actions.

Requirements

Python 3.12+
boto3
Docker
LocalStack 4.4 : installation
awslocal : https://github.com/localstack/awscli-local

Localstack setup

1. Run LocalStack on Docker :

  docker run \
  --rm -d \
  -p 127.0.0.1:4566:4566 \
  -p 127.0.0.1:4510-4559:4510-4559 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e DEBUG=1 -e SQS_ENDPOINT_STRATEGY=domain \
  localstack/localstack

2. Create FIFO SQS queues

awslocal sqs create-queue \
--queue-name monitoring-queue.fifo \
--attributes "FifoQueue=true" \
--region eu-west-1

awslocal sqs create-queue \
--queue-name monitoring-queue-dlq.fifo \
--attributes "FifoQueue=true" \
--region eu-west-1

3. Send a message to FIFO SQS

awslocal sqs send-message \
--queue-url http://sqs.eu-west-1.localhost.localstack.cloud:4566/000000000000/monitoring-queue.fifo \
--message-body '{"status":"delivered"}' \
--message-group-id message-group \
--message-deduplication-id 1

Configuration

Don't forget to install boto3 via pip

The script reads a few environment variables:

REGION = os.getenv("REGION", "eu-west-1")
SQS_ENDPOINT = os.getenv("SQS_ENDPOINT", "https://localhost.localstack.cloud:4566")

You're free to adjust these to match your LocalStack setup.
It uses boto3 and assumes test credentials for local use:

sqs = boto3.client(
    "sqs",
    region_name=REGION,
    endpoint_url=SQS_ENDPOINT,
    aws_access_key_id="test",
    aws_secret_access_key="test"
)

Features Breakdown

1. List Messages in a Queue

Prints up to 10 messages per call, with rich metadata:
Sent timestamp (human-readable)
Approximate receive count
Message group and deduplication IDs
Sequence number (for FIFO queues)
Parsed JSON body (if double-encoded)

def list_messages():
    print("Checking messages in queue...\n")    
    selected_queue_url = list_queues_and_choose()

    response = sqs.receive_message(
        QueueUrl=selected_queue_url,
        AttributeNames=['All'],
        MessageAttributeNames=['All'],
        MaxNumberOfMessages=10,
        WaitTimeSeconds=2,
        VisibilityTimeout=0
    )

    messages = response.get("Messages", [])
    if not messages:
        print("No messages available in the queue.")
        return

    for idx, msg in enumerate(messages, 1):
        print(f"\n--- Message #{idx} ---")
        print(f"MessageId: {msg['MessageId']}")
        print(f"ReceiptHandle: {msg['ReceiptHandle']}")

        # SQS attributes
        attrs = msg.get("Attributes", {})
        print(f"SentTimestamp: {human_time(attrs.get('SentTimestamp', '0'))}")
        print(f"ReceiveCount: {attrs.get('ApproximateReceiveCount')}")
        print(f"MessageGroupId: {attrs.get('MessageGroupId')}")
        print(f"MessageDeduplicationId: {attrs.get('MessageDeduplicationId')}")
        print(f"SequenceNumber: {attrs.get('SequenceNumber')}")
        print(f"VisibilityTimeout Active (IsVisible): {'false' if int(attrs.get('ApproximateReceiveCount', 0)) > 0 else 'true'}")

        # Raw Body
        body = msg["Body"]
        print(f"\nRaw Body:\n{body}")

2. Purge Queue

Deletes all messages from the selected queue after confirmation. Useful for clearing test data.

def purge_queue():
    print("Which queue do you want to purge: \n")
    selected_queue_url = list_queues_and_choose()
    if selected_queue_url:
        confirm = input("Are you sure you want to purge the queue? This deletes ALL messages. (yes/no): ")
        if confirm.lower() == "yes":
            sqs.purge_queue(QueueUrl=selected_queue_url)
            print("Queue purged.")
        else:
            print("Purge cancelled.")

3. List All Queues

Simply prints all available queues registered with your local SQS.

def list_queues():
    list = sqs.list_queues()
    print(json.dumps(list, indent=4))

Why This Is Useful

Faster feedback loop with local development.
No AWS charges or IAM concerns.
Understand SQS internals and debug message structures.
Great for simulating dead-letter queue behavior.

Final Thoughts

If you’re working with SQS and want to test locally with minimal overhead, this script will save you time and make debugging much more transparent.

Github Source : https://github.com/axeldlv/aws-sqs-debugging

Let me know in the comments how you’d improve it or if you want a follow-up post to add more features!

Deploy AWS Lambda Functions and Amazon DynamoDB with AWS CDK on LocalStack

Axel Dlv — Fri, 28 Mar 2025 08:29:07 +0000

Introduction

In my previous tech blog, I explained how to deploy AWS Lambda, API Gateway and DynamoDB using AWS SAM on LocalStack.

In this blog, I’d like to introduce AWS CDK. AWS CDK (Cloud Development Kit) is a powerful tool that allows you to define cloud infrastructure using code in languages like Java, TypeScript, or Python. It then generates a CloudFormation template, which can be deployed to AWS or, in this case, to LocalStack that is used for cost savings, offline development, and faster testing.

Prerequisites

Before starting this tutorial, ensure you have the following installed:

LocalStack (Download)
CDK Local (cdklocal) (Download)
AWS CLI Local (Dowload)
Node.js and NPM (Download)
Java 21
Python 3
Favorite IDE (Intellij IDEA or VS Code)

By default, AWS CDK commands like cdk synth and cdk deploy use the AWS CLI. However, in this tutorial, we’ll use cdklocal, provided by LocalStack, to deploy our infrastructure locally.

Creating a Java Project Using AWS CDK

Step 1: Install AWS CDK CLI

Ensure you have AWS CDK installed. If you haven't installed it yet, run:

npm install -g aws-cdk

Step 2: Create a New AWS CDK Project from scratch (Java)

If you want to create a project from scratch, follow this step below or directly git clone the GitHub Repository.

Run the following command to initialize a new Java AWS CDK project:

mkdir cdk-lambda-dynamodb && cd cdk-lambda-dynamodb
cdk init app --language=java

This will generate a Java-based AWS CDK project with a basic folder structure.

Step 3: Set Up the Project

By default, the CDK generates a blank structure automatically, but here is my custom structure:

cdk-lambda-dynamodb/
│-- infra/
│   ├── src/
│   │   ├── main/
│   │   │   ├── java/
│   │   │   │   ├── be/axeldlv/infra  (Java AWS CDK)
│   │   │   │   │   ├── CDKLambdaDynamoDBStackApp.java
│   │   │   │   │   ├── CDKLambdaDynamoDBStackStack.java
│   ├-- pom.xml (Maven configuration)
│   ├-- README.md
│   ├-- cdk.json
│-- lambda-getFlights/ (Python Application)
│   ├-- getFlights.py
│   ├-- requirements.txt
│-- lambda-insertFlights/ (Java Application)
│   ├── src/
│   │   ├── main/
│   │   │   ├── java/
│   │   │   │   ├── be/axeldlv
│   │   │   │   │   ├── InsertFlights.java

Step 4: Define Your Infrastructure Stack

If you are using the cdk init, remove java class previously create and rename it to CDKLambdaDynamoDBStackApp.java and CDKLambdaDynamoDBStackStack.java in src/main/java/be/axeldlv/infra to define your infrastructure.

In this example, we need to set up two AWS Lambda, AWS Secrets Manager and Amazon DynamoDB resources.

CDKLambdaDynamoDBStackApp.java

package be.axeldlv.infra;

import software.amazon.awscdk.App;
import software.amazon.awscdk.Environment;
import software.amazon.awscdk.StackProps;

public class CDKLambdaDynamoDBStackApp {
    public static void main(final String[] args) {
        App app = new App();
        new CDKLambdaDynamoDBStackStack(app, "CDKLambdaDynamoDBStack", StackProps.builder()
                .env(Environment.builder()
                .account(System.getenv("CDK_DEFAULT_ACCOUNT"))
                .region(System.getenv("CDK_DEFAULT_REGION"))
                .build()).build());

        app.synth();
    }
}

CDKLambdaDynamoDBStackStack.java

package be.axeldlv.infra;

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.dynamodb.Attribute;
import software.amazon.awscdk.services.dynamodb.AttributeType;
import software.amazon.awscdk.services.dynamodb.BillingMode;
import software.amazon.awscdk.services.dynamodb.Table;
import software.amazon.awscdk.services.lambda.*;
import software.amazon.awscdk.services.lambda.Runtime;
import software.amazon.awscdk.services.secretsmanager.Secret;
import software.constructs.Construct;

public class CDKLambdaDynamoDBStackStack extends Stack {
    public CdkFlightsFunctionsLambdaStack(final Construct scope, final String id) {
        this(scope, id, null);
    }
    public CdkFlightsFunctionsLambdaStack(final Construct scope, final String name, final StackProps props) {
        super(scope, name, props);

        Table flightTable = Table.Builder.create(this, "FlightTable")
                .tableName("FlightTable")
                .billingMode(BillingMode.PAY_PER_REQUEST)
                .sortKey(Attribute.builder().name("orderId").type(AttributeType.STRING).build())
                .partitionKey(Attribute.builder().name("bookingId").type(AttributeType.STRING).build()).build();

        Function insertFlightLambda = Function.Builder.create(this, "InsertFlightLambda")
                .functionName("InsertFlightLambda")       
                .runtime(Runtime.JAVA_21)
                .handler("be.axeldlv.InsertFlight::handleRequest")
                .code(Code.fromAsset("../lambda-insertFlights/jar/sendtodynamodb-1.0.jar"))
                .memorySize(1024)
                .timeout(Duration.seconds(60))
                .description("Lambda Function to insert data to DynamoDB")
                .tracing(Tracing.ACTIVE)
                .build();

        Function getFlightLambda = Function.Builder.create(this, "getFlightLambda")
                .functionName("getFlightLambda")   
                .runtime(Runtime.PYTHON_3_13)
                .handler("app.lambda_handler")
                .code(Code.fromAsset("../lambda-getFlights"))
                .memorySize(1024)
                .timeout(Duration.seconds(60))
                .description("Lambda function to retrieve data from DynamoDB")
                .tracing(Tracing.ACTIVE)
                .build();

        Secret addTableNameSecret = Secret.Builder.create(this, "addTableNameSecret")
                .secretName("table")
                .secretStringValue(SecretValue.unsafePlainText(flightTable.getTableName()))
                .build();

        addTableNameSecret.grantRead(insertFlightLambda);
        addTableNameSecret.grantRead(getFlightLambda);
        flightTable.grantReadWriteData(insertFlightLambda);       
        flightTable.grantReadData(getFlightLambda);
    }
}

Explanation of `CdkFlightsFunctionsLambdaStack` Code

This code defines an AWS CDK stack called CDKLambdaDynamoDBStack, which sets up a serverless architecture using AWS Lambda, Amazon DynamoDB, and AWS Secrets Manager. Below is a breakdown of the components defined in this stack:

DynamoDB Table (`flightTable`)

A DynamoDB table named FlightTable is created.
The table has two keys:
- Partition Key: bookingId (type STRING) - booking#1
- Sort Key: orderId (type STRING) - order#1
The table is set to Pay Per Request billing mode.

Lambda Function to Insert Flights (`insertFlightLambda`)

A Lambda function written in Java (Runtime.JAVA_21) is created to insert flight data into the DynamoDB table.
The Lambda function uses a locally stored JAR file (sendtodynamodb-1.0.jar).
The function is configured with:
- Memory: 1GB
- Timeout: 60 seconds
- X-Ray Tracing: Active for monitoring > Don't forget to move the build in the jar folder in the project.

Lambda Function to Get Flights (`getFlightLambda`)

Another Lambda function, written in Python (Runtime.PYTHON_3_13), is created to retrieve flight data from DynamoDB.
The function’s code is located in the folder ../lambda-getFlights.
It is configured with:
- Memory: 1GB
- Timeout: 60 seconds
- X-Ray Tracing: Active

Secrets Manager Secret (`addTableNameSecret`)

A secret is created in AWS Secrets Manager to store the name of the FlightTable.
The secret’s value is set to the table name (flightTable.getTableName()).
Both Lambda functions (insertFlightLambda and getFlightLambda) are granted read access to the secret.

Adding the table name in secret isn't really necessary, but it was done for the purpose of the tutorial.

Permissions

The Lambda functions are granted the necessary permissions to interact with the DynamoDB table and Secrets Manager secret:
- The insertFlightLambda is granted ReadWriteData access to the flightTable and Read access to the addTableNameSecret.
- The getFlightLambda is granted ReadData access to the flightTable and Read access to the secret as well.

Creating AWS Lambda Functions

We are creating two Lambda functions: one in Python for retrieving flight data and another in Java for inserting flight data. This will demonstrate how we can work with two different programming languages in AWS Lambda.

GetFlights Lambda Function - Python

getFlights.py

import boto3
import os
import json

region = "us-west-1"
localstackurl = "https://localhost.localstack.cloud:4566"

dynamoDBClient = boto3.client('dynamodb', 
             endpoint_url=localstackurl, 
             region_name=region)

dynamodb = boto3.resource('dynamodb', endpoint_url=localstackurl, region_name=region)

def lambda_handler(event, context):
    item_id = event.get('pathParameters', {}).get('bookingId')
    order_id = event.get('pathParameters', {}).get('orderId')

    if not item_id:
        return {
            'statusCode': 400,
            'body': json.dumps({'error': 'ID not provided in path parameters'})
        }

    table_name = "FlightTable"
    if not table_name:
        raise KeyError("TABLEName environment variable is not set")

    table = dynamodb.Table(table_name)

    # fetch todo from the database
    response = table.get_item(Key={'orderId': str(order_id), 'bookingId': str(item_id)}) 
    item = response.get('Item', {})   

    return {
        'statusCode': 200,
        'body': item
    }

InsertFlights Lambda Function - Java

This Java class implements an AWS Lambda function that processes API Gateway requests (not explain in this technical blog) and inserts data into a DynamoDB table. It retrieves the table name from AWS Secrets Manager and interacts with LocalStack for local testing.

InsertFlight.java

package be.axeldlv;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyRequestEvent;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyResponseEvent;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.PutItemRequest;
import software.amazon.awssdk.services.dynamodb.model.PutItemResponse;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;

import java.net.URI;
import java.util.HashMap;
import java.util.Map;

public class InsertFlight implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {

    String region = "us-west-1";
    String localStackUrl = "https://localhost.localstack.cloud:4566";

    @Override
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent apiGatewayProxyRequestEvent, Context context) {
        context.getLogger().log("Lambda function started...");

        SecretsManagerClient secretsManagerClient = createSecretsManagerClient();
        GetSecretValueRequest getSecretValueRequest = GetSecretValueRequest.builder()
                .secretId("table")
                .build();
        GetSecretValueResponse secretValue = secretsManagerClient.getSecretValue(getSecretValueRequest);
        DynamoDbClient dynamoDbClient = createDynamoDbClient();

        String tableName = secretValue.secretString();
        String requestBody = apiGatewayProxyRequestEvent.getBody();
        context.getLogger().log("requestBody " + requestBody);
        String bookingId = apiGatewayProxyRequestEvent.getPathParameters().get("bookingId");
        context.getLogger().log("bookingId " + bookingId);
        String orderId = apiGatewayProxyRequestEvent.getPathParameters().get("orderId");
        context.getLogger().log("orderId " + orderId);

        PutItemResponse response = insertItemToDynamoDb(dynamoDbClient, tableName, requestBody, bookingId, orderId);

        context.getLogger().log("Data saved successfully to DynamoDB: " + response);
        return createAPIResponse(requestBody);
    }

    private DynamoDbClient createDynamoDbClient() {
        return DynamoDbClient.builder()
                .endpointOverride(URI.create(localStackUrl))
                .credentialsProvider(DefaultCredentialsProvider.create())
                .region(Region.of(region))
                .build();
    }

    private PutItemResponse insertItemToDynamoDb(DynamoDbClient dynamoDbClient, String tableName, String requestBody, String bookingId, String orderId) {
        Map<String, AttributeValue> item = new HashMap<>();
        item.put("bookingId", AttributeValue.builder().s(bookingId).build());
        item.put("orderId", AttributeValue.builder().s(orderId).build());
        item.put("requestBody", AttributeValue.builder().s(requestBody).build());

        PutItemRequest putItemRequest = PutItemRequest.builder()
                .tableName(tableName)
                .item(item)
                .build();

        return dynamoDbClient.putItem(putItemRequest);
    }

    private APIGatewayProxyResponseEvent createAPIResponse(String body) {
        APIGatewayProxyResponseEvent responseEvent = new APIGatewayProxyResponseEvent();
        responseEvent.setBody(body);
        responseEvent.setStatusCode(201);
        return responseEvent;
    }

    private SecretsManagerClient createSecretsManagerClient() {
        return SecretsManagerClient.builder()
                .endpointOverride(URI.create(localStackUrl))
                .credentialsProvider(DefaultCredentialsProvider.create())
                .region(Region.of(region))
                .build();
    }
}

CDKLocal to deploy on LocalStack

Start LocalStack in background mode

localstack start -d

Generate the CloudFormation Template

To run commands, you need to go to the infra folder from your project.

Before deploying the Lambda function, you need to generate the CloudFormation template from the AWS CDK code. This step validates the stack definition and alerts you if there are any errors.

To synthesize the CloudFormation template, run:

cdklocal synth

This command will display the CloudFormation stack details in the console (example of output) :

It creates also a directory called cdk.out, which contains the generated template and related asset files.

Bootstrap the LocalStack Environment

Before deploying your project for the first time, you need to prepare the AWS environment. This process sets up a CloudFormation stack with essential resources, including an S3 bucket for storing deployment assets.

If you've already bootstrapped the environment, you can skip this step.

Run the following command:

cdklocal bootstrap

Example output:

⏳  Bootstrapping environment aws://000000000000/us-west-1...
CDKToolkit: creating CloudFormation changeset...
✅  Environment aws://000000000000/us-west-1 bootstrapped.

Step 3: Deploy the Lambda Function
Once the environment is ready, deploy the project by running:

cdklocal deploy

During deployment, the AWS CDK will prompt for confirmation. Press "y" to proceed.

InsertFlightLambdaStack: deploying... [1/1]
InsertFlightLambdaStack: creating CloudFormation changeset...
 ✅  InsertFlightLambdaStack
✨  Deployment time: 70.5s
Stack ARN:
arn:aws:cloudformation:us-west-1:000000000000:stack/InsertFlightLambdaStack/90270911

Test the lambda function with AWS CLI

To add data using AWS CLI :

awslocal lambda invoke 
     --function-name InsertFlightLambda 
     --cli-binary-format raw-in-base64-out 
     --payload '{ "pathParameters": {"bookingId": "booking#1","orderId": "order#1"},"body": "{\"testBody\": \"test\"}" }'

To get data :

awslocal lambda invoke 
     --function-name getFlightLambda
     --cli-binary-format raw-in-base64-out 
     --payload '{ "pathParameters": { "bookingId": "booking#1", "orderId": "order#1" } }' 
     --output json output.txt

You can see the output in the output.txt file create in the root of infra folder.

Here is the result of the request :

{"statusCode": 200, "body": {"orderId": "order#1", "requestBody": "{\"testBody\": \"test\"}", "bookingId": "booking#1"}}

Cleaning Up Resources

First, you need to destroy all resources :

cdklocal destroy

And stop the LocalStack :

localstack stop

Conclusion

Deploying AWS Lambda functions, AWS Secrets Manager and DynamoDB using AWS CDK provides a powerful way to manage infrastructure as code. By leveraging LocalStack, developers can efficiently test and deploy their applications in a local environment before pushing them to AWS, reducing costs and improving development speed. This approach ensures a seamless and automated deployment process, making it easier to maintain and scale serverless applications.

By following this guide, you now have a solid foundation for setting up Java-based Lambda functions, setting a secret string in Secrets Manager, integrating DynamoDB, and deploying them using AWS CDK. Keep exploring AWS CDK to enhance your infrastructure automation and optimize your development workflow! 🚀

Code source on Github Respository

AWS Tutorial: Using LocalStack for testing AWS SNS and SQS Locally with Filter Policies

Axel Dlv — Mon, 10 Feb 2025 08:54:59 +0000

Introduction

Working with AWS services locally can be challenging due to cloud dependencies, costs, and network constraints. LocalStack helps overcome these issues by providing a fully functional local simulation of AWS services. It allows developers to run and test cloud applications in a controlled environment without needing access to real AWS infrastructure.

With a simple Docker setup, LocalStack enables you to experiment with multiple AWS services, such as SNS (Simple Notification Service) and SQS (Message Queuing Service), making it ideal for sandbox projects and integration testing. However, it's important to remember that LocalStack is a simulated environment and not a fully integrated AWS service.

In this guide, we'll explore how to use LocalStack with AWS CLI commands to set up and test messaging services like SNS and SQS.

Environment Preparation

Installing AWS CLI for LocalStack

If you haven’t installed the AWS CLI yet, you can download it from: https://aws.amazon.com/fr/cli/

Verify the installation:

aws --version

Setting up LocalStack

Install LocalStack:
Create a LocalStack account here and follow the setup documentation for your environment (ensure Docker is installed).
Start LocalStack:

localstack start -d

Verify LocalStack Instance: Go to the LocalStack website UI: LocalStack Cloud, and under the "Status" section, check if the instance and services are running.

Configuring AWS CLI for LocalStack

Using AWS CLI Profile

To configure AWS CLI for LocalStack, create a new profile named dev-account:

aws configure set aws_access_key_id "dev-account-key-id" --profile dev-account
aws configure set aws_secret_access_key "dev-account-access-key" --profile dev-account
aws configure set region "eu-west-1" --profile dev-account

The information is stored in ~/.aws/credentials and ~/.aws/config.

This tutorial demonstrates how to send a message to an SNS topic and process it through an SQS queue using a FilterPolicy.

Architecture schema

Creating SNS Topic and SQS Queue in LocalStack

Create SNS Topic (FIFO)

Create `bookingairline.fifo`

aws sns create-topic \
  --endpoint-url=http://localhost:4566 \
  --name bookingairline.fifo \
  --profile dev-account \
  --region eu-west-1 \
  --attributes '{"FifoTopic": "true", "ContentBasedDeduplication": "true"}' \
  --output table | cat

Note: FIFO topics must end with .fifo.

Create SQS Queues (FIFO)

Create `airbus.fifo`

aws sqs create-queue \
  --endpoint-url=http://localhost:4566 \
  --queue-name airbus.fifo \
  --profile dev-account \
  --region eu-west-1 \
  --attributes FifoQueue=true,ContentBasedDeduplication=true \
  --output table | cat

Create `boeing.fifo`

aws sqs create-queue \
  --endpoint-url=http://localhost:4566 \
  --queue-name boeing.fifo \
  --profile dev-account \
  --region eu-west-1 \
  --attributes FifoQueue=true,ContentBasedDeduplication=true \
  --output table | cat

Note: FIFO queues must end with .fifo.

Attributes and Parameters

Parameters	Explanation
VisibilityTimeout	Time (in seconds) that a message is hidden after being received.
MessageRetentionPeriod	Time (in seconds) a message stays in the queue before deletion. Default: 345600 (4 days).
DelaySeconds	Delay (in seconds) before a message becomes visible. Max: 900 (15 minutes).
MaximumMessageSize	Max size (in bytes) of a message. Default: 262144 (256 KB).
ReceiveMessageWaitTimeSeconds	Long polling wait time (in seconds).
FifoQueue	true for FIFO queues. Cannot be changed after queue creation.
ContentBasedDeduplication	true to enable deduplication for FIFO queues.

Attributes	Explanation
FifoTopic	Indicating whether the topic is a FIFO (First-In-First-Out) topic. FIFO ensures message order is maintained.
ContentBasedDeduplication	Deduplication prevents messages with the same content from being processed twice for FIFO topics.

SNS Subscription to SQS Queue

Subscribe `airbus.fifo` to SNS

aws sns subscribe \
  --endpoint-url=http://localhost:4566 \
  --topic-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo \
  --profile dev-account \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:eu-west-1:000000000000:airbus.fifo \
  --region eu-west-1 \
  --output table | cat

Subscribe `boeing.fifo` to SNS

aws sns subscribe \
  --endpoint-url=http://localhost:4566 \
  --topic-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo \
  --profile dev-account \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:eu-west-1:000000000000:boeing.fifo \
  --region eu-west-1 \
  --output table | cat

Parameters and Protocols

Parameters	Explanation
topic-arn	The ARN (Amazon Resource Names) of the SNS topic that you want to subscribe to.
protocol	This specifies the protocol for the subscription (see below for more protocol).
notification-endpoint	This is the ARN (Amazon Resource Names) of the AWS Lambda function that will process the messages.
endpoint-url	In the case of LocalStack or a local setup, this would be http://localhost:4566. This is where the AWS CLI will send requests.

Protocols	Explanation
HTTP/HTTPS	Delivers messages to an HTTP(s) endpoint as a POST request.
Email/Email-JSON	Sends the message as an email or as a JSON-formatted email to the subscriber's email address.
SMS	Sends the message as a text message (SMS) to the subscriber's phone number.
SQS	Sends the message to an Amazon SQS queue.
Application	Sends the message to a mobile application (for example, a mobile push notification).
Lambda	Invokes an AWS Lambda function with the SNS message as the event.

Add a filter to the SNS Topic to filter messages

Note: For this part, you have to modify the subscription-arn with the subscription that you have created before for airbus and boeing.

Add FilterPolicy for airbus.fifo queue

aws sns set-subscription-attributes \
  --endpoint-url=http://localhost:4566 \
  --subscription-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo:a43062ba-55cd-4fd8-8c81-88480985a889 \
  --attribute-name FilterPolicy \
  --attribute-value '{"Type": ["AirbusAirline"]}' \
  --region eu-west-1 \
  --profile dev-account \
  --output json | cat

Add FilterPolicy for boeing.fifo queue

aws sns set-subscription-attributes \
  --endpoint-url=http://localhost:4566 \
  --subscription-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo:02d28ea6-e3ea-48e7-92cf-02ac0e8cf3b7 \
  --attribute-name FilterPolicy \
  --attribute-value '{"Type": ["BoeingAirline"]}' \
  --region eu-west-1 \
  --profile dev-account \
  --output json | cat

subscription-arn: The ARN (Amazon Resource Names) of the SNS topic subscription that you have created before for the airbus.fifo or boeing.fifo queue.
attribute-name: The attribute name to use FilterPolicy.
attribute-value: The type of message to filter. In this case, only AirbusAirline type will be forwarded to the airbus.fifo queue and the BoeingAirline type will be forwarded to the boeing.fifo.

Publishing Messages to SNS

This example is to show that the filterpolicy is working.

Create Message Attributes for AirbusAirline

You can create a JSON file (for exemple: message-attributes-airbus.json) to prepare a body to send via the AWS CLI :

`message-attributes-airbus.json`

{
  "Type": { "DataType": "String", "StringValue": "AirbusAirline" },
  "FlightID": { "DataType": "String", "StringValue": "AIRBUS_456" }
}

Publish Message

Now that we have attributes, we can send it to the SNS Topic :

aws sns publish \
  --endpoint-url=http://localhost:4566 \
  --topic-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo \
  --message 'New booking flight for Airbus has been created!' \
  --message-attributes file://message-attributes-airbus.json \
  --message-group-id 'MessageGroup1' \
  --region eu-west-1 \
  --profile dev-account \
  --output json | cat

We can see on Localstack that airbus.fifo has received the message.

Create Message Attributes for BoeingAirline

You can create a JSON file (for exemple: message-attributes-boeing.json) to prepare a body to send via the AWS CLI :

`message-attributes-boeing.json`

{
  "Type": { "DataType": "String", "StringValue": "BoeingAirline" },
  "FlightID": { "DataType": "String", "StringValue": "BOEING_962" }
}

Publish Message

Now that we have attributes, we can send it to the SNS Topic :

aws sns publish \
  --endpoint-url=http://localhost:4566 \
  --topic-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo \
  --message 'New booking flight for Boeing has been created!' \
  --message-attributes file://message-attributes-boeing.json \
  --message-group-id 'MessageGroup1' \
  --region eu-west-1 \
  --profile dev-account \
  --output json | cat

Parameters	Explanation
message-group-id	The message group ID to use for the message. This is used to group messages that are sent to the same queue.
message-attributes	The message attributes to use for the message. In this case, we are using the FilterPolicy attribute to filter messages.
message	The message to send to the SNS Topic.
topic-arn	The ARN (Amazon Resource Names) of the SNS Topic that you want to send the message to.
endpoint-url	In the case of LocalStack or a local setup, this would be http://localhost:4566. This is where the AWS CLI will send requests.

Consuming Messages from SQS Queues

aws sqs receive-message \
  --endpoint-url=http://localhost:4566 \
  --queue-url http://localhost:4566/000000000000/airbus.fifo \
  --profile dev-account \
  --region eu-west-1 \
  --output json | cat

aws sqs receive-message \
  --endpoint-url=http://localhost:4566 \
  --queue-url http://localhost:4566/000000000000/boeing.fifo \
  --profile dev-account \
  --region eu-west-1 \
  --output json | cat

Launching this command will display the messages in the SQS Queue for airbus.fifo and the other for boeing.fifo.

Cleaning Up Resources

aws sns delete-topic \
  --endpoint-url=http://localhost:4566 \
  --region eu-west-1 \
  --profile dev-account \
  --topic-arn arn:aws:sns:eu-west-1:000000000000:bookingairline.fifo

aws sqs delete-queue \
  --endpoint-url=http://localhost:4566 \
  --region eu-west-1 \
  --profile dev-account \
  --queue-url http://localhost:4566/000000000000/airbus.fifo

aws sqs delete-queue \
  --endpoint-url=http://localhost:4566 \
  --region eu-west-1 \
  --profile dev-account \
  --queue-url http://localhost:4566/000000000000/boeing.fifo

localstack stop

Conclusion

LocalStack provides a powerful way to simulate AWS services locally, making it an excellent tool for sandbox projects and integration testing. With a simple Docker setup, you can experiment with multiple AWS services without incurring costs or relying on cloud infrastructure.

By using AWS CLI commands, as demonstrated above, you can create and test essential services like SNS and SQS in a local environment. However, since LocalStack is a simulated tool and not a fully integrated AWS service, it's important to validate your workflows before deploying them to a real AWS environment.

With LocalStack, developers can streamline their testing process, improve efficiency, and reduce dependencies on live cloud resources, making development and debugging more seamless.

Thank you for reading!

If you have any questions, feedback, or suggestions, please feel free to leave them in the comments below. I'm eager to hear from you and respond to your thoughts!

AWS Landing Zone - AWS Services

Axel Dlv — Wed, 18 Dec 2024 08:07:54 +0000

Introduction:

In my previous post, AWS Landing Zone - Overview, I introduced the core concepts behind AWS Landing Zone and explained its purpose in setting up a secure, multi-account AWS environment. In this follow-up, we'll dive deeper into the key AWS services needed to implement a successful AWS Landing Zone. These services will help you automate governance, security, and networking while providing scalability and compliance in your multi-account setup. Whether you are just starting with AWS or looking to enhance your existing infrastructure, understanding these services will lay the foundation for a robust AWS environment.

AWS Services for Building a Landing Zone

Here’s a breakdown of the essential AWS services for setting up and managing an AWS Landing Zone:

Multi-Account Management

AWS Organizations: Used for centralized account creation, management, and applying governance through Service Control Policies (SCPs).
AWS Control Tower: Provides an easier alternative to AWS Landing Zone, helping you set up and govern a secure multi-account AWS environment. Control Tower automates the creation of accounts and applies pre-configured guardrails.

Identity and Access Management

AWS IAM: Configure roles, permissions, and policies across multiple accounts to manage access.
AWS SSO: Enables centralized user and role management with Single Sign-On capabilities, simplifying access control across accounts.
AWS Directory Service: Integrates with your on-premises Active Directory or creates a managed directory in AWS for user management.

Networking

Amazon VPC: Create isolated virtual networks for your workloads to enhance security and control traffic.
AWS Transit Gateway: A centralized hub for connecting multiple VPCs across AWS accounts and on-premises networks.
AWS Direct Connect: Establish private, high-speed connections between your on-premises data centers and AWS.
AWS PrivateLink: Enables secure connectivity between VPCs without exposing traffic to the public internet.

Security

AWS CloudTrail: Logs all AWS account activity to help with auditing and governance.
AWS Config: Tracks configuration changes and ensures compliance with defined policies.
AWS GuardDuty: Continuously monitors for threats and suspicious activity.
AWS Security Hub: Centralized security management and compliance monitoring.
AWS KMS (Key Management Service): Encrypt data to secure sensitive information.
Amazon Inspector: Performs automated security assessments and vulnerability scanning.

Logging and Monitoring

Amazon CloudWatch: Provides comprehensive monitoring for AWS resources and applications, helping you track performance and health.
AWS CloudTrail: Collects logs for auditing API calls and user activity across AWS services.
Amazon S3: Centralized storage for logs and other data.
AWS Lambda: Automate log processing, analysis, or respond to security events in real-time.

Governance and Compliance

AWS Service Catalog: Helps maintain consistency by allowing you to create and manage pre-approved resource templates.
AWS Trusted Advisor: Suggests best practices for cost optimization, performance, and security.
AWS Control Tower: Automates governance with built-in guardrails for compliance across accounts.

Automation and Deployment

AWS CloudFormation: Automates the provisioning of AWS resources based on predefined templates.
AWS CodePipeline: Provides continuous integration and delivery (CI/CD) workflows for deployment automation.
AWS Step Functions: Orchestrates workflows for various processes like account setup, compliance checks, and security tasks.

Data and Storage

Amazon S3: A highly durable and scalable object storage service for storing logs, data backups, and other artifacts.
Amazon RDS: Managed relational databases for workloads that require persistent, structured storage.
Amazon DynamoDB: A serverless key-value store for fast and flexible application data.

Conclusion

In conclusion, AWS Landing Zone offers a comprehensive approach to managing multi-account environments, and leveraging the key AWS services outlined in this post is essential to building a secure, scalable, and compliant AWS setup. By combining services like AWS Organizations, Control Tower, IAM and CloudTrail, you can automate much of the setup and governance while ensuring a high level of security. If you haven’t already, I recommend reading my previous post on AWS Landing Zone - Overview for a better understanding of the foundational concepts. With the services covered here, you’ll be well on your way to setting up an optimized, secure, and efficient AWS Landing Zone tailored to your organization’s needs.

AWS Landing Zone - Overview

Axel Dlv — Tue, 17 Dec 2024 06:48:49 +0000

Introduction

Setting up a multi-account AWS environment with robust security, governance, and compliance is crucial for large organizations. AWS Landing Zone provides a solution to help with this setup by offering best practices for organizing and managing your AWS accounts. This post will guide you through features you can use to build and manage your AWS Landing Zone environment effectively.

Overview of AWS Landing Zone

AWS Landing Zone is a solution designed to automate the setup of a secure, multi-account AWS environment. It is based on AWS best practices and ensures consistency, security, and governance across your AWS environment.

Multi-account setup: Allows you to organize your workloads into separate AWS accounts, ensuring better security and resource management.
Integration with AWS Organizations: Automates account management and governance using Service Control Policies (SCPs).
Security and compliance: Pre-configured IAM roles, AWS Config, CloudTrail and GuardDuty for security enforcement.
Networking setup: Creates a baseline network configuration to support isolation and communication between accounts.
Centralized logging: Aggregates logs from multiple AWS services into a Log Archive account for compliance and auditing.

Key Components of AWS Landing Zone

Multi-Account Setup

Landing Zone establishes a multi-account AWS structure based on AWS best practices, which helps with workload separation and better resource organization.

Common account types in the setup include:

Master Account: Manages billing and governance.
Shared Services Account: Hosts shared infrastructure like networking or logging resources.
Log Archive Account: Secures all audit and access logs for compliance.
Security Account: Houses central security tooling and monitoring.

AWS Organizations Integration

AWS Landing Zone integrates with AWS Organizations to automate account creation and management, applying Service Control Policies (SCPs) to enforce governance and security policies across accounts.

Security and Compliance

Landing Zone pre-configures AWS services such as IAM, CloudTrail, Config, and GuardDuty to enforce security and compliance requirements. This setup includes logging, encryption, and access controls by default.

Networking

Landing Zone automates network setup using Amazon VPC, ensuring secure isolation and communication across accounts. Subnets, VPC peering, and other essential networking configurations are pre-configured.

Centralized Logging

To comply with security and auditing requirements, AWS Landing Zone aggregates logs from CloudTrail, Config and other AWS services into a Log Archive Account.

Recommended Starting Set for AWS Landing Zone

If you're starting with a basic AWS Landing Zone setup, the following services should be your initial focus:

AWS Organizations for account management.
AWS Control Tower for secure account setup and governance.
AWS CloudTrail and AWS Config for continuous monitoring and auditing.
AWS IAM and AWS SSO for centralized access control.
Amazon S3 for centralized log storage.
AWS GuardDuty for threat detection.
Amazon VPC for network isolation and security.

These services provide a strong foundation for a secure, compliant, and well-governed AWS environment. As your organization's needs evolve, you can scale and integrate additional services.

Benefits of AWS Landing Zone

Consistency: Ensures a consistent, secure setup across all AWS accounts, reducing the risk of misconfigurations.
Automation: Automates the creation of AWS accounts, deployment of foundational resources, and enforcement of guardrails.
Scalability: Designed to scale as your organization grows, allowing for more accounts and users as needed.
Security: Incorporates AWS security best practices to protect sensitive data and ensure compliance.
Cost-Effective: By streamlining setup and governance, it saves time and resources during environment setup and maintenance.

Conclusion

If you're looking to establish a secure, scalable, and well-governed multi-account AWS environment, AWS Landing Zone or AWS Control Tower are both excellent solutions. These services help automate the setup of foundational AWS infrastructure, enforce security and compliance policies, and ensure consistency across your organization. By leveraging AWS's powerful suite of services, you can create a robust environment that scales as your organization grows.

Releasing Idle Elastic IPs on AWS to Save Costs

Axel Dlv — Thu, 14 Nov 2024 16:19:09 +0000

To automatically disable or shut down idle IPs on AWS, you can use AWS Lambda and Amazon EventBridge in combination. Here’s a guide on how to set up a solution that will periodically check for idle Elastic IPs and release them if they’re not in use.

Step 1: Define "Idle" IP Criteria

Define what "idle" means for your IPs. Typically, an Elastic IP (EIP) is considered "idle" if it's not associated with an active EC2 instance. AWS charges for idle EIPs, so releasing these IPs can save costs.

Step 2: Create an AWS Lambda Function

AWS Lambda will check for idle IPs and release them if they meet your criteria.

Go to the AWS Lambda console.
Create a new Lambda function:
- Name: release-idle-eip
- Runtime: Python 3.x

Add the following code to the Lambda function:

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    # Retrieves all Elastic IPs
    addresses = ec2.describe_addresses()

    for address in addresses['Addresses']:
        # Check if the EIP is not associated with any instance or network interface
        if 'InstanceId' not in address and 'NetworkInterfaceId' not in address:
            try:
                # Release the idle EIP
                ec2.release_address(AllocationId=address['AllocationId'])
                print(f"Released idle Elastic IP: {address['PublicIp']}")
            except Exception as e:
                print(f"Failed to release Elastic IP {address['PublicIp']}: {e}")

    return {
        'statusCode': 200,
        'body': 'Idle Elastic IPs released if found.'
    }

Save the function.

Step 3: Set Up Permissions

To add custom permissions (like ec2:DescribeAddresses and ec2:ReleaseAddress) to your Lambda execution role, follow these steps:

Find or Create the Lambda Execution Role

Go to the IAM Console in AWS.
In the left menu, select Roles.
Find the existing Lambda execution role (e.g. release-idle-eip-role-49zlj0sl).

Add a Custom Inline Policy to the Role

Select the role name (release-idle-eip-role-49zlj0sl) to open its details page.
On the role’s details page, go to the Permissions tab.
Click Add permissions, and then select Create inline policy.

Add EC2 Permissions in the Policy Editor

In the Create policy editor, switch to the JSON tab.

Delete any default text in the editor and paste the following JSON policy, which includes permissions for DescribeAddresses and ReleaseAddress:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeAddresses",
        "ec2:ReleaseAddress"
      ],
      "Resource": "*"
    }
  ]
}

Click Review policy.

Name and Save the Inline Policy

Give the policy a name, such as EC2DescribeAndRelease.
Click Create policy to save it.

Verify Permissions

Once saved, the policy will be attached to your Lambda execution role. The Lambda function now has permission to:

List all Elastic IP addresses (DescribeAddresses).
Release any Elastic IP address (ReleaseAddress).

With these permissions, your Lambda function should now be able to detect and release idle Elastic IPs automatically.

Step 4: Schedule the Lambda Function with Amazon EventBridge (CloudWatch Events)

To run the Lambda function on a schedule, use Amazon EventBridge.

Go to the Amazon EventBridge console.
In the left menu, Select Schedules.
Choose Create schedule.
Add a schedule name and choose Recurring schedule
To schedule each hour you can add a cron like 0 * ? * * * and choose "Off" for the Flexible time window.
Select the Lambda function and select your lambda function release-idle-eip.
Save the schedule.

Step 5: Testing and Monitoring

Test the Lambda function to make sure it releases idle IPs as expected.
Monitor CloudWatch logs to confirm that the function runs without issues and successfully releases idle IPs.

Additional Tips

Tagging and Filtering: If you have some EIPs that you want to keep idle, use tags to filter them out in your Lambda function.
Notification: Consider setting up an SNS topic to notify you whenever an idle IP is released.

FIFO Messaging with Azure Service Bus and Spring Boot

Axel Dlv — Tue, 12 Nov 2024 14:10:14 +0000

Introduction

Azure Service Bus is a fully managed message broker that facilitates reliable communication between distributed applications. For applications that require messages to be processed in a specific order, such as ensuring a First-In-First-Out (FIFO) sequence, sessions in Azure Service Bus provide an effective mechanism for message processing.

In the context of a Spring Boot application, leveraging sessions on an Azure Service Bus topic ensures that messages with the same Session ID are handled in the correct order by a single consumer at a time. This solution is particularly useful when handling high-throughput messaging scenarios while maintaining message order.

This guide outlines how to configure a Spring Boot application to consume messages from Azure Service Bus in FIFO order, ensuring reliability and scalability by using sessions without the need for complex infrastructure.

In practice

For a Spring Boot application deployed on multiple instances to consume messages from an Azure Service Bus topic in FIFO order, you can use sessions on the topic and configure the application to manage sessions in a coordinated between the authorities. Here's how to do it:

1. Enable Sessions on the Azure Service Bus Topic

Make sure the Azure Service Bus topic and subscription are configured to use sessions.
When sending messages, set a Session ID on each message to ensure that all messages with the same Session ID will be processed in order of arrival by only one consumer at a time.

2. Use the Azure SDK to Manage Sessions in Spring Boot

Azure provides libraries for Java that allow messages to be consumed with sessions in an orderly manner. Here is one approach:

Add Azure Service Bus Dependency

Add the dependency for Azure Service Bus in your Spring Boot project (in pom.xml for Maven):

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-messaging-servicebus</artifactId>
    <version>7.5.0</version> <!-- check for the last version -->
</dependency>

Configure the Service Bus in the Application

Configure the application to connect to the Azure Service Bus topic. Here is a basic configuration:

import com.azure.messaging.servicebus.*;

@Service
public class AzureServiceBusConsumer {

    private final String connectionString = "Endpoint=sb://<your-service-bus>.servicebus.windows.net/;SharedAccessKeyName=<key-name>;SharedAccessKey=<key>";
    private final String topicName = "<your-topic>";
    private final String subscriptionName = "<your-subscription>";

    public void startSessionProcessor() {
        ServiceBusClientBuilder clientBuilder = new ServiceBusClientBuilder()
                .connectionString(connectionString);

        ServiceBusProcessorClient processorClient = clientBuilder
                .sessionProcessor() // Using session mode
                .topicName(topicName)
                .subscriptionName(subscriptionName)
                .processMessage(this::processMessage)
                .processError(this::processError)
                .buildProcessorClient();

        // Start session processing in asynchronous mode
        processorClient.start();
    }

    private void processMessage(ServiceBusReceivedMessageContext context) {
        ServiceBusReceivedMessage message = context.getMessage();
        System.out.printf("Processing message from session: %s. Contents: %s%n",
                message.getSessionId(), message.getBody());

        // Process the message here, respecting the order of arrival in the session
        context.complete(); // Mark the message as processed
    }

    private void processError(ServiceBusErrorContext context) {
        System.err.printf("Error occurred while processing: %s%n", context.getException().getMessage());
    }
}

Ensure Unique Consumption per Session

Using sessionProcessor() ensures that each session is consumed by only one instance at a time, and messages in a session are always processed in FIFO order, regardless of the number of instances.

3. Concurrency Management between instances

When multiple instances of the application are connected to the topic:

Azure Service Bus dynamically assigns sessions to available consumers.
Each Spring Boot instance receives messages one session at a time, and Azure Service Bus ensures that an active session will not be assigned to multiple consumers simultaneously.
This allows for automatic scaling, with each instance supporting different sessions in parallel.

4. Deployment and Resilience

Deploy the Spring Boot application on your instances with identical Service Bus configurations, so they can dynamically manage sessions. If an instance goes down, Azure Service Bus will automatically reassign the waiting session to another connected instance.

5. Launch Example in Spring Boot

Launch the consumer when your Spring Boot application starts, using @PostConstruct to start in-session consumption:

import javax.annotation.PostConstruct;
import org.springframework.stereotype.Service;

@Service
public class AzureServiceBusConsumer {

    // ... (configuration and previous code)

    @PostConstruct
    public void init() {
        startSessionProcessor();
    }
}

Conclusion

In conclusion, integrating Azure Service Bus with a Spring Boot application for FIFO message processing can be effectively achieved by utilizing sessions. By enabling sessions on the Azure Service Bus topic and associating messages with a specific Session ID, you ensure that messages are processed in the exact order they arrive, within each session.

Using the Azure SDK for Java, the Spring Boot application can be configured to consume messages in a session-based manner, guaranteeing that each session is processed by a single consumer at a time. This eliminates the risk of message reordering, even in a multi-threaded environment, ensuring reliable and ordered processing.

This approach offers a scalable and resilient solution, ensuring that the application processes messages in strict FIFO order while maintaining efficiency and flexibility in managing distributed workloads.