Serverless Data Processing: DynamoDB Streams, SNS, SQS, and Lambda

1. Introduction

An event-driven architecture utilizing DynamoDB Streams, AWS Lambda, Amazon SNS, and Amazon SQS is considered the gold standard for designing highly decoupled, scalable, and resilient systems in the cloud. In this tutorial, you will construct a complete, asynchronous data flow from scratch. The process begins when modifications occur in a DynamoDB table, which automatically triggers a data stream. This stream is continuously captured by a producer Lambda function, which processes the change and publishes a notification to an SNS topic. The SNS acts as a high-availability message router, performing a fan-out distribution of this notification to an SQS queue. Finally, a second consumer Lambda function reads the messages from the SQS queue to perform the final business logic processing. By the end of this guide, you will be able to deploy this robust infrastructure using Terraform alongside official AWS community modules. This approach is extremely useful because it ensures your infrastructure is version-controlled, highly maintainable, and inherently adopts security best practices, such as the principle of least privilege and fault tolerance.

2. Prerequisites

Before starting the infrastructure build, you must ensure that your development environment is properly prepared. You will need an active AWS account with administrative permissions to provision compute, messaging, database, and Identity and Access Management (IAM) resources. The AWS Command Line Interface (AWS CLI) must be installed and configured with your access credentials on your local machine. Additionally, Terraform (version 1.3.0 or higher) needs to be installed and globally accessible from your command terminal. Fundamental knowledge of Terraform's declarative syntax (HCL) and a basic understanding of how cloud messaging systems operate are highly recommended for the best learning experience. A modern code editor will greatly facilitate the creation and manipulation of the configuration files.

3. Step-by-Step

Step 1: Initial Configuration and AWS Provider

What to do: Create the root Terraform configuration file to define the required Terraform version and establish the connection with the AWS provider, explicitly specifying the region where the resources will be allocated.

Why do it: Terraform operates through plugins called providers. The AWS provider block instructs the tool on which API to use for authentication and resource provisioning. Declaring the region explicitly avoids unexpected behaviors and ensures that all interconnected services, such as DynamoDB and SNS, are deployed in the same geographical environment, significantly reducing the network latency of your architecture. Setting default tags also helps organize your resources for billing and management purposes.

Example (main.tf):

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      Environment = "Production"
      Project     = "EventDrivenArchitecture"
    }
  }
}

Step 2: Creation of the DynamoDB Table with Stream Enabled

What to do: Utilize the official DynamoDB Terraform module to create the main application table, ensuring that the Streams feature is activated with the view type configured to capture both new and old images of the modified items.

Why do it: The official module drastically simplifies the definition of partition keys and table attributes. Enabling the Stream is the foundational step of this entire architecture, as it acts as the initial trigger for our event pipeline. Setting the stream_view_type to NEW_AND_OLD_IMAGES guarantees that the consuming Lambda function will receive both the previous state of the modified item and its new state. This provides complete context, allowing your code to perform complex validations or delta comparisons during data processing.

Example (dynamodb.tf):

module "dynamodb_table" {
  source  = "terraform-aws-modules/dynamodb-table/aws"
  version = "~> 4.0"

  name           = "application-data-table"
  hash_key       = "id"
  billing_mode   = "PAY_PER_REQUEST"

  attributes = [
    {
      name = "id"
      type = "S"
    }
  ]

  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
}

Step 3: Provisioning the SNS Topic and SQS Queue

What to do: Implement the official modules for Amazon SNS and Amazon SQS. You must configure the access policy of the SQS queue to securely allow the SNS topic to publish messages directly into it, and then create the subscription linking them together.

Why do it: Using SNS combined with SQS implements the robust "Fan-out" messaging pattern. SNS manages the immediate delivery and routing, while SQS ensures temporary storage, persistence, and message retention in case the consumer Lambda is temporarily unavailable. Utilizing the official modules abstracts away the complexity of manually crafting JSON IAM policies. By enabling create_queue_policy = true and providing the allowed ARNs, the SQS module automatically manages the IAM permissions required to receive payloads from SNS, keeping your security posture tight and error-free.

Example (messaging.tf):

module "sns_topic" {
  source  = "terraform-aws-modules/sns/aws"
  version = "~> 6.0"

  name = "data-processing-topic"
}

module "sqs_queue" {
  source  = "terraform-aws-modules/sqs/aws"
  version = "~> 4.0"

  name = "data-processing-queue"

  create_queue_policy = true
  queue_policy_statements = {
    sns_publish = {
      sid     = "AllowSNSPublish"
      actions = ["sqs:SendMessage"]
      principals = [
        {
          type        = "Service"
          identifiers = ["sns.amazonaws.com"]
        }
      ]
      conditions = [
        {
          test     = "ArnEquals"
          variable = "aws:SourceArn"
          values   = [module.sns_topic.topic_arn]
        }
      ]
    }
  }
}

resource "aws_sns_topic_subscription" "sns_to_sqs" {
  topic_arn = module.sns_topic.topic_arn
  protocol  = "sqs"
  endpoint  = module.sqs_queue.queue_arn
}

Step 4: Deploying the Producer and Consumer Lambda Functions

What to do: Configure two instances of the AWS Lambda module. The first function (Producer) requires strict permissions to publish messages to the SNS topic. The second function (Consumer) requires permissions to read and consume messages from the SQS queue.

Why do it: Lambda functions require highly granular and well-defined permissions (Policies) attached to their execution identity (Role). The Terraform Lambda module handles the packaging of your source code and the creation of these Roles simultaneously. By passing inline JSON policies directly into the module, we guarantee that each Lambda possesses strictly the minimum level of access necessary to perform its specific task. This prevents security vulnerabilities that arise from excessive or wildcard permissions.

Example (lambdas.tf):

module "lambda_producer" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.0"

  function_name = "dynamodb-stream-producer"
  handler       = "index.handler"
  runtime       = "nodejs20.x"
  source_path   = "../src/producer"

  environment_variables = {
    SNS_TOPIC_ARN = module.sns_topic.topic_arn
  }

  attach_policy_json = true
  policy_json = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["sns:Publish"]
        Effect   = "Allow"
        Resource = module.sns_topic.topic_arn
      }
    ]
  })
}

module "lambda_consumer" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.0"

  function_name = "sqs-message-consumer"
  handler       = "index.handler"
  runtime       = "nodejs20.x"
  source_path   = "../src/consumer"
}

Step 5: Mapping Event Sources to Triggers

What to do: Create the native aws_lambda_event_source_mapping resources to actively connect the DynamoDB Stream to the Producer Lambda, and the SQS Queue to the Consumer Lambda. You must also attach the AWS managed execution policies to the respective Lambda roles.

Why do it: While the Lambda module creates the compute resource, the "event source mapping" is the internal AWS polling service that actively reads the Stream or the Queue and invokes your function. This mapping mechanism requires specific operational policies (AWSLambdaDynamoDBExecutionRole and AWSLambdaSQSQueueExecutionRole) to function correctly. Integrating these components declaratively via Terraform centralizes the control of your end-to-end event flow, ensuring that infrastructure changes are applied consistently.

Example (triggers.tf):

# Attach Stream reading permissions to the Producer Lambda
resource "aws_iam_role_policy_attachment" "producer_dynamodb_policy" {
  role       = module.lambda_producer.lambda_role_name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaDynamoDBExecutionRole"
}

# Integrate DynamoDB Stream with Producer Lambda
resource "aws_lambda_event_source_mapping" "dynamodb_trigger" {
  event_source_arn  = module.dynamodb_table.dynamodb_table_stream_arn
  function_name     = module.lambda_producer.lambda_function_arn
  starting_position = "LATEST"

  depends_on = [aws_iam_role_policy_attachment.producer_dynamodb_policy]
}

# Attach SQS reading permissions to the Consumer Lambda
resource "aws_iam_role_policy_attachment" "consumer_sqs_policy" {
  role       = module.lambda_consumer.lambda_role_name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
}

# Integrate SQS with Consumer Lambda
resource "aws_lambda_event_source_mapping" "sqs_trigger" {
  event_source_arn = module.sqs_queue.queue_arn
  function_name    = module.lambda_consumer.lambda_function_arn
  batch_size       = 10

  depends_on = [aws_iam_role_policy_attachment.consumer_sqs_policy]
}

4. Common Troubleshooting

Even when following best practices, distributed systems can present challenges regarding network configuration and IAM permissions. Below are the most common problems you might encounter and how to efficiently solve them:

1. Producer Lambda is not invoked by DynamoDB: This almost always occurs due to IAM permission failures. The execution Role of the Lambda must possess the dynamodb:GetRecords, dynamodb:GetShardIterator, dynamodb:DescribeStream, and dynamodb:ListStreams permissions. Verify that the AWSLambdaDynamoDBExecutionRole policy was successfully attached in your Terraform state.
2. SQS Queue does not receive messages from the SNS Topic: The SNS subscription protocol dictates that the endpoint (the SQS queue) must explicitly authorize the reception of messages. Confirm that the SQS access policy contains a statement allowing the sqs:SendMessage action, with the principal set to the sns.amazonaws.com service, and a restrictive condition matching the exact ARN of your SNS topic. This prevents unauthorized cross-account access.
3. Duplicate processing or infinite retries in the Consumer Lambda: If your code within the consumer Lambda fails and throws an unhandled exception, the message will return to the SQS queue after the visibility timeout expires and will be reprocessed, creating an infinite loop of errors. To solve this, wrap your logic in try-catch blocks and implement a Dead Letter Queue (DLQ). Configure the maxReceiveCount property on your main SQS queue to safely route "poison pill" messages to the DLQ after a limited number of failed attempts.

5. Conclusion

In this tutorial, you have comprehensively understood and practically applied the creation of a highly scalable, asynchronous architecture on AWS. The combination of DynamoDB Streams, Lambda functions, SNS, and SQS forms a resilient, event-driven system where each isolated component performs its specific duty flawlessly. Utilizing Terraform with official modules ensured agility, security, and exceptional structural code organization. As next steps to evolve this ecosystem, it is highly recommended to implement KMS (Key Management Service) encryption for your SNS topics and SQS queues, build CI/CD pipelines to automate the provisioning process, and create specialized CloudWatch dashboards to monitor message age and function error metrics.