Creating an AWS Lambda function to process records in Amazon DynamoDB stream

In this article, I will show you how to create an AWS Lambda function which will process records in an Amazon DynamoDB Streams, as you do any data modification to the DynamoDB table - add, delete or update an item.

What is Lambda?

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or back-end service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and only pay for what you use.

Lambda Concepts used in this article

Function
You organize your code into Lambda functions. A function is a resource that you can invoke to run your code to process events in Lambda. You can invoke your function using the Lambda API, or you can configure an AWS service or resource like Amazon DynamoDB to invoke your function. Lambda runs your function only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume, and there is no charge when your code is not running.

Trigger
A trigger is a resource or configuration that invokes a Lambda function. Triggers include AWS services that you can configure to invoke a function and event source mappings.

Lambda event source mappings
An event source mapping is a resource in Lambda that reads items from a stream (DynamoDB) / queue (Amazon MQ or Amazon Simple Queue Service (Amazon SQS)) and invokes a function.

An event source mapping uses permissions in the function's execution role to read and manage items in the event source. Permissions, event structure, settings, and polling behavior vary by event source.

Deployment packages and .zip file archives
When you create a Lambda function, you package your function code into a deployment package.

Lambda supports two types of deployment packages: container images and .zip file archives.

The workflow to create a function depends on the deployment package type. A .zip file archive includes your application code and its dependencies.

When you create a function defined with a .zip file archive, you choose a code template, the language version, and the execution role for the function. You add your function code after Lambda creates the function.

You can use the Lambda console and the Lambda API to create a function defined with a .zip file archive. You can also upload an updated .zip file to change the function code.

What is DynamoDB stream?

A DynamoDB stream is a time-ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table. Whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record with the primary key attributes of the items that were modified. A stream record contains information about a data modification to a single item in a DynamoDB table. You can configure the stream so that the stream records capture additional information, such as the "before" and "after" images of modified items in near-real time.

• Each stream record appears exactly once in the stream.
• For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Please visit my GitHub Repository for DynamoDB articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

• Create a Lambda execution role
• Create the function
• Create a DynamoDB table with a stream enabled
• Add an event source in AWS Lambda
• Test the setup - Verify the logs in Amazon CloudWatch
• Cleanup

Pre-requisites:

• AWS user account with admin access, not a root account.
• Cloud9 IDE with AWS CLI.

Create a Lambda execution role

This role grants the function permission to access AWS services and resources. For example, you might create an execution role that has permission to send logs to Amazon CloudWatch.

When you invoke your function, Lambda automatically provides your function with temporary credentials by assuming this role. You don't have to call sts:AssumeRole in your function code.

From the Identity and Access Management (IAM) Console , Create a role using these parameters:

• Trusted entity type - Check AWS Service
• Use Case - Select Lambda
• Permissions policies - Search and select AWS Managed AWSLambdaDynamoDBExecutionRole
• Under Role details - Role name - lambda-dynamodb-role

AWSLambdaDynamoDBExecutionRole has the following permissions to list and read access to DynamoDB streams and write permissions to CloudWatch logs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeStream",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:ListStreams",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Create the function

To create the function

• Copy the sample code into a file named "index.js"

console.log('Loading function');

exports.handler = function(event, context, callback) {
    console.log(JSON.stringify(event, null, 2));
    event.Records.forEach(function(record) {
        console.log(record.eventID);
        console.log(record.eventName);
        console.log('DynamoDB Record: %j', record.dynamodb);
    });
    callback(null, "message");
};

• Zip up the sample code (index.js) to create a deployment package.

zip function.zip index.js

• Create a Lambda function with the create-function command

Create a DynamoDB table with a stream enabled

• To create a DynamoDB table

From the Amazon DynamoDB Console

Create table using these parameters

• Table name – lambda-dynamodb-stream
• Primary key – id (string)
• Create table

• Create table

• To enable streams

Open the Amazon DynamoDB Console

• Choose Tables

• Choose the lambda-dynamodb-stream table

• Under Exports and streams, choose DynamoDB stream details

• Choose Enable

• Choose Enable stream

Write down the stream ARN. You need this in the next step when you associate the stream with your Lambda function.

Add an event source in AWS Lambda

This event source mapping associates the DynamoDB stream with your Lambda function. After you create this event source mapping, AWS Lambda starts polling the stream.

Use the AWS CLI to map a function named ProcessDynamoDBRecords to a DynamoDB stream that its Amazon Resource Name (ARN) specifies, with a batch size of 100. After the command runs, note down the UUID.

No records processed (indicates that AWS Lambda has not started polling or that there are no records in the stream)

This creates a mapping between the specified DynamoDB stream and the Lambda function.

Test the setup - Verify the logs in Amazon CloudWatch

As and when you perform table updates, DynamoDB writes event records to the stream. As AWS Lambda polls the stream, it detects new records in the stream and invokes your Lambda function on your behalf by passing events to the function.

• In the DynamoDB console, add, update, and delete items to the table. DynamoDB writes records of these actions to the stream.

-- Delete ids 2000 and 2006

• AWS Lambda polls the stream and when it detects updates to the stream, it invokes your Lambda function by passing in the event data it finds in the stream.

• Your function runs and creates logs in Amazon CloudWatch. You can verify the logs reported in the Amazon CloudWatch console.

• deleted id 2000

• deleted id 2006

Cleanup

• delete the Lambda function
• delete the execution role
• delete the DynamoDB table

What we have done so far

Successfully created an AWS Lambda function to process records in an Amazon DynamoDB Streams, as you do add, delete or update an item in the DynamoDB table.