DEV Community: Rakesh Sanghvi

Enrich DynamoDB stream (List/Map) using EventBridge Pipes and Step Functions

Rakesh Sanghvi — Fri, 26 May 2023 21:56:21 +0000

Use Case:-

I am sure you all remember how much boilerplate code we used to write when we convert the DynamoDB stream to the business object. Here is a small snippet of the GoLang program.

    m := &EntitySearch{}

    for name, value := range recordImage {
        if name == "ExternalId" {
            m.ExternalId = value.String()
        } else if name == "Source" {
            m.Source = value.String()
        } else if name == "EntityId" {
            m.EntityId = value.String()
        } else if name == "EntityType" {
            m.EntityType = value.String()
        }

During ReInvent 2022, the introduction of EventBridge pipes brought about a significant transformation in how we utilize the DynamoDB stream by converting it into business objects. This breakthrough has nearly eliminated the need for writing Lambda code for the conversion process. But what if you have a DynamoDB List/Map object in your stream? EventBridge pipes don't support all JSON Path.

What if you have List or Map objects like this respectively?

"Skills": [
  "AWS",
  "GoLang",
  "Java",
  "TypeScript"
]

{
  "orderItems": [
    {
      "itemId": "1234567890",
      "quantity": 1,
      "price": 10.00
    },
    {
      "itemId": "9876543210",
      "quantity": 2,
      "price": 5.00
    }
  ]
}

if you’ve played with EventBridge Pipes you know that you can do a bit of a transform in target if you try to access Skills objects like this "$.dynamodb.NewImage.Skills.L[*].S", you will get an error

Here is the solution where I used the Pipeline Enrichment step with the Step Function. Step Function support fully JSON path.

The Architecture:

Step Functions:-

You can download the source code from here, which has been implemented using AWS CDK and SAM, whichever approach you prefer.

CDK Deploy:-

1) cd cdk
2) npm install
3) cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>

SAM Deploy:-

1) cd sam
2) sam deploy --stack-name EventBridgePipeEnrich --capabilities    CAPABILITY_NAMED_IAM --guided --profile <profile name>

Insert Data in DynamoDB:-

1) cd dynamodb
2) ./dynamodb.sh

You will see the below record in the DynamoDB table.

Here are the final Cloudwatch logs after an Enrichment.

Understand State Machine:-

I am utilizing straightforward Step Functions with an Inline Map to iterate over a DynamoDB Stream. It's worth noting that in this case, I am using the expression 'Skills.$' to refer to the 'Skills' attribute within the 'NewImage' object of the DynamoDB record, where the values are stored as a list of strings ('L[*].S').

{
  "Comment": "A description of my state machine",
  "StartAt": "DynamoDB Map",
  "States": {
    "DynamoDB Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        },
        "StartAt": "Transform",
        "States": {
          "Transform": {
            "Type": "Pass",
            "Parameters": {
              "details": {
                "meta-data": {
                  "correlationId.$": "$.eventID",
                  "eventName.$": "$.eventName"
                },
                "data": {
                  "PK.$": "$.dynamodb.Keys.PK.S",
                  "SK.$": "$.dynamodb.Keys.SK.S",
                  "Skills.$": "$.dynamodb.NewImage.Skills.L[*].S",
                  "Contact": {
                    "Home.$": "$.dynamodb.NewImage.Contact.M.Home.S",
                    "Phone.$": "$.dynamodb.NewImage.Contact.M.Phone.S"
                  }
                }
              }
            },
            "InputPath": "$",
            "End": true
          }
        }
      },
      "End": true
    }
  }
}

DynamoDB table and Stream:-

To ensure that EventBridge pipes can read it, we must activate the DynamoDB stream. Reading Both the new and the old images of the item.

   this._table = new dynamodb.Table(this, id, {
   billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
   removalPolicy: cdk.RemovalPolicy.DESTROY,
   partitionKey: { name: 'PK', type: dynamodb.AttributeType.STRING },
   sortKey: { 
            name: 'SK', type: dynamodb.AttributeType.STRING },
            tableName: `Employee`,
            stream: StreamViewType.NEW_AND_OLD_IMAGES
        });

EventBridge Pipes:-

Here is how I set up EventBridge pipes.

 const pipe = new pipes.CfnPipe(this, 'pipe', {
            name: 'employee-app-pipe',
            roleArn: props.employeePipeRole.roleArn,
            source: props.table.tableStreamArn!,
            target:  props.employeeEventBus.targetArn,
            sourceParameters: {
                dynamoDbStreamParameters: {
                    startingPosition: 'LATEST',
                    batchSize: 1,
                    deadLetterConfig: {
                        arn: props.employeeDLQueue.queueArn,
                    },
                    maximumRetryAttempts: 1,
                },
                filterCriteria: {
                    filters: [{
                     pattern: '{ "eventName": ["INSERT","MODIFY"] }',
                    }],
                }
            },
       enrichment: props.employeeAppStateMachine.stateMachine.attrArn,
            targetParameters: {
                eventBridgeEventBusParameters: {
                    detailType: 'EmployeeDetailsChanged',
                    source: 'employee-app',
                },
            },
        });

Reading LATEST from the DynamoDB stream.
Reading 1 Item in batch due to Sample Example.
Make sure you set "maximumRetryAttempts" value. By default value is -1 and it will keep retrying for the next 24 hours. you can read more here.
For Enrichment steps, call Step Functions props.employeeAppStateMachine.stateMachine.attrArn

Wrap-Up:-

In the provided example, we demonstrate the utilization of EventBridge Enrichment for processing complex JSON structures, specifically DynamoDB List/Map data types. This approach is necessary because EventBridge pipes lack complete support for JSON path expressions. Employing this example will prove useful when working with intricate JSON structures.

I hope you enjoy reading this post, and I sincerely hope it proves beneficial to you during your implementation process.

EventBridge Scheduler- Start/Stop ECS instance

Rakesh Sanghvi — Fri, 05 May 2023 03:03:14 +0000

As I mentioned in my prior post, we used the ECS Fargate instance in my organization. During off hours, we shut down the ECS Fargate instance every day in our lower environment to cut costs.

Here is a prior article where I have explained about Start/Stop EC2 Instnace. This is a simple pattern that starts and stops the ECS Fargate instance using Universal Target.

Important:- This code will shut down 1 ECS instance as API doesn't support multiple ones. In a future article, I will show you how to shut down multiple instances using Step Functions.

API Target and Parameters.

Target :- arn:aws:scheduler:::aws-sdk:ecs:updateService

{
  "Service": "ecs-service",
  "Cluster": "MyCluster",
  "DesiredCount": 0
}

This example uses AWS CLI and CDK (TypeScript). Assuming you already Bootstrapping your account.

Here is the main stack which creates the nested stacks of role, ECSStart and ECSStop.

import {Stack, StackProps} from 'aws-cdk-lib';
import {Construct} from 'constructs';
import {SchedulesRole} from "./scheduler-role";
import {ECSStart} from "./ecs-start";
import {ECSStop} from "./ecs-stop";

export class ECSInstanceStartStopStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);


    // Create Role
    const schedulesRole = new SchedulesRole(this,"SchedulerRoleStack")

    // Create ECSStart schedules
    const ecsStart = new ECSStart(this,"EcsStart", {
      roleArn: schedulesRole.roleArn,
    })

    // Create ECSStop schedules
    const ecsStop = new ECSStop(this,"EcsStop", {
      roleArn: schedulesRole.roleArn,
    })

  }
}

Give minimal permission to start and stop ECS instances.

import { StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import {Effect, PolicyStatement, Role, ServicePrincipal} from "aws-cdk-lib/aws-iam";

export class SchedulesRole extends cdk.NestedStack  {
    private readonly _role: Role;
    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props);


        // Add scheduler assumeRole
        this._role  = new Role(this,  "scheduler-ecs-start-stop", {
          assumedBy: new  ServicePrincipal('scheduler.amazonaws.com'),
          roleName: "scheduler-ecs-start-stop"
        })

        // Add policy
        this._role.addToPolicy(  new PolicyStatement( {
            sid: 'ECSStartStopPermissions',
            effect: Effect.ALLOW,
            actions: [
                "ecs:List*",
                "ecs:Describe*",
                "ecs:UpdateService"
            ],
            resources: ["*"], //Give the least privileges
        }))
    }

    get roleArn(): string {
        return this._role.roleArn;
    }
}

Create an ECS start schedule. ECS instance will start at 8 AM CST time.

import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import { Role } from "aws-cdk-lib/aws-iam";
import {CfnSchedule} from "aws-cdk-lib/aws-scheduler";

interface ECSStartProps extends cdk.NestedStackProps {
    roleArn: string
}

export class ECSStart extends cdk.NestedStack {
    private readonly role: Role;

    constructor(scope: Construct, id: string, props: ECSStartProps) {
        super(scope, id, props);

        // Start ECS Instance 8 am Central Time
        new CfnSchedule(this,"ecs-start-scheduler", {
            name: "ecs-start-scheduler",
            flexibleTimeWindow: {
                mode: "OFF"
            },
            scheduleExpression: "cron(0 8 ? * * *)",
            scheduleExpressionTimezone: 'America/Chicago',
            description: 'Event that start ECS instances',
            target: {
                arn: 'arn:aws:scheduler:::aws-sdk:ecs:updateService',
                roleArn: props.roleArn,
                input: JSON.stringify
                    ({ 
                       Service: "ecs-service", 
                       Cluster: "MYCluster", 
                       DesiredCount: 1 
                    }),                           
            },
        });
    }
}

Create an ECS stop schedule.

import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import { Role } from "aws-cdk-lib/aws-iam";
import {CfnSchedule} from "aws-cdk-lib/aws-scheduler";

interface ECSSopProps extends cdk.NestedStackProps {
    roleArn: string
}

export class ECSStop extends cdk.NestedStack {
    private readonly role: Role;

    constructor(scope: Construct, id: string, props: ECSSopProps) {
        super(scope, id, props);

        // Start ECS Instance 8 am Central Time
        new CfnSchedule(this,"ecs-stop-scheduler", {
            name: "ecs-stop-scheduler",
            flexibleTimeWindow: {
                mode: "OFF"
            },
            scheduleExpression: "cron(0 20 ? * * *)",
            scheduleExpressionTimezone: 'America/Chicago',
            description: 'Event that start ECS instances',
            target: {
                arn: 'arn:aws:scheduler:::aws-sdk:ecs:updateService',
                roleArn: props.roleArn,
                input: JSON.stringify(
                  { 
                    Service: 'ecs-service', 
                    Cluster: 'MyCluster', 
                    DesiredCount: 0 
                  }),
            },
        });
    }
}

Caution:- Please make sure about your ECS Auto Scaling policy. In my case, ECS instances were kept recycling during start/stop due to the auto-scaling policy was not configured properly. This caused Docker image to keep downloading and costing more $$$.

Deployment Instructions:-

1) git clone git@github.com:awsmantra/eventbridge-schedules-ecs-start-stop.git

2) cd eventbridge-schedules-ecs-start-stop

3) modified ecs-service name and cluster name ecs-start.ts and ecs-stop.ts

4) npm install

5) cdk deploy --all -a "npx ts-node bin/app.ts" --profile dev

Cleanup:-

cdk destroy --profile dev

You can download the source code form here.

EventBridge Scheduler- Start/Stop EC2 instance

Rakesh Sanghvi — Sun, 30 Apr 2023 20:19:23 +0000

It has been always challenging to bring down Cloud costs. In my organization, we have EC2, RDS, and ECS instances that we can shut down during off hours in our lower environment. This is a simple pattern that starts and stops ec2 instances using Universal Target. Using a universal target we can call the underlying AWS API without writing a single line of code using AWS SDK. As of now, it's having more than 270+ API calls.

This example uses AWS CLI, and CDK (Typescript). Assuming you already Bootstrapping your account to deploy the CDK app.

Here is the main stack which creates nested stacks of role, EC2Start and EC2Stop.

import {Stack, StackProps} from 'aws-cdk-lib';
import {Construct} from 'constructs';
import {SchedulesRole} from "./scheduler-role";
import {Ec2Start} from "./ec2-start";
import {Ec2Stop} from "./ec2-stop";

export class EC2InstanceStartStopStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);


    // Create Role
    const schedulesRole = new SchedulesRole(this,"SchedulerRoleStack")

    // Create EC2Start schedules
    const ec2Start = new Ec2Start(this,"Ec2Start", {
      roleArn: schedulesRole.roleArn,
    })

    // Create EC2Stop schedules
    const ec2Stop = new Ec2Stop(this,"Ec2Stop", {
      roleArn: schedulesRole.roleArn,
    })

  }
}

Give minimal permission to start/stop EC2 instances

import { StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import {Effect, PolicyStatement, Role, ServicePrincipal} from "aws-cdk-lib/aws-iam";

export class SchedulesRole extends cdk.NestedStack  {
    private readonly _role: Role;
    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props);


        // Add scheduler assumeRole
        this._role  = new Role(this,  "scheduler-ec2-start-stop", {
           assumedBy: new ServicePrincipal('scheduler.amazonaws.com'),
            roleName: "scheduler-ec2-start-stop"
        })

        // Add policy
        this._role.addToPolicy(  new PolicyStatement( {
            sid: 'EC2StartStopPermissions',
            effect: Effect.ALLOW,
            actions: [
                "ec2:DescribeInstances",
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            resources: ["*"], //Give the least privileges
        }))
    }

    get roleArn(): string {
        return this._role.roleArn;
    }
}

Caution:- if the underlying EBS volume attached to EC2 is encrypted, please make sure you give appropriate KMS permission in the above policy.

Create an Ec2Start schedule. This will start all EC2 Instances at 8 AM CST time. AWS EventScheudler automatically adjusts DST time. See AWS StartInstances API

import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import { Role } from "aws-cdk-lib/aws-iam";
import {CfnSchedule} from "aws-cdk-lib/aws-scheduler";

interface Ec2StartProps extends cdk.NestedStackProps {
    roleArn: string
}

export class Ec2Start extends cdk.NestedStack {
    private readonly role: Role;

    constructor(scope: Construct, id: string, props: Ec2StartProps) {
        super(scope, id, props);

        // Start all EC2 Instance 8 am Central Time
        new CfnSchedule(this,"ec2-start-scheduler", {
            name: "ec2-start-scheduler",
            flexibleTimeWindow: {
                mode: "OFF"
            },
            scheduleExpression: "cron(0 8 ? * * *)",
            scheduleExpressionTimezone: 'America/Chicago',
            description: 'Event that start EC2 instances',
            target: {
                arn: 'arn:aws:scheduler:::aws-sdk:ec2:startInstances',
                roleArn: props.roleArn,
               input: JSON.stringify(
               { InstanceIds:['i-05c757e84518Z1234',
                              'i-0e8cf751ca6dD9678']}),
            },
        });
    }
}

Create an EC2Stop schedule. See AWS StopInstances API

import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import { Role } from "aws-cdk-lib/aws-iam";
import {CfnSchedule} from "aws-cdk-lib/aws-scheduler";

interface Ec2SopProps extends cdk.NestedStackProps {
    roleArn: string
}

export class Ec2Stop extends cdk.NestedStack {
    private readonly role: Role;

    constructor(scope: Construct, id: string, props: Ec2SopProps) {
        super(scope, id, props);

        // Start all EC2 Instance 8 am Central Time
        new CfnSchedule(this,"ec2-stop-scheduler", {
            name: "ec2-stop-scheduler",
            flexibleTimeWindow: {
                mode: "OFF"
            },
            scheduleExpression: "cron(0 20 ? * * *)",
            scheduleExpressionTimezone: 'America/Chicago',
            description: 'Event that start EC2 instances',
            target: {
                arn: 'arn:aws:scheduler:::aws-sdk:ec2:stopInstances',
                roleArn: props.roleArn,
                input: JSON.stringify(
                      { InstanceIds: ['i-05c757e84518Z1234',
                                      'i-0e8cf751ca6dD9678']}),
            },
        });
    }
}

Deployment Instructions :-

1) git clone git@github.com:awsmantra/eventbridge-schedules-ec2-start-stop.git

2) cd eventbridge-schedules-ec2-start-stop

3) npm install

4) cdk deploy --all -a "npx ts-node bin/app.ts" --profile dev

Cleanup :-

cdk destroy --profile dev

You can download the source code form here

All Complex systems that work evolved from simpler systems that worked --Gall's Law

How to save money on AWS Transfer Family(SFTP) Server- Non-Production use

Rakesh Sanghvi — Fri, 21 Apr 2023 22:24:39 +0000

Use Case:-

Our company has an SFTP server in the lower environment, but we don't have regular traffic on it. Our QA team only uses it once every two weeks during production release testing. If you create an AWS SFTP server, you will be charged $0.30 per hour, and AWS will continue to charge you until you delete the server. When attempting to stop the SFTP server, you will receive the following message.

Our expenditure on SFTP servers in lower environments is nearly $750 per month, despite our infrequent usage throughout the entire month. Can we tear up/down during off hours or can we create SFTP sever on Demand? Here is what I came up with a solution.

The Architecture:

To make it simple I created a simple Step Function, we can extend this functionality if we want to call Step Function from API Gateway or create an EventBridge schedule.

Create SFTP serve step function:-

Delete SFTP server step function:-

Steps:-

Configured EventBridge Scheduler.
Create a Step Function State Machine using AWS ASL language.
Add user information necessary for SFTP user creation, including the user's ID, S3 bucket path, and SSH public key, to the static data stored in DynamoDB. Additionally, once the SFTP server is successfully created, store its ID in this table. This id will use to delete the SFTP server.

You can download the source code from here, which has been implemented using AWS CDK and SAM, whichever approach you prefer.

CDK Deploy:-

1) cd cdk
2) npm install
3) cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>

SAM Deploy:-

1) cd sam
2) sam deploy --stack-name SFTPAppStack --capabilities    CAPABILITY_NAMED_IAM --guided --profile <profile name>

Populate Static Data in DynamoDB:-

1) cd dynamodb
2) ./dynamodb.sh

You will see the below records in the DynamoDB table.

you can use this link to generate your private key and public key. Modify your public key in the DynamoDB script.

Populate Dummy Data in S3:-

1) cd s3
2) ./s3.sh

Understand State Machine:-
I am using the AWS SDK integration API to create an SFTP server with the help of the endpoint "arn:aws:states:::aws-sdk:transfer:createServer". There are various parameters available for creating the server, but it's important to ensure the security aspect is verified if you plan to create an SFTP server in your account.

 "CreateServer": {
      "Type": "Task",
      "Next": "Save ServerId",
      "Parameters": {
        "Domain": "S3",
        "EndpointType": "PUBLIC",
        "IdentityProviderType": "SERVICE_MANAGED",
        "LoggingRole": "${SFTPBaseRole}",
        "Protocols": [
          "SFTP"
        ],
        "SecurityPolicyName": "TransferSecurityPolicy-2018-11"
      },
      "Resource": "arn:aws:states:::aws-sdk:transfer:createServer"
    },

Above execution will return sever-id which I am saving into the DynamoDB table. This server-id will be used to delete the SFTP server. Here is the code snippet to save server-id into the DynamoDB table using SDK endpoint "arn:aws:states:::dynamodb:putItem"

"Save ServerId": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:putItem",
      "Parameters": {
        "TableName": "${SFTPTable}",
        "Item": {
          "PK": {
            "S": "SFTP"
          },
          "SK": {
            "S": "ServerId"
          },
          "ServerId": {
            "S.$": "$.ServerId"
          }
        }
      },
      "Next": "Get SFTP User",
      "ResultPath": null
    }

Now I am fetching all SFTP users from the DynamoDB table. This users list we can iterate using Map and pass it to endpoint "arn:aws:states:::awssdk:transfer:createUser".

"Get SFTP User": {
      "Type": "Task",
      "Parameters": {
        "TableName": "${SFTPTable}",
        "KeyConditionExpression": "PK = :v1 AND begins_with(SK,:v2)",
        "ExpressionAttributeValues": {
          ":v1": {
            "S": "SFTP"
          },
          ":v2": {
            "S": "USER"
          }
        }
      },
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:query",
      "Next": "Iterate User",
      "ResultPath": "$.context"

Iterate all users list using the map.

    "Iterate User": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        },
        "StartAt": "Create SFTP User",
        "States": {
          "Create SFTP User": {
            "Type": "Task",
            "End": true,
            "Parameters": {
              "ServerId.$": "$.ItemSelector.ServerId",
              "HomeDirectory.$": "$.ItemSelector.Value.S3Path.S",
              "HomeDirectoryType": "PATH",
              "Role": "${SFTPBaseRole}",
           "SshPublicKeyBody.$":"$.ItemSelector.Value.SSHPublicKey.S",
           "UserName.$":   "States.ArrayGetItem(States.StringSplit($.ItemSelector.Value.SK.S, '#'), 1)"
            },
            "Resource": "arn:aws:states:::aws-sdk:transfer:createUser"
          }
        }
      },
      "End": true,
      "ItemsPath": "$.context.Items",
      "ItemSelector": {
        "ItemSelector": {
          "ServerId.$": "$.ServerId",
          "Value.$": "$$.Map.Item.Value"
        }
      }
    }

The $$.Map.Item.Value.Value context object contains the value of each data item. You can use $$.Map.Item.Value if you want to add/modify JSON elements in iteration. You can see I am adding the new attribute "ServerId.$": "$.ServerId"

Login to SFTP Server:-

You should be able to log in using the following hostname. I am using the us-west-2 region. Modify according to your region. Use any user you have created with your private key.

.server.transfer.us-west-2.amazonaws.com

Delete SFTP Server:-

Getting ServerId from DynamoDB and passing this value to endpoint "arn:aws:states:::aws-sdk:transfer:deleteServer"

{
    "Comment": "A description of my state machine",
    "StartAt": "Get SFTP ServerId",
    "States": {
      "Get SFTP ServerId": {
        "Type": "Task",
        "Resource": "arn:aws:states:::dynamodb:getItem",
        "Parameters": {
          "TableName": "${SFTPTable}",
          "Key": {
            "PK": {
              "S": "SFTP"
            },
            "SK": {
              "S": "ServerId"
            }
          }
        },
        "Next": "DeleteServer",
        "OutputPath": "$.Item"
      },
      "DeleteServer": {
        "Type": "Task",
        "End": true,
        "Parameters": {
          "ServerId.$": "$.ServerId.S"
        },
        "Resource": "arn:aws:states:::aws-sdk:transfer:deleteServer"
      }
    }
  }

Cleanup:-

sam delete --stack-name SFTPStack --profile <your profile name>
                               OR
cd cdk
cdk destroy --profile <your profile name>

Wrap-Up:-

By creating or deleting a server during off hours, we can observe how much money can be saved. At my company, we have made the decision to generate an SFTP server on an as-needed basis. I'm confident that our costs will be less than $10 per month, in contrast to the $750 expense for a non-production environment. As I mentioned earlier, this post is not about how to create a secured SFTP server, please ensure the security aspect if you are creating an SFTP server in your account. Hope you like this post and helps you to reduce your AWS $$$ bill.

EventBridge Schedule to Step Functions - No-Code/No-Lambda

Rakesh Sanghvi — Fri, 14 Apr 2023 13:00:00 +0000

Use Case:-

We have a business use case where we need to call our nightly job for multiple tenants. Initially, I planned to use EventBridge Bus and Rules to achieve this goal. However, we discovered that EventBridge Rules has a limit of 5 targets, which would prevent us from adding more tenants as input for the nightly job. I want to avoid maintaining multiple rules for the same use case, so I came up with this solution. I believe No-Code/Less-Code approach using AWS Step Functions and EventBridge is the best way forward for our development needs.

The Architecture:-

Steps:-

Configure EventBridge Schduler. You can read my prior 2 posts here and here.
Populate DynamDB static data.
Create a Step Function State Machine using AWS ASL language.

You can download the source code from here, which has been implemented using AWS CDK and SAM, whichever approach you prefer.

CDK Deploy:-

1) cd cdk
2) npm install
3) cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>

SAM Deploy:-

1) cd sam
2) sam deploy --stack-name LegacyAppStack --capabilities    CAPABILITY_NAMED_IAM --guided --profile <profile name>

Populate Static Data in DynamoDB:-

1) cd dynamodb
2) ./dynamodb.sh

You will see the below records in the DynamoDB table.

Understand State Machine:-

AWS step functions are integrated with more than 200+ AWS services and you can call 9K+ API from step functions. There are 2 ways you can integrate step functions with other services, either using AWS SDK Integration or you can use Optimized integration. In this example, I am using Default Response (Request/Response) optimized integration pattern with Express workflow.

"GetTenantList": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:getItem",
      "Parameters": {
        "TableName": "LegacyApp",
        "Key": {
          "PK": {
            "S": "METADATA"
          },
          "SK": {
            "S": "METADATA"
          }
        }
      },
      "ResultPath": "$.context",
      "OutputPath": "$.context.Item.TenantList.L",
      "Next": "IterateTenantList"
    }

I am utilizing "arn:aws:states:::dynamodb:getItem" API to retrieve data from DynamoDB by passing the primary key (PK) and sort key (SK). Step functions offer access to all CRUD API operations available with DynamoDB. After filtering output with the ResultPath and OutputPath, the output of the aforementioned operation would be as follows.

[
  {
    "S": "TENANT#123456"
  },
  {
    "S": "TENANT#456789"
  }
]

Now I am iterating TenantList using Map. The Map states concurrently iterate over a collection of items in a dataset, such as a list of S3 Objects, CSV file or JSON object. it repeats a set of steps for each item in the collection.

 "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        }

The Map states have an "Inline" mode where they can only accept input in the form of a JSON array. When operating in this mode, the map states can handle up to 40 concurrent iterations. However, the Map states also offer a "Distributed" mode that can support up to 10,000 parallel child workflows. I am planning to write a blog about these features soon.

Next, I am getting messages from DynamoDB and transforming them before sending them to SNS.

"StartAt": "GetTenant",
        "States": {
          "GetTenant": {
            "Type": "Task",
            "Resource": "arn:aws:states:::dynamodb:getItem",
            "Parameters": {
              "TableName": "LegacyApp",
              "Key": {
                "PK": {
                  "S.$": "$"
                },
                "SK": {
                  "S.$": "$"
                }
              }
            },
            "InputPath": "$.S",
            "Next": "TransformMessage",
            "ResultSelector": {
              "message.$": "States.StringToJson($.Item.Message.S)"
            }
          },
          "TransformMessage": {
            "Type": "Pass",
            "Parameters": {
              "version.$": "$.message.version",
              "source.$": "$.message.source",
              "eventType.$": "$.message.eventType",
              "details": {
                "tenantId.$": "$.message.details.tenantId"
              },
              "correlationId.$": "States.UUID()"
            },
            "Next": "SendMessageToLegacyApp"
          }

Step functions provide several intrinsic functions. You can see here I am using "States.StringToJson" and "States.UUID". Intrinsic functions will help you to perform basic data processing operations without using a Task state. Here is the output of the above 2 tasks.

{
  "details": {
    "tenantId": 123456
  },
  "correlationId": "f43e0b5b-c187-483f-800e-44fb71486494",
  "source": "SampleApp",
  "eventType": "LegacyApp",
  "version": "1.0"
}

In the last step, I am sending messages to the SNS topic using the following code.

 "SendMessageToLegacyApp": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sns:publish",
            "Parameters": {
              "Message.$": "$",
              "TopicArn": "${LegacyAppSNSPath}"
            },
            "End": true
          }

Cleanup:-

sam delete --stack-name LegacyAppStack --profile <your profile name>
                               OR
cdk destroy --profile <your profile name>

Wrap-Up:-

We have not utilized any Lambda code in our implementation, relying entirely on AWS API integration. I believe that each line of code in any service is a liability, it carries a certain amount of risk, and it requires ongoing maintenance, monitoring, and the creation of unit/integration test cases. Let me know if you have any better way to solve this use case

Code is more art to me. I don’t want computer art - Ben Pyle

100 millions Lambda execution and guess our AWS $$$ Bill - GoLang

Rakesh Sanghvi — Thu, 06 Apr 2023 13:00:00 +0000

Little Background:-

in 2019 we started moving from server to serverless and started adopting AWS Lambda + GoLang for any new development. My chief architect and current CTO Ben Pyle(Blog) came up with the GoLang pattern which we used in all our Microservices. It was one of the best architectural decisions we have made in recent times.

As of now, we have more than 300+ lambdas deployed in each of our environments. 99% of those lambdas are written in GoLang and the rest 1% in either Node, Python or Java. Most of us had a Java background in my organization when we started in 2019. A couple of GoLang training sessions have helped us in a smooth transition.

This post will help you to understand and implement AWS Lambda written in GoLang.

AWS Bill:-

Here is a screenshot of our recent month's bill. You can see we had 98+ million lambda requests and our total AWS bill for lambda was $108 after our compute savings plan.

Keep 3 things in mind when you write Lambda.

1) Runtime
2) Package Size
3) Startup logic.
The majority of the Go Lambda that we have in my organization is configured with 128 MB memory (that's the minimum memory we can configure for AWS Lambda).
Go executable binary size is approx 8 MB for most of our Lambda. Fewer footprints for runtime.
Here are a couple of runtime examples. You can see the COLD START time in this example is 210.65 ms and it used 45 MB of memory. As GoLang compiles code to native code and executes pretty fast, most of the time cold start duration will be a few milliseconds to 1 second.

 Duration: 3.93 ms Billed Duration: 4 ms Memory Size: 
 128 MB Max Memory Used: 44 MB Init Duration: 210.65 ms 

 Duration: 1.36 ms Billed Duration: 2 ms Memory Size: 
 128 MB Max Memory Used: 45 MB 

 Duration: 16.44 ms Billed Duration: 17 ms Memory Size: 
 128 MB Max Memory Used: 45 MB

Examples:-

We follow industry best practices for implementing AWS Lambda.

1. Optimizing static initialization:-
This is the "INIT" code that happens outside of the handler. Initialized your variable or connection to other services.

2. Single-purpose lambdas:-
each lambda has a single responsibility. For any CRUD operation, we implement 4 lambdas (Create, Update, Delete and Get)

This example is implemented using AWS CDK and AWS SAM, whichever is your preferred way, clone the repo from GitHub and try it...

You can download the source code from here

AWS SAM Deployment Instructions:-

1) change the region name in the template.yaml file if you are deploying other than us-west-2

  Region:
    Type: String
    Description: AWS Region Name
    Default: us-west-2

2) Changed directory to pattern directory sam

cd sam

3) From the command line, use the AWS SAM CLI command to deploy the app.

sam deploy --stack-name employee-functions --capabilities CAPABILITY_NAMED_IAM --guided --profile <your profile name>

4) Once the stack deploys successfully, grab the following API Gateway URL. We will use this URL to execute our API.

Key            APIEndpoint
Description    endpoint of EmployeeApi
Value          https://fy6fpson09.execute-api.us-west-2.amazonaws.com/

AWS CDK Deployment Instructions:-

1) change the region name in the config.ts file if you are deploying other than us-west-2.

 export const getConfig = (): Options => {
    return {
        defaultRegion: "us-west-2",
        apiStageName: "main",
    }
}

2) Changed directory to pattern directory cdk

cd cdk

3) From the command line, use the AWS CDK CLI command to deploy the app.

cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>

4) Once the stack deploys successfully, run the following AWS CLI command to get the API URL. it's using JQ.

aws cloudformation list-exports --query "Exports[?Name=='employee-api-url'].Value" --output text --profile <your profile name>

API Execution:-

1) Add Employee:-

curl -X POST -H 'Content-type: application/json'
https://fy6fpson09.execute-api.us-west-2.amazonaws.com/main/employee \
--data '{
    "Age": 30,
    "FirstName": "Kyle",
    "LastName" : "Gray"  
}'

You will see the following record in DynamoDB.

2) Get Employee:-

curl -X GET -H 'Content-type: application/json' https://fy6fpson09.execute-api.us-west-2.amazonaws.com/main/employee/2414 | jq

Response:-
{
  "Id": 2414,
  "firstName": "Kyle",
  "lastName": "Gray",
  "age": 30,
  "createdDatetime": "2023-03-06T05:23:59.471574185Z",
  "modifiedDatetime": "0001-01-01T00:00:00Z"
}

3) Update Employee:-

curl -X PUT -H 'Content-type: application/json' https://fy6fpson09.execute-api.us-west-2.amazonaws.com/main/employee/2414
--data '{
    "Age": 31,
    "FirstName": "Kyle",
    "LastName" : "Gray"
}'

4) Delete Employee:-

curl -X DELETE -H 'Content-type: application/json' https://fy6fpson09.execute-api.us-west-2.amazonaws.com/main/employee/2414

Cleanup:-

sam delete --stack-name employee-functions --profile <your profile name>
                               OR
cdk destroy --profile <your profile name>

Wrap-Up:-

As you can see how it's easy to understand, implement and deploy GoLang lambda in an AWS environment. I would highly recommend trying it yourself and for your organization. We have a couple of legacy applications where we use an ECS container with 1 CPU and 2 GB RAM, this cost us around $32 per month. Even though the traffic pattern is not consistent, we end up paying $32 per month per task. In the case of serverless, we pay per use and don't pay anything when there is no traffic. In the next post, I will implement the same example with low code or less code using Step Function + universal target.